How to improve academic performance?

Paper Title: Incentives and services for college achievement: Evidence from a randomized trial

Responsible researcher: Viviane Pires Ribeiro

Authors: Joshua Angrist, Daniel Lang and Philip Oreopoulos

Intervention Location: Canada

Sample Size: 1600 students

Big theme: Education

Variable of Main Interest: Academic results

Type of Intervention: Assessment of the main strategies used to improve academic performance

Methodology: Randomized study

Angrist, Lang, and Oreopoulos (2009) report an experimental evaluation of strategies designed to improve academic performance among college freshmen. One treatment group received academic support services. Another received financial incentives for good grades. A third group combined both interventions. Use of the service was greater for women and individuals in the combined group. The combined treatment also raised grades and improved the women's academic standing. These differentials persisted until the end of the second year, although the incentives only occurred in the first year.

Assessment Context

In recent decades, there has been increasing interest in interventions designed to increase higher education attendance and completion in the United States, especially for low-income students. This interest has resulted in efforts to increase enrollment, and among these efforts are student aid based on student need and merit, tax exemption programs, tuition subsidies, employment assistance, and infrastructure improvements. Thus, the resulting expenditures are justified in part by empirical evidence that suggests there are substantial economic returns to college education and degree completion.

Student academic performance is not always satisfactory. Many students take longer to complete the course than the expected completion time – and one of the possible reasons for this poor performance is a lack of preparation. In particular, many students have poor study skills. However, motivated by the view that the return on these skills is high, the traditional response to academic performance problems has been a series of academic service strategies (e.g., time management workshops, goal setting, academic advising and etc.). Furthermore, in some specific cases, student support services are combined with psychological support services.

Another strategy that seeks to improve student performance is merit scholarship programs – which have focused primarily on a small number of high-performing students. Seeking to increase the number of students benefiting from these programs, an attempt has emerged in the field of scholarships to use prizes and financial incentives to motivate students with good academic performances and not just those with spectacular performances. These programs are, in part, an effort to attract better students to public institutions. But they are also motivated by the view that merit aid increases interest in school and makes students more willing to develop good study habits.

Non-experimental evidence on the effectiveness of student services is mixed. More rigorous studies with designs , particularly for high school students, present a more promising picture. However, to the extent noted in the literature, neither academic support strategies nor financial incentives have been the subject of large-scale evaluations using random assignment research designs So this is the gap that Angrist, Lang and Oreopoulos (2007) seek to fill.

Intervention Details

Angrist, Lang, and Oreopoulos (2009) report a large randomized field experiment that was designed to evaluate key strategies used to improve students' short- and long-term academic performance. Specifically, the study seeks to identify whether students with low academic performance perform better when they are offered additional university services, merit scholarships, or both, and whether the effects of these interventions extend beyond the period in which they were made available.

The study reports on the Student Achievement and Retention Project (STAR), a randomized evaluation of academic services and incentives that was conducted at one of the satellite campuses of a large Canadian university. In American terms, this institution can be considered a large state school, with heavily subsidized tuition. Thus, approximately 1,600 first-year students participated in the project, the majority of whom were from the local area and had a common secondary education background.

The STAR demonstration involves three types of treatment: a service strategy known as the Student Support Program (SSP), an incentive strategy known as the Student Grant Program (SFP), and an intervention that offers both strategies, called the SFSP. SSP offered 250 students access to peer counseling service and supplemental instruction service in the form of Facilitated Study Groups (FSGs). Peer advisors are students trained by the study program to offer academic suggestions to support the student in the first year of school. While FSGs are specific class sessions designed to improve students' study habits and learning strategies, without focusing solely on specific course content.

SFP offered 250 students the opportunity to earn merit scholarships to maintain good grades, but not necessarily the best, in their first year of school. Participants in this program received $5,000 cash for a grade of B (an average grade of 3.0) or higher, or $1,000 cash for a C+ or B- (an average grade of 2.3 to 2.9). However, to be eligible for a scholarship, students had to take at least 4 courses per term and enroll to attend the second year of their program.

The third treated group of 150 students was offered both the SSP and SFP. Finally, the STAR demonstration included a control group of 1,006 students, with whom the program operators had no contact.

Methodology Details

The study by Angrist, Lang, and Oreopoulos (2009) reports on a randomized field experiment involving two strategies designed to improve academic performance among full-time undergraduate students at a large Canadian university. A treatment group (“services”) received peer counseling and organized study groups. The other group (“incentives”) were offered merit scholarships so that the first-year student maintains good grades, but not necessarily excellent. A third treatment group combined both interventions.

For the purposes of analysis, all first-year students entering September 2005, except those with average grades in the top quartile, were randomly assigned to one of three treatment groups or a control group. One treatment group received a range of support services, including access to mentoring by other upperclassmen and supplemental instruction to promote critical thinking. A second group received cash prizes for achieving a certain average score. Additionally, a third treatment group received a combination of services and incentives, an intervention that has not been previously analyzed. The control group was eligible for standard college support services but received nothing extra.

Results

The first result of the study is that young women use services much more than young men. The second finding related to the use of the service is the appearance of a strong interaction between the offer of scholarships and adherence to the service; students in the combined group were more likely to use the services than those offered without the opportunity to earn grants. The incentives therefore had the immediate short-term effect of increasing the rate at which students sought academic support.

The effects of the STAR intervention on students' academic performance are more varied. Parallel to the gender differential in acceptance rates, results in academic performance show a significant impact on grades only for women. The effects on women are greater in the combined group (scholarship and services). The combined group also earned more credits and had a significantly lower rate of academic probation at the end of the year.

The authors point out that women in the combined group continued to outperform the rest of the STAR population in the second year – despite the grants and services being available only in the first year. This result suggests that study skills or study habits acquired during the first year helped increase subsequent performance.

Student interest in support services was lower than expected. On the other hand, interest in services, reflected in rates of adherence and service use, was markedly higher in the group that also received monetary incentives. Interest in services and use of services was also much higher for young women than for young men. Peer counseling was considerably more popular than supplemental instruction for both sexes. The peer counseling intervention clearly deserves further exploration, as does the use of achievement incentives to increase interest in services.

Several patterns emerge from the STAR results. First, students who received peer counseling and supplemental instruction services without grants did not perform better than students in the control group. This may be because compliance rates were relatively low in the service treatment groups, as low consent rates dilute intention-to-treat effects. On the other hand, a 2SLS analysis that adjusts for intention-to-treat effects for nonparticipation reveals a level of precision sufficient to detect theoretical service effects equal to 0.25σ in the combined sample of men and women. The results therefore suggest that the benefits of the services alone are relatively modest.

Public Policy Lessons

Based on the results of the study, it is possible to raise some questions, such as: why the effects were greater in the combined group; and why women had better responses to the programs than men.

Angrist, Lang, and Oreopoulos (2009) suggest two possible reasons for the greater impact of the combined program. The significant impact of fall grades on the SFP and SFSP groups suggests that scholarships alone were a strong motivating force. The SFSP group had the advantage, however, of ongoing guidance and support. These students received more frequent contact in the form of biweekly emails from peer counselors. Greater interaction with classmates in the final year may have facilitated adaptation and integration into the new university environment. The service-only group (SSP) had similar access, but acceptance rates into the SSP were low. For the counseling service, acceptance rates for SFSP women were almost double the acceptance rates for SSP women. Thus, the SFSP appears to have successfully combined increased motivation with a well-defined channel for sustained improvement in results.

Although there is no simple explanation for gender differences in response to incentives and services, the authors point out that women generally appear more committed to postsecondary study than men. Part of this difference appears to be due to better study habits among women, suggesting that they may be more motivated to perform well in school and therefore take advantage of programs like STAR.

Previous studies have also confirmed gender differences in response to incentives and services, with women showing better responses to programs.

References

ANGRIST, Joshua; LANG, Daniel; OREOPOULOS, Philip. Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics , vol. 1, no. 1, p. 136-63, 2009.