Examining the Long-Term Effects of the Horizons Student Enrichment Program on Student Academic Outcomes

• In Event: 47.082 - Digging Deeper: Using Mixed-Methods Research to Assess Promising Practices in Out-of-School Time

Abstract

Objectives
This study explored the long-term effects of the Horizons Student Enrichment Program (Horizons) on school outcomes. Horizons is a multi-year, intensive summer learning program that also includes school year supports. Horizons serves low-income youth in over 50 affiliates across 18 states. The study examined 15 affiliates partnering with 13 school districts in seven states. Unlike most summer learning impact studies that study short-term effects, this study explored long-term, sustained effects by comparing youth who participated in Horizons for at least four summers to similar non-participants who attended the same schools.
Theoretical framework
The Horizons program offers a unique opportunity to measure long-term effects across many grades. Horizons enrolls children in pre-K or kindergarten and continues to invite them back each summer through at least grade 8. Horizons provides a combination of high quality academic and enrichment programming, and the majority of youth (88 percent) return from one summer to the next. This intensive, multi-year programming is hypothesized to improve long-term academic performance and behaviors.
Methods
The study employed a quasi-experimental design to compare the outcomes of long-term Horizons participants with similar non-participants. Within each program site, propensity score matching techniques were used to match long-term Horizons students to comparison students who attended the same feeder schools.
The study examined outcomes for multiple cohorts of youth. Specifically, an elementary analysis examined effects on test scores, attendance, steady grade retention and disciplinary incidents for fourth- and fifth graders. A middle school analysis examined the same outcomes for youth in grade 6 through 8. The high school transition analysis explored effects on GPA, credits earned, attendance, steady grade progression and disciplinary referrals for students in grades 9 or 11. Impacts were estimated using multiple regression techniques, controlling for site-level fixed effects as well as background characteristics.
Data sources
The study relied on administrative records provided by the school districts and Horizons participation data. Annual school records data were collected from school years 2010–11 through 2015–16, including school attended, demographic characteristics and the academic outcomes noted above.
Results
Horizons students who participated in programming for at least four summers demonstrated stronger 2015-16 academic outcomes than comparison students. For example,
• Attendance rates were higher and chronic absenteeism was less prevalent among long-term Horizons participants versus comparison students. Horizons students had the equivalent of two extra days of schooling in the elementary and middle school analyses and 6.5 additional days in the high school transition analysis.
• Long-term Horizons students had stronger academic achievement outcomes, including, higher standardized assessments in math and science at the end of elementary school and higher GPAs and the equivalent of one full-year course credit by the end of grade 9.
• Long-term Horizons students were less likely to repeat a grade or receive a disciplinary referral during middle and high school.
These promising results provide evidence regarding program impacts in a timely manner but rely on retrospective, quasi-experimental methods. An important next step would be to engage in a rigorous multi-year study, exploring academic as well as social-emotional outcomes.

Authors

• Sherri Christine Lauver, Synergy Enterprises Inc.

• Lauren S. Scher, Concentric Research & Evaluation