The Implementation of Summer Programs in North Carolina and the Impact on Student Outcomes

• In Event: Roads to Recovery: State and District Approaches to Post-Pandemic Education Renewal

Abstract

To promote student academic recovery and re-engagement in school the Office of Learning Recovery and Acceleration (OLR) at the North Carolina Department of Public Instruction (NCDPI) has sponsored a series of summer school extension programs each summer from 2021 to present. The purpose of this study is to look at the implementation of these programs across districts in the state and to estimate the impact on a range of student outcomes.

In particular, we seek to answer the following questions:
1. How do districts across North Carolina implement summer school extension programs in 2021, 2022, and 2023?
2. How does participation in summer programs impact student academic and engagement outcomes?
3. How do impacts vary by characteristics of the students and the summer programs?

Summer school extensions are a promising strategy for improving student academic outcomes and increasing engagement with school following the COVID-19 pandemic. In addition to providing additional time for direct instruction, these programs can offer opportunities for acceleration, enrichment, and socioemotional learning. Over the course of three successive summers, districts in North Carolina have had the opportunity to offer three distinct summer programs. These programs also provide flexibility for districts to adapt the programs to meet the unique needs of their students.

In order to examine the implementation of the programs, the research team has coded program plans and post-program surveys for each program in each summer. Through this coding process, we have created variables measuring differences between districts in approaches to instruction, socioemotional learning, enrichment, and identifying students for participation. We find a number of differences in district approaches to summer programming. For example, some districts focused more tightly on at-risk students while others invited all students districtwide to participate. Some of these differences corresponded with district characteristics. Smaller districts, for instance, were more likely to use teacher and parent recommendations to identify students for participation compared to larger districts which more exclusively focused on measures such as test scores.

To examine the impact of summer programs on student academic and engagement outcomes, we use propensity score weighting approaches to compare students who attended the programs to similar students within the same districts with similar demographics and prior outcomes. Using doubly robust regression models, we examine the impacts of summer program participation on test scores, attendance, discipline, and grades. A potential limitation of this approach is selection into program participation based on unobservables. To address this limitation, we also compare students who enrolled in the program but attended different numbers of days, as we would expect greater impacts of program participation among those who attended more days.

Results from summer 2021 participants suggest that students who attended summer programs had better attendance in the year following participation but had similar or slightly lower test scores. Among high schoolers, students were less likely to repeat previously failed courses and had higher grades than similar students. These results will help to identify characteristics of successful summer programs.

Authors

• Sarah Crittenden Fuller, University of North Carolina - Chapel Hill

• Kevin C. Bastian, University of North Carolina - Chapel Hill

• Douglas L. Lauen, University of North Carolina - Chapel Hill

• Calen Clifton, Arizona State University

• Helen Rose Miesner, University of North Carolina - Chapel Hill

• John McDonnell, University of North Carolina - Chapel Hill