COVID-19, School District Reopening Operations, and Student Math Performance in Virginia

• In Event: Sunday Roundtable Session 7:45 am
  In Roundtable Session: Unequal Impacts of COVID-19, Reopening Operations, and Recovery Efforts on Student Outcomes Across Four States (Table 30)

Abstract

The pandemic had unprecedented effects on PK-12 students’ academic achievement and disparately affected already vulnerable groups (U.S. DOE, 2021). Post-pandemic onset declines in test-based performance have been greater in math as compared to reading, among racially and ethnically minoritized and economically-disadvantaged students, and among students in districts that spent more time in remote instruction (Jack et al., 2023; Lewis & Kuhfeld, 2023). Yet, Fahle and colleagues (2023) found that within districts demographic groups were affected similarly, suggesting that it was mechanisms at the district-level driving differences in outcomes. Existing research on district-level factors has largely focused on learning modality (in person vs. remote) but there were other decisions districts made that could have affected student learning that have yet to be explored.

This study examines the impact of the pandemic and differences in a range of district-level responses to the pandemic on 3rd to 8th grade test-based mathematics performance in Virginia. We address the following research questions: (1) How did student math outcomes in the first three recovery years (2020-2021 through 2022-2023) differ from pre-shutdown trends? (2) To what extent did any deviations from pre-shutdown trends narrow or exacerbate inequality based on race/ethnicity, social class, special education status, and English learner status? (3) How did variation in district reopening operations explain differences in the pandemic’s impact on math outcomes?

We leveraged student-level administrative data from the Virginia Department of Education that allowed us to follow students from 2011-2012 through 2022-2023. Additionally, we built a unique dataset tracking multiple dimensions of district operations in the reopening year and used principal component analysis to identify eight factors of operations (family engagement, SEL support, health protocols, assessment use, expectations, teacher PD, technology support, learning needs support) by coding 1,032 public-facing documents. We developed measures of district-by-grade-by-day opportunity for in-person learning and use interrupted time-series models (Bloom, 2003) to estimate the pandemic’s effect on math outcomes and heterogeneous effects by district operations and in-person learning. Models include student-level covariates, time variant school-level controls, and school fixed effects.

Preliminary findings suggest that math pass rates and scale scores declined in the post-pandemic onset period across all grades with the greatest declines among Black, Hispanic, economically-disadvantaged, and English Learner students, deepening existing achievement disparities (Authors, 2022). We find variation in how districts operated in 2020-2021 across the eight factors and by in-person learning opportunity. Like others (Camp & Zamarro, 2022), we find districts with greater shares of Black students spent less of the reopening year in person after controlling for other district-level characteristics. Reopening operations varied less with district-level characteristics than modality, but we find that districts with more economically-disadvantaged students focused less on teacher professional development.

This research informs the work of policymakers and educators in Virginia and beyond as they support student academic recovery. We also make a unique contribution to the growing literature on differences in test-based outcomes by exploring district-level policy decisions that may have led some districts to experience less-negative impacts of the pandemic with important implications for future disaster response.

Authors

• Beth Schueler, University of Virginia

• Luke C. Miller, University of Virginia

• Amy L. Reynolds, University of Utah