Changes in Public School Composition During the Pandemic and Their Relationship With NAEP Performance

• In Event: Using NAEP Data to Understand Student Experiences and Achievement During the Pandemic

Abstract

The 2022 Main NAEP results were the first following the start of the COVID-19 pandemic in 2020. The fourth- and eighth-grade mathematics and reading results provided evidence of how achievement has changed since 2019, showing declines among national public school students, including an eight-point decline in grade 8 mathematics. Most states and Trial Urban District Assessment districts (TUDAs) also showed declines in their results.

It may be tempting to attribute these results to COVID-19 policies and variations in policies across states, but a myriad of factors likely contributed to the results, including, potentially, changes in school composition. According to the National Center for Education Statistics (2022), enrollment in U.S. public schools shifted dramatically between fall 2019 and fall 2020, with the number of prekindergarten through grade 8 public school students dropping by 1.5 million nationwide. It is unclear to what extent such shifts were temporary and to what extent students returned to school by the spring 2022 Main NAEP testing window.

This study uses NAEP 2019 and 2022 eighth-grade mathematics data to examine two research questions:
1. How did the characteristics of the NAEP-assessed population change between 2019 and 2022?
2. Are observed compositional changes between 2019 and 2022 related to changes in NAEP national, state, and TUDA scores?

To answer the first question, we conducted a descriptive examination of trends in the composition of the NAEP-assessed population. Of particular interest are the trend lines for the major NAEP reporting groups: race/ethnicity, students eligible for the National School Lunch Program (NSLP), English learners (ELs), and students with disabilities (SDs). Changes in the trajectory of the trend line between 2019 and 2022 signify how much the groups changed during this period as proportions of the total assessed population. Our initial trend analysis results did not show major demographic shifts at the national level (e.g., Figures 1 & 2), meaning any demographic trend changes from 2019 to 2022 were expected, given pre-2019 population shifts. The one exception is ELs, which showed a marginal uptick from 2019 to 2022.

To answer the second question, we implemented Beaton and Chromy’s (2007) partitioning method that standardizes the 2022 population to the 2019 proportions of the population on the reporting group variable. In this method, the results are partitioned between a population effect, a performance effect, and a joint effect residual. The effect of interest is the population effect, which represents the variance in scores related to changes in the population. The national results showed generally negligible population effects (Figure 3), with only a national population effect near one point for EL status. However, variance existed across states and TUDAs in the results for race/ethnicity, NSLP eligibility, and EL status (Figure 4–6). The results imply that the observed decline in grade 8 mathematics achievement is primarily performance related.

Overall, the national public school population changes were less than expected and had little relationship with the differences in scores between 2019 and 2022. However, there were greater population changes for states and TUDAs.

Authors

• Martin Hooper, American Institutes for Research

• David Bamat, American Institutes for Research

• Rebecca C. Shipan, American Institutes for Research

• George W. Bohrnstedt, None