Data Processing and Analysis of the 2021 NAEP Monthly School Survey Results

• In Event: 2021 NAEP Monthly School Survey: Learning From U.S. Schools on Their Response to Pandemic

Abstract

Successful analysis and reporting for the 2021 Monthly School Survey (MSS) were incumbent upon taking advantage of the existing NAEP data processing/quality assurance and analysis infrastructure while also addressing unique challenges to meet the timely release of the survey results. In this paper we will discuss the work on data quality assurance and analysis. We will discuss response rates and implications for reporting, summary statistics of survey responses and margin of error calculation considering complex sampling of schools/sampling weights, and dissemination of survey data for secondary research.

The MSS is comprised of both multiple-choice questions and matrices of grade-level enrollment, for the total population and by subpopulations (e.g., Race/Ethnicity). Many of the survey questions are also broken down by mode of instruction (Remote, Hybrid, or In-person).

For 2021 MSS, most of the statistics were computed as percentages. When applicable, the percentages are reported both in terms of schools (weighted by the school-based weights), and of students (weighted by the product of the school-based weights and the student enrollment for a given grade within school). The margin of errors of the associated percentages due to school sampling were estimated using jackknife resampling approach. For a given reporting unit (region, locale, school type, state, or urban district), the results were released if the minimum response rate standards were met. The response rates were computed for each item individually because a jurisdiction could have sufficient responses for some items but not others. Statistical comparisons of the results across months were also made available.

One critical data quality assurance effort is on the estimation of the number of students within sampled schools. In this survey, the school enrollment data were collected for the sampled schools via one of the survey questions. As described above, the school self-reported enrollment data were then used as the number of students for a given grade (grades 4 or 8) for that school (e.g. the enrollment of Hispanics in a “remote or online setting only” for grade 4 in school A). So, it was vital to ensure the numerical entries by the respondents were valid. Given that the timespan from receipt of each month’s data to publishing of the results was limited, a set of rules was developed on the exclusion of responses that were deemed implausible. The databases from the Department of Education, including the Common Core of Data (CCD) were used to substitute the responses to school enrollment that were deemed implausible or were missing.

These summaries were formatted to support populating the public-facing MSS dashboard and the accompanying downloadable appendix tables. The survey response data and the accompanying school sample weights for all five months of data collection (and supporting documentation) have been packaged for dissemination to interested secondary users via the standard NCES procedures. This paper will provide more details of data processing/quality assurance, the estimation of the statistics, and reporting standards.

Authors

• Laura Jerry, Educational Testing Service

• David S. Freund, ETS

• Bingchen Liu, Educational Testing Service