Survey Data Quality: Educator Survey Participation During the Pandemic

• In Event: Provision of Curriculum and Remote Instruction in the United States During COVID

Abstract

Objective. We examine potential nonresponse bias in teacher and principal surveys and factors associated with educator survey participation during the COVID-19 pandemic that may undermine confidence in the results of educator surveys.

Perspectives and Theoretical Framework. This analysis draws on the total survey error paradigm (Groves et al., 2004), which emphasizes multiple potential sources of non-sampling error that can undermine the fidelity of survey estimates, including coverage error, nonresponse error, and measurement error. And while counterintuitive, response rates can be inversely correlated with nonresponse bias (Groves and Peytcheva, 2008). Nonetheless, systematic differences in response patterns can be an important source of bias.
Methods and Data Sources. In this study, we analyzed survey completion rates for two surveys, one to teachers and one to principals, that were fielded to the American Educator Panels (AEP) during the pandemic in April 2020. We compared completion rates in 2020 with the same surveys—the American Instructional Resources Survey—fielded to the AEP in 2019 to model educator response behavior based on multiple factors including district, school, and personal educator characteristics. We estimated potential bias in surveys fielded during the pandemic which did not adjust for potential inequalities in response opportunities. We also examined responses of early and late survey respondents to identify potential response bias due to systematic differences in response patterns of compliant and reluctant survey participants.

Results. TBD

Significance. Given the sudden and continuously shifting learning environment, ambiguity, and multiple demands on educators’ attention in spring of 2020, were they able to respond to survey requests? And can survey results during this tumultuous time be trusted to provide a trustworthy picture of what was happening on the ground in virtual classrooms across the county? Were teachers and principals able to respond to survey invitations in spring 2020 as they have in the past, or were responses conditional based on factors such as urban vs rural areas, school racial demographics, poverty levels, and district closure policies? Answers to these questions assess the data quality of educator surveys such as those fielded to the AEP and their ability to represent state and national teacher and principal perspectives.

Authors

• David Grant, RAND Corporation

• Christopher Young, RAND Corporation

• Joshua Snoke