Assessing Data Quality in Large-Scale Assessment: Prevalence and Impact of Insufficient Effort Responses Across Economies (Poster 12)

• In Event: AERA Poster Session Two
  In Poster Session: Measurement Theory and Application 1

Abstract

In education, large-scale surveys are increasingly used for education-quality monitoring and policy-decision making. Yet, researchers are concerned about the quality of these low-stake assessments and whether the data quality varies across countries. In this research, we used the Programme for International Student Assessment (PISA) 2018 to examine the seriousness and impacts of different types of insufficient effort responses (IER, including students randomly answered) across economies. Specifically, we (a) ranked the 78 economies in their IER seriousness, (b) showed how two IER types (random responses and overly consistent responses) affected data quality, and (c) how economy ranking might change after the IER removal. In sum, monitoring and removing IER is essential in ranking economies in international or similar low-stake survey studies.

Authors

• Dan Wang, The Chinese University of Hong Kong
  Presenting Author

• Leifeng Xiao, Chinese University of Hong Kong
  Non-Presenting Author

• Luyang Guo, Chinese University of Hong Kong
  Non-Presenting Author

• Kit-Tai Hau, Chinese University of Hong Kong
  Non-Presenting Author