Can We Trust Published PISA Research Studies? A Review of Reporting Practices in Published PISA Articles From 2014 to 2023

• In Event: Surveys in Action: From Technical Refinement to Real-World Influence

Abstract

Secondary data analysis of national data is commonly used to address educational outcomes. As the frequency of these analyses continues to increase, omission of elements such as sample weights or design effects can lead to incorrect inferences. This study addresses the prevalence of researchers reporting the use of statistical techniques that consider the complex sampling structure of survey data in peer-reviewed academic journal articles between 2014 and 2023. A total of 945 articles using data from two education surveys (PISA and ECLS-K) were selected for a systematic review. Preliminary analysis of PISA articles (n = 252) indicates that 23.4% of the reviewed articles reported accounting for design effects and 46.3% reported using sample weights.

Authors

• Yitong Yue, University of Virginia

• Rujun Xu, University of Virginia

• Bethany A. Bell, University of Virginia