Assessing Methodological Transparency in TIMSS, PIRLS, and PISA Studies: Reporting Practices from 2014–2023

• In Event: AERA Roundtable Session Sunday 11:45 am - Gold 4
  In Roundtable Session: Contextualizing Educational Data

Abstract

Education researchers often use international large-scale assessments to inform education policy. Omission of required analytic elements such as sample weights, variance adjustments, to adjust for complex sampling, and/or plausible values can lead to incorrect estimates and inferences. This study addresses the prevalence of researchers reporting the use of statistical techniques that consider the complex sampling structure of TIMSS, PIRLS, and PISA data and the use of plausible values in peer-reviewed academic journal articles published in 2014-2023. A total of 1532 articles using TIMSS, PIRLS, and PISA were selected for a systematic review. Preliminary analysis (n = 216) indicates that it was not common (often less than 30%) for researchers to report their use of sample weights and replicate weights.

Authors

• Yitong Yue, University of Virginia

• Rujun Xu, University of Virginia

• Bethany A. Bell, University of Virginia