Reporting of Design Effects and Sample Weights: A Review of Published Early Childhood Longitudinal Study, Kindergarten Cohort and NAEP Articles

• In Event: National Databases, Science and Math Student Achievements, and a Methodological Assist

Abstract

Secondary data analysis of nationally representative data is commonly used to address educational outcomes. As the frequency of these analyses continue to increase, omission of elements such as sample weights or design effects can lead to incorrect inferences. This study addresses the prevalence of researchers reporting the use of statistical techniques that take into account the complex sampling structure of survey data in published, peer-reviewed articles between 2000 and 2010. A total of 266 articles using data from two national education surveys (ECLS-K and NAEP) were selected for a systematic review. Preliminary results indicate that across the two data sources, 62.5% of the reviewed articles reported accounting for design effects and 75% reported using sample weights.

Authors

• Mihaela Ene, University of South Carolina

• Kshawna Askew, University of South Carolina

• Bethany A. Bell, University of South Carolina