Improving Score Precision in NAEP With a Multivariate Bayesian Lasso

• In Event: 35.020 - Excellence in Education Research: Early-Career Scholars and Their Work

Abstract

Large-scale assessments, such as the National Assessment of Education Progress (NAEP), provide a snapshot of educational achievement by collecting item responses and student, teacher, and school background characteristics. NAEP is designed to broadly sample content and does not collect enough data from any given examinee to report results for individual students, which is different from district and state standardized testing programs. Instead, statistical procedures are used to predict the performance for groups of students. The ability of NAEP to satisfy its mission of providing data about education relies upon the accuracy and performance of the employed statistical methodologies. This project offers four benefits to existing statistical procedures employed in large-scale assessments.

First, current statistical procedures employ a two-stage strategy that separately estimates measurement and structural parameters. This project developed a latent multivariate Bayesian Lasso regression (LMBLR) model to jointly estimate measurement and structural parameters to account for multiple sources of variability. The Bayesian Lasso feature is designed to model high-dimensional data, as found in large-scale assessments, and can possibly achieve greater parsimony by shrinking regression coefficients. Furthermore, show how to estimate model parameters with a more efficient Gibbs sampler.

Second, existing approaches for estimating student achievement uses principal components to reduce the number of student, teacher, and school administrator background questionnaire variables. One consequence is that variables are grouped into components and researchers and test-developers are unable to readily assess which predictors most relate to achievement. In contrast, test-developers can employ the high-dimensional LMBLR model to include 100s of predictors in an effort to assess which background questions uniquely contribute to the prediction of student achievement. Only a subset of theoretically relevant questions can be included in the background questionnaires and the LMBLR model can be used to aid redesign of student, teacher, and school instruments to include questions that relate most to achievement.

Third, this paper offers new methodology for high-dimensional Bayesian modeling. Specifically, the Bayesian group Lasso was extended to multivariate contexts and to research settings that involve modeling latent dependent variables. In fact, we show how a generalized Laplace distribution can be used to achieve greater parsimony in latent variable models. The LMBLR model is directly relevant to large-scale testing programs and it is also important to highlight that the developed LMBLR model is applicable in other research domains. Namely, researchers across the social sciences model latent variables, and the multivariate Bayesian Lasso is a general methodology that can be readily employed by researchers in other substantive areas.

Fourth, The LMBLR model is applied to 175,200 8th graders’ mathematics achievement data from the 2011 NAEP. The results in this study offered additional information regarding correlates of 8th graders mathematics achievement in a national random sample. In particular, the LMBLR model results offer insight about which student and school variables related most to achievement. Interestingly, the included teacher characteristics were less related to mathematics achievement after controlling for student and school factors.

Authors

• Steven Andrew Culpepper, University of Illinois at Urbana-Champaign

• Trevor Park, University of Illinois at Urbana-Champaign