Developing Career-Readiness Indices Through Quantitative Psychometric Modeling

• In Event: Developing Career Readiness Indicators: A Focus on Feature Analysis

Abstract

This paper describes the process and results from the integration of qualitative features regarding career and college readiness, obtained by coding [State]’s high school ELA and math items, into a multidimensional explanatory item response model. Results of this integration demonstrate the utility of the features in explaining item- and test-level psychometric characteristics, providing initial validity evidence of the features. The application of the explanatory statistical model also showcases the feasibility of such models in the analysis of data arising out of large-scale operational educational assessments. The model can be viewed as a nonlinear extension of linear models used in typical generalizability theory analyses.
Item response data from computerized adaptive administration of the [State] high school assessment and qualitative item features will be used in the analysis. The main analytical approach involves the following random effects item response model:
η_ij=μ+α_i+θ_j,
where η_ij stands for the value of the linear predictor, μ is the overall (mean) item location, α_i is the random item effect, and θ_j is the latent proficiency variable (a random person effect). The linear predictor enters into an appropriate link function, depending on the type of item scores. For instance, for dichotomously scored items, the following logistic link function might be appropriate:
P(Y_ij=1)=1/(1+exp⁡(-η_ij ) ).
The random effects are assumed to have zero means. Their variances can be estimated from data. The random effects capture the observed variability in student performance. The main difference between this model and a typical item response model used in educational testing is the treatment of items as a random effect crossed with the individuals. In typical item response models, items are treated as fixed effects. Here the items are random effects.
As is typical in random effects modeling (e.g., hierarchical linear model), one can then enter covariates into the model to further explain the random variation. For instance, an item feature X_i may be viewed as an explanatory variable in the following regression model:
α_i=β_0+β_1 X_i+ϵ_i,
where the β’s are regression coefficients expressing the magnitude of association between the item features and the item random effect. In other words, a feature may be perceived as contributing to a more or less difficult item, holding individual proficiency constant. This model enables more direct content, cognition, and linguistic interpretations of the features and relates them to observed performance. All parameters can be estimated in a single stage from student item response data using [Author]’s (2010) Metropolis-Hastings Robbins-Monro algorithm. The model generalizes the widely used explanatory linear logistic test model to the case of crossed random effects.
The integrated analysis combines qualitative data and quantitative information to yield more interpretable and more statistically sound conclusions regarding career and college readiness features. The model also provides statistical results that may be used in future test design, analysis, and reporting.

Author

• Li Cai, University of California - Los Angeles