The Criterion-Related Validity and Classification Accuracy of Computer Adaptive Testing on Reading: A Meta-Analysis

• In Event: AERA Roundtable Session Wednesday 3:45 pm - Gold 3
  In Roundtable Session: Computational Assessment in Education: Validity Evidence from Automation to Adaptivity

Abstract

This meta-analysis evaluated the criterion-related validity and classification accuracy of Computer adaptive tests (CATs) in reading assessment. Results synthesizing 32 studies showed that the criterion-related validity for CATs is r = .67, 95% CI [.64, .70] (concurrent validity r = 0.66, 95% CI [.61, .71], predictive validity r = 0.68, 95% CI [.64, .71]). A bivariate meta-analysis accounting for the correlation between sensitivity and specificity yielded an overall sensitivity of .70 (95% CI [.66, .74]) and specificity of .76 (95% CI [.72, .80]). These values fall below recommended benchmarks for universal screeners (e.g., sensitivity ≥ .80). Moderator analyses revealed that validity and accuracy varied by grade level, test duration, item response theory (IRT) framework, and cut-score selection method.

Authors

• Linling Shen, University of Texas at Austin

• Yixian Huang, University of Texas at Austin

• Zixuan Ma, University of Texas at Austin

• Man Chen, University of Texas at Austin

• Nathan H. Clemens, University of Texas at Austin