Evaluating the Classification Accuracy of an Automated Writing Evaluation System as a Universal Writing Screener in Middle School

• In Event: Innovations in Validity Evaluation and Measurement Development

Abstract

Many middle school students struggle to achieve grade-level writing proficiency, yet traditional screeners face limitations due to subjective scoring and inefficiency. This study evaluates the automated writing evaluation (AWE) tool MI Write as a universal screener, examining its accuracy in predicting Smarter Balanced ELA performance and whether integrating MI Write with the i-Ready reading screener enhances predictive accuracy. ROC analyses showed MI Write consistently performed well (AUC > .75), highest in Grade 7 and lowest in Grade 8. The combined model (MI Write + i-Ready) achieved superior accuracy (AUCs .91–.94), especially with a 90% sensitivity cutoff to reduce false negatives. Limitations, practical implications, and future directions for universal screening in middle schools are discussed.

Authors

• Fan Zhang, University of Delaware

• Joshua Wilson, University of Delaware