Evaluating the Construct Validity of an Automated Writing Evaluation System With Manipulations of Masterwork Narratives (Poster 18)

• In Event: AERA Poster Session 8
  In Poster Session: Division C Section 1A Poster Session 2. Literacy: Reading and Writing Across Contexts (San Diego)

Abstract

This study evaluated the construct validity of six scoring traits of an automated writing evaluation (AWE) system called MI Write. Masterwork narratives (N=14) were selected to control for errors that are typical of student writers and to ensure high-quality writing in regard to the traits of interest. A paragraph randomization algorithm was utilized to assess the high-level traits of idea development and organization for sensitivity to such a manipulation. Texts were randomized 35 times each, and randomized iterations (n = 490) were compared to the control text across all traits. Randomizations did not significantly impact high-level trait scores, indicating a disconnect between MI Write’s formative feedback and its underlying constructs. Implications for consumers and developers of AWE systems are discussed.

Authors

• Matthew Myers, University of Delaware

• Joshua Wilson, University of Delaware