A Validation Framework for Automated Essay Scoring Systems

• In Event: Roundtable Session 6
  In Roundtable Session: Research on Rater and Scoring Issues

Abstract

In order to promote a more rigorous approach to construct validation for Automated Essay Scoring (AES) systems, this study proposes a framework to guide the collection of validity evidence in five areas: writing traits assessed by AES, scoring procedures used by AES, structural and measurement properties of AES scores and the consequential aspect of validity. The framework is applied in this study’s investigation of score validity for the Intelligent Essay Assessor (IEA). Evidence collected includes differences in the structural patterns within the human and IEA trait scores and measurement evidence challenging the assertion that IEA traits are measuring a single ability. This leads to identification of issues related to the IEA scoring of writing traits and discussions of implications.

Authors

• Lucy Lu, NSW Department of Education

• James Tognolini, Pearson plc