Developing a Generic Essay Scorer for Online Practice Writing Tests of Statewide Assessments

• In Event: AERA Roundtable Session Thursday 1:45 pm Four Seasons Ballroom 4
  In Roundtable Session: Machine Learning Techniques II: Advancing Coding/Essay Scoring and Estimation of Treatment Effects (Table 2)

Abstract

This study utilizes transfer learning technique in natural language processing (NLP) to develop generic automated essay scoring (AES) models for instant online scoring in writing practice tests of statewide writing assessments. Using Google’s BERT (Bidirectional Encoder Representations from Transformers), the research aims to build AES models that are generalizable to essays on any prompts. The study analyzes three groups: a control group, a group pre-trained on ASAP essays, and a group pre-trained on 500 SWAS essays. Results showed that further pre-training does not necessarily improve scoring performance, with the control group often performing as well or better. The findings suggest the importance of further pre-training’s quality and the need for further research with balanced datasets and more diverse essay prompts.

Author

• Yi Gui, The College Board