A Comparison of the Generalizability of Scores Produced by Human Scorers and an Automated Scoring System

• In Event: 14.046 - Poster Session 1
  In Poster Session: 14.046-5 - Assessment, Evaluation, and Research in the Schools

Abstract

The use of constructed response items and the implementation of automated scoring in large-scale assessments have increased over recent years. This study compared scoring results between human raters and an automated scoring system named KASS, by reporting on generalizability analyses. The results of this study indicate that automated scoring system offers outcomes nearly as reliable as those produced by human scoring. Further inspection of the results reveals that the scores generated by the automated system vary across the item clusters. Upon completion of the study, a more comprehensive set of the results along with in-depth discussion will appear in the full paper.

Authors

• KyungHee Sung, Korea Institute for Curriculum and Evaluation

• Eunhee Noh, Korea Institute for Curriculum and Evaluation

• Kyong Hee Chon, Western Kentucky University