Evaluating Rater Accuracy and Cognition for Document-Based Literacy Assessments Using a Mixed Methods Approach

• In Event: Roundtable Session 6
  In Roundtable Session: Research on Rater and Scoring Issues

Abstract

Document-based literacy assessments have become prevalent since the adoption of the Common Core State Standards. These new assessments underscore the connection between reading comprehension and writing skills. Since these assessments are rater-mediated, there have been new challenges introduced for rater training. A key concern centers on the degree to which training is comprehensive so that raters are adequately trained to score diverse types of student responses accurately. One approach for evaluating frame-of-reference training is based on benchmark essays that illustrate and anchor the most common categories of each rating and location on the underlying scale. This study focuses on the examining of rater accuracy and investigating of rater cognition in the context of document-based literacy assessments.

Authors

• Jue Wang, University of Georgia - Athens

• George Engelhard, University of Georgia

• Kevin Raczynski, University of Georgia

• Tian Song, Pearson

• Edward W. Wolfe, Pearson