Exploring Rater Errors and Systematic Biases in Language Assessment Using Mokken Scale Analysis

• In Event: Roundtable Session 6
  In Roundtable Session: Research on Rater and Scoring Issues

Abstract

The assessment of English language proficiency (ELP) is a significant issue in numerous settings around the world. Many ELP assessments include essays that require human scoring with the potential for introducing a variety of rater errors and biases. Mokken scale analysis (MSA) offers a probabilistic-nonparametric approach for evaluating the psychometric quality of ratings based on the principles of invariant measurement (Author, 2008). This study describes an application of MSA to ratings obtained from an assessment of ELP in southeastern Asia. We explore the use of MSA identify rater errors and systematic biases. A comparison is conducted between indices based on Rasch Measurement Theory and MSA. Implications for research, theory, and practice within the context of assessment of ELP are discussed.

Authors

• Stefanie Anne Wind, The University of Alabama - Tuscaloosa

• George Engelhard, University of Georgia