A Hierarchical Cross-Classified Model on Children’s Rated Performance in Reading and Math: Exploring Rater Effects

• In Event: Roundtable Session 97
  In Roundtable Session: Multilevel Modeling Methods and Applications III

Abstract

In rating scores, rater effects, a systematic variation associated with raters yet independent of the “true scores”, can become a major source of measurement error. Using a subsample of the ECLS-K national dataset (5910 responses, 985 children, 2880 raters), this paper builds up a multilevel cross-classified model to examine the variations in rating scores on children’s performance in reading and math. Results suggest that only 16.73% of the total variation in the observed rating scores lies between children, while 38.52% lies between raters. Parents' tend to give higher ratings than teachers do. Children's self-rating is yet higher, particularly for boys. Only teachers' rating is significantly associated with IRT subject scores.

Authors

• Hui Jiang, The Ohio State University

• Sui Huang, The Ohio State University

• Richard G. Lomax, The Ohio State University

• Ann A. O'Connell, The Ohio State University