Interrater Reliability of a Pool of Evaluators Working With Early Childhood Educators

• In Event: Multiple Perspectives on Accountability in Schools

Abstract

The Early Educator Support Office (EES) serves as the statewide education agency for early childhood educators who are required to hold the North Carolina Birth to Kindergarten licensure. Trained Evaluators employed by EES conduct ongoing performance evaluations of these teachers. These Evaluators recently served as participants in an interrater reliability study in which they provided summative ratings for 10 fictitious profiles using the North Carolina Teacher Evaluation Rubric. The purpose of this study was to analyze the level of agreement between an expert panel of administrators and EES evaluators. The exact agreement rate was 68.7% and the adjacent agreement rate was 87.6%. Several additional methods of evaluating inter-rater agreement are also compared. Implications for evaluator training are discussed.

Authors

• Bryndle Bottoms, University of North Carolina - Charlotte

• Amanda Vestal, University of North Carolina - Charlotte

• Timothy Scott Holcomb, University of North Carolina - Charlotte

• Richard G. Lambert, University of North Carolina - Charlotte