How Do Teacher Rating Scales Reflect the Teacher Doing the Rating?

• In Event: Challenges and Considerations for Creating Equitable Identification Systems

Abstract

Teacher rating scales (TRS) are often used as part of the screening or identification process in gifted education. There is a general belief that teacher ratings add unique information to the identification process, improve multi-criteria systems, and provide a more nuanced assessment of giftedness that should increase diversity. However, TRS and nominations could be counterproductive because the score a student receives depends, at least in part, on the teacher who provides the rating. Wide variability across teachers in their use of the TRS could introduce inconsistency into TRS, which has implications for both the reliability and the validity of multiple measure identification systems that use TRS as part of the process. Our prior study found that between 10% and 25% of a students’ TRS score could be attributed to the teacher doing the rating. Ability, math achievement, reading achievement, and demographics explained little to none of the between-teacher variability. Even after controlling for ability, achievement, and demographics, 10-24% of the total variance in TRS is explained by the teacher, and between teacher standard deviations represent an effect size of one-third to one-half standard deviation unit. In other words, even after equating for ability, achievement, and demographics, some teachers tend to give higher ratings and some teachers tend to give lower ratings.
This follow-up study examines the following research questions: How consistent are teachers’ ratings over time? Do teachers who tend to give higher (or lower) ratings during one academic year also tend to give systematically higher or lower ratings to different classes of students during subsequent academic years?
Using data from three cohorts of students from one district, we estimated a mixed model in which students were nested within teachers and schools. We included cohort as a fixed effect in the model. In addition, we allowed between-student (within-teacher) variance and between-school variance to vary by cohort. (Because there was little to no between-school variance in the TRS across the three cohorts, we did not estimate the between-cohort covariances for the between-school variance.) We allowed between-teacher (within school) variance to vary by cohort, and we estimated the between-teacher covariances across the three cohorts. This allowed us to estimate the between-teacher correlations in TRS across the three academic years. For the unconditional model, the between-teacher correlations were .65 (cohorts 1 and 2), .54 (cohorts 1 and 3), and .57 (cohorts 2 and 3.) After controlling for ability and achievement, the between-teacher correlations were .73 (cohorts 1 and 2), .64 (cohorts 1 and 3), and .54 (cohorts 2 and 3). These correlations are statistically significant and moderately large in magnitude. Across academic years, there is a moderately high degree of consistency in teachers’ rating behavior, suggesting the differences in teachers’ scores are fairly stable across time. In the presentation, we will discuss the implications of the findings from both studies of between-teacher variance.

Authors

• D. Betsy Mccoach, University of Connecticut

• Anthony J. Gambino, University of Connecticut

• Scott Joseph Peters, NWEA

• Daniel A. Long, NWEA

• Del Siegle, University of Connecticut