Year-to-Year Stability in Measures of Teachers and Teaching

• In Event: Developing Reliable and Valid Measures of Classroom Instruction: Challenges and Innovations From the Field

Abstract

A major issue in the implementation of teacher evaluation systems is the degree of cross-year stability for key metrics. Many assume teaching quality is a relatively stable underlying trait; if teacher scores prove to vary substantially from year to year without explanation, stakeholders may call into question the validity of those scores. However, the existing evidence on the stability of teacher measures suggests that cross-year stability is low to moderate. While the process-product research suggested cross-year correlations in observed teacher behaviors are generally above 0.5 (Brophy, 1975), Politikoff (2013) shows that the stability of observational measures from the more recent Measuring Effective Teaching study (Kane & Staiger, 2012) is roughly 0.3-0.4. Value-added scores appear to have cross-year correlations between 0.2 and 0.5 (Goldhaber & Hansen, 2012; McCaffrey, Sass, Lockwood & Mihaly, 2009). This implies that teacher scores may shift markedly between years. The extent of these shifts, and the degree to which such shifts can be explained, are the focus of this paper.

For our analysis, we use mathematics data from 139 fourth and fifth-grade teachers from four urban districts over two years. Each year, teachers responded to two surveys, allowed three videotaped mathematics lessons, and allowed their students to answer a survey regarding experiences in math class. We also captured state test scores for the students. We used correlations to establish the relationship between teacher-level scores in 2010-2011 and 2011-2012. We list the constructs and source of the measures below, with between-year correlations in brackets.

Assessments of teacher knowledge
• Mathematical knowledge for teaching, Hill, Rowan and Ball, 2005 [.69]
• Knowledge of students’ likely performance on a low-stakes test, Based on Sadler et al., 2013 [.34]

Observations of instruction
• Mathematical Quality of Instruction (MQI): Common Core-aligned instruction, Learning Mathematics for Teaching, 2008 [.51]
• MQI: Teacher errors and imprecision, Learning Mathematics for Teaching, 2008 [.28]
• Classroom Assessment Scoring System (CLASS) Classroom organization, Pianta et al., 2008 [.22]
• CLASS Emotional Support, Pianta et al., 2008 [.25]

Teacher surveys
• Effort in preparing for mathematics teaching, Project-developed [.67]
• Implementation of standards-based instruction, Kunter & Baumert, 2006 [.46]
• Test preparation activities, Project-developed [.48]
• Test preparation beliefs, Project-developed [.64]

Value-added scores
• Teacher contribution to student outcomes; models include student background and peer effects [.53]

Student surveys
• Student reports on math classroom and teacher, TRIPOD, Ferguson, 2012 [.42]

Most measured characteristics vary at least moderately between years. In a future analysis, we will add a third year of data as well as correct these correlations for measurement error.

Next, we assessed the association between changes in these indicators and the following:

• Changes in classroom composition
• Changes in student reports of math tutoring
• Prior year teacher professional development
• Teacher reports on the quality of students in his/her class compared to the prior year

A preliminary analysis finds that decreases in the proportion of free-lunch eligible and limited English proficient students are associated with improved scores on the CLASS, MQI and TRIPOD.

Authors

• Heather C. Hill, Harvard Graduate School of Education

• Mark Chin, Vanderbilt University