Professional Practice, Student Surveys, and Value Added: Multiple Measures of Teacher Effectiveness in the Pittsburgh Public Schools

• In Event: 35.043 - Toward Improvement of Teacher Evaluations: Methodological and Substantive Issues

Abstract

Responding to federal and state prompting, school districts across the country are implementing new teacher evaluation systems that aim to increase the rigor of evaluation ratings, better differentiate effective teaching, and support personnel and staff development initiatives that promote teacher effectiveness and ultimately improve student achievement. States and districts are implementing richer measures of professional practice alongside “value- added” measures of student achievement growth and in some cases are incorporating additional measures, such as student surveys. Pittsburgh is a leader in the nationwide movement to evaluate, enhance, and reward effective teaching. The analyses presented in this report were conducted to assist Pittsburgh Public Schools in refining its multiple measures of teacher effectiveness, to create a rich, valid, and comprehensive combined measure. In addition, Pittsburgh’s work is based on an approach that is being used or considered elsewhere, the findings have important implications for districts and states across the country.

The Pittsburgh Public Schools teacher evaluation system includes three types of measures. The first—the Research-based Inclusive System of Evaluation (RISE), based on Charlotte Danielson’s Framework for Teaching (Danielson, 2013)—is an observation-based professional practice measure that relies on principals’ assessments. The second measure is based on a student survey called the 7Cs, which incorporates students’ perceptions of teachers’ practices and was developed by Ronald Ferguson of Harvard University as part of the Tripod Project and administered by Cambridge Education. The third measure is a value- added measure that uses changes in student test scores to estimate each teacher’s contribution to student achievement over up to three years of teaching. This study used 2011/12 data to describe how the ratings on the three measures are distributed across teachers and how the ratings are correlated.
It is found that all three measures have the potential to differentiate among teachers. While all three composite measures suggest a wide range of teacher effectiveness, only the district’s value-added measures have been shown to reliably differentiate among teachers (Johnson et al., 2012); the reliability of the RISE and 7Cs composites cannot be determined without multiple ratings per teacher (ideally by multiple raters). However, the components of each of these measures are highly correlated, indicating that the composites have acceptable levels of internal consistency.

Teachers with high RISE ratings tend to have high 7Cs ratings and high value- added measure estimates as well. The correlations are moderate but statistically significant—consistent with other research on similar measures of teacher effectiveness. These results suggest that the measures capture teaching skills that overlap but are not identical—as the district intended in creating multiple measures.
Systematic differences in RISE ratings remain between school even after accounting for differences in value- added and 7Cs measures, suggesting that some principals are tougher or more lenient than others in applying RISE. Using an additional rater for each teacher could help principals better calibrate their RISE ratings, thus enhancing the consistency, fairness, and validity of the ratings (particularly if the additional raters work in more than one school).

Authors

• Duncan D. Chaplin, Mathematica Policy Research, Inc.

• Brian Gill, Mathematica Policy Research, Inc.

• Allison Thompkins, Mathematica Policy Research, Inc

• Hannah K. Miller, University of Wisconsin - Madison