Composite Measures of Teacher Effectiveness

• In Event: Educator Effectiveness: Measurement, Leadership, and Policy Processes

Abstract

The purpose of this study is to develop methods for combining diverse measures of teaching and teacher qualifications into one or more composite measures. At the core of the growing movement to improve teacher evaluations are measures of teacher effectiveness. There is a rising consensus that multiple measures of effectiveness are required to capture the diverse behaviors and dimensions of teaching (BMGF, 2010). Measuring student outcomes via a single measure like value-added models will be insufficient. Moreover, value-added measures are imprecise and additional measures can improve the precision of the teacher performance measures (Goldhaber and Hansen, 2008; McCaffrey et al. 2009).

Using an empirical approach, we conduct a study of methods to combine multiple measures of teaching into a composite performance measure. The goal of measuring teacher effectiveness is to develop the best predictor of a teacher’s effects on his or her future students’ outcomes. However, teachers may have different effects on different student outcomes. Moreover, activities that improve effects on one outcome might not yield improvements in effects on other outcomes. Hence, prediction of teacher effects on multiple student outcomes may be desirable.

Thus, one approach to creating composite estimates is to use them as predictors of various outcomes. This approach might lead to multiple composites if there are multiple outcomes. Predicting outcomes may be insufficient if the available outcomes are limited. An alternative approach is to combine the measures to yield the most accurate estimate of the latent constructs that they measure.

Using data from 1,947 mathematics and English language arts teachers of students in grades 4 to 8 from six large urban school systems, we test both approaches to composite measures. The measures of teaching include: value-added on state mathematics and English language arts tests and supplemental tests administered for the project; scores of videotaped observations using the Framework for Teaching, the Classroom Assessment Scoring System, the Mathematical Quality of Instruction, Protocol for Language Arts Teaching Observations, and UTeach Observation Protocol; student responses to survey items about the classroom and teacher, and scores on tests of content knowledge. The data also included teacher experience and degrees earned and student self reports on college aspirations, enjoyment of school, and effort.

For the outcome prediction we use teacher measures on one class to predict outcomes in another class within the same year or across years. We use a hierarchical factor analytic model to allow for pooling information across grades.

We will describe the underlying factor structure of the extensive array of observations. We will compare composites based on estimating the underlying factor or factors to those based on predicting outcomes. Preliminary results suggest that multiple measures improve prediction and that the factor structure is complex and may not be unidimensional.

As the school districts across the nation struggle to develop better measures of teacher performance they are looking for guidance on combining multiple measures of teaching into composites. We present two approaches the problem.

Author

• Daniel McCaffrey, RAND Corporation