The Validity and Reliability of Teacher Value-Added Models: Findings From the Measures of Effective Teaching Data

• In Event: 35.043 - Toward Improvement of Teacher Evaluations: Methodological and Substantive Issues

Abstract

VAMs are increasingly being integrated into teachers’ evaluation, often with high stakes attached. At the local level, more school districts have started to incorporate value added measures in teacher personnel decisions. Two large-scale federal programs, namely Race to the Top and the Teacher Incentive Fund encourage states and school districts to design teacher evaluation systems that incorporate student achievement data. However, recent empirical studies found that VAMs vary substantially across achievement measures, subjects, and years, which raises concerns about the use of VAMs in teacher personnel decision-making.

Drawing on the Measures of Effective Teaching (MET) data of 2,512 elementary and middle school teachers from six large school districts, this study examined (1) the correlations of teacher value-added scores and student composition characteristic variables, classroom observation measures, and student perception of teacher instruction; (2) the stability of value-added scores across high- and low-stakes assessments, across years, as well as across subjects within the same type of assessment; and (3) Agreement of teacher decile ranks based on VAMs across alternative assessments, years and subjects.

This study is guided by the potential outcomes framework for causal inference (Rubin, 1974) and covariate-adjustment approaches in observational studies (Rosenbaum & Rubin, 1983). A causal inference analysis is biased if the treated and comparison groups differ prior to treatment in ways that matter for the outcomes under study. The VAM scores were estimated from a covariate-adjustment model that controls for the demographic characteristics and prior achievement scores for both the individual student and his or her peers in the same classroom. VAM scores for the same teachers were estimated based on both state assessments and supplemental assessments that include more cognitively demanding test content. A three-way mixed model with double within-subjects factors was applied to measure the relative influences of between-teacher factors (district) and within-teacher factors (type of test and subject) on value-added scores for the elementary teachers. The authors found no or negligible relationship of teacher VAMs with classroom composition characteristics. Results indicate that value-added scores derived from high-states state assessments are moderately related to those derived from low-stakes assessments of the same skills. This correlation seems to be somewhat weaker for ELA. The variability of VAM scores across types of assessment was found to be greater than the variability across subjects. The agreement rate of teacher decile ranks based on the VAM scores across subjects and assessments were below 20%. About a quarter of teachers were classified in the adjacent deciles from those based on a different set of VAM estimates. Over three-quarters of teachers were misclassified across test instruments and subjects. This pattern persisted when the data were disaggregated by grade and subject. It was even more difficult to identify the bottom 10 percent of teachers. There was also noticeable variability of the low-high stakes correlation across districts. It is recommended that multiple measures and information be considered in conjunction with VAMs while making personnel decisions to minimize potentially costly classification errors.

Authors

• Ning Rui, Westat

• Matthew Finster, Westat