Falsification Tests and Value-Added Models

• In Event: 35.043 - Toward Improvement of Teacher Evaluations: Methodological and Substantive Issues

Abstract

In his influential paper, Jesse Rothstein (2010) finds standard value-added models (VAMs) suggest implausible and large future teacher effects on past student achievement. This is the basis of a falsification test that appears to indicate bias in typical VAM estimates of current teacher contributions to student learning. Rothstein’s critique of value-added methods used to estimate teacher effectiveness has been cited by both researchers and policymakers as a reason to doubt the wisdom of using VAMs for high-stakes purposes.
In this paper the Rothstein test is investigated using both theoretical considerations and simulation methods. The findings call into question whether the Rothstein falsification test (RFT) provides accurate guidance regarding the magnitude or existence of bias of teacher effect estimates. Ideally, the RFT could be used to identify VAMs that produce biased estimates of current teacher effects. It is shown, however, that the RFT identifies important control variables left out of a VAM only under conditions that are not plausible. More precisely, we find that one cannot use the RFT to reject the hypothesis that students were effectively randomly assigned conditional on lagged achievement. In addition, we find that when data are generated that appear similar to the data analyzed in Rothstein (2010), estimated future teacher effects, from his tests, are similar in magnitude to the true teacher effects, but the bias for current teacher effects is extremely small, suggesting that the magnitude of the future teacher effects does not provide useful information about the magnitude of the bias. In a nutshell, the RFT can be used to identify the existence of tracking, but the tracking could well be a function of lagged achievement, the variable that is included in most VAMs. It does not appear that the RFT can be used to tell us much more.
The authors think that Rothstein’s 2010 paper raised important concerns about the ability of VAMs to produce unbiased estimates of teacher effectiveness, but the RFT itself does not provide useful guidance regarding VAMs. Given this, more work needs to be done to understand the potential reasons why VAMs might produce biased teacher effect estimates. This will likely involve a closer look at the various factors affecting student sorting into classrooms so that one can better account for student sorting when estimating teacher effects.
From a policy perspective, the important question may not be whether there is any bias, but the potential magnitude of any bias. It is quite likely that teacher effectiveness estimates generated from VAMs are biased, at least to some small degree but, as shown in Rothstein (2009), Kinsler (2012), and our simulations, the magnitude of bias may be inconsequential. Decisions about using VAMs should consider how this bias compares to potential information that value-added models can provide about teacher effectiveness over, or in addition to, other means of assessment.

Authors

• Dan Goldhaber, American Institutes for Research

• Duncan D. Chaplin, Mathematica Policy Research, Inc.