The Evolving Conversation Around the Validity of Using Student Test Scores to Measure Teacher Effectiveness

• In Event: Evaluating the Validity Evidence Surrounding the Use of Standardized Test Scores for Teacher Evaluation/Accountability

Abstract

Purpose

The use of student test scores in measuring teacher effectiveness is a major source of contemporary U.S. education policy (e.g., Race to the Top, No Child Left Behind waivers) and a contentious topic of academic debate. While the value-added model (VAM), in particular, is the source of most recent and controversial attempts to apply student achievement scores to the measurement of teacher effects, the conversation about the relationship between the two is extensive and complex.

Perspectives

To systematically gauge the conversation regarding the validity of using student test scores to measure teacher effectiveness, there is a need to compare the validity evidence upon which the surrounding conversations rest. Rather than focusing on the who or the why of the debate, then, researchers in this study focused on the what, or the empirical evidence that has provided the grounds for decades-long debate. Researchers anchored the debate in the validity evidence, as informed by Kane’s (2013) interpretation/use argument (IUA) framework discussed in Paper 1 (see also American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME), 2014).

Methods

Analyses of validity evidence included 92 purposefully selected peer-reviewed articles covering 100 years that, in some way, directly addressed and provided evidence related to the IUA in question. Researchers coded the studies using the aforementioned IUA validity framework (see Figure 1, Paper 1), while also capturing direct quotations from study authors that represented each category of evidence. Researchers used these to draw inferences about each category, independently, and then about the debate as a whole. Researchers also traced authors’ external citations to record when and how authors addressed others’ previous work in the area.

Results

The scholarly conversation has changed over time in some ways, and in some ways not. While there has been an increase in the sophistication for which teacher effectiveness has been both conceptualized and measured, (100 years of) evidence to support the validity of the IUA in question is (still) grossly lacking. As for the issues at the core of the debate, the longest running conversation is related to the level (and measureable) impact that teachers actually have on student test scores (see for example Berliner, 2014; Good & Beckerman, 1978; Shavelson & Dempsey-Atwood, 1976). Other major conversations surround issues of bias (Chetty, Friedman, & Rockoff, 2014b; Koedel, 2009; Rothstein, 2010), year-to-year stability (Goldhaber & Hansen, 2010; Lefgren, 2012), and the evidence based on intended/unintended consequences (Chetty, Friedman, & Rockoff, 2014a; Winters & Cowen, 2013).

Significance

Given the millions of students, teachers, and schools now impacted by the use of student test scores in teacher evaluations, it is of utmost importance to understand the decades of research related to this topic. Dating back 100 years, scholars have continuously cautioned test-users about the potential problems associated with measuring teacher effectiveness using test scores. Researchers analyzed this evolution via a choice set of these literatures and offer a systematic review of the extant evidence.

Author

• Jessica Holloway-Libell, Kansas State University