Critics of and Advocates for the Use of Standardized Test Scores in Teacher Evaluation

• In Event: Evaluating the Validity Evidence Surrounding the Use of Standardized Test Scores for Teacher Evaluation/Accountability

Abstract

Purpose

For this section, researchers’ analyzed the differences in validity evidence provided by the proponents of the use of students’ standardized test scores to evaluate teachers and those who challenge or critique their use for the same purposes, and as manifested within their peer-reviewed publications. Conceptually, researchers employed the validity framework they developed (see Figure 1, Paper 1) to frame diverse authors’ arguments around validity (American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME), 2014; Kane, 2013; Messick, 1995).

Perspectives

Both proponents and opponents of the use of standardized tests to evaluate teachers expressed opinions across all aspects of validity supported by quantitative or qualitative evidence. However, each of the two “camps” of authors of these articles tended to be selective in what specific aspects of the validity framework they used to support their oppositional or supporting arguments (see also Authors, 2015a, 2015b). Researchers, therefore, framed, identified and analyzed the same six sources of validity evidence mentioned in Paper 1 (i.e., 1) test content, 2) response processes, 3) internal structure, 4) relations to other variables, 5) related consequences, and 6) overall measurement and methodology) (see also Figure 1, Paper 1), but researchers did this, this time, by author/author stance.

Methods

Researchers analyzed the coded results of the same 92 peer-reviewed articles, by author stance, that researchers also independently coded and collectively discussed if/when disagreements on stance occurred. Researchers also coded the collected qualitative data (e.g., quotations taken from articles as evidence of validity) to support, with text, overall trends (Miles & Huberman, 1994; Miles, Huberman, & Saldana, 2014). Researchers employed tests of statistical significance to analyze the differences between groups as well.

Results

Researchers found that opponents and proponents tended to select and analyze different aspects of the validity construct in order to support their (possibly a priori) points of view. For instance, proponents seemed to more often overlook evidence on response processes and internal structure, but they more often studied areas of general measurement and methodology (56% vs. 27%), related consequences (31% vs. 15%), and relations to other variables, or criterion-related evidence of validity (19% vs. 10%). Opponents directly studied test content (10% vs. 6%) and internal structure (5% vs. 0%) significantly more often, and they provided a significantly larger share of the evidence surrounding related consequences, or consequential-related evidence of validity (63% vs. 31%).

Significance

This is the first study to quantify and position the validity evidence provided by both opponents and proponents on the use of test scores for measuring teacher effects. Researchers found significant divergence in the type of evidence provided by the scholarly members of the two “camps,” which also suggests that the absence of consensus likely stems from methodological and conceptual beliefs, as well as foundational disagreements. This lack of agreement has considerable implications for what aspects of validity might matter or count, and which aspects of validity might be of consequential importance when high-stakes are attached to teachers’ students’ test scores.

Author

• Margarita Pivovarova, Arizona State University - Tempe