Validity Evidence to Support an Interpretation and Use Argument

• In Event: Evaluating the Validity Evidence Surrounding the Use of Standardized Test Scores for Teacher Evaluation/Accountability

Abstract

Purpose

For this study, researchers examined the six elements of the proposed validity framework (see Figure 1, Paper 1) to assess the interpretation/use argument (IUA) that students’ standardized test scores can be used to evaluate teacher effects/effectiveness. Researchers addressed the following three research questions: (1) Did the validity evidence support or challenge the IUA? (2) Was the evidence of validity predicted by whether author(s) constructed or mentioned evidence of validity? (3) Was the validity evidence provided in any of the six validity elements more supportive or challenging?

Perspectives

As mentioned prior, while traditional validity literature has conceptualized validation as including different types of validity evidence (e.g., construct, criterion; AERA, et al., 2014; Messick, 1995), Kane (2013) argued that validity evidence is established when many types of validity evidence support the claim, or IUA.

Methods

Researchers searched for articles using JSTOR and EBSCOhost, and selected journals using 2013 Journal Citation Reports® Social Sciences Edition metrics (JCR; Thomson Reuters, 2014). This yielded 965 peer-reviewed articles published within the last 100 years (see Figure 2). Researchers reviewed abstracts for final inclusion, screening out 841/965 (87%) of the articles that were not empirical or directly related to the IUA, leaving 92 articles for review and analysis. Two researchers read each article, yielding 78% initial agreement, and researchers discussed discrepancies to consensus.

Results

As per research question one, researchers found that of the 92 articles reviewed, authors of 89% (n=82/92) of the articles reported on at least one type of validity evidence. Across these articles, evidence provided was at least somewhat challenging of the IUA (M=-.49, SD=.60). More specifically, a plurality of authors (collapsed at the article level) were critical of the IUA (45%, n=41/92), fewer were more balanced in their empirical work/perspective (24%, n=22/92), and the minority were supportive of the IUA (17%, n=16/92) (see Table 1).

As per research question two, researchers used an ordinal multilevel model to examine these studies, with studies nested within publication year. Researchers found that studies that directly examined (versus mentioned) validity evidence had a 20% lower probability of being supportive of the IUA (p=.009).

As per research question three, researchers found that within these articles, authors addressed at least one (38%, n=35/92) or two validity elements (37%, n=35/92), respectively, and authors of no articles addressed all six validity areas within the same article (see Table 2). Validity evidence that authors presented based on relations to other variables was less challenging to the IUA, on average, than the evidence presented in the other key validity areas (F(5,155)=2.83, p=.018), except for when authors addressed/examined actual test content. When researchers analyzed studies that directly examined at least one area otherwise, there was no difference in how the evidence was rated between framework (F(5,71)=1.23, p=.303).

Significance

The researchers did not find sufficient validity evidence to support using student standardized test scores to evaluate teacher effectiveness, especially when evaluations are tied to high-stakes personnel decisions. This is significant in that this has not been done prior, and holds implications for accountability policy.

Author

• Debbie L. Hahs-Vaughn, University of Central Florida