Validity Evidence for Measures: Understanding Both Construct and Consequential Validity

• In Event: Measuring and Supporting the Improvement of Common Core State Standards–Aligned Mathematics Teaching at Scale

Abstract

Objective: The long-term objective of our work is a measurement system to aid the improvement of teaching at-scale. In order to achieve this, we examine the extent to which internal validity evidence collected on our developed measures allow for valid inferences about teaching and the extent to which these inferences have potential utility for improving mathematics teaching (consequential validity).

Theoretical Framework: While there is a growing consensus that measuring teaching is important, there is far less consensus on the methods researchers should utilize to measure it (e.g., different researchers have used value-added models (Rivkin et al 2005); subject-specific observation protocols (Grossman et al, 2014; Hill et al, 2008); logs (Rowan & Correnti, 2009); and portfolios (Martinez et al, 2012). However, all of these research-based measures are focused more on the internal (mostly predictive) validity of their central constructs and have paid less attention to the consequential validity of how measures of teaching could be incorporated into a measurement system for improving teaching. Like others (Bryk, et al, 2015), we argue that a measurement system must be an integral partner in both monitoring and supporting improved instruction. Measures become a measurement system, however, only when they are embedded in tools and in practitioners’ routines for collecting, analyzing, and feeding back data to appropriate stakeholders. The entire measurement system must, in turn, be informed by theory that gives meaning to what is being learned (see papers 1 and 2).

Methods/Data sources: In 50 classrooms we have collected intensive measures of mathematics teaching (videos, artifacts and surveys), while in another 50+ classrooms we have collected moderately intensive measures (artifacts and surveys). We have an additional 300 classrooms where teachers self-reported teaching on our survey only. We have state achievement data (TCAP) in all classrooms and student performance-based task sets in 175 of the 400 classrooms.

Results: We develop a validity argument triangulating our different measures of teaching (videos, artifacts and surveys). The argument depends on inferences we make from each measure of teaching vis-à-vis the other measures. In addition, our theory suggests there should be a relationship to project-generated measures of social resources as well as predictive validity for our teaching measures on student learning – skills efficiency measured on the state standardized test and conceptual understanding as measured through performance-based task sets. Preliminary empirical analyses suggest that teaching profile (as suggested by Quadrant placement) predicts achievement scores in the manner theorized and that reliable coding of videos and artifacts using protocols aligned with the theory is possible.

Scholarly Significance: Based on prior interactions, TN leaders appear primed to make productive use of the measures we have developed (e.g., leaders have already asked to incorporate our survey measures in their statewide teaching surveys and they have asked us to present our quadrant theory to the new commissioner). We discuss the validity evidence we have generated to date, as well as the tools and routines that would also need to manifest in order to envision a measurement system.

Authors

• Richard James Correnti, University of Pittsburgh

• Mary Kay Stein, University of Pittsburgh

• Jennifer L. Russell, University of Pittsburgh

• Debra W. Moore, University of Pittsburgh