Conceptualizing Validity Arguments for Practical Measures for the Improvement of Instruction at Scale

• In Event: Using a System of Practical Measures, Routines, and Representations to Support Instructional Improvement at Scale

Abstract

Objectives: In this paper, we describe a validity argument for the use of particular practical measures in efforts to improve middle school mathematics teaching and learning, and provide implications to support others to conceptualize validity arguments for other practical measures of instructional improvement efforts more generally.

Theoretical Perspectives: Practical measures for the purpose of improving instruction at scale require constructing validity arguments around particular interpretations and use (AERA, APA, & NCME, 2014; Haertel, 2013; Kane, 2013). Validity is not a property of a particular practical measure but how that measure is interpreted and used by specific users. While there are a range of ways to construct validity arguments for any measure (see for example, Carney et al., 2019), there are few examples in improvement science and related fields (Takahashi et al., 2020). Validity is assessed with respect to its intended use for a particular purpose, by particular users, in a given context (Moss, 2016). Our approach is guided by questions around how the use of the measures are “shaping and shaped by the local learning environment and its learners” (Moss, et al., 2006, p. 111).

Methods and Data Sources: We analyzed how the classroom measures were used in three districts, each of which aimed to support mathematics teachers’ development of ambitious and equitable instructional practices. Qualitative analysis of multiple measures included practical measures data, field notes of classroom observations, and audio recordings of professional learning opportunities.

Results: On the basis of our analysis, we propose a framework for conceptualizing validity of practical measures designed to support instructional improvement at scale. The first component of the framework concerns the technical rigor of the measures. This component provides confidence to the users that the measure actually assesses what it purports to assess and predicts what it purports to predict. The second component concerns the actual uses of the measure in particular contexts, and involves identifying: 1) appropriate purposes for using the measure support inquiry into practice; 2) key aspects of school and district contexts that enable measure to be used for the purpose of improvement rather than accountability; and 3) key aspects of users’ perspectives, knowledge, and current practice that enable them to use the measures productively for improvement purposes.

Significance: This conceptualization of validity for practical measures investigates the extent to which users working in particular contexts are confident that the resulting data can and should be used to inform actions in the service of the improvement effort. Minimum conditions for use of the measure and suggestions for considering users’ current perspectives in ongoing validity efforts will be discussed.

Authors

• Marsha M. Ing, University of California - Riverside

• Thomas M. Smith, Vanderbilt University