Using Validation as a Research Methodology to Advance Theory in Mathematics Education

• In Event: Informing Perspectives on Validation: Narratives From Mathematics Education Scholarship

Abstract

The applied practice of instrument validation in mathematics education has lagged behind recommendations for best practice from methodologists and professional organizations. In this paper presentation, I argue that this shortfall represents a missed opportunity. Rather than a burdensome requirement, validation can be viewed as a research methodology to systematically advance knowledge (Symposium author_C, 2016; Symposium author_C, 2019), not just about a specific instrument but also about the broader theory which the instrument operationalizes. However, not all perspectives on validation are equally suited to this goal. In the first part of the paper, I explain why I selected a theoretically-oriented framework for developing and evaluating validity arguments for a novel instrument to measure mathematics teacher knowledge (Symposium author_C, 2017). In the second part, I share how this framework enabled us to leverage validation studies to advance the substantive theory that had informed the instrument development.
In particular, I use a theory-focused validity framework (Schilling 2004, 2007; Schilling & Hill, 2007) to discuss examples from the development of a novel, topic-specific instrument of teacher content and pedagogical content knowledge for teaching fraction and decimal concepts and operations. The framework reconfigures Kane’s (2004, 2006, 2013) two-part approach to validation involving an interpretive argument and a validity argument by explicitly articulating the underlying theoretical assumptions on which the interpretive argument is based. Next, empirically falsifiable inferences based on these assumptions are identified that can be examined in validation studies. Finally, empirical evidence is collected to address the assumptions and inferences in three complementary domains of validation. Elemental assumptions and inferences focus on the test questions, and his category intersects both test content and response process, two validity categories described in the Standards (AERA et al., 2014). Structural assumptions and inferences deal with constructs and the relation of constructs with questions. This category is similar to the internal structural validity category described in the Standards. Schilling’s third category includes ecological assumptions and inferences, and this category intersects both the relations with other variables and test consequences categories in the Standards.
This approach to validation was useful because it provided structure the validity argument by helping us focus on the relationship between theory and evidence. We report our assumptions and inferences in each category for the novel teacher knowledge instrument, the studies conducted to obtain evidence pertaining to these inferences, and our conclusions about the intended use. For example, in one structural validation study, we compared fit indices for two conjectured log-linear cognitive diagnostic models (LCDM; Henson, Templin, & Willse, 2009) with each model corresponding to competing theoretical assumptions about the relationship between mathematical content knowledge and pedagogical content knowledge. The findings from this analysis contribute to theory by clarifying the structure of teacher knowledge for teaching mathematics. The other examples I discuss further illustrate our use of validation studies as a research methodology to advance theory in mathematics education.

Author

• Erik D. Jacobson, Indiana University - Bloomington