Validation as Design Science-Based Research: Implications for Practice and Theory

• In Event: Informing Perspectives on Validation: Narratives From Mathematics Education Scholarship

Abstract

The presentation’s purpose is to convey a broad perspective on validation that has been applied to mathematics education assessments, which may impact validation related to other content assessments. Kane argues that “validation may not be easy, but it is generally possible to do a reasonably good job of [it] with a manageable level of effort” (2016, p. 79). However, there is evidence of a lack of validation-related studies discussing quantitative instruments used mathematics education (Symposium Author_B, 2019). This lack may imply that scholars across disciplines that intersect with mathematics education are unsure how to conduct such work.
Design-science research has a history in engineering and educational research that can be applied to assessment development (e.g., Middleton et al., 2003). Design research can: (a) address theoretical questions about the nature of learning in context, (b) provide a methodological approach for studying learning phenomena in an authentic setting as opposed to laboratory settings, (c) go beyond a singular measure of learning, and (d) derive justifiable findings from formative evaluation (Collins et al., 2004). Design-science research incorporates several opportunities for data gathering and analysis, which leads to making slight adjustments in future assessment development (Cobb et al., 2001; Symposium Author_B, 2017). Some prior research has drawn upon a design-science approach to develop a series of problem-solving measures and observation protocols for mathematics education contexts (see Symposium Author_B, 2017a, 2017b, 2019). It has been through these development experiences that similarities between design-science research and validation were noticed.
There is no prescribed way to gather evidence for quantitative instruments and there are numerous approaches for framing validation arguments (e.g., Kane, 2001; Schilling & Hill, 2007; Symposium author_A, 2005). Some simply frame an argument around the five sources of validity evidence (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). Recently, others have suggested hybridized approaches for conveying validation arguments (e.g., Symposium author_A, 2019; Symposium author_B, 2019). As a central point for agreement, Merriam-Webster defines an argument as a “coherent series of reasons, statements, or facts intended to establish a point of view” (“argument”, 2018). Therefore, a validity or validation argument serves to inform readers of the validity evidence and why it justifiably grounds the implications and results from an instrument. The validation process is cyclical (see figure 1), much in the same way a design-science approach encourages instrument developed to examine new data in light of an intended goal: developing an instrument for a specified purpose (Symposium Author_B, in press).
The purpose of this presentation is to articulate a process for constructing a validation argument using design science as a methodological research framework. Attendees will be briefed about one example from mathematics education scholarship used this process. This process may provide scholars with a broadly recognized and shared language to talk about validation and engage in it. A shared language connecting validation to a known methodological framework has potential to increase attention to validity across several bodies of literature and foster collaboration across disciplines.

Authors

• Jonathan David Bostic, Bowling Green State University

• Erin Elizabeth Krupa, North Carolina State University

• Jeffrey C. Shih, University of Nevada - Las Vegas