Building a Validity Argument for a Measure of Whole Number Problem Solving

• In Event: Roundtable Session 62
  In Roundtable Session: New Approaches to Validity Judgments for Emergent Research Methods

Abstract

Objectives:

To show that the normal process of measure building presented by the psychometric community when engaged iteratively maps well to the iterative “engineering model” used by design researchers.
To present and test a validity argument for a measure of young students understanding of whole number word problems.

Perspectives:

To address objectives I and II above, we examine the iterative processes and procedures used to establish a measure of early elementary students’ problem solving patterns of word problems in whole number. Psychometricians offer an iterative framework for the development of a measure (see figure 1 below).

Figure 1. Steps for developing a measure

We used this framework, aligned with Kane’s validity argument (2006), to iteratively design, test and build our measure of whole number problem solving (WNPS). We drew on cognitive studies of student modeling of word problems to examine student strategy selection (e.g., direct modeling, counting up from, derived fact and recall) when attacking word problems. Moreover, we accounted for strategy change over time in the context, and the semantics, of particular types of word problem (Carpenter, 1985; Carpenter, Carey, & Kouba, 1990, Carpenter, Ansell, Franke, Fennema, & Weisbeck, 1993; Fuson, 1992).

Methods & Data Sources:

Our methods included item development based on the extant research literature, cognitive protocol analyses, data collection via interviews, iterative analyses that follow the framework described in figure 1 above. In addition, a validity argument is presented and tested throughout the iterative process. Data were gathered in inner city K-2 classrooms from nearly 400 students in each of the first three years of the study.

Results:

The work has resulted in a well calibrated measure of young students’ knowledge of WNPS, a measure that takes into consideration student change over time (based on the likely “learning progressions” inferred from in the extant literature). Moreover, we are able to show how a validity argument can be built and tested.

Significance for Validity and Design Research:

All research studies that do not draw on previously published instruments have to “build” them from the floor up. In essence then the act of developing a measure is a design act where the measure becomes a designed artifact. The artifact, in this case, a measure of WNPS “uses samples of observations to draw general and abstract conclusions” about students’ learning of mathematics (Kane, p. 17, 2006). To validate the “interpretative use” of this tool is to evaluate the rationale, or argument, for the claims being made. Drawing on the traditional framework for the development of measures, this study offers critical insights into the ways one can generate and test a validity argument about the meaning and modeling of change in student knowledge of WNPS.

Authors

• Finbarr C. Sloane, National Science Foundation

• Brandon Helding, Arizona State University