Developing a Series of Problem-Solving Measures for Elementary Students

• In Event: Igniting Discussions About Measures for K–12 Mathematics Education Contexts

Abstract

The presentation’s purpose is to share results from investigating validity evidence for a series of Problem-solving Measures (PSM). PSMs are designed for grades 3-5 and address mathematics content and practices described in the Common Core State Standards for Mathematics (CCCSI, 2010) and are designed for use by schools and researchers interested in learning about students’ content knowledge and/or problem-solving performance. They are low-stakes measures, which may be used as formative or summative assessments by researchers and practitioners.
Literature and Frameworks
Many problem-solving measures ask students to apply a known procedure to a task and few measures connect to mathematics content described in curricular standards (Author_A, 2015). Thus, problem-solving measures are needed that include problems, connect to curricular standards (e.g., CCSSI, 2010), and have necessary validity evidence.
Several frameworks guide PSM development. Problems are tasks such that they are (a) realistic and plausible, (b) complex in nature, and (c) may be solved in multiple ways (Schoenfeld, 2011; Verschaffel et al., 1999), which differ from exercises meant to promote procedural efficiency (Kilpatrick et al., 2001). PSM development drew upon the Standards’ five sources of validity evidence (AERA et al., 2014; see table 1 for more information) as well as a design science-based framework for creating assessments (Middleton et al., 2003; see Figure 1). The five sources are test content, response processes, relations to other variables, internal structure, and consequences from testing.
Methods and Data
This presentation focuses on data and results from year two. The aim of year two work was item revision and broader implementation. Qualitative data (i.e., expert panel and student data) were analyzed using inductive analysis (Hatch, 2002). Quantitative data were analyzed using Rasch (Rasch, 1960/1980) and inferential statistics.
Results and Their Implications
Findings indicated experts and students agreed that PSM items addressed mathematics content and practices described in the CCSSM. Experts and students reported problems drew upon realistic contexts, could be solved in multiple ways, and were complex in nature. Quantitative results suggested there were appropriate and expected relationships to other variables. Rasch results indicated improved item fit from previous PSM administrations. Qualitative and quantitative suggested bias was limited and areas for improvement in future administrations. Qualitative and quantitative results suggested item revisions improved overall test quality. Test reliability for the PSMs exceeded the minimum threshold for use in low-stakes testing situations.
While many K-12 mathematics measures have been used in peer-reviewed published research, few have gathered and reported substantial validity evidence (Author_A, 2019; Author_E, 2009). It may be that mathematics education researchers are unaware of how to build quantitative measures and attend to validity. This presentation provides scholars with an opportunity to learn about a process for gathering validity evidence. Further, problem-solving measures that address mathematics curricular standards used by classroom teachers can provide meaningful data to make instructional adjustments. The PSMs have potential to fill a critical gap of meaningful mathematics measures that address curricular standards and be used as part of scholarly outcomes.

Authors

• Jonathan David Bostic, Bowling Green State University

• Gabriel Matney, Bowling Green State University

• Toni A. May (Sondergeld), Drexel University

• Gregory E. Stone, University of Toledo