Evaluating Item Quality in Large-Scale Mathematics Assessments

• In Event: 16.023 - Developing Design Principles for Improving State Assessment Items to Support More Valid and Equitable Measurement

Abstract

Purpose & Significance
One of the driving principles of new standards development and adoption has been to support students and teachers in exploring mathematical ideas with more rigor, depth, and coherence across the grade levels. Yet in this era of standards-based accountability, there remains enormous pressure on teachers to “cover” the “right” standards, meaning those that will be on the test. As a result, curricula and assessment measures continue to be atomized into bite-sized math, with the persistent unintended consequence of deeply fragmented mathematical experiences for students and teachers alike. The challenge of assessing conceptual mathematical understanding–together with procedural fluency–has persisted through decades of large-scale test development efforts, including those of NAEP, TIMSS, PISA, and state assessments (Yuan and Le, 2014). The purpose of this review is to highlight item design features that offer promise for meaningfully measuring conceptual understanding. Our intention is to support those who want to prioritize assessments that can help steer instructional time toward more coherent, more cognitively rigorous and conceptually robust learning experiences. The selected items are all publicly available from current (or recent) large-scale assessments, and therefore reflect approaches to standardized assessment that are very much within reach.
Theoretical Framework
The primary tool used in the cognitive complexity analysis of assessment items was an adapted version of Webb’s Depth of Knowledge framework (adapted by Herman, Buschang, & La Torre Matrundola, 2014). This DOK framework has been used widely to identify the cognitive demand of assessment items, and serves in the present review as a starting point for identifying item design features that support meaningful measurement of conceptual understanding and mathematical proficiency (NRC, 2001a). In addition, our review relied upon two other frameworks for qualitative analysis of the selected items: (1) the PISA 2015 Draft Mathematics Framework (OECD, 2016), and (2) the Cognitive Rigor Matrix/Depth of Knowledge table developed by Hess, Carlock, Jones, and Walkup (2009).

Methods and Sources
To find examples of promising items among those publicly available from existing large-scale mathematics assessments, we convened a team of experts to review a wide range of released and practice items from all publicly available state and new consortium assessments. Selected items represented a variety of item types, grade levels, and content foci. The goal of the review process was to identify cognitively rigorous items that assess core disciplinary skills, such as extended reasoning, problem-solving, and coordinating across mathematical representations. The content of each item was considered, together with the response format, prompt wording and structure, and any specific design features that support item quality.

Results
In order to craft cognitively rigorous items that are both student-friendly and mathematically precise, are both clear and concise, and focus on mathematical ideas that are teachable, learnable, and important, we propose focusing on the following design considerations:
• A focus on core disciplinary processes and ideas
• Support for multiple entry points
• Support for multiple solution strategies
• Considerate presentation
• Technological enhancements that support student reasoning
• Engaging contexts

Authors

• Vinci Daro, Stanford University

• Kari Kokka, University of Pittsburgh