Designing, Developing, and Validating Assessments of Complex Thinking in Mathematics for the Middle Grades

• In Event: Assessment of Complex Thinking in Mathematics, Reading, Science, and History

Abstract

Designing an assessment system for complex thinking in mathematics involves challenges and decision points at every stage, from how to represent the target competencies to how to interpret performance data with respect to a student’s knowledge state. I will draw on examples from middle school mathematics, but the arguments are generalizable to other levels of study. Learning mathematics with understanding involves perceiving connections among mathematical ideas (e.g., NCTM, 2000). One persistent challenge in characterizing mathematical competency is to capture not only the variety of skills and concepts, but also their connections. Research related to meeting this challenge will be discussed, as will lessons learned during the course of developing successive iterations of the mathematics competency model (Graf, 2009; Graf, Harris, Marquez, Fife, & Redman, 2010) for the Cognitively Based Assessment of, for, and as Learning (CBAL) project (Bennett, 2010; Bennett & Gitomer, 2009). The development for CBAL mathematics was based on an evidence-centered design approach (e.g., Mislevy, Steinberg, & Almond, 2003).
In order to assess complex thinking, a model that represents connections among mathematical ideas and across stages of development is needed. Developmental models, sometimes referred to as learning progressions, can inform the design of tasks so that they are more likely to be both suitable and informative at a particular level: they can also suggest how to scaffold a task that will be applied to a formative purpose. I will refer to work on developmental models for mathematics (e.g., Harris, Bauer, & Redman, 2008), as well as for other scientific disciplines.
Another question concerns the design of suitable tasks. Some tasks that assess complex thinking have complex prompts (e.g., descriptions of real world scenarios, one or more data displays). Others have short prompts but require a complex constructed response. Each task type provides different information about a student’s knowledge state. Developing tasks for complex thinking requires several rounds of piloting and revision. The prompts must meet at least two criteria: 1) the mathematical question must be well specified, and 2) the language must be accessible to students. These two criteria can work in opposition, especially in the design of complex prompts involving real-world scenarios: including advanced vocabulary or qualifiers may render a prompt technically precise, but if taken too far can result in a mathematics task that makes too many demands on vocabulary knowledge or language comprehension. The use of graphical displays can be used to meet both criteria. Research and pilot work related to this question will be discussed.
A final question I will address concerns the question of validity and strategy selection. Mathematics tasks that require complex thinking often lend themselves to alternate correct strategies, which is part of what makes mathematics interesting but which can also pose challenges to validity. Which strategy should be considered optimal? The most efficient strategy? The one that is most suggestive of conceptual understanding? The answers are not cut and dried, but are important to the design of both tasks and rubrics.

Author

• Edith Aurora Graf, ETS