Evaluating Item Quality in Large-Scale History Assessments

• In Event: 16.023 - Developing Design Principles for Improving State Assessment Items to Support More Valid and Equitable Measurement

Abstract

Purpose & Significance
History/social studies education in the United States is currently undergoing change as standards, instructional materials, and educators focus more on disciplinary skills and content, rather than solely on knowledge of historical specifics. In history, this is sometimes characterized as a move towards historical thinking rather than memorization, or a focus on not only what we know about the past, but also how we know it. At least 40 states’ history/social studies standards include historical thinking, as do national and cross-state frameworks such as the C3 Framework for Social Studies State Standards and the Common Core State Standards for Literacy in History/Social Studies (American Historical Association, 1997; College Board, 2015; Martin, Maldonado, Schneider and Smith, 2011; National Council for the Social Studies, 2013.)
This paper aims to help stakeholders know more about how to craft and select standardized history items that align with these complex disciplinary competencies. We explain a tool developed by the author for evaluating the cognitive demand of existing history items and take a close look at a small set of those items to help readers identify design features of items that measure important disciplinary competencies.
Theoretical Framework
The Evaluating Cognitive Complexity in History Items tool was developed to analyze the cognitive demand of history assessment items. Key ideas that informed its design included: (1) The integration of historical knowledge, skills, and thinking is at the crux of historical understanding; (2) Disciplinary literacy is key to historical understanding; and (3) Assessment item format shapes cognitive demand. The tool evaluates assessment items along two aspects, with each aspect including multiple sub-dimensions:
Aspect 1: Design Features
Aspect 2: Disciplinary Demands
Methods & Sources
We started with the 50 State Assessment Collection compiled by our research team. We worked to identify items in the collection that show promising approaches to assessing historical understanding. We looked for items that went beyond testing the recall of single historical specifics and that represented a variety of item formats.
While other core subjects identified and used existing tools to code the cognitive complexity of items, a widely used, validated tool is not available in history. We decided to create a discipline-specific tool that would identify criteria for analyzing the cognitive demands and complexity of large-scale history test items --“Evaluating Cognitive Complexity in History Items”.
Findings
Our analysis of items generated a set of five questions that can help test developers and other stakeholders aim for more cognitively complex assessments:
1. Does the item prioritize domain-specific skills and knowledge?
2. Does the item represent a facsimile of disciplinary work?
3. Are the materials used in the item carefully selected and prepared to minimize confounding factors such as reading ability and necessary background knowledge?
4. Are there multiple pathways to a correct answer or multiple correct responses, or does the
item assess knowledge of one specific?
5. Is the item “balanced” in construction?

Author

• Daisy A. Martin, Stanford University