Setting Quality Benchmarks for Next-Generation Assessments: Lessons Learned in Assessing Deeper Learning

• In Event: 25.024 - Next-Generation Alignment Methodologies for Next-Generation Tests

Abstract

Setting Quality Benchmarks for Next Generation Assessments: Lessons Learned in Assessing Deeper Learning
Purpose
This presentation illustrates challenges in setting quality benchmarks for the alignment of standards and assessments, based on an effort to establish targets for the representation of deeper learning. The effort derives from one Foundation’s “simple” question: Given that educators and students focus teaching and learning on what’s tested, deeper learning must be adequately represented in accountability tests. But what is adequate attention to deeper learning?
To answer this question, the presentation summarizes data on deeper learning in:
• Common Core State Standards (CCSS),
• state tests prior to the CCSS.
• respected national and international tests, and
These analyses surface a number of challenges and inconsistencies in current alignment methodologies, echoed in other presentations, and suggest additional issues that will need attention in evaluating the alignment of more complex ESSA assessment systems with state standards.
Theoretical Framework
Norman Webb’s four-level Depth of Knowledge (DOK) framework (Webb, Alt, Ely, & Vesperman, 2005) is used to operationalize evidence of deeper learning. Levels DOK3 and DOK4 were considered to represent deeper learning, they require students to apply and synthesize their knowledge and engage in critical thinking and reasoning (Herman & Linn, 2013).
Methods and Data Sources
DOK studies examined included:
• 17 leading states’ assessment programs prior to the adoption of CCSS (Yuan & Le, 2012)
• Synthesis of DOK studies of English language arts and mathematics CCSS (Herman et al, 2014)
• Respected national and international tests (Yuan & Le, 2014)
• PISA’s reading and mathematics literacy tests (Herman et al, 2016).
• Fordham and HumRRO studies of the quality of PARCC, Smarter Balanced, MCAS and ACT/Aspire.
Results
Selected results include
• Analyses of deeper learning evident in the CCSS varied widely across studies
• Prior to CCSS, on average that 28% of the ELA items and 3% of the math items on leading state assessments addressed deeper learning.
• Analyses of respected national and international tests revealed that only AP and IB tests met or surpassed the deeper learning expectations of the CCSS. Moreover the two studies of PISA’s DOK in reading literacy showed divergent results.
• Fordham’s analysis of the proportion of total score points at higher DOK levels revealed that PARCC and Smarter Balanced generally met the expectations of CCSS.
• Across all sources, higher levels of DOK were more evident in ELA than in math.
Significance
Variation in DOK results is striking across studies, both in their characterizations of the CCSS and in PISA analyses, raising questions about the accuracy and meaning of DOK and what appropriate DOK benchmarks should be. For example, if DOK distributions should be consistent with standards, whose characterizations should govern? Using what procedures and which metrics? --as they vary across studies as well. Tensions between measurement, deeper learning and feasibility concerns and questions about weighting of components and of qualitative and quantitative perspectives also are apparent, particularly for more complex ESSA systems. Study results yield possible parameters and suggested solutions.

Author

• Joan L. Herman, University of California - Los Angeles