Use of Fine-Grained Learning Maps to Develop a Large-Scale Alternate Assessment

• In Event: Systems Thinking Approaches to Learning and Assessment Design

Abstract

Objectives
This paper will demonstrate how a very-large, fine-grained learning map can be used in the development of content specifications, dynamic routing of examinees, and score reporting. Theoretical rationale and empirical results will be provided.

Theoretical Framework
Learning maps are a form of research synthesis that provide a powerful structure for developing assessment systems. Shepard, Daro, and Stancavage (2013, p1) said “Learning progressions are one of the most important assessment design ideas to be introduced in the past decade.” Because fine-grained, multi-pathway learning maps bridge content, cognition, and psychometrics, we believe that to be an understatement.

Methods
The Dynamic Learning Maps Alternate Assessment System began with the development of a fine-grained, multi-pathway learning map. We started by reviewing existing research syntheses about learning of mathematics and English language arts, creating portions of a learning map, identifying the placement of common core standards, connecting the portions of maps, and using a team of content experts and cognitive scientists to flesh out the map. Figure 1 shows a portion of the map dealing with constructing understandings of text.

Because the map was large and hard to assimilate additional organizing structures were created: Claims, Conceptual Areas, and essential elements (the cognitively right-sized content standards for students with significant cognitive disabilities). For each essential element a subset of the map was extracted – a simpler learning progression identifying five levels within a common pathway to achievement of the essential element. An example relating to identifying the feelings of characters in a story is shown in Figure 2.

By adding additional information, the essential element concept maps served as a form of evidenced centered design for the actual assessment development. Score reports were organized around these same organizing structures.

Data Sources
Data sources include the learning map, essential element concept maps, and student responses to items tagged to the nodes of the map.

Results
Interviews and surveys of test developers and educators provided evidence of the power of learning maps as an organizing structure. Test development was much more efficient than other reported uses of evidence centered design. Comparison of the BIC for a multi-dimensional Rasch model (ELA and Mathematics) versus a diagnostic classification model based on the learning map structure showed a much better fit with the learning map structure.

Significance
The use of learning maps as an organizational structure provided coherence throughout the assessment system from test design through score reporting. This included use in instructionally embedded assessments as well as an end-of-year summative assessment. While more research is necessary, results of this study were extremely promising as a method for developing assessments that better support student learning.

Author

• Neal M. Kingston, The University of Kansas