Illustratability of Mathematics Items in a Large-Scale Mathematics Assessment

• In Event: 66.049 - Designing Illustrations Glossaries for Special Populations in a National Large-Scale Mathematics Assessment

Abstract

This paper describes how to determine when terms can be illustrated and the percentage of terms within and across items that are illustratable in a large-scale nationally administered mathematics assessment. We ask:
● How can assessment developers determine the illustratability of construct irrelevant terms on mathematics assessments measuring the Common Core State Standards (CCSS)?
● For what percentage of unique words can illustrations glossaries be created as an accessibility resource?
First, we present the guidelines for identifying construct-irrelevant contextual constituents to illustrate. Utilizing the multi-disciplinary outlined in the first presentation based in linguistics, semiotics, and cognitive science, the guidelines facilitate identifying specific terms that can be illustrated. Based in previous research (Authors, 2014), the guidelines consider affordability, illustratability, and allowability. First, affordability examines the extent to which test takers are familiar with a given textual constituent to be illustrated. Second, illustratability centers on whether elements of the term can be adequately represented by an Illustrations Glossary. Finally, allowability ensures that the textual constituent will not interfere with the construct measured and can be illustrated in a manner consistent with the conceptual framework.

The data examined in this study include 112 nationally administered test secure items from grades 4, 7, and 11. Two independent coders were trained in the guidelines for selecting terms to illustrate. Training included reading research on the use of illustrations in large-scale assessment and applying the guidelines to sets of common items to compare codes. Once training was complete, each coder identified when a unique word was illustratable according to the guidelines and within the context of the item. Using consensus coding, the coders arrived at final codes indicating whether each unique word could be illustrated. Next, we calculated the percentages of unique words in an item and identify trends by grade and claim (CCSS).

Our analysis indicates that illustratability ranges from 0.0% to 48.0% of unique words in an item. Preliminary results indicate consistency in the percentages of unique words that are illustratable across grade levels. However, it appears that significant differences exist across claims (CCSS). This presentation will describe these differences and relate them to the effectiveness of this accessibility resource as determined in the cognitive labs. That is, secondary analysis will answer, are the illustrations that students in cognitive labs identified as most helpful for those terms that are frequently found in items aligned to specific claims? By examining how illustratability trends related to the effectiveness of an illustration, we can examine the overall likelihood that illustrations will support students’ understanding of terms typically found in items aligned to specific claims.

These results provide important information for the development of illustrations as an accessibility resource. First, it estimates the number of unique illustrations that assessment systems need to develop when offering Illustrations Glossaries on mathematics assessments. Second, it expands the field’s understanding of how this innovative accessibility resource supports students by understanding the trends of illustratability and how those trends relate to effectiveness of the accessibility resource as reported in cognitive labs.

Authors

• Rachel R. Kachchaf, Smarter Balanced

• Magda Chia, UCLA