Paper Summary
Share...

Direct link:

Illustrations Glossaries for English Learners: Findings From Cognitive Labs

Mon, May 1, 8:15 to 9:45am, Grand Hyatt San Antonio, Floor: Fourth Floor, Crockett B

Abstract

To provide valid assessments for EL students, the effects of unnecessary linguistic complexity of assessments should be mediated by either reducing the unnecessary complexity or by providing accessibility resources that help EL students with their linguistic needs. Accessibility resources are meant to help EL students overcome the language barriers they may face in understanding and responding to content assessments. Prior meta-analysis studies suggested that language-support accommodations (e.g., glossary, linguistic modification, read-aloud, translation) have overall positive impact on EL students’ performance on content assessments (Kieffer, Lesaux, Rivera, & Francis, 2009; Pennock-Roman & Rivera, 2011). With the advancement of technology and use of computer-based assessments, these types of accommodations are readily embedded in assessments. Yet, it is important to understand how EL students interact with accommodations on computer to ensure that the design of given accommodations is appropriate for target users.

The purpose of the study was to explore EL students’ interactions with Illustration Glossaries embedded in a mathematics content assessment designed to measure rigorous college and career readiness standards. With the aim of implementing Illustration Glossaries in large-scale standardized assessments, we attempted to investigate student interaction and optimal design features in designing appropriate illustration to describe the given word or phrase to increase the accessibility of test content for EL students.

We conducted cognitive labs with 43 EL students in elementary, middle, and high schools from two urban school districts. We sampled students to have various home language backgrounds and a variety of English language proficiency levels (beginning to early advanced). During the cognitive labs, individual students were first presented with illustrations with no context and asked to describe what each illustration was intended to describe. Subsequently, students took an abbreviated mathematics test form with illustration-based glossaries embedded in the items. Students were interviewed by researchers in one-on-one cognitive labs. Researchers collected observational data related to students’ interaction with the glossaries and item performance to determine if the glossaries helped the students access the tested content. The observational notes and interview data were analyzed using a coding scheme, focusing on alignment, clarity, helpfulness (at the word level and the item level) of illustrations, and prior knowledge of the glossary term.

Findings suggested that EL students seemed to benefit from having illustrations in context of an item, rather than a stand-alone illustration. The EL students’ interpretations of the illustrations were mostly aligned with the intended meaning, although the students experienced some challenges with polysemous terms. On average, students reported that the illustrations were helpful for them to understand the terms used in the items.

To validly and fairly assess EL students’ content knowledge and skills, it is critical to provide appropriate accommodations that help EL students to show what they know and are able to do. The findings of this study will contribute to the development of a new type of accessibility resource, illustration-based glossaries which can be widely applicable to EL students from different language backgrounds. In this presentation, we will discuss practical recommendations to consider based on our empirical findings.

Authors