Using Scenario-Based Assessment (SBAs) to Deliver Culturally Relevant Test Content

• In Event: Scenario-Based Assessment (SBAs) in the Age of Gen-AI: Developing Tools for Postsecondary Instructors

Abstract

Objectives or purposes
This study serves two purposes. First, it provides a rationale and method for examining the cultural relevance of reading materials in a comprehension assessment. Second, it provides evidence supporting the potential of using scenario-based assessments (SBAs) to deliver culturally relevant test materials.

Theoretical framework
Standardized assessments have been criticized for not effectively addressing inequities (Koljatic et al., 2021; Randall, 2021). Under-performance of certain student groups may reflect a disconnect between students’ experiences and the cultural expectations embedded in assessment tasks (Hood, 1998). To address these concerns, socioculturally relevant assessments have been proposed (Bennett, 2023; Walker et al., 2023).
SBA provides a paradigm to deliver culturally relevant test materials. Each SBA focuses on a different reading topic, yet student performance across SBA test forms can be compared due to a common scale established via test linking (O'Reilly et al., 2014). Do forms that differ in cultural relevance impact student success?

Methods
We conducted secondary data analysis of student performance on three SBAs that differ in cultural relevance for Black students. We examined the interaction between test form and student group on SBA scores, controlling for students’ prior achievement.

Data sources
Data from 6,965 pre-college students from the U.S. Black students accounted for about 6-7% of the sample. Students took one of three SBAs. The high cultural relevance SBA test form for Black students focused on Harlem Renaissance, a movement in the 1930s that celebrated Black culture and heritage. The two lower cultural relevance SBA test forms focused on invasive species in a community or European immigration to the U.S. at the end of the 19th century. Students’ state test scores in ELA were obtained as a measure of prior academic achievement.

Results
Black and non-Black had comparable topic knowledge on Harlem Renaissance, but Black students had less topic knowledge than non-Black students on invasive species and immigration to the U.S. (p’s < .01).
Non-Black students were more likely than Black students to spend adequate time on the two low cultural relevance SBAs (p’s < .05), but not so on the high cultural relevance SBA (p = .48).
A significant group by test form interaction was identified, even after controlling for students’ prior academic achievement, F(2,5869) = 7.23, p < .01.
All three SBAs had high and comparable reliability (Cronbach’s alpha .86 - .88). They were similarly correlated to students’ prior academic achievement. No differential item functioning was identified.

Scientific significance
These results demonstrate that group differences can vary as a function of the cultural relevance of testing materials across test forms that had similar psychometric properties. These results suggest it is necessary to examine the cultural relevance of test materials, especially when interpreting performance differences between student groups. Results of this study also support the promise of using SBAs to deliver culturally relevant test materials. Other presentations in this session demonstrate how we can leverage generative AI to enable more people to develop SBAs.
Full References Available upon request.

Authors

• Zuowei Wang, Educational Testing Service

• Tenaha P. O'Reilly, Educational Testing Service