Developing an NLP (Natural Language Processing) Model for Student Explanations of Social Justice Science Issues (Poster 6)

• In Event: AI and Formative Assessment: The Train Has Left the Station

Abstract

Objective
This study reports on the development of an automatic scoring model that assesses students’ ability to explain social justice science issues (SJSI; Morales-Doyle, 2017). We explore using the model in two different curriculum contexts and how teachers use the generated scores.
Perspective
We leveraged the Knowledge Integration Framework (KI; Linn & Eylon, 2011) to develop a natural language processing (NLP) model to automatically score student explanations of SJSI (Morales-Doyle, 2017). We aimed to measure the degree to which students explain a science phenomenon, the intersecting justice issues, and how well they connect ideas from those domains to demonstrate integrated understanding. We designed tools to convey scores to teachers. In this study we ask:
● Does an NLP model that captures students’ integrated understanding of social justice science issues generalize to multiple contexts?
● How do teachers make sense of and use knowledge integration scores?
● How does student understanding of SJSI progress as their teachers enact the units?
Data Sources and Methods
We developed a NLP model to automatically score an item, Impacts, where students explain whether all people are impacted by climate change in the same way. The item is featured in two units: Global Climate Change and Urban Heat Islands (UHI)) and Chemical Reactions and Asthma (Asthma). In both units, students connect their science understanding to the role of race, socioeconomic status, and local policies. We designed a KI rubric (scale 1-5) (Liu et al., 2016) and two subscore rubrics (scale 1-3) for disciplinary and justice-focused ideas. The KI rubric measures the integration across the two subscores. We created training data by applying the rubrics to 2000+ responses from previous classroom studies. We embedded the Impacts item with scoring models in the units and as a pre/post-test item. We displayed the scores for the teachers (Figure 2).
----------------------------------------------Figure 2---------------------------------------------------------
Findings
We trained models (see Riordan, et al., 2020) using data from each unit and combined data from both units. We tested the models on data from each unit. The combined model performed best for all scores in both unit contexts (Table 7).
----------------------------------------------Table 7---------------------------------------------------------
Teachers attended to differences in disciplinary and justice scores during instruction. One noted that the students started with vague ideas about the SJSI but shared more detailed ideas orally. Another reflected that the progress on the science topic was greater than the progress on the SJSI. The teachers used the score reports to reflect on improvements to the unit.
Students made significant gains in their KI scores from pretest to posttest on justice (mean difference=0.371 t(150)=6.91, p<0.001), disciplinary (mean difference=0.291 t(150)=5.13, p<0.001), and KI scores (mean difference=0.583 t(150)=8.26, p<0.001).
Significance
We successfully designed an NLP model for an assessment item measuring the integration between disciplinary and justice ideas that worked across unit contexts. The scores supported plans for unit revision. Teachers endorsed the plan to refine the unit and use the scores to guide students to revise their responses.

Authors

• Allison Bradford, University of California - Berkeley

• Troy Vincent Wilson, University of California - Berkeley

• Brian Riordan, Outshift by Cisco

• Kenneth Steimel, Educational Testing Service

• Libby F. Gerard, University of California - Berkeley

• Marcia Linn, University of California - Berkeley