Applying Idea Detection in Dialogue Designed to Support Integrated Revision (Poster 3)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Objective
The ideas and experiences students express during sensemaking are powerful resources for developing integrated science understanding (e.g., diSessa, 2006; Linn, 2006; Hammer, 2000) when taken up a built upon by teachers (e.g., Fulberg & Silseth, 2022). We explore the impact of an adaptive dialog that responds to specific ideas detected in student explanations, mirroring an effective teacher responding to students’ ideas.
Perspective
Advanced natural language processing (NLP) techniques enable detection of individual ideas in students’ written explanations (Riordan et al., 2022). Adaptive dialogs using such techniques can help students value and build on their initial ideas. A recent review of adaptive dialogs in education found few studies in K12 classrooms and the impact on learning remains negligible (Wollny et al., 2021). We designed an adaptive dialog aligned with the Knowledge Integration framework (KI; Linn & Eylon, 2011) and examined how it elicited students’ thinking and supported them to integrate their ideas.
Data Sources and Methods
Data were collected during an end-of-year assessment featuring four explanation items (Table 2). We designed adaptive dialogs to further elicit student ideas and scaffold the development of integrated understanding (Figure 1). Students wrote initial explanations, responded to two rounds of adaptive guidance tailored to their detected idea(s), and revised their explanations.
------------------------------------------------Figure 1---------------------------------------------------
------------------------------------------------Table 2----------------------------------------------------
To develop adaptive dialogs, we built idea-detection and KI NLP models. Idea-detection models are based on rubrics capturing the range of student ideas for each item (Table 3; see Gerard et al., 2022; Riordan et al., 2020 for details). The KI rubric captures the links students make between normative science ideas (Linn & Eylon, 2011). We examined the frequency of new ideas elicited during each round of adaptive guidance and the change in KI score (Liu et al., 2008; scale 1-5) from initial to revised explanation using a paired t-test.
Findings
Students significantly improved the accuracy and coherence of their explanations after the dialog, for all items (Table 3), likely because our adaptive approach was effective at eliciting additional student ideas that they used to elaborate their explanations. For example, the Car item elicited 1095 ideas across students’ initial explanations. Students added another 579 ideas in response to the first round of idea-based guidance and 528 ideas in the second. The second round of idea-based guidance elicited more student science ideas compared to a generic second prompt, tested in a prior study (Gerard, et al., 2022).
------------------------------------------------Table 3---------------------------------------------------
Students expressed more normative ideas in their revised explanations (Table 3). For Speed and Bowls students maintained the number of non-normative ideas from initial to final explanations, and for Car and Energy Story they increased; suggesting students need further opportunities to engage evidence to distinguish among the ideas elicited.
Significance
Our results indicate the promise of idea-detection NLP models combined with KI-aligned, personalized, idea-based guidance, for engaging students in sustained revision and reasoning. Future work will explore how to support students to distinguish among their ideas once their repertoire has been made more visible by the dialog.

Authors

• Allison Bradford, University of California - Berkeley
  Presenting Author

• Weiying Li, University of California - Berkeley
  Presenting Author

• Libby F. Gerard, University of California - Berkeley
  Presenting Author

• Kenneth Steimel, Educational Testing Service
  Presenting Author

• Jonathan Lim-Breitbart, University of California - Berkeley
  Presenting Author

• Brian Riordan, Cisco
  Presenting Author

• Marcia Linn, University of California - Berkeley
  Presenting Author