How Automated Feedback Supported Students' Written Science Explanations (Poster 1)

• In Event: Automated Analysis of Students' Science Explanations: Techniques, Issues, and Challenges

Abstract

Introduction
Written science explanations are central to learning and practicing science (Berland et al., 2016). However, there are two main challenges: first, students struggle to explain their ideas (Berland & Reiser, 2009; Braaten & Windschitl, 2011) and use supporting data appropriately (Manz et al., 2020). Second, it is not realistic for teachers to provide real-time, comprehensive feedback to each student (Duschl & Bybee, 2014; McNeill et al., 2016). However, natural language processing (NLP) technologies can provide timely, personalized, automated feedback (Gerard & Linn, 2022; Kucirkova et al., 2021; Zhu, Liu, & Lee, 2020). This study investigated how an NLP technology (Authors, 2019) identified students’ main ideas in a scientific essay. We hypothesized that students would include more ideas in their essays as a result of the automated feedback.

Methods
We used the NLP technology [name blinded] in middle school classrooms as students learned physics (e.g., energy transfer and conservation) by designing roller coasters. Students used a roller coaster simulation to manipulate variables. The simulation was integrated with a digital notebook, where students could take notes, answer questions, and write essays. The unit was implemented by three 8th-grade teachers (N = 264).
Students’ essays were analyzed using the NLP technology to identify the presence of key ideas, referred to as Content Units (CU). 15 CUs were identified as the most important ideas for students to learn during this unit, which were then applied using the NLP program to automatically assess students’ essays. Students wrote two essays during the unit. After Essay 1, students received feedback in their digital notebooks. Students then had to incorporate the feedback and write Essay 2.

Results and Discussion
We hypothesized that students’ written explanations would improve as a result of timely and personalized feedback on their essays. We conducted a repeated measures analysis to examine changes in students’ essays. We summed the 15 CUs into a CU total score for each student for Essay 1 and 2 and used these as the dependent outcomes. Our analysis showed that students included significantly more CUs in Essay 2 than Essay 1 (Table 1). We also ran Wilcoxon signed-rank tests to understand the changes in students’ essays for each CU. We found significant differences for six CUs (Table 2); i.e., more students included these CUs in Essay 2, compared to their Essay 1. Conversely, there were no significant differences for the remaining CUs. It should be noted that CU 1, 7, and 12, also had p<α0.05, but were not significant using a stringent Holm’s sequential Bonferroni adjustment.
Although students included significantly more ideas in Essay 2 (Table 3), they had difficulty understanding the feedback and revising their essays. This is consistent with other studies (Gerard et al., 2016; Tansonboom et al., 2017). Teachers revealed that understanding the feedback may have overwhelmed students. We also found that CUs with little-to-no punctuation, or ideas spread over multiple sentences, were not identified by the NLP technique. Future feedback will check sentence length and ask students to explain ideas more concisely.

Authors

• Indrani Dey, University of Wisconsin - Madison
  Presenting Author

• Dana Gnesdilow, University of Wisconsin - Madison
  Non-Presenting Author

• Sadhana Puntambekar, University of Wisconsin - Madison
  Non-Presenting Author

• Rebecca J. Passonneau, Pennsylvania State University
  Non-Presenting Author