Developing Automated Feedback to Improve Teachers' Formative Assessment of Student Work in Technology-Enhanced Science Curricula (Poster 6)

• In Event: Artificial Intelligence and the Future of STEM Instruction: Designing New Models to Automate Feedback to Teachers

Abstract

The current call for teaching disciplinary core ideas through science practices (NRC, 2012) presents a challenge for teachers who strive to provide ongoing and timely support for their students (Ruiz-Primo & Furtak, 2006). Teachers need to be aware of “how judgments about the quality of student responses (performances, pieces, or works) can be used to shape and improve the student’s competence by short-circuiting the randomness and inefficiency of trial-and-error learning” (Sadler, 1989, p.120). However, teachers rarely have either the time or the pedagogical, domain, and assessment knowledge to provide students with individualized feedback in real-time or before the next class.
Developing automated teaching feedback to assist teachers’ formative assessment is to be carried out in phases. In the first phase, this study is carried out in the context where student assessment information from an online student module is fed into a teacher dashboard that is connected to a teacher edition of the student module. See Figure 3. We hypothesize that Machine Learning (ML) algorithms can identify how teachers’ interactions with the dashboard relate to changes in student learning behaviors individually and collectively. Identifying these patterns is important to develop data-driven automated teaching feedback that would be directly integrated into the teacher dashboard in the following phase.
We recruited 40 middle school teachers across the US to implement an online science curriculum module on wildfire risks and impacts. The Wildfire Module has three components: the student version of the module, the teacher edition with educative curriculum materials layered on top of the student version, and the real-time teacher dashboard. See Figure 3. The data will be collected in the fall of the 2021-2022 school year.
The three components of the Wildfire Moule will log all user interactions and artifacts. Our analysis will focus on eight scientific argumentation tasks embedded in the module where students use real-world data and simulation results to make and support claims or predictions about risks and impacts caused by wildfires. In particular, we will examine to what extent the teachers use the three main features of the dashboard:
1) Main view to track student progress throughout the module: How often do teachers click on student responses?
2) Question view to read student responses: When do teachers read full responses--during class or between classes?
3) Feedback view to give students feedback: How often do teachers provide feedback? Did feedback affect the student’s actions? Did scores increase after teacher feedback?
ML has been applied to finding patterns in unstructured data or to developing detection algorithms based on experts’ judgments (Zhai et al. 2020). The most well-known ML applications are automated scoring of student responses made available on a teacher dashboard. This study goes one step further by employing ML to identify data-driven pathways to systematically develop automated teaching feedback to enhance teachers’ formative assessment practice.

Authors

• Trudi Lord, Concord Consortium

• Hee-Sun Lee, Concord Consortium

• Christopher Lore, The Concord Consortium

• Amy R. Pallant, Concord Consortium