Generating Science Analogy With the Generative Pretrained Transformer: Preservice Teachers’ Epistemic Dependence During Formative Self-Assessment (Poster 4)

• In Event: AI and Formative Assessment: The Train Has Left the Station

Abstract

Objectives
Science teachers use analogy as an instructional strategy to respond to formative assessment results and advance student learning. However, teachers need to develop this skill via professional learning. As part of teacher professional learning for formative assessment, this study focused on how pre-service teachers use epistemic resources to generate science analogies and how ChatGPT can shape the process.
Theoretical Background
Analogies can be used to explain challenging and abstract science concepts, connecting those with familiar experiences (Heywood, 2002). But teachers usually lack the epistemic resources to generate quality science analogy. Zhai & Nehm (2023) have shown that AI is already being used for formative assessment, in which large-language models such as ChatGPT can be used to support teachers’ generative practices. However, research has warned that epistemic dependence may occur, especially when teachers rely on a certain epistemic agent or technology (Allchin, 2022). Investigating how and why a learner shows epistemic dependence to ChatGPT given its ubquitousness is crucial for supporting each teacher’s epistemic agency.
Methods
Participants. Twenty-seven pre-service elementary teachers enrolled in a professional learning workshop (9 males, 28 females). They were mostly sophomores and had a 3-week on-site teacher training in schools.
Procedures. The participants were given one workshop on using ChatGPT and two 2-hrs development sessions to generate and revise their science analogy. In Phase 1, they were limited to use epistemic resources such as themselves, textbook/curriculum, internet, peer feedback, or instructor feedback, but ChatGPT. In Phase 2, they were allowed to additionally use ChatGPT. At last, teachers discussed how to apply the analogies in formative assessment practices.
Results
Teachers-chosen target concepts and corresponding analogies are summarized in Table 3. Between the two phases, 13 revised their analogy and 14 reinvented it.
----------------------------------------------Table 3---------------------------------------------------------
The participants averagely asked ChatGPT 11.52 times in Phase 2 (Table 4) - For example, they directly asked ChatGPT to “revise the analogy,” “evaluate the analogy,” or “suggest a new analogy” to best explain the target scientific concept.
----------------------------------------------Table 4---------------------------------------------------------
In Phase 1, without ChatGPT, they consulted themselves by 48.91% and used various other resources too (Table 5). However, in Phase 2, they responded that they consulted ChatGPT by 48.62%, while the references to other resources decreased.
----------------------------------------------Table 5---------------------------------------------------------
It was revealed that participants who improved their analogy (n = 18) used ChatGPT by 40.99%, who worsened (n = 5) by 71%, and who retained (n = 4) by 55% (Table 3).
Also, it was found that two learners’ analogies for ‘dissolving of materials’ converged in Phase 2 while depending on ChatGPT by 70% or 100%, which was not the case in Phase 1 (s5 and s23, Table 1).
Scholarly Significance
This study explored how AI can be used for analogy-generating. It was clear that ChatGPT is a powerful epistemic resource for teachers in generative activities for formative assessment purposes. However, findings also revealed that learners might fall into overdependence to powerful AI tools. Future studies to guide instructors to appropriately use AI formative assessment tools are needed.

Authors

• Gyeonggeon Boaz Lee, National Institute of Education - Nanyang Technological University

• Xiaoming Zhai, University of Georgia