How to Use ChatGPT: Strategies to Learn With Large Language Models in Physics Education

• In Event: Accepting the Challenge: Technology-Based Performance Assessment of University Students’ Critical Thinking in Times of ChatGPT

Abstract

Objectives
AI-based language models (LMs) offer the opportunity of an automated response to a user’s verbal input. With the recent development of the transformer models with self-attention, language models that have been trained on a large corpus of data, so-called large language models (LLMs) like ChatGPT, show compelling responses to complex inputs, such as writing coherent essays, programming, reasoning, and even a good performance on zero-shot learning problems, i.e., problems that the LLM has not been trained on[18].
During learning or solving problems with ChatGPT, students need to evaluate the output of ChatGPT regarding correctness and potential biases. At the same time, students may apply certain strategies, such as few-shot learning, to avoid or at least reduce the chances of an incorrect output of ChatGPT[22]. Therefore, the free use of large language models for learning and problem-solving presents certain affordances for learners. In this context, the study presented here aims to investigate whether students can learn and apply strategies to improve ChatGPT’s output and successfully solve problems and learn with ChatGPT.
Prior Research
While LLMs have made significant advances in recent years and keep an excellent promise for education, there are still many limitations and challenges that need to be addressed. One major limitation is the difficulty of controllability, as the developers and providers of LLM as well as teachers if used in a classroom, need to verify that the output is correct, safe, and lawful. Apart from that there is a challenge of interpretability, as it is difficult to understand the reasoning behind the model’s output and how it is generated. Additionally, there are ethical concerns about bias and the impact of these models, e.g., on employment, racial injustice, risks of misuse and inadequate or unethical deployment, loss of integrity, and many more[17].
Methods & Data
In this work, we demonstrate how 9th and 10th grade students (N=114) solve physics problems with ChatGPT. In a pre-post design with three groups, we investigated the problem-solving process of students who were instructed how to use a prompting strategy (intervention group (IG) 1) in comparison to a group that did not receive a strategy instruction (IG 2). The physics problems were designed in the way that ChatGPT provides incorrect answers when the questions were entered without additional prompting. To verify the correctness of the solution, the students had a control window next to the chat window which also allows them to learn from ChatGPTs output. The learning gains from the two groups (IG 1, IG 2) and the cognitive load during the problem-solving process were compared to a control group that learned with worked examples (CG).
Results
The presented results show that solving problems in physics with ChatGPT causes significant challenges to learners if the output is not immediately correct.
Significance
The results in terms of the opportunities and challenges of large language models in education are critically discussed within a framework of learning with Generative Artificial Intelligence.

Authors

• Stefan Küchemann, Ludwig Maximilian University of Munich

• Karina Avila, Technical University of Kaiserslautern

• Steffen Steinert, Ludwig Maximilian University of Munich

• Jochen Kuhn, Ludwig Maximilian University of Munich

• Yavuz Dinc, Ludwig Maximilian University of Munich