Prompt Characteristics and Student Essay Scores in a ChatGPT Writing Task

• In Event: AERA Poster Session Friday 1:45 pm
  In Poster Session: Instructional Design, Feedback, and Learning Strategies

Abstract

Although ChatGPT has been in use by post-secondary students since 2022, the research base on its effects on student work is still small. We coded prompt characteristics of 31 consented students using ChatGPT to write an argumentative essay, describe the most commonly used prompt components and relate them to essay scores. Participants mostly focused on the provided essay directions to write their prompts, under-utilized some dimensions of the prompts, but the number of prompt components from the provided essay directions positively predicted essay scores even though the essay scoring rubric scored for general quality. Although some prompt components were repetitive, this was associated with ChatGPT producing a better essay. Total number of prompt components was unrelated to essay quality.

Authors

• Jennifer G. Cromley, University of Illinois at Urbana-Champaign

• Matt Gadbury, University of Illinois at Urbana-Champaign

• Pooja Roy, University of Houston