Evaluating the Efficacy of Generative AI in Automating Assessment in Introductory Computer Science Courses (Poster 47)

• In Event: AERA Poster Session Sunday 8:00 a.m.
  In Poster Session: Integrating Generative AI in Education: Challenges and Opportunities

Abstract

Despite ongoing debates about the challenges and affordances of Generative Artificial Intelligence (GenAI), its presence is increasingly felt in various aspects of daily life, including education. This paper documents our experiences using GenAI to assess student performance on coding and writing assignments in large Computer Science courses. Aiming to automate the labour-intensive task of grading while providing high-quality, immediate feedback, we conducted a comprehensive evaluation of GenAI for assessment using student data from three different assignments, comparing its performance to that of teaching assistants. Our findings suggest that while GenAI may be valuable for providing immediate individual formative feedback and identifying student errors in programming assignments, it is not yet suitable for automated summative assessment or evaluating writing assignments.

Authors

• Preeti Raman, Toronto Metropolitan University

• Harjot Singh, Ryerson University

• Atif Ahmad, Toronto Metropolitan University