Detecting AI-Generated Essays: What You Should Know Before Making Important Decisions

• In Event: Authentic Responses From Students in the Age of ChatGPT: Guidelines for Dos and Don’ts

Abstract

The introduction of ChatGPT and other generative AI technologies has ushered in a new era for education, profoundly influencing teaching, learning, and assessment. These advancements have revolutionized the learning landscape, offering personalized tutoring and instant feedback to students, while easing the burden of content creation for educators. The dynamic and interactive nature of AI-driven educational tools has also enhanced student engagement and motivation. However, amid these transformative changes, educators face a critical challenge – ensuring the authenticity of student responses in an environment where AI-generated content can be indistinguishable from human work.

The authenticity of student responses has become a significant concern for educators in the age of generative AI. With AI models capable of producing human-like essays, traditional methods of detecting plagiarism may not suffice in identifying AI-generated submissions. As a result, it becomes increasingly difficult to discern originality from generated content, potentially undermining the credibility of the assessment process. This challenge calls for novel strategies to preserve academic integrity while leveraging the benefits of generative AI in education.

Various detectors have emerged to address academic. These tools employ sophisticated algorithms to analyze language patterns and identify potential misconduct. While promising, caution must be exercised when utilizing such detectors. Over-reliance on AI-generated essay detectors may lead to false positives, unjustly penalizing students and creating an atmosphere of mistrust. To strike the right balance, educators must adopt a thoughtful approach, employing these detectors as complementary tools to guide the assessment process rather than as a sole determinant.

In this presentation, we summarize the status of the detection of AI-generated essays in an educational context and highlight a few critical issues, such as performance metrics, bias, and evidence for high-stakes actions, educators should be aware of before using the results of from the detectors for decision making. To ensure fair and equitable evaluation, guidelines for the proper utilization of AI-generated essay detectors will be presented. Emphasizing best practices and acknowledging potential limitations, these guidelines will encourage educators to wield these tools judiciously. A balanced approach will foster a learning environment that embraces the potential of generative AI while upholding academic integrity.

As examples, we share an array of detectors of AI generated essays developed based on our large sample of high-quality data. By leveraging large language models and innovative machine learning techniques, these detectors have demonstrated robust results based on our datasets. Most importantly, rather than relying solely on detector results, we have a carefully designed human-in-the-loop mechanism to have human experts review the cases by considering the context and individual capabilities of students.

In summary, this presentation targets an urgent topic in educational evaluation in the age of advanced generative AI. The guidelines and the empirical examples we show in this presentation could help educators and educational practitioners to better understand the challenges from generative AI, come up with sensible strategies, and make smart decisions when they encounter the misconducts using generative AI.

Authors

• Jiangang Hao, Educational Testing Service

• Michael Fauss, Educational Testing Service

• Chen Li, Educational Testing Service