Proactive communication for STEM success: Impact of a course-specific chatbot in introductory undergraduate chemistry

• In Event: What Matters for Majors: Program Choice in Higher Education

Abstract

Background: Key gateway university courses often are not optimized for success. Many first-year students are taught in large, impersonal lectures (Labaree, 2017), despite the negative effect of class size on student performance (Bettinger & Long, 2018). Although additional student services can positively influence outcomes (Webber & Ehrenberg, 2010), such supports typically are not integrated into courses. Proactive communication via a text-based chatbot can improve students’ navigation of college-related processes (Page et al., 2025). Here, we ask whether this same technology can improve undergraduates’ academic experiences and outcomes.

Purpose: We adapt and test a text-based chatbot with AI capability to provide course-specific, proactive outreach to students in “Principles of Chemistry” (CHEM1211) at Georgia State University (GSU). Chatbot communication addresses three domains with which previous students have struggled: time management, course content, and chemistry belonging. We ask: what is the impact of assignment to receive course-specific chatbot communication on course performance and engagement, and continuation in STEM for CHEM1211 students? 

Setting: GSU is a large, public university that serves a diverse student population. CHEM1211 is GSU’s first-semester chemistry course for STEM majors and serves approximately 1,000 students each semester.

Intervention: CHEM1211 students receive 2–3 messages weekly, including: (1) deadline reminders, (2) encouragement messages promoting a sense of belonging and inviting connection with the instructional team, and (3) resource reminders highlighting additional learning supports. Messages are customized to each student’s performance so that outreach encourages completing outstanding assignments and taking up additional supports, as needed, to promote course success. Students can respond to the chatbot. Drawing on a trained knowledge base, the chatbot AI replies instantaneously to many questions. When it cannot answer a query, the system notifies a teaching assistant who engages with the student via the chatbot.  The chatbot also includes a novel feature called #quizme through which students can access course material and exam questions.

Research Design: CHEM1211 is taught in lectures of 200 students each.  Over the course of the two years, we have experimentally tested the chatbot using student-level randomization in 10 sections to achieve a sample of approximately 2,000 students. Spring 2026 is our final experimental semester.

Data: From student-level administrative data, we draw baseline covariates and five types of outcomes, including: completion of and performance on each course milestone; summative course performance; take-up of supplemental course supports; overall academic performance; and continuation in STEM.

Findings: Results here are partial, based on two experimental semesters. Assignment to chatbot outreach reduced withdrawal from CHEM1211 by 3 percentage points. Impacts on the remaining academic outcomes are imprecise here but suggestive of modest improvement in numeric course grades and in earning a C or higher. Outreach increased attendance of any tutoring by 7 percentage points and of STEM-specific tutoring by 6 percentage points. Throughout the semester, treatment students participated in nearly 40 additional minutes of tutoring, on average. Although these are significant increases, the additional educational time may not have been sufficient to improve overall course performance. Our APPAM presentation will pertain to the full experimental sample.

Author

• Lindsay C. Page, Brown University
  Presenting Author