Exploring the impact of artificially intelligent chatbots on undergraduate STEM student success: Experimental evidence from a developing country

• In Event: The impact of generative artificial intelligence (AI) on students and teachers

Proposal

The integration of artificial intelligence (AI) technologies in education is rapidly transforming the landscape of higher education, especially in developing countries where digital advancements are creating new opportunities and challenges. This study investigates the potential of a machine learning-based chatbot to enhance the educational experience of university students in a developing country context, with a specific focus on Kazakhstan.

As digital technologies become increasingly embedded in educational settings, understanding their impact on student outcomes is crucial. This research evaluates the effectiveness of an AI-driven chatbot designed to support undergraduate students majoring in Computer Science at Suleyman Demirel University, a private and selective institution in Kazakhstan. The study is grounded in behavioral science theories, which suggest that reminders, nudges, and timely information can significantly influence decision-making processes and enhance individual well-being.

A randomized controlled trial (RCT) was conducted to assess the chatbot’s impact on both academic and non-academic outcomes. The research questions addressed include: 1) What are the effects of a machine learning-based chatbot on students' academic and non-academic outcomes in higher education? 2) To what extent can a chatbot improve students' satisfaction with a course in a Computer Science program?

Data from 103 students enrolled in a Computer Networks course were analyzed using ordinary least squares (OLS) regression, controlling for factors such as gender, age, prior academic performance, and language of instruction. The outcome variables included pre-final grades, final exam scores, attendance, and course satisfaction as measured by an end-of-course survey.

The findings reveal that while the chatbot intervention did not significantly impact academic outcomes, such as grades or attendance, it had a notable positive effect on students' course satisfaction. These results suggest that AI chatbots, while limited in their ability to directly improve academic performance, can play a valuable role in enhancing the overall educational experience by increasing student engagement and satisfaction.

This study contributes to the growing body of literature on the application of AI in education by providing empirical evidence from a developing country context, an area that remains underexplored. The originality of this research lies in its focus on a region where access to advanced educational technologies is often limited, thus offering insights into how digital tools can be utilized to bridge educational gaps and improve student outcomes.

The implications of this research are significant for educators, policymakers, and technologists aiming to harness the potential of AI in education. By demonstrating the chatbot's ability to enhance student satisfaction, this study highlights the importance of interdisciplinary approaches that combine technology with insights from behavioral science to create effective educational tools. Future research should explore the scalability of such interventions and their long-term impact on both academic and non-academic outcomes, particularly in developing countries where the digital divide presents unique challenges.

Study underscores the potential of AI-driven educational tools to contribute to more engaging and supportive learning environments, even in contexts where technological infrastructure may be less developed. As the global education landscape continues to evolve, these findings offer valuable guidance for the development and implementation of digital technologies that can support student success and equity in education.

Authors

• Galymzhan Momonov, L.N. Gumilyov Eurasian National University

• Aizat Nurshatayeva, Brown University

• Kuandyk Tleuzhanuly, Narxoz University