A Meta-Analysis on the Effectiveness of AI-Based Edtech on Students’ Academic Achievement: Empirical Evidence from Primary to Secondary Education

• In Event: Educational Paradigms in the Age of AI: Policy, Efficacy, Application, and Digital Citizenship in Korea

Proposal

The proliferation of Artificial Intelligence (AI) carries far-reaching implications for future generations, specifically in the domains of decision-making, labor, and employment. Concomitantly, it is also poised to reshape the educational landscape by generating a heightened demand for a comprehensive understanding and effective utilization of digital technologies as instructional tools and substantive curricular content. This paradigm shift, recently, has led to the integration of research and practice for the digital transformation in the field of education as a major national directive in Korea, initiating an array of deliberations on the deployment of AI-based EdTech in diverse aspects of schooling, including curriculum design, instructional strategies, assessment techniques, and student support mechanisms.

The prevailing discourse surrounding AI-based EdTech pivots on the assumption of its inherent efficacy in improving student learning outcomes. However, there is a conspicuous lack of empirical evidence substantiating this presupposition. In this context, our research endeavors to bridge this knowledge gap by employing meta-analytical techniques to ascertain the tangible impacts of AI-based EdTech on student learning. More specifically, we scrutinize the overall effect size emanating from the implementation of AI-based EdTech services on academic performance and probe into the variance in effect sizes depending on the moderating variables, through the process of meta-analysis for the comprehensive perspective on a large collection of individual studies.

In pursuit of our research objectives, we have applied a rigorous meta-analytical approach to an extensive array of prior investigations that studied the influence of AI-based EdTech on student outcomes based on the following four criteria. First, we focus on studies investigating student population at the level of primary and secondary education. Second, we review literature examining the effect caused by the intervention of AI-based EdTech services on student outcomes. Third, we set traditional lessons taught by an instructor and lessons using non AI-based EdTech services as comparison groups. Fourth, considering differences among various subjects, we concentrate on the mathematical achievement.

Our empirical findings reveal a relatively more pronounced positive effect in the primary school cohort as compared to the middle and high school clusters. Moreover, the study identifies certain conditions under which the impact of AI-based EdTech is amplified: specifically, when the usage period exceeds one month and the duration of each session remains within an hour.

Importantly, analysis also indicates that certain features of AI-based EdTech contribute more substantially to learning outcomes. Real-time feedback and automated assessment, in particular, were found to yield higher effects as compared to strategies centered on learning resource recommendations and psychological support. Thus, our research not only validates the efficacy of AI-based EdTech in promoting academic achievement but also provides crucial insights into optimizing its use to maximize student learning. Through our comprehensive meta-analysis, we aspire to contribute significantly to the scholarly discourse on AI’s role in education and provide a robust empirical foundation to guide future policy and practice in the realm of AI-based EdTech.

Authors

• Sunbok Lee, Dept. of Education, Ewha Womans University

• Hyunmyung Jo, Ewha Womans University

• Myunghyun Moon