Enhancing Meta-Analyses with LLM-Assisted Literature Screening: A Case Study on AI-STEM Education and Critical Thinking

• In Event: AERA Poster Session Wednesday 3:45pm
  In Poster Session: Systematic Review and Meta-Analysis: Methods, Contexts, and Impacts

Abstract

The rapid advancement in artificial intelligence (AI) presents both challenges and opportunities for traditional academic research paradigms. This study investigates the potential of large language models (LLMs) in automating the literature screening task for meta-analyses. Using a database of 2,198 articles, we compared the performance of two popular LLMs, GPT-4.0 and DeepSeek-V3 with manual verification in automated screening. Results showed DeepSeek-V3 achieves 88.2% accuracy in abstract screening (vs. GPT-4.0’s 63.3%) with faster processing speed. The accuracy was improved in both models when screening full-text (30 articles), reaching 93.9% (DeepSeek-V3) and 92.0% (GPT-4.0). These findings underscore AI’s transformative role in evidence synthesis, enabling researchers to focus on higher-order intellectual tasks.

Authors

• Xuechun Wang, The Chinese University of Hong Kong

• Alan C. K. Cheung, The Chinese University of Hong Kong