MetaMate: Large Language Model to the Rescue of Automated Data Extraction for Educational Systematic Reviews

• In Event: Leveraging Artificial Intelligence and Automation Tools for Systematic Reviewing

Abstract

Systematic reviews and meta-analyses are useful but time-consuming, especially during data extraction. To address this challenge, we developed MetaMate, an open-access web-based tool leveraging large language models for automated data extraction. MetaMate utilizes a hierarchical schema and divide-and-conquer approach in its extraction chain, and a from-global-to-local lens and example retriever in its verification chain. Evaluated on 32 empirical studies, it extracted 20 elements related to participants and interventions, achieving high precision, recall, and F1 scores, comparable to human coders. Notably, MetaMate demonstrated advanced mathematical reasoning and semantic comprehension, surpassing keyword-based approaches and avoiding common human errors. As the first LLM-powered tool for educational research, MetaMate can significantly streamline the systematic review process, reducing time and effort for researchers.

Authors

• Xue Wang, Johns Hopkins University

• Gaoxiang Luo, University of Pennsylvania