Evaluating an AI Tool for Researchers: Do the Benefits Outweigh the Risks?

• In Event: Future-Oriented Dialogues on AI in Education

Abstract

We evaluated a purpose-built systematic review tool from an academically-focused artificial intelligence (AI) system to assess its utility for educational research. Tool functions included automated source searching based on research questions, inclusion/exclusion recommendations, and data extraction with an AI-generated “first draft” report. The search function struggled with basic criteria such as publication date, over-suggested irrelevant results, and missed papers found through traditional search methods. The library function was unable to retrieve many papers in full or receive PDF uploads effectively. The tool rejected corrective feedback during extraction and reporting phases. We conclude these tools require significant improvement before consistently reducing systematic review time and effort without introducing significant risks to research quality.

Authors

• Michael Ahlf, University of Houston

• Sara G. McNeil, University of Houston