Expanding the Scope of Systematic Literature Reviews with AI: Evidence from a Large-Scale Application

• In Event: Large Language Model (LLM) Evaluation and Use in AI Policy Research

Abstract

The exponential growth of academic research in recent years poses challenges to researchers and academic institutions, with implications for those in broader society who wish to benefit from research. In 2022 alone, there were 2.82 million articles published, constituting a 47% increase relative to 2016 (Hanson et al., 2024). The popularization of generative AI and AI-assisted research is expected to supercharge this phenomenon.

Systematic literature reviews (SLRs) have been an important tool for synthesizing large bodies of research. However, given the speed of research activity, conducting broad reviews has become increasingly challenging. In response, researchers have turned to AI tools such as large language models (LLMs) to facilitate the process. A small but growing methodological literature shows that AI can help reduce screening workload and improve scalability (Delgado-Chaves et al., 2025; Feng et al., 2022; Thode et al., 2025). Yet, extant studies are mainly concentrated in fields such as medicine and informatics, with limited evidence and guidance on using AI assistance for conducting SLRs for social sciences or interdisciplinary research.

This paper develops and empirically evaluates an approach for AI-assisted screening of scholarly work, combining ensemble-based screening procedures to leverage principles of reliability and reproducibility, and reducing stochasticity in model judgment. This approach builds on our experience integrating LLMs into a SLR on AI’s impacts on workers, which initially identified over 150,000 abstracts across dozens of fields. As our second contribution, we provide transparent procedural guidance for conducting AI-assisted SLRs, addressing a gap in the existing literature regarding replicable workflows. We present and validate a structured protocol for integrating LLMs into large-scale abstract screening pipelines, grounded in the PRISMA framework for systematic reviews, including the iterative testing and reasoning that shaped its development. Third, we advance the literature by explicitly integrating humans and AI in SLRs. Rather than pursuing fully automated review pipelines, our approach adopts a “humans-in-the-loop” design that combines the scalability of LLM-based screening with human judgment in validation and refinement stages. In doing so, our paper contributes to supporting an effective and responsible transition to AI assistance for systematic reviews and provides clear guidance and examples for researchers in the social sciences and beyond.

Author

• Luisa Nazareno, Virginia Commonwealth University
  Presenting Author