Using AI to Support Systematic Evidence Reviews for DOL’s Clearinghouse for Labor Evaluation and Research

• In Event: Leverage Artificial Intelligence to Speed Research and Evaluation Projects

Abstract

Systematic evidence reviews are a critical component of government efforts to engage in evidence-based policy making, but are often labor-intensive and expensive to complete. Recent advancements in the use of artificial intelligence (AI) could potentially lower the costs of systematic evidence review work and enable more federal and state decision makers to use systematic evidence reviews to inform policy and funding decisions. This paper presents findings from two AI-related pilots for the U.S. Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR). The pilots explored the feasibility of using AI to streamline evidence review processes at the screening, reviewing, and dissemination phases, while still maintaining human quality controls. Since 2019, CLEAR has conducted an annual systematic review of the published and grey literature on labor-related interventions known as the Systematic Annual Search and Review (SASR). The first pilot examined the feasibility of using AI-assisted abstract screening with Abstrackr to support the SASR. The second pilot evaluated the feasibility of using generative AI (ChatGPT) to support reviewing and dissemination efforts for the SASR by prepopulating initial drafts of study reviews in Excel and study profiles in Word. Across both pilots, we collected data on the time, accuracy, and correction burden involved in AI-assisted systematic review processes. We then compared these data to benchmarks from prior SASR efforts that solely relied on human-coded data and completed blinded quality assurance checks. Overall, across both pilots, we found that AI-assisted processes decreased the total number of staff hours required for those efforts. We also found that with human oversight and verification we were able to maintain the same level of quality as in prior SASR efforts. These findings suggest AI advancements have the potential to make systematic evidence review efforts more accessible and affordable for policymakers not only at the federal level but also at the state level where the costs of such efforts have historically been prohibitive. We will conclude by highlighting where AI can meaningfully improve efficiency, where human oversight remains essential, and what governance and transparency practices are needed for responsible AI adoption in evidence-building efforts.

Author

• Radha Roy, Abt Global
  Presenting Author