Leverage Artificial Intelligence to Speed Research and Evaluation Projects

Session Submission Type: Panel

Abstract

Artificial Intelligence (AI) has the potential to both automate or accelerate portions of research and evaluation studies, leading to more efficient activities and shorter study timelines. Project teams must be thoughtful, skeptical, and reflective regarding ways they leverage AI, however, to ensure that studies use AI ethically and are able to continue to produce accurate, thorough, and rigorous results.The papers in this panel will describe the ways in which project teams are leveraging AI for studies conducted for the U.S. Department of Labor. We focus on decisions that any research or evaluation project contemplating the use of AI needs to address, including identifying which portions of a project could benefit from AI, what AI techniques and approaches are most relevant potential solutions, how to program, control, and evaluate the AI solution to address concerns of accuracy, reproducibility, thoroughness, and hallucinatory behavior. We will also discuss the results of the studies, our assessment of the feasibility of applying these solutions to future, similar projects, and the extent to which AI ultimately improved the quality of our projects or decreased the project timeline to results.  Studies in this panel include the use of natural language processing query tools, AI abstract screening, and natural language processing to scrape unstructured information into an analysis-ready dataset. Presenters will discuss which elements of their programs they though were suitable for AI enhancement, how they selected AI models or approaches that met their studies needs, and barriers and challenges they encountered in implementing AI solutions in their work. They will discuss the extent to which they were able to use AI to speed their study activities without sacrificing relevance, rigor, or quality, and what recommendations they have around developing AI governance procedures. Attendees will walk away with potential ideas for ways to leverage AI in their research and evaluation projects as well as lessons learned from these case studies to apply in their future work.

Policy Area

• Methods and Tools of Analysis

Chair

• Tara Martin, U.S. Department of Labor

Discussant

• Allison Hyra, Westat

Organizer

• Allison Hyra, Westat

Individual Presentations

• Developing an AI-based query and synthesis tool to synthesize grantee materials - Presenting Author: Gizem Korkmaz, Westat

• Using AI to Support Systematic Evidence Reviews for DOL’s Clearinghouse for Labor Evaluation and Research - Presenting Author: Radha Roy, Abt Global

• Building analysis-ready data from unstructured filing: An AI approach to Form 5500 data - Non-Presenting Co-Author: April Yanyuan Wu, Mathematica; Presenting Author: Amy Newman, Mathematica

• When Does NLP Help? Human-AI Collaboration in Public Opinion Topic Modeling: A Study of E-Scooters - Presenting Author: Dafeng Xu, University of Washington