Does AI Deliver? Evidence from Training Programs, the Workforce, and Career Pathways

Session Submission Type: Panel

Abstract

Generative AI and AI-enabled technologies are rapidly being integrated into employment and training systems, including workforce development programs and workplace practices. States are central in shaping how these tools are adopted through funding decisions, program design, and implementation across policy priorities, industry demands, and service delivery systems. As states pursue different approaches to workforce innovation, understanding how AI affects training quality, service delivery, and individual outcomes has become increasingly important for evidence-based employment and training policies that meet regional skill needs. Yet, as AI tools proliferate across workforce systems, evidence about their real-world effects remains limited. Policymakers and practitioners are making high-stakes decisions about adoption and scale with little clarity on whether these tools meaningfully improve service delivery and outcomes—or simply introduce new complexity.

This panel examines a core question at the intersection of employment and training policy and technological innovation: How does the integration of customized AI and AI-enabled tools into employment and training programs affect service delivery, learning outcomes, and workforce system performance across varied contexts? In particular, the panel focuses on whether these tools improve learner, job seeker, and employee outcomes, or risk adding complexity without clear benefits.

Presenters will share pilot, implementation, and impact evaluation findings of AI use across three distinct employment and training settings. Together, the studies draw on rigorous mixed‑methods approaches—including a randomized controlled trial (RCT), administrative outcomes data, user engagement metrics, and qualitative data. Collectively, these evaluations examine AI integration in an IT sectoral training program, a multi-state employment navigation and career development service, and a self-directed career navigation platform.

Jaiden Butler from the American Institutes for Research will share findings from an RCT evaluation of an AI-enabled Intelligent Tutoring System (ITS) embedded within Per Scholas’ nation‑wide IT Support training curriculum. Impact results showed a 25% increase in CompTIA certification rates. Implementation results documented instructor and learner experiences and use of the ITS tool (employment and earnings impacts forthcoming).

Martelle Esposito from Nava Labs will share findings on how a generative AI powered referral generator enhanced caseworker workflows. By integrating the tool into day-to-day practice for staff supporting clients in Texas and Pennsylvania with education, training, job placement, and resource navigation, caseworkers saw a 9% reduction in administrative burden and increased confidence and speed in matching clients to appropriate resources.

Kelsey Schaberg from MDRC will share findings from an evaluation of a Career Coaching AI Chatbot implemented by SkillUp Coalition, a nation‑wide career navigation platform. The evaluation found chatbot users engaged less with SkillUp’s offerings than users of its traditional catalog. Lessons include the potential benefits and limitations of using AI chatbots in career navigation, as well as ways to use these tools effectively.

Together, this panel underscores the importance of rigorous evaluation as employers, sectoral training, and career coaching providers experiment with AI driven innovations. By examining varied state and organizational contexts, the panel offers actionable insights for policymakers and practitioners seeking to responsibly scale AI tools in employment and training systems to improve wellbeing, efficiency, and program effectiveness.

Policy Area

• Employment and Training Programs

Chair

• Samia Amin, American Institutes for Research

Discussant

• Yu-Fu Michael Chen, Nava PBC

Organizer

• Jaiden Samantha Butler, American Institutes for Research (AIR)

Individual Presentations

• Evaluating AI-Enabled Intelligent Tutoring Systems in Information Technology Sectoral Training - Presenting Author: Jaiden Samantha Butler, American Institutes for Research (AIR)

• Piloting an AI-Referral Tool to Reduce Administrative Burden and Improve Resource Matching in Workforce Services - Presenting Author: Martelle L Esposito, Nava PBC

• Using AI to Help Job Seekers: Lessons from SkillUp AI - Presenting Author: Kelsey Schaberg, MDRC

• How AI May Reshape Career Pathways to Better Jobs and What Regional Pathway Health Tells Us - Presenting Author: Joseph Henry Siegmund, Opportunity@work; Non-Presenting Co-Author: Shriya Reddy Methkupally, Brookings Institution