Evaluating AI-Enabled Intelligent Tutoring Systems in Information Technology Sectoral Training

• In Event: Does AI Deliver? Evidence from Training Programs, the Workforce, and Career Pathways

Abstract

As artificial intelligence (AI) reshapes workplace skill demands, workforce training programs face growing pressure to scale high‑quality instruction without overburdening instructional staff. Intelligent Tutoring Systems (ITS)—adaptive, AI‑enabled tools that provide personalized, real‑time feedback—offer a promising approach, yet evidence on their effectiveness in adult workforce training remains limited. This study examines the early impacts and implementation of an ITS embedded within Per Scholas’ nationwide IT Support training program, which serves predominantly low‑income and majority‑minority learners preparing for CompTIA A+ certification.

We conducted a mixed‑methods evaluation from 2024 to 2026, anchored by a randomized controlled trial (RCT) and a complementary implementation study. A total of 2,543 learners across 134 cohorts were randomly assigned to either receive access to 13 ITS simulations aligned with course content (treatment) or to receive standard instruction only (control). The impact study examined credential attainment, course completion, and additional certifications, with employment and earnings outcomes forthcoming. Implementation analyses drew on ITS usage data, learner surveys, and interviews with learners and instructors to understand patterns of adoption, user experiences, and perceived benefits and barriers.

Early results show that learners with access to the ITS were 25% more likely to complete the CompTIA A+ certification within one month of course completion, a statistically significant improvement. ITS access did not affect course completion rates or the number of additional (non‑CompTIA) certifications earned. Across 21 moderation analyses, impacts were largely consistent by race, gender, age, income, education level, caregiver status, and nativity. One statistically significant interaction by race emerged for additional certifications, but the effect size was very small, suggesting limited practical significance.

Despite strong impacts on certification outcomes, only 54% of learners with access used the ITS at least once. Among users, engagement was substantial, with many returning multiple times. Usage patterns suggest higher engagement among hybrid learners than fully remote learners, while lesson length remained consistent across users. Learners most frequently used simulations tied to complex or high‑stakes content areas and often revisited lessons when preparing for certification exams.

Qualitative findings indicate strong support for the tool’s usefulness. Learners and instructors viewed the simulations as a valuable complement to instruction, particularly for hands‑on practice, exam preparation, and applying technical concepts. Variation in instructor guidance and framing—whether the tool was presented as optional or required—appeared to influence adoption. Together, findings underscore both the promise of ITS in improving credential outcomes and the importance of intentional integration and instructor support to maximize equitable uptake as programs scale AI‑enabled tools in workforce training systems.

Author

• Jaiden Samantha Butler, American Institutes for Research (AIR)
  Presenting Author