Experimental Evidence on the Learning Impact of Generative AI

• In Event: The Emerging Role of AI in Education and Workforce Preparation

Abstract

Generative artificial intelligence has transformed workplace productivity, but its effects on human capital accumulation remain an open empirical question. We study whether off-the-shelf generative AI access affects immediate learning and knowledge retention through a randomized experiment with undergraduate students. The experiment unfolds over two sessions, approximately one week apart. In the first session, students are randomly assigned to an AI-allowed or AI-forbidden condition during a 35-minute learning phase in which they learn an unfamiliar topic (blockchain, carbon capture, or CRISPR gene editing) and write an analytical essay. Treated students receive off-the-shelf ChatGPT without customization, training, or safeguards; control students learn without any AI access. All participants then complete unaided assessments both immediately and approximately one week later.

AI access raises immediate test scores by 0.25 standard deviations, with gains concentrated among middle-performing students and smaller effects at the tails of the performance distribution. These learning gains persist approximately one week later, when all students work without AI. AI access also changes how students write: treated students produce higher-quality essays that are longer and easier to read, and these writing improvements partially persist in the delayed, unassisted assessment. Commercial AI content detectors flag substantially more treated-student text as AI-generated, yet AI-allowed essays are no more homogeneous than control essays, contrasting with the textual convergence documented in workplace settings.

Two changes in the learning production function help explain the gains. First, AI access does not alter total time on task but reallocates it: students spend less time drafting and more time reading and searching for information. Second, AI access substantially increases task enjoyment without affecting perceived effectiveness. A less benign mechanism also emerges: exposure to AI during learning raises rule-breaking on subsequent assessments, suggesting that AI access may erode compliance with academic-integrity norms.

Our findings provide proof of concept that off-the-shelf generative AI can build durable human capital in an academic task common to undergraduate education, identifying a channel through which AI may raise long-run productivity beyond its well-documented effects on immediate task performance.

Author

• Germán Reyes, Middlebury College
  Presenting Author