Imagining the Future of AI Literacy: Leveraging Theory Toward Principled Design of LLM-Enabled Scenario-Based Assessments

• In Event: AERA Poster Session Wednesday 11:45am
  In Poster Session: Poster Session I -- AERA Division D Section 1: Measurement, Psychometrics, and Assessment

Abstract

The rise of generative artificial intelligence tools has elevated AI Literacy as a critical priority in education; however, variability in school policies and curricula underscores the need for reliable assessments to identify students’ current skills. This paper presents a novel AI literacy framework explicitly designed to support assessment, distinguishing it from existing frameworks. We describe initial work applying this framework to develop a prototype scenario-based assessment through an evidence-centered design approach. The prototype integrates LLMs to simulate realistic AI interactions, including flawed outputs, requiring students to evaluate sources, apply verification strategies and demonstrate responsible AI use. We discuss the iterative design process, highlight challenges in assessment development, and outline future directions for refining and scaling AI literacy assessments in education.

Authors

• Jesse R. Sparks, Educational Testing Service

• Caitlin Tenison, ETS

• Teresa M. Ober, Educational Testing Service

• Tenaha P. O'Reilly, Educational Testing Service