Paper 1: Using Implementation Data to Improve a Curricular Unit to Develop AI Literacy

• In Event: Learning from Implementation Research: How Curriculum, Professional Learning, AI Tools and Classroom Studies Can Support AI-Literacy

Abstract

Purpose
This paper describes the key features of a curricular activity system (Roschelle et al., 2010) comprising: (1) an AI literacy unit; (2) AI partners to support collaboration within the unit, and (3) professional development to support implementation. We characterize how implementation data led to the re-design of this system to enhance both teachers’ ability to enact the unit and the depth of students’ engagement in AI literacy.

Perspective
Co-design is often used to enhance the usability of an innovation by improving the fit of an innovation to its context (Authors, 2007). But initial designs can fall short of the goal of ensuring that intended users can implement innovations. Implementation data—including data on teachers’ difficulties with using an innovation, feedback from leaders, and student learning data—can all be used to inform revisions of an innovation.

Methods
We present a design narrative (Hoadley, 2002). Design narratives are retrospective, collaborative accounts of design decisions, which is often used in design research to inform both the iterative design of innovation and develop theory (Gravemeijer & Cobb, 2013). Our narrative relies on implementation data (e.g., observations, student learning data) that speaks to teacher learning needs, usability, and student learning.

Results
In 2022, we partnered with teachers to co-design a curricular system comprising a nine-lesson unit for middle schoolers and a week-long professional learning (PL) experience for teachers. The unit engaged students in investigating how games are designed and how to create more inclusive game worlds. During PL, teachers experienced lessons as students and explored issues of representation and justice online moderation of gaming communities.

In two 8th-grade classrooms during the 2023–2024 school year, the unit’s initial implementation met some goals but revealed key challenges. The teacher noted the unit was too long and implemented it post–state testing, limiting time for collaborative discussions on AI ethics. Student artifacts suggested growing awareness of algorithmic bias, but depth was uneven.

Based on these findings, we made several revisions. First, we shortened and streamlined the unit to focus more tightly on AI and human moderation of online gaming spaces. Second, we strengthened lessons on algorithmic bias, adding opportunities to explore sources of machine learning bias throughout. Third, we designed AI partners to support collaboration and used usability data from a preliminary study to create a new introductory lesson that integrated these partners across the unit. We also revised our PL to include instructional practices that help teachers introduce, interpret, and reflect on AI partner output. These changes enhanced teacher learning and supported more effective implementation of the revised unit, which we refer to as the “Moderation Unit” for the remainder of this submission.

Significance
Implementation data can inform iterative redesign of curricular activity systems, and as subsequent papers in this session will illustrate, yield redesigns that enhance usability and teaching and learning outcomes.

Authors

• William R. Penuel, University of Colorado - Boulder

• Mon-Lin Ko, University of Colorado - Boulder

• Nga Hoang, University of Colorado - Boulder

• Brooklyn Clines, University of Colorado - Boulder