On-the-Fly Context Personalization with Generative AI in Mathematics Assessments: Co-Designing with Teachers

• In Event: Designing AI-Powered Personalized Educational Tools by Incorporating Teachers’ Perspectives

Abstract

Objectives

We have developed a digital assessment prototype called Context-AI, that leverages generative AI to personalize the context of the mathematics story problems during the assessment. Our goal with Context-AI is to effectively engage all students to better assess what they know and can do and to help them develop and maintain interest in mathematics. Here, we aim to co-design with teachers to ensure their perspectives are reflected in Context-AI.

Theoretical and Empirical Rationale

Traditional, one-size-fits-all assessments are increasingly less culturally relevant for most students, leading to lower motivation, disengagement, and poorer performance among those from traditionally underserved students (Bennett, 2023). Previous approaches of personalization in mathematics investigated adjusting the context manually, in advance, without giving students agency to personalize the context of the problems (e.g., Sinharay & Johnson, 2024; Walkington & Bernacki, 2019).
Context-AI leverages generative AI (GenAI) to allow on-the-fly context personalization of classroom assessments at scale and gives students agency through choice (see self-determination theory; Ryan & Deci, 2000; and the Four-Phase Model of Interest Development; Hidi & Renninger, 2006) to personalize the context of mathematics problems based on students’ cultural identity and interests during the assessment in a way that personalization can accommodate the heterogeneity within diverse groups of students. Our research questions are:
1)What do teachers think about giving students agency about context personalization?
2)What are the teachers’ perspectives about Context-AI and its expected effects?

Methods

Two online focus-group sessions were conducted with n=5 grade 8 mathematics teachers (1 male, 4 female) who taught in public schools in New Jersey, New York, and Philadelphia in the United States. The teachers filled in a survey questionnaire before the focus group sessions. The teachers were racially diverse and had an average of 15.2 years teaching experience. We conducted a thematic analysis of the interview data identifying themes related to the benefits and challenges with using a technology like Context-AI.

Data Sources

One of the survey questions was whether they think middle school students should or should NOT have choice on context of math story problems. During the focus group sessions, teachers were presented with a Context-AI demo. They were asked if they would expect any difference in student engagement with the assessment or performance on it along with other questions.

Findings

Four teachers responded to the survey question as middle school students should have choice on context of math story problems (one teacher responded as maybe). During the focus group sessions, teachers agreed that Context-AI is an efficient and effective formative, classroom assessment tool to increase students’ motivation, engagement, or performance (Table 1). Teachers also noted several challenges with using this technology (Table 2).

Significance

Our initial investigation of teachers’ perspectives indicates that giving students agency to personalize mathematics story problems and leveraging GenAI for on-the-fly context personalization perceived as beneficial and effectful for increasing students’ motivation, engagement, or performance. Teachers also suggested some challenges to be addressed. This study emphasizes the importance of involving teachers as co-designers when designing classroom solutions that leverage technology.

Authors

• Burcu Arslan, Educational Testing Service Global

• Caitlin Tenison, ETS

• Reginald Marshall Gooch, Educational Testing Service