Using AI to Support Generalizability of Research

• In Event: Toward Responsible Use of AI in Education Research and Development

Abstract

Generative AI holds the promise of helping us answer questions about the generalizability of findings in educational research and about the impact of personalization. The social sciences have been wrestling with the problem of replication (Perry et al., 2022). Failures to replicate may reinforce practitioners’ beliefs that their own educational environment is unique and that there is limited value to using research findings for practical guidance. In this presentation, we will describe the potential for generative AI to address the replication crisis and contribute to a more robust knowledge base to inform the work of educators.
Generative AI dramatically reduces the cost of creating content to be used in field testing and can automate content in ways that allow us to study the impact of personalization at a large scale, with diverse groups of learners. We will discuss several examples based on a partnership between developers and researchers.

We have been testing the impact of word problem phrasing on the math performance of students with reading difficulties. We rewrote and field-tested word problems in two mathematics topics with over 12,000 students from diverse districts. In one, students identified as having reading difficulties were able to complete the topic in 30% less time while improving on their mathematics performance. To see if the result generalizes, we are using generative AI, trained on our word problem style guide. Our process involves iteration on prompts to the LLM that better match the style guide and across mathematics topics.

Another experiment prompts students for their interests and generates a word problem that reflect those interests. Students solve the problem with AI-based support. One focus of the research is the extent to which the act of authoring problems is essential to the process. Do self-authored problems produce a larger impact on students than problems authored by other students? We are also exploring the use of AI for feedback. We are using AI to refine a chatbot prompting strategy; when students ask for help, they get randomly assigned to a particular version of the chatbot. Using “multi-armed bandit” algorithms, we use student feedback and performance to automatically shift the probability of assignment to the best-performing chatbot version. Future work will identify both problem and student characteristics that favor one version of the chatbot over another.

Finally, we are field testing the use of generative AI in video-based instruction. We produce videos that are personalized to the student, including responding to the specific errors that the student made and the student’s preference for instructor avatar. The technology has advanced to the point where generated videos are approaching the quality of filmed segments.

These examples will help the audience understand specific ways in which generative AI can be used to improve the quality and reach of education R&D. During the Q&A, we will engage with the other panelists and audience members to explore additional use cases and discuss the benefits and potential drawbacks of these tools.

Author

• Steven Ritter, Carnegie Learning, Inc.