AI as a Team Member for Designing Educational Interventions

• In Event: Blending Human and AI Feedback: Impacts on Student Engagement and Performance

Abstract

Objective
This study investigates the impact of integrating ChatGPT, a large language model (LLM), into authentic collaborative learning activities for teachers specializing in supporting students with specific learning needs and disabilities. The research aims to understand how ChatGPT, if used as a team member, affects authentic learning and design processes when used at different stages of a collaborative instructional design task.

Theoretical Framework
The theoretical framework draws on concepts of artificial intelligence in education (AIED) (Holmes, 2024; Koedinger & Corbett, 2006; Roll & Wylie, 2016), augmented intelligence (Englebart, 1962; Rouse & Spohrer, 2018), and human-in-the-loop machine learning (Monarch, 2021; Mosqueira-Rey et al., 2023). The study is grounded in perspectives that view AI as a tool to enhance human cognition and decision-making rather than replace it. It builds on work examining how large language models can support various educational tasks like group interactions, formative assessment, and personalized learning (Agostini & Picasso, 2024; Miao et al., 2021; Tamkin et al., 2021).

Methods and Data Source
Using a quasi-experimental mixed-methods design, 103 teachers enrolled in a course on instructional design with ICT for inclusion were divided into small groups of 4-5. All the groups were instructed to use ChatGPT 3.5 as a group member and keep it in the loop of the conversation. Only half of the groups could use ChatGPT, while the other half should do the task without it. Half the groups could use it in the first task, which was to select appropriate ICT tools for their lesson design, while the other half used it during the actual lesson planning process. Data were collected through individual questionnaires administered over three moments (beginning, end of first task, and end of second task). Open-ended feedback was also gathered during a plenary discussion session. Finally, products of each group have been assessed to understand which strategy and use of ChatGPT worked best.

Results and Significance
Content analysis of qualitative feedback revealed several key themes, including reflection on AI use (40% of comments), the use of ChatGPT as an "oracle" for ICT tool selection (20%), and ChatGPT's role in formative design (15.5%). Students reported both benefits and challenges of integrating AI. Benefits included reduced discussion time, facilitation of task structuring, and access to diverse ideas and resources. Results suggest that ChatGPT can serve as a valuable collaborative partner in instructional design tasks, if used as a team member, particularly in tool selection and idea generation phases. However, its integration requires careful consideration of how to balance AI input with human interaction and critical thinking. The study highlights the importance of developing skills in crafting effective prompts and critically evaluating LLM-generated content.
Future research directions include examining long-term impacts on teacher competencies, exploring optimal strategies for human-AI collaboration in educational contexts, selecting the best LLMs for these tasks, and investigating how different prompting techniques may influence learning outcomes. Additionally, further study is needed on how to best prepare teachers to critically engage with AI tools while maintaining a focus on pedagogical goals and student needs.

Authors

• Daniele Agostini, University of Trento

• Anna Serbati, University of Trento

• Federica Picasso, University of Trento

• Anastasiya A. Lipnevich, City University of New York