Using AI-Generated Insights to Support Collaborative Decision-Making

• In Event: Making the Invisible Visible: Dynamics, Processes, and Outcomes of Research-Practice Partnerships

Abstract

Research-practice partnerships (RPPs) position practitioners not as consumers but producers of knowledge. Strengthening RPPs presents unique needs to make visible the needs, values, and expertise of different stakeholders. We report on the design of an analysis tool, empowered by large language models (e.g., GPT-4o) and visual analytics, to generate insights from RPP meetings and invite designers, researchers, and stakeholders to reflect upon the design decisions. We examine: How can we design an automated tool to support collaborative decision-making? We generate design conjectures drawing from research on RPP and participatory design. First, the design should highlight diverse perspectives and expertise from participants (Campano et al., 2016; Gutiérrez & Penuel, 2014). Second, given that power imbalance might position researchers as authoritative knowledge producers (Cober et al., 2015), the design should promote equitable contributions. Third, it should demonstrate the fluid roles that stakeholders play (Ishimaru & Takahashi, 2017). Fourth, collaborative design is dynamic, and the tool should trace how participants’ voices and expertise develop in relation to the design goals (Zavala, 2016). We leverage Design-Based Research (Hoadley & Campos, 2022) to document our grounding, initial design conjectures, and two iterations of building and refining conjectures (Sandoval, 2014). The first iteration involved interviews with four designers with extensive experience with RPPs in education. Five educators and designers participated in the second iteration, where we used the tool to analyze real-world design transcripts and uncover insights within participants’ contexts. The interviews were audio-recorded and lasted 45-60 minutes. We report on the initial design goals, the design features mapping onto these goals, and user feedback. The tool processes design meeting transcripts and offers four main features: Idea Summary, Idea Evolution, Talk Duration, and Chatbot Query. Idea Summary synthesizes the key themes from the transcripts and connects them to the speakers, thus highlighting contributions from each participant. Participants can switch to Idea Evolution , to see how themes unfold over the course of the meeting. They can reflect on Talk Duration, including who is talking and for how long, what roles they play, and how they respond to others. Finally, they can attach other documents, such as other transcripts and design materials, in Chatbot Query, to generate insights that facilitate the next design decisions. Each design makes transparent the speakers’ ideas, roles, and interactions in connection to other stakeholders and contributions. Overall, the interviewees remarked that the AI-generated insights were accurate. They mentioned different use cases for these insights: self-reflection to examine one’s contributions, documentation to share design processes with other stakeholders, and generative decision-making by synthesizing insights across transcripts and design artifacts. Manually coding transcripts requires extensive resources and often occurs at the completion of design cycles. More immediate insights about meeting dynamics can help designers and researchers better facilitate RPPs. Compared to commercial products agnostic to the needs of education designers, our tool supports informed, equitable, and shared decision-making at the core of partnerships (Penuel et al., 2020).

Authors

• Ha Nguyen, University of North Carolina - Chapel Hill

• Fabio Campos, New York University