BackHome
Refresh: Off
Paper Summary
Share...

Direct link:

A Modeling and Simulation Transdisciplinary Approach to the Design of Human Digital Twins for Learning

Sun, April 14, 11:25am to 12:55pm, Philadelphia Marriott Downtown, Floor: Level 4, Room 406

Abstract

Objectives: Human-digital twins (HDTs) are virtual representations of real-life entities that provide real-time user feedback as they serve as a real-time replica of a human (Miller & Spatz, 2022), and may include the cognitive, affective, metacognitive, motivational, social functions and processes that inform decision making, self-regulation, and collaborative efforts during learning and training. The future design and development of these real (robotic) and virtual (avatar) HDTs will transform the way that we think about AI in education and pedagogical agents’ roles within and between digital immersive and real-world learning environments. In our session, we will present a theoretical and empirical based framework for designing HDTs.

Conceptual Perspectives: Pedagogical agents (PA) act as learning facilitators that support learners’ self-regulatory process (Azevedo et al., 2022; Azevedo & Wiedbusch, 2023; Schroeder et al., 2013; Siegle et al., 2023). The future design of some PAs may be HDTs, taking on the role of a (near) peer, tutor, or expert. PAs could become a (near) real-time replica of the learner themselves and be utilized as a modeling and simulation “sandbox” to determine which pedagogical approaches may work efficiently given a challenging topic. An educator’s HDT of could provide after-hours help to students anywhere using specialized and contextualized knowledge of their human counterpart using physiological devices, eye trackers, NLP, including student motivations, goals, (meta)cognitive strategies, and emotion-regulation processes. Our approach challenges current conceptions of PAs and their roles and functions in supporting human learning across tasks, domains, and contexts.

Proposed Method and Data Sources: This research builds upon previous efforts by designing a HDT blueprint from multimodal data collected from learners and educators (e.g., see Azevedo, 2022; Wiedbusch et al., 2023). Learners’ eye movements, physiological responses, behavioral traces, facial expressions, and concurrent verbalizations will determine how the HDT may alleviate extraneous cognitive load and maximize physiological, (meta)cognitive, and behavioral responses by providing the learner with real-time verbal commands, instruction, or scaffolding to optimize self-regulation.

Our team has started modeling and simulating an approach utilizing Unreal’s MetaHumans as a communication agent that relies on GPT 4.0’s large language model and other natural-language processing techniques to inform pedagogical responses (see Figure 1). A learner’s verbal input activates the HDT and other HDTs to collaborate, monitor, regulate, and share the cognitive load associated with key learning goals and instruction. While we continue to design these HDTs, we will begin to incorporate some approaches found in traditional cognitive architectures such as knowledge storage and buffering (e.g., ACT-R [Ritter et al., 2018] and SOAR [Laird et al., 2017]) or metacognitive cycles (e.g., MIDCA; Cox et al., 2016).

Significance & Conclusion: The future of PAs remains a rapidly changing open field in the current digital environments and technology landscape. This research will develop a novel learner and educator HDT prototype capable of conducting after-action reviews using explainable AI techniques to emphasize learners’ strengths, weaknesses, and subsequently propose optimal new methods of planning, communication, collaboration, monitoring, strategizing, and performing using verbal, nonverbal, and holographic simulations to prepare for workforces and educational challenges.

Authors