MetaTutor as an Innovative Technology Environment to Assess Students’ Self-Regulatory Processes

• In Event: Knowing What Students Know and Feel: Innovative Technology-Rich Assessments

Abstract

Objectives/Purposes: Understanding the complex nature of cognitive, metacognitive, motivational, and affective processes during learning with multi-agent learning environments is key to understanding how these processes impact learning about complex domains. Our approach has been to use MetaTutor (an intelligent, hypermedia multi-agent learning environment) as an innovative technology-rich assessment tool with which to collect trace data of cognitive, metacognitive, and affective processes during learning.

Theoretical Framework: We integrate Winne and Hadwin (2008), Zimmerman and Schunk’s (2011) socio-cognitive model of SRL, and Pekrun and colleagues’ (2006, 2010) control value theory of academic achievement emotions. From a measurement perspective, we treat SRL as an event and therefore make the fundamental assumption that cognitive, metacognitive, and affective SRL processes can be detected, traced, and modeled during learning with MetaTutor. By integrating these models of SRL, we are able to use technology rich assessments to measure and better understand what students know and feel and why they behave in so many different ways.

Methods: 72 college students took part in a two-day experiment with MetaTutor to learn about the circulatory system. Participants were randomly assigned to one of three conditions—control (no prompt or feedback), prompt (receive only prompt), or prompt and feedback (receive both).

Data sources: During the two-hour lesson session with MetaTutor, we collected the following data from each participant: concurrent think-alouds, eye-tracking, video recording of the face, text log files and notes and drawings. We also collected pretest and posttest data and several self-report measures on agent likeability and metacognitive knowledge about specific SRL processes.

Results: Results will focus on the technology-rich assessments, and describe, using multiple processes, participants’ learning experience with MetaTutor. In order of granularity, micro-level data provides information on: (1) fluctuations in affect, (2) eye-tracking processes which reveal participants’ selection, organization, and integration of multiple representations of information (which are indicative of cognitive and metacognitive processes), and (3) log-file data which details the duration and sequencing of specific behaviors (e.g. navigational profiles). Mid-level data (1) represents learners’ accuracy in making metacognitive judgments; (2) provides information on the deployment of cognitive and metacognitive processes from the concurrent think-aloud protocols; (3) illustrates their emotion regulation during different phases of learning; (4) provides information on their regulatory processes associated with adaptive changes during the learning session; (5) reveals their knowledge integration across representations of information; and, (6) exemplifies changes in their self-regulatory processes based on learner-agents dialogue moves. Macro-level data provides information on changes in students’ learning based on their pretest-posttest scores.
Scientific contribution: The results will demonstrate the advantage of using technology-rich learning environments to advance our understanding of what, how, when, and why students’ know and feel during complex learning episodes. The data sources will provide evidence that has the potential to advance current conceptual, theoretical, methodological, and analytical frameworks related to SRL processes. These advances will in turn allow researchers to design more effective multi-agent learning environments that are sensitive and responsive to students’ cognitive, metacognitive, and affective needs during learning.

Authors

• Roger Azevedo, McGill University

• François Bouchet, McGill University

• Reza Feyzi Behnagh, McGill University

• Jason Matthew Harley, McGill University

• Melissa Duffy, McGill University

• Gregory Trevors, McGill University