Measuring Students’ Self-Regulated Learning in Teachable Agent Environments

• In Event: Measuring Self-Regulated Learning With Multi-Agent Learning Environments

Abstract

We have developed a computer-based learning environment that helps students to learn about a variety of science topics by constructing causal concept maps. The system builds upon research in Learning-by-Teaching, and has each student take on the role and responsibilities of being the teacher of a virtual student named Betty. The environment is structured so that successfully instructing their teachable agent requires the students to learn and understand the science topic for themselves. The teachable agent’s performance is a function of how well it has been taught by the student, which provides the student with an unthreatening way of assessing their own understanding and areas of confusion. Based upon the student’s level of progress and pattern of activities, the system triggers responses at appropriate times from Betty or a Mentor agent, Mr. Davis, who provides guidance on problem-solving and metacognitive strategies. As a result, the students are more likely to both increase their knowledge of the specific domain content and develop more sophisticated problem-solving and metacognitive strategies, which in turn helps their preparation for future learning.
To effectively design, test, and refine a system promoting SRL skills requires the ability to identify and measure metacognitive processes. The traditional approach to measuring students’ self-regulated learning (SRL) has been through the use of self-report questionnaires. The underlying assumption in these questionnaires is that self-regulation is an aptitude that students possess. However, recently researchers have argued that questionnaires provide valuable information about the learners’ self-perceptions, but they fail to capture the dynamic and adaptive nature of SRL as students are involved in learning, knowledge-building, and problem-solving tasks. Increasingly, researchers have begun to utilize trace methodologies, such as computer logs, to examine complex temporal SRL patterns.
We adopt this approach, and discuss analyses from several studies that were conducted in middle school science classrooms. One of our goals was to determine the degree to which the agents’ metacognitive and SRL prompts could help improve students’ learning. Within this framework, we have developed analytical methods to identify and interpret students’ learning strategies based on their activity traces on the system. Such analyses can shed light on students’ underlying learning processes and the strategies they employ in achieving their learning tasks.
Our analysis of students’ activity sequences captured as computer logs is based on a novel methodology that combines the use of sequence mining methods and the derivation of hidden Markov models (HMMs) to quantify and assess student learning and use of metacognitive and self-regulated learning strategies. In addition, we report the results from verbal protocol analyses to determine students’ acceptance of the strategies discussed by the two agents, and how the feedback provided by the agents influenced their subsequent learning activities.
Our approach to analyzing students’ activity sequences using HMMs has produced good results. We were able to characterize students’ activity patterns into a number of (good and bad) knowledge construction and monitoring strategies. The interpretation of student behaviors with the HMMs also matched the SRL feedback model we implemented in the Betty’s Brain system.

Authors

• Gautam Biswas, Vanderbilt University

• John Kinnebrew, Vanderbilt University

• Kirk Loretz, Vanderbilt University