Assessing Learning With MetaTutor, a Multiagent Hypermedia Learning Environment

• In Event: Innovative Practices for Assessment in Computer-Based Learning Environments

Abstract

Computer-based learning environments (CBLEs) are complex systems capable of a myriad of advanced functions, including measuring, tracking, and, in some cases, adapting to learners based on information such as their learning assessments (e.g., pre-test scores), experience of positive emotions, and use of effective self-regulatory skills (Azevedo et al., 2013; Kinnebrew, et al., 2013; Lajoie et al., 2013; Shute & Kim, 2013). However, as CBLEs have become increasingly more sophisticated (Azevedo & Aleven, 2013) the methodological reporting and empirical scrutiny regarding their assessments of learning have commonly become a secondary consideration (Mislevy, 2010).
The purpose of this paper is to discuss the ways in which assessments of learning are implemented in MetaTutor, a multi-agent, hypermedia learning environment about several human body systems (Azevedo et al., 2012, 2013). We focus on lessons learned from assessing learning with the system across three different universities and three studies.
The presentation will revolve around three topics regarding the assessments of learning used in MetaTutor: (1) the types of assessments (e.g., multiple-choice page quizzes, and pre and post-tests); (2) their pedagogical characteristics (e.g., text vs. inference-based questions), and alignment with learners’ personally set learning objectives); and (3) their psychometric properties (e.g., sample and data distributions, inter-item correlations, and effective/ineffective distractor items).
The assessment framework will be presented in the context of learning outcomes across three studies (N = 324 students) and three different universities . Comparing results from Study 1 we found that students at University B learned significantly more (F (1, 145) =6.20, p < .05, n2p = 0.04) from MetaTutor than students in University A when prior circulatory system knowledge (pre-test scores) was controlled for. Students from university B not only learned more, but also had significantly higher prior knowledge levels, (F (1, 150) = 47.10, p < 0.01, n2p = 0.24). Although significant differences in learning were not observed in a subsequent study (Study 2) between University B and C, (F (1, 121) =1.94, p > .05, n2p = 0.02), high prior knowledge levels persisted in University B and extended to C. However, University B students had significantly higher pre-test scores (F (1, 122) =6.90, p < .05, n2p = 0.05) than University C students. These results provided evidence of a pervasive skew and ceiling effect in learning assessment measures and prompted us to examine the psychometric properties of the learning assessment items and to subsequently make changes to the level of difficulty of some questions and to re-write others entirely. We will discuss the positive effects these and other changes had on students’ prior knowledge and interaction with MetaTutor in Study 3.
This research is significant because it provides a detailed example of the importance of examining how learning is assessed in CBLEs. More specifically, the methods and results of this presentation can be used to motivate other CBLE researchers to examine the psychometric and pedagogical features of their learning assessments.

Authors

• Jason Matthew Harley, McGill University

• François Bouchet, McGill University

• Niki Papaioannou, Illinois Institute of Technology

• Cassia Carter, Illinois Institute of Technology

• Gregory Trevors, McGill University

• Reza Feyzi Behnagh, McGill University

• Roger Azevedo, North Carolina State University

• Ronald Landis, Illinois Institute of Technology