Ecological Assessment Frameworks for Learning in Computer-Based Learning Environments

• In Event: Innovative Practices for Assessment in Computer-Based Learning Environments

Abstract

The 21st century digital revolution demands a reconceptualization of learning and assessment in computer based learning environments (CBLEs). It is conjectured that CBLEs can provide a means to innovate instructional design and promote learning to prepare students to be cognitively competent, emotionally mature, reflexive, and self-regulating. Fulfilling the CBLEs’ potential requires a well-conceived theoretical framework that successfully articulates the dynamic relationship between learners and digital tasks, and their interaction through a feedback loop. Assessing cognitive growth in such dynamic and complex environments requires re-conceptualized principles of assessment.
In this paper, we discuss assessment principles for learning in CBLEs by highlighting a shift in view of its target measurement construct from discrete content domain knowledge to cognitively high-functioning skills; that is, competencies required to process content-domain knowledge for successful task completion. Secondly, learners and tasks are viewed from an ecological perspective rather than statically. In other words, learners’ current state of knowledge and skill mastery are constantly changing as a result of interactions with elements of CBLEs, contradicting a static view which assumes that learners remain the same, and that tasks should be controlled to avoid measurement error.
CBLEs also provide researchers the opportunity to retrieve multiple types of behavioural, affective, cognitive, and contextual data that can be systematically traced over time. Such information can be applied in various latent class profiling approaches, including cognitive diagnosis modeling (Jang, 2009, 2010; Leighton & Gierl, 2007; Roussos et al., 2008) or covariate-based latent class regression modeling (Vermunt & Magidson, 2005), which may be viable for identifying distinct change patterns attributable to the CBLEs interventions. Triangulating multiple data sources would provide rigorous evidence for warranting the claims about the effectiveness of the CBLEs’ intervention and assessment. As discussed, tasks and feedback are considered crucial variables. A CBLE task can be conceptualized as a covariate in terms of its cognitive complexity, time requirement, and task flexibility, while feedback can be viewed as a mediator in terms of its grain size, timing, intensity, and degree of scaffolding.
We take one specific CBLE, BioWorld (Lajoie, 2009), as a case to illustrate the development of the assessment framework for learning in CBLEs. The development of this framework is grounded in 26 medical students’ think-aloud data from their diagnosis of virtual patients in BioWorld. Each student engaged with three problem-solving tasks, received feedback during and at the conclusion of each, and completed a survey that sought their emotional responses to the feedback.
Using multiple data sources, we identified key covariate and mediating variables for modeling individual students’ growth. Students exhibited a range of cognitive skills demonstrating their engagement with tasks and the creation and evolution of their knowledge structures. Furthermore, students’ use of feedback was examined to determine how it influenced students’ self-regulation. We discuss the viability of the ecological assessment framework for learning in CBLEs in terms of its potential as an evaluation tool as well as an intervention tool to identify the best mechanism for providing students with feedback tailored to individual’s’ profiles.

Authors

• Eunice Eunhee Jang, University of Toronto

• Maryam Wagner, OISE/University of Toronto

• Zhenhua Xu, University of Toronto