Assessing the Disengaged Behaviors of Student Teachers With Network-Based Tutors

• In Event: Data-Intensive Approaches to Studying Engagement in Education: Exploring Their Current Potential

Abstract

Student teachers’ engagement is hypothesized to influence their efforts to regulate their own learning of the affordances of educational technologies and, subsequently, their use in the classroom to enhance student learning (Poitras & Fazeli, 2016). We demonstrate the applications of educational data mining techniques in assessing student teacher disengagement while designing lessons in the context of an intelligent tutoring system.

The existing body of research on engagement suggests a multifaceted construct, with behavioral, affective, cognitive, motivational, cognitive-behavioral, and social-behavioral dimensions (Fredericks, Blumenfeld, & Paris, 2004; Pekrun & Linnenbrink-Garcia, 2012). A great deal of previous research into student engagement from the perspective of intelligent tutoring systems (for a review, see Baker & Rossi, 2013) has focused on the development and applications of detectors of disengagement, including off-task behaviors (talking to other students about unrelated topics, disrupting others, and web surfing), gaming the system, carelessness, and WTF behaviors (Without Thinking Fastidiously). This had led to the development of automated detectors to gain real-time insights into the challenges faced by students and how tutors should intervene to support them in achieving instructional goals.

Data were collected from 68 student teachers who created a lesson plan while navigating the web with nBrowser, an intelligent web browser with tools designed to support them in their learning of the affordances of educational technologies. Student teachers completed the task during a 1-hour session under one of two conditions: dynamic network or static network. The dynamic version of nBrowser continually updated the underlying representation of the domain model on the basis of student teachers’ ratings for the usefulness of online resources. In contrast, the static version of the system did not update the network of online resources.

We conceptualize behavioral engagement with network-based tutoring systems in terms of observable indicators that are suggestive of investment, effort, and thoughtfulness (see Table 1). The most striking result to emerge from a series of studies reviewed here is that student teachers seldom report the usefulness of online resources in designing a lesson plan, which we conceive as a form of off-task behavior synonymous with inactivity/non-compliance with instructions. The teachers rated 57% of online resources harvested from the web and featured on the network, and 8% of all websites visited during a session. Table 2 shows the learner behavioral detection models trained to detect usefulness ratings in the event of such off-task behaviors.

We discuss the implications of the ensemble model for the performance of the recommender system in nBrowser and how to remedy issues related to student teachers’ off-task behaviors. Our discussion will examine the challenges inherent to conceptualizing engagement on the basis of log trace data and the development of detectors that are scalable to a broad range of learning environments and situations.

Authors

• Eric G. Poitras, University of Utah

• Tenzin Doleck, McGill University

• Lingyun Huang, McGill University

• Shan Li, McGill University

• Susanne P. Lajoie, McGill University