A Clearer Picture: Integration of Multiple Data Sources in MOOC Research

• In Event: Data-Intensive Approaches to Studying Engagement in Education: Exploring Their Current Potential

Abstract

In an effort to affordably scale self-directed learning to larger audiences, Massive Open Online Courses (MOOCs) have been implemented by many education providers, including universities and professional development organizations. Unfortunately, most MOOCs have been fraught with learner disengagement challenges (cf., Jordan, 2014; Milligan & Littlejohn, 2014), while the distal nature of online learning has made it difficult to understand the challenges and needs of learners. Prior studies have examined patterns in log trace data, assessment data, or self-report data in depth, but limited integration of data sources has made interpretation difficult (Bergner, Kerr, & Pritchard, 2015; DeBoer, Ho, Stump & Breslow, 2014). Do long periods of time between clicks mean a user was reflecting on content or taking a break to eat a snack (or both)? Why do some people who intend to earn certificates drop out, and why do some people who only intended to explore end up completing?

Only recently has the field started to triangulate multiple data sources over time to form stronger conclusions about goal-oriented behavior and self-regulation (Bauer & Shanahan, 2007; Littlejohn, Hood, Milligan, & Mustain, 2016). The presenters will discuss recent experiences integrating multiple data sources in distal professional development MOOC research at the [institution], including various combinations of pre-course surveys, weekly surveys, end-of-course surveys, focus groups, follow-up surveys (months afterward), pre-course knowledge assessments, during-course assessments, participant generated artifacts, end-of-course projects, certificate progress and completion, and various resource use metrics from trace data. Special attention given to strengths, weaknesses, and challenges of using multiple measurements in different phases of a project.

Discussion of strengths will include topics such as resolving discrepancies between intentions (indicated through self-report) and behavior (indicated through trace data), examining different types of participation, and capturing a more holistic picture of engagement over time. Some specific project examples include identifying learners that changed their goals mid-course through the triangulation of self-report and trace (log) behavioral data, understanding the impact of badging systems by looking at both participation quantity and quality, tracking self-reported engagement throughout each unit of courses, and combining multiple participation quantity metrics (frequency, variety, and duration) to better understand learner’s engagement trajectories.

Discussion of limitations and challenges will include topics such as: participant compliance with self-report requests, data analytic time-efficiency, random error sources and typical logistical challenges, and analytic challenges of integrating multiple data sources. Some specific examples from projects include, selecting the most important surveys for your research, shortening surveys to reduce time, developing abbreviated screeners and assessments based on pilot data, collecting important meta-data information from course administrators, and working around competing opinions and goals of stakeholders of parent institutions and funding agencies.

The presentation will conclude with a discussion of future directions for research and application in MOOCs and, more generally, any research studying goal-directed behavior and self-regulation. By the end of the presentation, the audience should be informed and encouraged to tackle new problems in their own research by integrating multiple data sources.

Authors

• Eric N. Wiebe, North Carolina State University

• James H Creager, North Carolina State University