Practical and Scientific Benefits of Experiment-Guided Instructional Design of MOOClets and Educational Modules

• In Event: 65.063 - Theory-Driven Design of Online Learning Systems, Data Structures, and Analytic Techniques

Abstract

Objective
Differences between online education platforms pose challenges for coherent scientific and practical progress. This presentation examines experiments conducted across several online learning platforms to consider whether shared insights might be obtained by focusing on randomized micro-experimental comparisons that manipulate features of modular components common to most online learning (Williams, 2013).

Theory
Research of platforms or massive open online course (MOOC) involves complex interactions of its specific pedagogical assumptions and technological constraints that it might require design-based research (Brown, 1992) yielding conclusions specific to the context. However, a critical affordance of a digital online environment is that its components can be randomized at a very fine grain, and large-scale participation allows statistical power to detect subtle differences, providing a broad range of potential in vivo experiments (Koedinger, 2011). The challenge is therefore to delve into the components of a MOOC platform to identify “MOOClets” that have the right scope and structure for systematic relationships between design choices and learning.
The presentation explains the process and criteria used to search for these target educational modules or MOOClets. An analysis of multiple platforms – Khan Academy, Assistments, Intelligent Tutoring Systems, EdX, Udacity, & Coursera MOOCs – was conducted with a search for modular components/MOOClets that were ubiquitous within and across platforms, elicited reliable measures of learning and engagement from participants’ interactions, and contained features that were technically feasible to modify. The search for appropriate modules was also guided by a review of published cognitive and social psychology experimental findings (e.g., Dweck, 2008), which provided insights as to how features of modules could be changed to influence educational outcomes across variations in context.

Methods
The module considered in this work is an individual exercise or problem. The design involves randomizing types of motivational information and reminders students are provided with – none at all, vs. broadly encouraging, vs. emphasizing that intelligence is malleable (Dweck, 2008) – to influence engagement in attempting, persisting, and accurately solving exercises.
The experiments using this module include varying which motivational messages appear above mathematics exercises on the www.KhanAcademy.org platform, and how feedback was incorporated after unsuccessful problem attempts on www.Assistments.org. Across all contexts, the effects of information about intelligence being malleable through effort were compared to both broadly positive and encouraging messages and to nothing at all.

Data, Results & Significance
The choice of the individual exercises as a module/MOOClet furnished a paradigm with rich data about engagement: the time students spent, how many problems they completed, repeated attempts after initial failure, requests for hints, measurable progress in learning, and success/accuracy rate. Benefits for engagement were found across experiments and platforms from messages emphasizing the malleability of intelligence, but not for merely encouraging messages, which did not differ from no messages at all, although the magnitude and nature of these benefits did vary.
This presentation provides a theoretical and conceptual contribution by formulating an approach to identifying MOOClets or modular components that are shared across online platforms, and which are sufficiently circumscribed to support experimental research to identify systematic instructional design principles.

Authors

• Joseph Jay Williams, Stanford University

• Jascha Sohl-Dickstein

• Neil T. Heffernan, Worcester Polytechnic Institute

• Candace Thille, Stanford University

• John Mitchell