Enhancing Design Through Instructor-Centered Approaches to Experimentation

• In Event: 81.025 - Innovating MOOC Pedagogies

Abstract

Objectives
Our goal is to help instructors figure out how to provide useful feedback messages and explanations to students who are learning by solving online problems. We aim to lower the barriers for instructors to systematically test out hypotheses about which feedback messages or explanations are helpful for students, by conducting randomized experimental comparisons and obtaining data about what students find helpful. Moreover, to ensure that data from such randomized comparisons has practical impact, we explore the use of algorithms from statistical machine learning to automatically analyze data and provide the better conditions (messages or explanations) to future students. We therefore aim to investigate an instructor-centered approach to designing tools for experimentation, that reduce the barriers (e.g., time and programming experience) for instructors to do randomized comparisons, and that increase the practical impact of randomized comparisons by ensuring the data collected can be used to improve instruction for future students.

Perspectives
Our approach takes inspiration from work in human-computer interaction and education on action research (e.g., McLean, 1995) and design-based research (e.g. Barab & Squire, 2004; Penuel et al., 2015). We help instructors solve pedagogical problems, as they take a research-oriented approach to formulating questions and hypotheses, and collecting data about what works in their own classrooms. To rapidly turn data from experiments into practical improvements, we also draw on work in statistical machine learning (Chapelle & Li, 2011) for algorithms that analyze students’ ratings of which conditions are helpful, and present conditions to future students in proportion to the evidence they are higher rated.

Methods
We explored this instructor-centered approach to experimentation by designing and deploying a proof-of-concept tool for experimentation on components of digital problems, with three university instructors who conducted randomized comparisons. They used the interface to author experiments on explanations, hints, and learning tips, as well as to interpret the data from the experiments. We deployed this tool in three on-campus courses (the focus of this paper), and more recently in a MOOC environment.

Data Sources
We evaluated the system by collaboratively deploying experiments in the courses of three mathematics instructors. The first author observed the instructors throughout design and deployment, and conducted semi-structured interviews with instructors upon completion of the experiments.

Results
The interviews provided evidence that the tool lowered the barriers to experimentation substantially in making it possible for instructors to deploy experiments, helped instructors reflect on how to improve pedagogy, and provided a data-driven pathway for enhancing their online problems and helping future students.

Scientific Significance
To help realize the promise of experimentation to help instructors enhance the design of digital educational resources, this paper presented design guidelines for more instructor-centered tools for experimentation. These reduced the programming knowledge and time that instructors need to conduct experiments, and helped instructors more rapidly use data to enhance the learning experience of the next student, enabling a systematic approach to helping instructors test design decisions on components of online resources.

Authors

• Joseph Jay Williams, Harvard University

• Anna Rafferty, Carleton College

• Dustin Tingley, Harvard University

• Andrew Ang, Harvard University

• Walter Lasecki, University of Michigan - Ann Arbor

• Juho Kim, Korea Advanced Institute of Science and Technology