1. Feedback in Massive Open Online Courses: How It Works and How It Doesn’t

• In Event: Tech-Driven Feedback: Leveraging Technology for Enhanced Learning and Assessment

Abstract

This presentation will report results of two experimental studies that examined the effects of pre-tests, practice tests, and feedback on learning outcomes, persistence, and course completion in a five-week massive open online course (MOOC). In study 1, the participants (N = 399) were adults from around the world who self-enrolled in the American Museum of Natural History’s (AMNH) climate change MOOC (called Our Earth’s Future) offered on the Coursera platform. Participants were randomly assigned to one of four conditions. Learners in the first treatment group took pre-tests without receiving feedback. Learners in the second treatment group took pre-tests and received basic (correct/incorrect) feedback. Learners in the third treatment group took pre-tests and received elaborate feedback. The fourth group was the control. Post-tests were administered to measure learning outcomes.
Results of this study indicated that among all users in a MOOC, pre-tests and feedback did not affect their performance on a test. Furthermore, the presence of pre-tests significantly and negatively affected persistence and completion, deterring many participants from progressing through the course. However, when analyzing sub-samples of the participants, different patterns emerged. For example, among those who did persist and completed the course, those who took pre-tests achieved higher learning outcomes than those who did not. And among those who took pre-tests, there was a positive, cumulative effect of persistence (module completion) on learning outcomes.
In study 2, the participants (N = 1000) enrolled in the same course took practice tests instead of pre-tests, with the rest of the design staying the same. Results showed that participants did improve their performance when offered detailed extensive feedback on their practice tests. The results, however, differed from what we typically see in the traditional educational settings.
These findings highlighted both the challenges and rewards of teaching and learning in massive open online courses. The instructional design implications of these studies are complex. In summary, these studies represents a new contribution to the literature on assessment and feedback, expanding the field to include adult participants from around the world who enrolled in a self-paced, not-for-credit online science course. The results pave the way for future research in this area with this population and have a direct practical application for online course developers, offering them information to help improve student learning outcomes and engagement.

Authors

• Anastasiya A. Lipnevich, City University of New York

• Maria Janelli, Scratch Foundation

• Carolina Lopera-Oquendo, City University of New York

• Mi Jin Park, Graduate Center - CUNY

• Felix Esser, Graduate Center - CUNY