Investigating the Value of Evidence-Based Instructional Design for Online University Courses

• In Event: Cognitive Science Goes to College: Scaling Up What We Know About Learning and Instruction

Abstract

Objectives or purposes

The purpose of the study was to investigate whether applying a set of evidence-based instructional design principles to online university courses would improve student outcomes, including learning, retention, and satisfaction.

Perspective(s) or theoretical framework

As noted in the introduction to the recent Handbook of Research on Learning and Instruction (Mayer & Alexander, 2011), there have been significant advances in the research on how people learn (learning science) and how to help people learn (instructional science) in the past 20 years. The Pittsburg Science of Learning Center’s (PSLC) Knowledge-Learning-Instruction framework (Koedinger, & Corbett, & Perfetti, 2010), integrates principles and practices for which we have the most robust evidence. We applied principles from the PSLC framework related to the amount and type of guidance, practice and feedback, motivational support, and multimedia use, to redesign three high volume courses at a large online university. We compared the new evidence-based designs with the existing versions of the courses which consisted primarily of textbook readings, discussions, and written assignments.

Methods

We used a quasi-experimental design which assigned approximately 700 students and approximately 20 faculty to control and experimental versions of 3 courses delivered over two 10-week terms. Dependent variables were student learning (on common assessments), withdrawal and retention rates, grades, student satisfaction, and instructor variability in student outcomes. Learning and motivation process data were collected from the experimental group to help explain differences in outcomes.

Data sources

Data came from two sources: 1) existing university databases that track student background variables, course outcomes, scores on graded activities, student activity time, and 2) a new database that tracks time, perception and performance data in the experimental courses. Online content in the new courses was delivered in a guided learning sequence of Prepare, Practice, and Perform for each learning (performance) objective (see sample screen below). Instructor training included a written guide, eight hours of training, and weekly coaching sessions during the term.



Results

We anticipate that student outcomes (learning, retention, satisfaction) will be significantly better in the experimental versions of the courses, and that the results will be consistent across multiple courses and multiple instructors.

Significance of the work

The work is significant in that it tests evidence-based instructional principles at scale in entire courses with hundreds of students. The instructional techniques we incorporated were validated, for the most part, in short duration lab studies with smaller samples. We were able to leverage the scale and speed of a large online university to demonstrate their impact in a real-world setting.

Authors

• Brenda Mary Sugrue, Kaplan

• Erik Ellefsen, Kaplan Inc.