Large-Scale Collaborative Science: The ManyClasses Approach to Experimental Educational Research

• In Event: Increasing the Credibility of Educational Research

Abstract

Complex questions require complex teams that leverage the expertise of professionals across a variety of disciplines (NRC, 2015). The science of education is no exception. To understand which instructional practices work for whom and under what conditions, projects need to expand beyond a single investigator and beyond a single educational context (https://ies.ed.gov/director/remarks/12-17-2018.asp).

The goal of ManyClasses is to provide a large-scale collaborative approach to education research in which we investigate the generalizability of educational practices in a wide range of contexts. One collaboration is among researchers with different areas of expertise. The core ManyClasses team includes six researchers with experience teaching various courses and with expertise in psychology, education, data science, and technology. Another collaboration is among researchers and teachers. One unique goal of a ManyClasses study is to maintain the rigor of a randomized experiment while also allowing teachers the flexibility to prepare materials that are authentic to their discipline. The first ManyClasses study launched in Fall 2019 and was be represented in classes from 17 disciplines ranging from agriculture to business, online and in-person, and with enrollments ranging from 20 to 140 students. A third collaboration is among researchers and administrative officials. The first ManyClasses study was at four universities across eight campuses. The multi-institutional nature of the project requires precise attention to scientific integrity, student rights, and data security. As such, our current collaborations have included data stewards, chief information security officers, and associate general counsels.

There are at least three factors that make the current ManyClasses collaborations successful: transparency, communication, and perspective-taking. For large-scale education research to work, study protocols need to be clear to all parties involved. To that end, we created an osf webpage that documents the IRB protocol, the analysis plan, and instructor resources (https://osf.io/q84t7/). Further, the manuscript, which outlines the method and data collection details, was submitted as a registered report (Fyfe et al., 2019). Open communication is also a key priority. This includes biweekly meetings for the research team, a series of open webinars, calls for applications via twitter and listservs, and a central contact for instructors. Finally, perspective-taking has shaped the ManyClasses approach. Conducting an experiment in authentic courses has ramifications for many different people. For scientists, the experimental design increases the credibility of the findings. For teachers, changes to course materials influence course evaluations. For students, the target manipulation influences learning outcomes. For university personnel, sharing of student data influences legal and security issues.

In sum, the ManyClasses approach to education research is a unique large-scale collaboration that answers the calls for team science (e.g., NRC, 2015), open science (e.g., Peters, 2009), and systematic replications (https://ies.ed.gov/director/remarks/12-17-2018.asp). This session will share ideas for how researchers can engage in large-scale collaborative science, like ManyClasses, as well as present an argument for how such collaborations yield more-credible knowledge in the field of education.

Authors

• Emily R. Fyfe, Indiana University

• Joshua R de Leeuw, Vassar College

• Paulo Carvalho, Carnegie Mellon University

• Robert Goldstone, Indiana University

• Janelle Sherman, Indiana University

• Ben Motz, Indiana University - Bloomington