Improving ASSISTments: Scaling Up Research Capabilities and Advancing the Platform as a Scientific Tool

• In Event: 54.028 - Efficacy of an Online Mathematics Homework Intervention: Rigorous Research Transforming Knowledge to Action

Abstract

Objective
This paper focuses on efforts to scale up the research capabilities of the ASSISTments platform and expand its use by multiple investigators as a scientific tool for educational research.

Perspective
The gold standard in determining causality, the randomized controlled experiment, has been difficult and costly to conduct in real-world classrooms. Teacher or school level randomization is often complicated to achieve and maintain, and conventional “intervention” research may be perceived by participants as disruptive. The ASSISTments Testbed, an extension of the platform dedicated to research, aims to improve upon the status quo. Online learning platforms can automate randomization at the student-level, delivering different content to students within the same classroom (i.e., different types of feedback). Additionally, when a learning platform is designed to support existing school policies and practices, intervention research can be conducted at scale without intrusion (i.e., through homework assignments).

Modes of Inquiry
Through a mix of methods spanning the fields of education, psychology, human computer interaction, and data science, a new approach to educational research has been devised that leverages the ASSISTments platform. ASSISTments is provided as a free service of Worcester Polytechnic Institute, increasing its accessibility for schools. Teachers already use ASSISTments to assign classwork and homework to over 50,000 students around the world. Over 10 million problems were solved in the 2013-2014 school year, making the platform a valuable tool for learner analytics, especially when considering the capabilities for embedded experimental research. To scale these capabilities, the ASSISTments TestBed has been established as a project that parallels and leverages ASSISTments to enhance educational research by making it simple for researchers to design and implement low cost, high power, noninvasive randomized controlled experiments that feature student level randomization and access ASSISTments’ pre-existing content and user population.

Data Sources
The ASSISTments TestBed is a unique collaborative funded by an NSF SI2 grant that brings researchers together with students and teachers to study effective learning interventions while iteratively improving ASSISTments content and the user experience. Currently, over 130 randomized controlled experiments are running in the TestBed. In parallel, data is being collected to assess and improve how researchers and participants external to the ASSISTments team access and leverage the TestBed.

Results
Recent publications highlight randomized controlled trials conducted using the ASSISTments TestBed with to advance the clarity and implications of educational data mining results (Koedinger & McLaughlin, 2016) and add to the literature in educational psychology regarding pedagogical practices (Fyfe, 2016). Additionally, observations collected from researchers using the ASSISTments TestBed and the Assessment of Learning Infrastructure (or ALI), a data retrieval and analytics tool that provides TestBed users with rich reports detailing the results of their experiment, were recently published to help validate these tools (Ostrow, et al. 2016).

Significance
The ASSISTments TestBed can lower the stakes of educational research by reducing costs, promoting validated universal measures of achievement, and assisting researchers through the process of designing, implementing, and analyzing randomized controlled experiments within authentic educational environments.

Authors

• Neil T. Heffernan, Worcester Polytechnic Institute

• Korinn S Ostrow, Worcester Polytechnic Institute