When Research Proposal Meets Reality: Challenges of a Multiyear National Randomized Trial

• In Event: The Challenges and Successes of Conducting Large-Scale Educational Research

Abstract

Objectives/Perspective. This project brought together researchers from our institution and five university partners with two aims; 1) to redesign an existing middle school math curriculum by applying research-based cognitive principles, and 2) to test the efficacy of the redesigned curriculum in a large-scale randomized control trial. Over the two-year study of the revised materials, the team overcame significant challenges in a number of areas including recruiting, offering professional development to geographically diverse participants, retaining participants over multiple school years, managing and processing massive amounts of paper data, and addressing missing data in our analyses.

Methods/Data Sources/Results. Recruitment challenges began after the first year of the study. When the project was initially conceived, the study was to take place in two large, urban school districts. However, during the lag between proposal writing and preparing the revised curriculum, there were shifts in district administration and priorities leading one district to opt out of the study and the other to participate in a more limited capacity than initially planned. These shifts led us to recruit nationally and our final sample involved teachers across 22 states.
Once recruitment went national, offering face-to-face professional development institutes was no longer feasible. To address this concern, the project team created a synchronous, online version of the institute that allowed teachers to engage in real time with teachers from around the country. The professional development sessions were designed to be interactive, with teachers engaged in polls, Q & A, and breakout sessions.
As adopting any new curriculum takes time, our study design involved a “practice year” in which teachers became comfortable with the new versions of the materials and approaches to classroom practice. However, as this necessarily requires multi-year participation, participant retention was an additional challenge. Schools and districts routinely re-assign teachers to other subjects and grade levels over the summer and many participating teachers became ineligible to continue as a result. In addition, the publisher of the curriculum we revised released a new version after the study revisions had already been created, and many schools opted to use this newer version, becoming ineligible to continue with the study. Some participants left for study-related reasons: they were unable to attend the professional development workshops or found the process of completing logs and returning data cumbersome.
Finally, data collection issues emerged due to the large-scale nature of the work. As we were required to collect massive amounts of paper data, (over 1,000,000 sheets of paper), we leveraged technology including bar code scanners and automated visual recognition software to speed data entry. Further, at the end of the study many districts were unable or unwilling to provide student level demographic data resulting in non-random patterns of missing data that we needed to account for in our analyses.

Significance. Our poster will further detail the challenges of the center and share our solutions and lessons learned to inform future large-scale research efforts.

Authors

• Jodi Davenport, WestEd

• Yvonne Kao, WestEd

• Aleata Hubbard, WestEd

• Kimberly Viviani, WestEd

• Kathleen L. Lepori, WestEd

• Steven Arnold Schneider, WestEd