Lessons Learned About Designing a Randomized Controlled Trial for the "Real" World

• In Event: Math for All: Lessons Learned From Piloting a Randomized Controlled Trial in a Large Urban District

Abstract

The MFA study is a cluster RCT with schools serving as the unit of analysis and treatment; this is because teachers are to collaborate their instructional efforts under coordinated leadership. MFA’s focus on collaboration requires general and special education teachers to work together in a planned and coordinated manner, rendering infeasible a design that assigns teachers within schools.
Tables 1 and 2 summarize the research questions, measures, and planned analyses. Figure 1 displays the draft CONSORT diagram for the study, and Figure 2 illustrates the decision points for determining a school’s eligibility.
Lessons learned from the pilot
• Life is what happens when making design plans. Although a strong study design was conceptualized that would meet What Works Clearinghouse (WWC) evidence standards, aspects of the plan needed to be revisited because the district in which we are working requires all teacher PD to occur on Saturdays (this was not clear to any parties during initial negotiation). This means that full teacher participation cannot necessarily be expected among schools randomly assigned to the treatment, as this would require that they willingly give up part of their weekend. A number of design options were conceptualized to handle a possible scenario where teacher participation is low, but for now we believe we can maintain the original design by attending to teacher participation rates during recruitment and identifying a lower threshold for PD that will still allow for school-level inference. We will describe the importance of remaining flexible with design plans while maintaining study rigor.
• Be prepared to highlight the importance of being assigned to a control condition. Early recruitment efforts have found that many teachers are not willing to join the study if there is a chance that they may not be in the MFA treatment. We think this is in part because the phrase “control condition” denotes the idea that nothing will happen with respect to mathematics instruction among schools that form the counterfactual. But practically any trial in education settings essentially entails comparing two treatments (a new approach versus normal instructional routines), yielding a policy-relevant question: is MFA yielding better outcomes than your current approaches? This is a question of inherent interest to teachers in the district, and of course the ethics of withholding treatment from students are addressed by the fact that we do not yet know if MFA can yield learning outcomes superior to the best instructional efforts taken to date. Hence, the messaging around design and recruitment has proven to be of critical importance.
• A priori statistical power is fluid when recruiting. We operate in a large district with a wide variety of schools. Hence, recruiting is an organic process and we found it necessary to routinely revisit the power implications for recruiting one school over another as we consider school size, teacher volunteer rates, and the likelihood that MFA can be implemented as intended. As such, we will describe aspects of power that are largely static and what needs to be continuously revisited during live recruitment.

Authors

• John H. Hitchcock, Indiana University - Bloomington

• Teresa Garcia Duncan, ICF International

• Rosemarie O'Conner, ICF International