Trials and Tribulations: Challenges and Adjustments in a Multisite School-Based Randomized Controlled Trial

• In Event: Educational Responsibilities in Problem-Based Education (PBE): Principles, Practices, and Networks Creating and Supporting PBE

Abstract

Purpose
This study examines how a team of researchers utilized problem-based learning principles to analyze the opportunities and challenges posed in conducting a multi-site randomized controlled trial (RCT) of an innovative internship experience program for teacher candidates. We discuss how collaboratively we addressed problems in assessing program effectiveness and adjusting the program to align with local realities. Analyses demonstrate how the research team, in consultation with local stakeholders, addressed methodological and program implementation problems in the field.

Perspectives
Growth of randomized controlled trials (RCTs) within educational studies has widely been documented, with over 760 unique studies published in English journals and reports between 2006-2016 (Connolly, Keenan, & Urbanska, 2018). RCTs are the ‘gold standard’ in many disciplines since the random assignment mechanism theoretically allows confidence in attributing group differences to the intervention itself–provided known and unknown differences are controlled for. While RCTs are valuable and enjoy growing use, RCTs within education have received criticism for challenging implementation and for ignoring contextual circumstances (Connolly, Biggart, Miller, O'Hare, & Thurston, 2017). In light of this, it has become clear that open and honest conversations about solutions, as well as failures, are needed.

Methods and Data
The Teacher Quality Partnership (TQP) program is a five-year U.S. Department of Education funded pre-baccalaureate program designed to recruit, prepare, and sustain urban teachers. Proposed as a randomized block design, the project was conceived to have multiple layers of random assignment.
This study uses evaluation coding, domain analysis, and statistical analyses of daily activity logs, observation rubrics, and interviews with administrators, teachers, and teacher educators to illustrate problems faced in RCT and program implementation. The team worked closely with both stakeholders and participants, continuously updating their research design to better accommodate emerging circumstances and unique, local demands.

Results
RCT implementation challenges included, among others, lack of clearly defined intervention, inadequate numbers and procedural barriers for block randomization, and treatment diffusion. Using the problem-based education principles of engaging in locally situated and collaborative problem-solving, we show the challenges faced and the solutions the team developed. Challenges included inadequate numbers resulting from the district’s desire to meet the needs of multiple schools, and postsecondary institution’s needs for placing interns. Treatment diffusion resulted not from ill intentions but just the opposite—the desire of the interventionists to provide support, amplified for interns in high-need schools, and recognition that the partnership with the school and supervising teachers need nurturing to develop over time. Solutions revolved around RCT implementation and the need for a collaborative, iterative, adaptive research designs to account for professional concerns and contextual circumstances of partnering schools and district.

Significance
This study experienced the same challenges that have made scholars critical of RCTs in the past (Connolly et al., 2017), but we worked to adapt to local realities. Thus, our presentation outlines some of our more challenging experiences in conducting an RCT and demonstrates solutions to some of the practical barriers to RCT implementation, which may be useful to others conducting RCTs in schools.

Authors

• Debbie L. Hahs-Vaughn, University of Central Florida

• Mary E. Little, University of Central Florida

• Christopher D Charles, University of Central Florida

• Christine D DeStefano, University of Central Florida