Identifying the Design Methods of Highly Effective Improvement Science Interventions: Incorporating Them Into Leadership Programs

• In Event: Integrating Improvement Science Into Leadership Programs: Opportunities, Challenges, and Adaptations

Abstract

A key element of improvement science is the design and implementation of an intervention to solve a problem of practice. However, there is little research on what the characteristics of good intervention design are. In addition, research testing the effectiveness of interventions using randomized controlled trial (RCT) trials or quasi experimental methods overwhelmingly finds miniscule effect sizes (Lipsey, et al., 2012; Kraft, 2020; Evans & Yuan, 2020). Lortie-Forgues and Inglis (2019) suggested that these disappointing findings may result from “the insights from basic research on which the trials are based were not adequately translated into an effective intervention…the skills required to successfully translate insights from laboratory research into effective interventions that are possible to implement successfully are relatively rare…” (p. 164). But what are these translation/design skills? An even more fundamental question is: Is translating insights from basic laboratory research the best way to design interventions that can produce noticeable real-world benefits at scale—and if not—what are the best design methods?

Improvement science offers opportunity for leaders and leadership programs to identify key problems of practice and collaboratively design interventions to solve them. This opportunity will be wasted in the absence of new knowledge about how to design highly effective interventions. This paper reports on the analysis of the design process for two very different interventions that were highly effective at scale in solving two very different problems of practice. The first is the Higher Order Thinking Skills (HOTS) project for accelerating the academic performance of Title I and Learning Disabled Students in grades 4-8. The second is the Statway program developed by the Carnegie Foundation for reducing the dropout rate of community college freshman students.

An autoethnography process was used to study the design process that HOTS engaged in over its first 12 years, and an extensive set of qualitative interviews were conducted with the lead designers of Statway. This paper examines the common design elements and processes across these highly successful interventions. This paper will describe research on the effectiveness of these programs and what the common key design elements and processes were. This study identified 13 common design elements across the two interventions. A number of the common design elements were very non-traditional—e.g., neither design was (actually) primarily based on theory or basic laboratory research.

This paper provides a way to explain the small effect sizes in RCT and quasi -experimental research as resulting from interventions whose designs were simplistic and underspecified. Indeed, it appears that several of the characteristics of good design cannot be implemented within the strictures of RCT methodology. As a result, this paper will suggest switching from research methods that seeks to establish causation to ones that seek what the presenter calls “implicit causation” (Author, 2022)—i.e., consistent patterns of replication across contexts which become both the basis of evaluation and of design.

Finally, this paper will discuss how to incorporate these intervention design practices into leadership programs, leadership practice, and improvement science EdD dissertations.

Author

• Stanley Pogrow, San Francisco State University
  Presenting Author