Formative Experimentation for Educational Innovation

• In Event: Entrepreneurship in Education: An Academic Debate

Abstract

What is the difference between a fad and an innovation? An innovation has staying power that systematically improves benefits for adopters, while a fad may hold promise but soon fades. Thus, one difference between a fad and an innovation is the evidence that supports the promise of an idea. Further, research plays an important role in separating fads from innovations, because it helps to distinguish between ideas that produce outcomes and those that do not.
But research can play another role too, that of helping to advance and refine innovations during their development process. Thus research may become a tool to systematically refine ideas by bringing the science of systematic and continuous improvement to the innovation development process.
In this symposium presentation, I will explore the role of experimentation in education innovation. Many view formal experimentation – with the random assignment of subjects to treatment and control conditions – as a situation reserved for testing the effects of mature, stable interventions that needs a final rigorous assessment before they can be scaled up or stamped with a thumbs up or thumbs down by the What Works Clearinghouse. Experiments of this type are often ungainly affairs because they require rigid adherence to strict conditions, withholding treatment from a subgroup of participants, within which any changes to the pre-specified treatment once the experiment has begun is taboo. Their purpose is to determine the ultimate effect of an intervention, not to provide useful feedback to interveners that can be used to improve the intervention.
In this presentation I am interested in examining experiments in a different light. Can experimentation be used more intimately as a mechanism for intervention improvement? What value do they provide and in what ways does a more formative use compromise the experiment? In what ways are the contrasts provided by an experiment helpful in providing feedback and adjusting program design? Do program changes really compromise the experimental situation? In essence, can experiments be used for program learning?
First, the presentation will briefly summarize the dominant position on the role of experimentation in educational research and contrast it with a parallel movement of design experiments, in which the term ‘experiment’ means a very different thing. Merging these two conceptions, I conceive of a new way of thinking about experimentation in research, which I call formative experimentation, whose goal is as much to provide feedback to interveners for the purpose of learning and improvement as it is to provide summative evidence of impact.
Second, using a case study experience of an experiment I have been engaged in with a local district, I explore the opportunities and constraints of developing an intervention within an experimental design. I follow the program adjustments over a year in the program development process and identify two sources of learning that provoked program redesign: problem-driven changes and data-driven changes. Using this case, I explore the merits of formative experimentation as an innovation improvement device.

Author

• Jonathan A. Supovitz, University of Pennsylvania