Measure Once, Cut Twice: Using Data for Continuous Improvement and Impact Evaluation in Summer Programs

• In Event: 47.082 - Digging Deeper: Using Mixed-Methods Research to Assess Promising Practices in Out-of-School Time

Abstract

Frustration and confusion often occur when practitioners require detailed information about programmatic functioning for continuous improvement while policy-makers require evidence of impact for accountability and funding. Impact studies are often preferred over continuous improvement studies, but they seldom offer useful information to practitioners. Per the conference theme, this situation leads to a social science worldview that emphasizes the limitations of social science method for achieving practical purposes and welcomes arbitrary decision making (i.e., Type-2 error) in the absence of better arguments. The Summer Learning Program Quality Intervention Study (SLPQI Study) was a multi-year multi-site design and development study that included impact evaluation in one site. Conducted with 152 providers in seven cities over four years, the study exemplifies a performance measurement design that meets the purposes of both continuous improvement and impact evaluation i.e., “measure once and cut twice.”

Traditional definitions of rigor entailed in psychometrics and experimental design often put these purposes – improvement and impact – on different sides of a methodological fence. For example, data produced for improvement is more useful (a) when disaggregated to the lowest level of measurement (i.e., the item level) using formative measurement principals, (b) with information disaggregated for each case (i.e., each specific micro setting and/or participant), and (c) with identification of multivariate subgroups where performance is similar so that improvement responses can be targeted. Conversely, impact evaluation designs emphasize (a) statistical power necessary to differentiate two or more groups achieved through scale-level measurement using reflective items to maximize reliability, (b) aggregation across cases (often nested) to produce average effects, and (c) using one outcome variable at a time.

The SLPQI Study’s “quality-outcomes design,” achieves both ends. The quality of staff’s instruction is assessed on measures of prevalence for over 70 specific staff behaviors that support students’ social and emotional skills. These measures are then used to stratify summer classrooms into quality subgroups. Students are then matched across subgroups to address impact questions: Do summer students exposed to higher-quality have greater academic gains during the summer and school year? Do students who enter the program with lower social, emotional, or academic skills have greater gains in higher-quality settings?

The quality-outcomes design was implemented with over 60 summer classrooms and 1,040 student scores in math and literacy during the summer and school year. Results include high teacher satisfaction with improvement data, year-to-year improvements on instructional quality, and impact evidence suggesting that students attending higher-quality summer classrooms had greater gains in academic skills compared to peers in lower-quality settings.

The implications of the study suggest that researchers can produce improvement value for clients, while advancing aspirations to greater certainty and generalizability. Further, the application of social science to social problems does not have to leave practitioners frustrated or doing guesswork about findings. By integrating expert practitioners and their concerns into the performance measurement process the validity of the overall enterprise is improved.

Authors

• Charles Smith, QTurn Group LLC

• Steve Peck, University of Michigan

• Leanne Roy, David P. Weikart Center for Youth Program Quality