A Flexible Mixed-Methods Approach to Evaluation in Afterschool: Benefits and Considerations

• In Event: Square Peg, Round Hole: Negotiating the Appropriateness of "Gold Standard" Research Design for Out-of-School Time

Abstract

This presentation examines the benefits and limitations of a mixed-methods, non-experimental evaluation design in OST settings. The focus is on an early literacy program for early elementary school students which implements read alouds to improve student vocabulary acquisition and text comprehension. This program draws heavily on evidence-based practices that have been reinforced by decades of research in the literature, limiting the need to reproduce results through a rigorous experimental design. Thus, the research design incorporates program-, educator-, and student-level measures that are developed in collaboration with school and community-based organization members and adapted from year to year to respond to changing contexts and priorities. Measures include a fidelity rubric, semi-annual program observations, educator surveys, and student surveys. This flexible design has allowed for the development of practitioner-generated measures that are uniquely targeted to the specific goals of the program, instead of limiting research to best-fit normed and validated scales. While this limits the ability to draw longitudinal or causal conclusions, this flexibility allows for a true reflection of program practice and has resulted in a fluid continuous improvement model that responds with agility to ever-changing contexts. Evaluation results demonstrate improved educator practice, strong student outcomes, and document the continuous improvement process over time. Recommendations and considerations for comparable OST settings will be discussed.

Authors

• Joel Nunez, ExpandED Schools

• Isabella Fante, ExpandED Schools