A National Perspective on Collective Impact: Methods and Results From the Communities in Schools National Evaluation

• In Event: Community Schools: Collective Trust, Collective Action, and Collective Impact

Abstract

Communities In Schools, Inc. (CIS) is a nationwide initiative designed to connect needed community resources with schools to help students, particularly those identified as at-risk, successfully learn, stay in school, and prepare for life. ICF International recently completed a five-year national evaluation of Communities In Schools, and this comprehensive study included eight components, including a school-level quasi-experimental study and three randomized controlled trials. The purpose of this study was to estimate the impact of CIS and determine how CIS could strengthen as an organization to achieve progressively stronger impacts.

In this presentation, the presenter will describe the comprehensive mixed-method evaluation design and describe how it fits into a growing body of work that supports the development of community-based integrated student services. For example, the National Research Council (2004) suggested that “programs for youth offered by more than one organization – in schools, community centers, or both – that focus on different areas of interest and through different kinds of curricula provide the greatest opportunity for young people to acquire personal and social assets”. Schools also benefit from the involvement of families and the community in youth education because they receive additional support and input, enabling schools to identify and provide more comprehensive services for students (Epstein and Salinas, 2004). In short, support has been growing in the youth development field for community-based integrated student services (CBISS), which are designed to provide a more comprehensive support structure for at-risk students struggling with academic, behavioral, health, and/or other issues.

The presenter will describe the impact analyses, which included a school-level quasi-experimental study and three student-level randomized controlled trials. Altogether, the school-level study included 602 CIS schools and 602 matched comparison schools that were identified using propensity score matching techniques. The three RCTs included 573 students across three sites in Jacksonville, Austin, and Wichita. Outcome data were drawn from student records, and included measures of dropout, graduation, standardized test scores in math, standardized test scores in reading, attendance, behavior, credit completion, and GPA.

ICF found from the school-level quasi-experimental study that CIS schools had positive effects on both dropout and graduation rates, which is a unique finding in dropout prevention research. Moreover, effects on dropout and graduation roughly double in magnitude when the CIS model was implemented with fidelity, which provided validation of the CIS model itself. The RCTs produced statistically significant impacts at the middle school level on standardized test scores for reading (ES=.26) and grade promotion (ES=.50). At the high school level, statistically significant impacts were found on dropout rates (ES=.90), attendance (ES=.45), credit completion (ES=.38), and grade point average (ES=.38).

This research is important because it is unprecedented in its comprehensiveness and it provides a model for conducting complex multi-method, mixed-method evaluations. Findings from this study have already been used to transform practice at more than 3,000 CIS schools across the country.

Author

• Allan W. Porowski, ICF International