How Well Do Group Mentoring Programs Adhere to Best Practices and What Are the Implications? A Fidelity of Implementation Analysis

• In Event: A Comprehensive Evaluation of Group Mentoring at the Boys & Girls Clubs: Results from a Four Year Study

Abstract

To increase the potential for obtaining causal evidence, fidelity of program implementation (FOI) should be emphasized in the evaluation design. Even a true experiment with random assignment of participants to groups produces weak evidence of causality if implementation is compromised. Although statistical attempts to control for such covariates as site and mentor characteristics have been made, poor program implementation has been linked to finding a small effect size of mentoring. Program integrity is a complex phenomenon and requires a sound theoretical framework to guide the entire process and outcome evaluation of mentoring programs. Assessing FOI of prescribed programs allows for examination of project outcomes across multisite settings and helps to identify possible confounding elements prohibiting the attainment of project goals. Measuring FOI of abstract constructs, such as whether a mentor is facilitating a positive mentoring experience, is elusive. The resulting FOI rubric must be grounded in theory and best practices and provide sufficient guidance so that inter-rater reliability and validity of inferences are strong. This paper illustrates the development of the FOI Group Mentoring Rubric, shares findings from individual clubs, and presents a multivariate analysis that includes data from the quantitative study and the FOI analysis.

Author

• Svetlana Beltyukova, Inference LLC