Evaluating the Implementation of the Increasing College Access Network

• In Event: Improving Student Success in Online Dual Enrollment Courses through the Increasing College Access Network

Abstract

Objectives
The goals of the implementation study were to (a) assess whether key components of the Increasing College Access Network (ICAN) were implemented with fidelity, (b) identify contextual factors that may facilitate or inhibit the scaling of ICAN, and (c) describe what students experienced in their dual enrollment (DE) courses.
The implementation study was guided by the following research questions.
1. To what extent was ICAN implemented as intended?
2. What obstacles inhibit, and what factors enable, the successful implementation of ICAN?
Perspectives
The implementation study draws from guidance provided by the Institute of Education Sciences for conducting implementation research for education interventions (Hill, Scher, Haimson, & Granito, 2023) which specifies two categories of implementation components: support and direct. Support components include the training and resources provided by the intervention developers that establish and sustain the intervention over time. Direct components consist of the activities experienced by the “end users” of the intervention that more directly influence outcomes. Because intervention providers may not have full control over direct components, measures of implementation fidelity often focus on support components. However, information about direct components is key to understanding differences in student experiences that may relate to differences in outcomes.
Methods
We used the ICAN logic model (see Exhibit 3) to create an implementation fidelity matrix, which allowed us to assess whether support components were implemented in way that achieved pre-specified thresholds for fidelity of implementation. Throughout the two years of ICAN implementation, the study team engaged in qualitative and survey data collection and analysis.
Data Sources
To measure fidelity of implementation, we collected attendance logs for technical assistance sessions, annual site visits, and communities of practice, and we tracked the development and dissemination of resources from implementation partners (support components). We also collected weekly instructor logs to document use of the social-emotional learning (SEL) strategies highlighted by ICAN (direct components). To capture insights on facilitators and barriers to implementation, we conducted instructor interviews, instructor surveys, and student interviews each semester.
Results
We found that ICAN was implemented with fidelity during the two years of implementation. All instructors implemented each of the SEL strategies at least once each semester, though the frequency with which they used strategies varied (see Exhibit 4). For example, use of “emotions check-ins” decreased across semesters, while use of “team building exercises” increased. Through interviews, we learned that most instructors strengthened and systematized their use of the SEL strategies over time and continue to show interest in further growth in their practice through participation in the implementation supports. Although students were not consistently able to recall experiences with specific strategies, they generally reported positive experiences in their online DE courses.
Scholarly Significance
The field is lacking in evidence-based strategies to address students’ motivation to participate and succeed in online learning. The implementation study addresses this gap in research by examining ways in which teachers can improve students’ performance in online courses, SEL skills, and college and career readiness by providing a welcoming, engaging learning environment.

Author

• Kellie Mayer, American Institutes for Research