Assessment of Fidelity of Implementation: The STEM21 Digital Academy High School Program Development

• In Event: How the Investing in Innovation (i3) Program Supports High-Quality Implementation Evaluation and Program Development From Initial Stage to Scale-Up

Abstract

The STEM21 Digital Academy provides a standards-based, 9th-12th grade course sequence in mathematics, science and technology delivered in an interactive “blended learning” environment. Students are engaged in a progression of online coursework guided by teachers and infused with authentic learning experiences in twelve schools in Connecticut. Coursework consists of blended learning, experiential learning, a digital portfolio and proficiency assessments. Teachers receive training through a summer institute and ongoing support via onsite and online coaching. This study is framed by social cognitive theory (Bandura, 1986) and examines how the STEM 21 program influences student and teacher outcomes. During 2011-2012, the first cohort of 9th grade students participated. During 2012 – 2013, the second cohort of 9th grade students joined the program.
A logic model describes linkages between STEM21 program components (professional development, curriculum materials, instruction), mediators and expected outcomes. Desired teacher outcomes are the use of instructional practice related to 21st century and inquiry skills and the use of the digital environment. Student outcomes are use of a digital environment, increased 21st century and inquiry skills, interest in STEM and science and mathematics knowledge.
FOI evaluation informs program improvement and assesses the degree to which the program is implemented as intended including:
• Variation in implementation fidelity across sites.
• Degree to which PD was provided.
• Degree to which students received intervention.
• Degree to which students were “underrepresented”.
• The relationship between specific fidelity components and student outcomes.
Formative data includes teacher and student focus groups and surveys. Summative data sources include: (1) teacher fidelity (attendance records), (2) grade level fidelity (attendance records, portfolio grades, records of on-line forum use and a student survey )and (3) program fidelity (PD agendas and observations, course schedules and student registration).
Key model components are teacher, grade and program evaluation. An FOI components metric and calculation table using scores from 0-2 were developed. Thresholds for key implementation components at teacher, grade and program levels were set and the percentages of these elements and schools meeting these thresholds assessed. Data has successfully been collected for two years.
Results indicate an overall high degree of fidelity across sites. Fidelity is being achieved by component and by school. Data indicate variability across sites. Challenges have included the accurate collection of data in a time effective manner, staffing and funding limitations, education of school staff related to the purpose and use of FOI data, and a need to modify thresholds and improve measurement as new information has been obtained. Strategies to address challenges included the development of on-line data collection techniques, an FOI manual for school staff, the use of multiple funding sources to augment data collection, the use of formative measures to provide just-in-time feedback on key program components, and the use of FOI data in program improvement. The successful implementation of FOI has been facilitated by the close relationship between the program evaluators and the program developer. This relationship is critical to address challenges and successfully identify appropriate measures and data collection techniques.

Authors

• Mhora Lorentson, EDUCATION CONNECTION

• Youn Joo Oh, Independent Researcher

• Frank LaBanca, EDUCATION CONNECTION