Using Fidelity of Implementation to Advance Project-Based Learning Curricular Design, Professional Learning, and Assessment

• In Event: 15.052 - Promoting and Measuring Student Learning and Engagement Through Project-Based Learning: Leveraging Coherence

Abstract

Objectives
This study describes the design and testing of a fidelity of implementation instrument for measuring the implementation of PBL in twenty 3rd grade classrooms. More than a traditional implementation checklist, the instrument was created to reflect PBL design principles with innovative measures that illustrate sensemaking and figuring out big ideas, ownership, and collaboration. Classroom observations (N=122) were measured and analyzed to examine variation in implementation, accounting for classroom and teacher effects.

Perspective
This study, similar to others in the implementation literature (Century, 2010; O’Donnel, 2008; McNeill, 2017) was chiefly designed to measure enactment of ML-PBL lessons relying on in-classroom observations by independently-trained raters. Many of the multi-faceted features of this intervention call for behaviors that are difficult to observe and are rarely measured. Not only was the primary focus on whether teachers exhibited certain behaviors but also on how well the design principles of PBL were articulated in each lesson, including use by the teacher and student of the driving question which frames the goals of each lesson; support for sensemaking and the figuring out process; use of discourse that promotes language and literacy development; development of a lesson product or artifact that reflects three-dimensional learning goals; and support for the development of social and emotional learning goals, e.g., equity, agency, persistence, and collaboration. The fidelity instrument measured these components using 4 main constructs including: driving question, sensemaking, lesson artifact, social-emotional learning; and classroom management. These constructs were measured by the following: (1) Lesson as Written - Level of fidelity of implementation of the components enacted, and (2) PBL Practices – Level of use of PBL features to support students in figuring out phenomena, and scored on a 4-point scale. Guides were added to assist raters in their observations.

Method and Data Sources
Teachers who participated in the ML-PBL 2017‒18 field test were observed for several weeks by two raters for a total of 122 observations. Raters were trained with video and in-classroom observations and strongly encouraged to use running records of their experiences in the classroom. Two rating analyses were conducted: (1) rater scores were transformed with a log solution and standardized to account for rater bias; (2) random fixed effect two-level analysis was conducted to estimate which PBL classroom experiences predicted the highest levels of fidelity of implementation accounting for variation among classroom ratings on specific PBL design features and teacher enactment.

Findings
Preliminary results show that teachers earned higher fidelity of implementation scores for teacher support of equity, student persistence, and artifact development. Use of driving questions and discourse to improve sensemaking show significant teacher differences, but rater-free effects. Certain instructional practices were sensitive to different raters, particularly when assessing teachers’ support for collaboration, students’ work, and reading and writing activities.

Significance
Findings of this study suggest: (1) fidelity of implementation can be measured even for complex PBL constructs, including sensemaking; and 2) multivariate analyses linking ratings with specific lesson components can highlight specific measures that need further training and clarification for teacher participants and raters.

Authors

• Susan Codere, CREATE for STEM at MSU

• Emily Adah Miller, University of Georgia

• I-Chien Chen, Michigan State University

• Joseph S. Krajcik, Michigan State University

• Barbara Schneider, Michigan State University

• Israel Touitou, Michigan State University