Evaluating the Fidelity of Implementation of Cognitive-Based Principles in the Classroom

• In Event: 66.026 - Scaling Up Cognitive Learning Principles to Redesign a Mathematics Curriculum for Improved Learning

Abstract

How do research-based principles translate to the classroom? As cognitive research necessarily focuses on the individual and typically involves study-specific materials, research findings must be translated into concrete recommendations that apply to dynamic classroom environments. In the current study we examine whether teachers’ self-reported fidelity measures are correlated with student outcomes and whether and how teachers’ beliefs about implementation are aligned with researchers’ intentions.

The current analyses are a component of a large-scale study to test the efficacy of applying research-based principles (e.g., Pashler et al. 2007) to the Connected Mathematics Project 2 (CMP2) curriculum to improve students’ mathematics learning. Student materials, teacher materials, and teacher training for two 6th grade units and two 8th grade units were redesigned to incorporate the following principles: 1) verbal-visual mapping, 2) interleaving worked examples, 3) spacing learning over time, and 4) using formative assessment.

One hundred and twenty 6th and 8th grade teachers, experienced with CMP2, participated in the study. Teachers worked in a wide range of school contexts across seventeen states. Using a within-teacher design, each study teacher provided data for both units in his/her grade level: one revised (CMP2-R) and one control (CMP2). Whether a given unit was used in its original or redesigned format was counterbalanced across participants. All teachers attended a two-day professional development workshop that introduced the research-based principles and reviewed implications for instructional materials, practice, and the classroom environment. Teachers also completed weekly instructional logs. Logs included closed-ended items for teachers to indicate what unit was taught and whether they had applied each principle, and requested open-ended explanations of how the principles were implemented in class that week. Student data was collected from pretests and posttests for each unit.

MANOVA analyses explored whether treatment assignment impacted the treatment fidelity for that unit. Large and statistically significant effects of treatment assignment were found for treatment fidelity in all four units. Teachers reported using the research-based principles in instruction more often for treatment units than for control units. However, 2-level hierarchical linear models found that self-reported treatment fidelity was not associated with student outcomes.

In ongoing analyses, researchers are coding open-ended responses in teacher log entries to establish whether and how teachers’ descriptions of their use of the principles aligned with researchers’ intended use. We expect the detailed coding of open-ended responses will allow us to identify areas where clarification about the instructional use of the principles is needed. Early results suggest that researchers’ intentions regarding the instructional use of the principles were not fully translated into the classroom. Teachers describe actions that are consistent with the principles, but there are important differences between the focus of teachers’ descriptions and the critical aspects of each principle as determined by researchers (e.g., “going over the answers [to worked examples] in class” vs. prompting students to explain the correct and incorrect reasoning in the worked examples). More broadly, the work will provide researchers and professional development providers with recommendations about how to translate research from the lab to the classroom.

Authors

• Yvonne Kao, WestEd

• Jodi Davenport, WestEd

• Sarah Warner, WestEd

• Shandy Hauk, WestEd

• Aleata Hubbard, WestEd

• Steven Arnold Schneider, WestEd