Connecting the Dots Between Teacher Use of Educative Curriculum Materials and Student Curricular Learning Outcomes

• In Event: When Nothing Is Something: Null Results in Informal Learning Research

Abstract

Objectives. Enacting curricula equipped with innovative pedagogical ideas is challenging. To support teacher learning of a technology-enhanced plate tectonics unit on their own and at their own pace, we developed online educative curriculum materials (ECMs). See Figure 1. These ECM features focused on how to help students formulate causal mechanistic explanations related to plate tectonics. Our hypothesis was that the more ECM features the teachers accessed, the greater learning gains the students would achieve.

Theoretical Framework. In developing ECMs, Ball and Cohen (1996) identified several factors to consider including students’ prior knowledge, teacher understanding of the content, the materials available, the environment of the classroom, and the support of the broader community. Researchers have used these factors to craft design heuristics (Davis & Krajcik, 2005). To actualize student gains, teachers need not only well-designed curriculum materials but also training that helps them implement these materials (Penuel & Gallagher, 2009).

Methods and data sources. Under a mixed methods research design, we analyzed automatically generated log data to identify how teachers interacted with ECMs. To measure student learning gain in each teacher’s classrooms, we calculated Effect Size (ES) based on Cohen’s d, i.e. mean difference between pretest and posttest divided by the pooled standard deviation. The test consisted of 16 multiple-choice items with a maximum score of 26. The test reliability was 0.74 in Cronbach’s alpha. We studied 15 middle school teachers and 11 high school teachers. Among them, 88% taught in public schools; 15% of the schools were in urban settings, 50% in suburban settings, and 35% in rural settings. An average of teaching years was 17.14 ranging from 5 to 35. The average grade level of students (n = 1,098) was 8.40; 52.9% were female; 12.5% spoke English as a second language.

Results. The students across teachers made significant gains, ranging from ES= 0.36 SD to 2.59 SD. We found a non-significant, but positive, correlation between the total amount of teacher interactions with ECMs and student learning gains, r = 0.20, p = 0.32. To explain the non-significant correlation, we further analyzed teacher interviews and surveys. We found that each classroom environment was unique in terms of student, teacher, and school characteristics, which confounded the relationship between teacher use of ECMs and student learning gain. Middle school teachers accessed significantly more ECM features than high school teachers, t(24) = 3.15, p < 0.01. The student learning gain was significantly higher when ECMs were used during class with students when not, t(24) = 2.42, p < 0.05.

Scholarly significance. The path between teacher access to ECMs and student learning gain is not straightforward. Focusing on simple significant relationships in studies like ours as a sole evaluation criterion can be misleading and often deny an important opportunity to examine diverse ways in which educational interventions work. Regardless, if online ECMs are freely available, unconstrained by the limitations of travel, logistics, and cost that accompany the person-to-person modes traditionally used for in-service teacher training, a much greater number of teachers and their students can benefit.

Authors

• Hee-Sun Lee, Concord Consortium

• Trudi Lord

• Amy R. Pallant, Concord Consortium