Identifying and Measuring Factors Related to Student Learning: Promises and Pitfalls of Teacher Instructional Logs

• In Event: K-12 Student Success: Complexity in Mathematics and Science Education Research

Abstract

Objectives
The purpose of this study is to investigate the relationships among teacher content knowledge, classroom instruction, and student learning of science content. Among other findings, the study revealed the complexity of deriving measures of instruction from teacher logs.

Theoretical Framework
Discussions of the relationship between science teacher characteristics and student learning tend to be theoretical and divisive. Despite the lack of consensus, there is broad agreement on one premise—teacher knowledge of disciplinary content directly and positively affects classroom practice and, ultimately, student learning. Interestingly, although this premise is logical, the empirical support is inconsistent, largely because of a lack of suitable measures. Studies rely mainly on proxies of teacher content knowledge, e.g., certification type (Goldhaber and Brewer, 2000), undergraduate major (Monk, 1994), and courses taken (Druva and Anderson, 1983).

Research Design and Data Collection
We recruited roughly 200 teachers nationally for each of two studies: one on force and motion and one on plate tectonics. Each teacher:
● Completed the relevant teacher assessment on-line shortly before the unit of instruction;
● Administered the relevant student assessment (paper/pencil) just before and just after the unit; and
● Completed a daily web-based instructional log during the unit.

The teacher and student assessments were developed as pairs of psychometrically rigorous multiple-choice assessments. Data collection took place from January to June 2009. There was substantial attrition at the teacher level from both studies, which we attribute to the burden associated with the teacher log. Of the teachers in the Force and Motion study, 79 completed all components, compared with 107 in the plate tectonics study.

Results
Data were analyzed using hierarchical linear modeling (HLM) to account for the variation shared by students in the same class. In the Force and Motion study, amount of instruction by itself did not predict student learning. Teacher knowledge was a significant predictor of student learning, both by itself and with amount of instruction in the model. In Plate Tectonics, neither amount of instruction nor teacher knowledge predicted student learning, either by themselves or in combination. In an attempt to clarify what is obviously a murky relationship between teacher content knowledge and student learning, we explored the possibility of mining the unit instructional logs for more information about student opportunity to learn. Our analysis showed no relationship between these ratings and changes in student assessment scores.

Scholarly Significance
Our findings suggest the relationships among teacher knowledge, amount of instruction, and student learning depend on the content students are studying. Most surprising are the findings that (1) amount of instruction by itself does not predict student learning of force and motion concepts, and (2) neither teacher knowledge nor amount of instruction predict student learning in plate tectonics. We found instructional materials to be a powerful mediating influence in the relationship between teacher knowledge and student learning. This study also suggests that teacher instructional logs do not provide sufficient evidence to rate the quality of instruction and that direct observations of instruction may be necessary.

Author

• Sean Smith, Horizon Research, Inc.