The Utility of Teacher Logs in Assessing Teacher Implementation of the Instructional Conversation Pedagogy

• In Event: Exploring Best Practices to Promote English Language Learner Achievement in the New South: The Instructional Conversation Pedagogy

Abstract

Objectives: The purpose of this paper is, using exploratory factor analysis, to assess the utility of using self-reported teacher logs as a measure of fidelity of implementation of the Instructional Conversation pedagogy.

Theoretical Lens/Theory of Change: Within this larger IES-funded study of the Instructional Conversation, one of our purposes has been to identify the most effective measures of fidelity of implementation for techniques taught through professional development. External ratings of video classroom data (as discussed in the previous paper), student achievement scores and self-reported teacher logs are the primary data that have been thus considered. The focus of this paper are self-reported teacher logs, in which teachers answer a series of open-ended and Lichert-scale questions related to their implementation of the IC pedagogy over a two-year period.

Research questions: To what extent are self-reported teacher logs an effective measure of teacher fidelity of implementation of the Instructional Conversation pedagogy?

Methods/data: Our primary method was a factor analysis of teacher responses to questions related to 29 indicators on their bi-weekly self-reported teacher logs. We conducted factor analysis as a method of data reduction, seeking to bring out underlying unobservable (latent) variables that are reflected in the observed variables (manifest variables). As is typical for this procedure, we were looking for simple structure, or in other words 1-2 high-loading variables, to assess the utility of these logs in measuring fidelity of implementation.

The teacher log dataset included teacher responses to dozens of different items on bi-weekly teacher logs administered over a two year period--we selected 29 indicators that lent themselves to factor analysis.

Results & Conclusions: Using our original 29 indicators, we had a three-factor solution, with few themes or trends noticeable among the factors. In an attempt to further exclude extraneous variables, we reduced the number of indicators to 19, focusing on those indicators which best lent themselves to counterfactual analysis. Using this set of 19, we had a solution with two factors, but again with little thematic consistency to suggest the factors that were loading together were doing so for non-random reasons.
We argue that the self-reported nature of the dataset is the primary driver of its ineffectiveness in measuring fidelity of implementation.

Scholarly significance: No previous studies have evaluated the utility of self-reported teacher log data in evaluating fidelity of implementation of the Instructional Conversation.

Authors

• Rolf Straubhaar, University of Georgia

• Rebecca K. Gokee, University of Georgia

• Pedro R. Portes, University of Georgia

• Manuel S. Gonzalez Canche, University of Georgia