Using Classroom Video to Determine Teacher Instructional Conversation Competency

• In Event: Exploring Best Practices to Promote English Language Learner Achievement in the New South: The Instructional Conversation Pedagogy

Abstract

Objectives: The objective of this portion of the study was determination of counterfactual evidence through the objective observation of video-recorded classroom sessions of both experimental and control group teachers. Condition-blind reviewers, using a rubric aligned with the 5 CREDE standards, scored teachers on their delivery of the Instructional Conversation (IC) pedagogy. As a broadly-tailored pedagogical intervention, the IC merits a comprehensive assessment of baseline differences in teaching approach, one that can not be assessed without holistic review of classroom practice. We undertake this review of a sample (n = 67, 34 exp group, 33 control) videos each of a different teacher in spring of the efficacy year.

Theoretical Lens: The theory used to guide the study is a necessary synthesis of existing work on disparate topics. The CREDE rubric was designed with the intent of measuring fidelity of implementation (FOI) and relied heavily upon the work of Dusenbury et al (2003), Mowbray et al (2003) and O'Donnell (2008) among others. Objectivity in video review was determined through operationalization of the work of Ho & Kane (2013) in efforts to minimize bias and the Tinsley & Weiss (1975; 2000) guidelines for interrater reliability.

Research questions:
To what degree do the a priori factors capture IC competency?

Given the rubric adheres to the 5 CREDE standards, there might be an expectation of 5 distinct factors. The counter to this line of thinking is that, due to our design, we should only see true differences in the factor associated with the IC.

How well does IC proficiency predict treatment status?

This research question also implies examination as to whether non-IC factors predict treatment status as well. Significant differences in favor of the treatment group implies that the treatment group may indicate that they were superior (in our measures) regardless of professional development and subsequent training and monitoring.

Methods: To first determine interrater reliability, intraclass and tetrachoric correlation coefficients were generated to improve upon simpler methods of reliability determination such as percent agreement, Scott’s pi, or Cohen’s kappa. Once rater reliability was determined, factor analysis was conducted - first as principal components analysis, second as principal axis analysis, and finally factor analysis of the tetrachoric coefficients. The results across the three methods were consistent and are presented below. Binary logistic regression was then used to test if factor scores predict treatment status.

Results & Conclusions: The results show, first, that the ratings were reliable (ICC = .87, p < .001) and that factor analysis of the results was feasible. The factor analysis results consistently loaded the IC indicators into one group. The IC factor predicted treatment status (p < .001) and, perhaps most importantly, treatment teachers showed no significant differences with the control group with regard to other factors.

Scholarly significance: We believe that our approaches to interrater reliability and determination of implementation fidelity are critical to the unbiased assessment of any pedagogical intervention. These results will play a larger role in the ultimate evaluation of IC effects on student test outcomes.

Authors

• Rob Stollberg, University of Georgia - Athens

• Pedro R. Portes, University of Georgia

• Manuel S. Gonzalez Canche, University of Georgia

• Albert Manuel Jimenez, Kennesaw State University

• Rolf Straubhaar, University of Georgia