Internal Consistency of Observation Protocols in Measuring Teaching Effectiveness: The Sheltered Instruction Observation Protocol

• In Event: Measurement in the Second Language Classroom

Abstract

The reliability of widely used observation protocols must be empirically measured if their results are to be used in teacher evaluation. This study examines overall and component-based reliability of the Sheltered Instruction Observation Protocol (SIOP) in evaluating teaching effectiveness in mainstream classes with ELLs. It also examines if the instrument’s reliability yields consistency across different classrooms and what challenges the instrument poses for raters. Survey and qualitative data were collected from three raters and two science teachers. Results revealed several weak to moderate levels of component-based and overall inter-rater reliability to evaluate teaching effectiveness across two K-12 settings. Qualitative data revealed several challenges related to component-based reliability of the protocol regarding linguistic clarity, consistency, and potential for measuring observable behavior.

Authors

• Nihat Polat, Duquesne University

• James B. Schreiber, Duquesne University