Validation Evidence for Observation Protocols

• In Event: Practical Guidance and Best Practices for Gathering Validity Evidence Based on Assessment Type

Abstract

The final paper will explore validity evidence in the context of observation protocols are a type of performance assessment in which raters use a predefined rubric to provide real-time scores to examinees on specific tasks. In educational settings, common types of observations include principal observations of teachers (e.g., Ho & Kane, 2013; Jones & Bergin, 2019) and observation of preservice teachers (e.g, Star & Strickland, 2008).
The context of this study will be The Network for Educator Effectiveness (NEE) which is a Classroom Observation Rubric (COR) is the context for this portion of the symposium. The NEE Rubric is an analytic scoring rubric used by principals to evaluate teaching effectiveness across a variety of teaching practices. An outreach unit of the University of Missouri, NEE provides teacher evaluation services to roughly half of Missouri’s 500+ school districts. The NEE rubric is designed for use in K-12 classrooms, and is based on a condensed version of the Interstate Teacher Assessment and Support Consortium (InTASC) core teaching standards (Council of Chief State School Officers, 2011). The NEE rubric is modular, in that principals may select from 26 distinct teaching practices, although districts typically select 3 to 6 teaching practices each year for their teacher evaluations. Teachers receive a score of 0 (“not present”) to 7 (“perfect exemplar”) on each selected teaching practice. The rubric provides anchor points at scores of “1”, “3”, “5” and “7”. Participating principals are required to complete annual calibration trainings, which include shared ratings of video-recorded teachers, which can be used to provide connectivity in order to compare ratings across settings (Wind & Jones, 2018).
As a subset of rater-mediated performance assessments, observation protocols share many of the validity characteristics, and validity frameworks are therefore similar (e.g, Engelhard & Wind, 2018; Jones & Bergin, 2019). However, one characteristic that sets observation protocols apart from other performance assessments is that observation protocols often occur in real time. Because of this, raters must be present in order to provide ratings for the examinees. Therefore, validity and reliability are often assessed alongside the level of standardization in the rating process. Standardization methods include the use of standardized protocols (Pianta & Hamre, 2009) and building consistency through rater calibration exercises (Cash, Hamre, Pianta & Myers, 2012).This paper will explore the unique issues surrounding the validity of observation protocols, such as stability of estimates across rating occasions, the necessity and challenges of working with sparse rating networks, and a discussion regarding the merits of different methods of obtaining validity evidence (e.g, Generalizability Theory, Rasch measurement).

Author

• Eli Andrew Jones, The University of Memphis