Developing a Validity Argument for Teacher Observation Tools

• In Event: Validity Arguments for K–12 Teacher Observational Measures: Issues and Future Directions

Abstract

The validation of an assessment system is a process that begins with a theory of action, followed by a statement of the proposed use, a carefully planned argument defining its key assumptions, and then proceeds with the analysis of evidence into a substantiated validity argument (Kane, 2006; Kane 2013). The interpretive/use argument (IUA), presents the “network of assumptions leading from the observed performances to the conclusions and decisions based on the performances” (Kane, 2013, p. 23). The validity argument evaluates those assumptions through the collection of empirical data and analytic reasoning. In this paper, we present the theory of action and the IUA for RESET, then describe the series of studies undertaken to collect and examine evidence to support those assumptions. The purpose of this paper is to describe an approach and provide an example for the validation of teacher observation systems.

The proposed uses of RESET are to provide valid observations and feedback to special education teachers to improve their implementation of evidence-based practices (EBPs). To support these uses, several assumptions related to scoring, generalization, extrapolation and decisions need to be met. The scoring and generalizability assumptions were examined through the use of generalizability theory and MFRM studies, in which teachers provided video recorded lessons evaluated by trained raters. The extrapolation assumptions were examined through studies investigating the relationship of teacher performance to growth in student scores. The decision assumption was examined through a feedback study, in which growth on RESET for teachers who received feedback was compared to growth for a comparison group.

Across studies, a total of 143 special education teachers from four states contributed video recorded lessons. A total of 112 raters were trained to apply the RESET scoring criteria. Progress monitoring data from 117 students with disabilities were collected. Our generalizability studies indicated acceptable reliability, with a generalizability coefficient of .74. MFRM results consistently indicated that all fit statistics were within acceptable ranges, suggesting that the scoring criteria were consistently applied to determine teachers’ implementation of the relevant EBPs. Our feedback study’s overall results were not significant. However, there was a significant interaction effect for lesson number (time) * group, F (1, 28) = 2.386, p = .034. Follow up contrasts examining the interaction effect revealed that the treatment group made significantly greater gains on one component of RESET than the comparison group, F (1, 28), p = .049. Our regression analyses indicated that teacher performance on RESET accounted for significant, unique variance in student growth.

To defend the use of system-based results, comprehensive validity efforts based on the collection of evidence aligned to system-based claims must be defined and conducted for observation systems prior to implementation, once they are in place, and for the subsequent years of use (Hall, 2014). This paper provides an example of the comprehensive validity efforts needed to support the use of teacher observation systems.

Author

• Evelyn Johnson, Boise State University