Developing New Measures of Teachers' Standards Implementation

• In Event: College- and Career-Readiness Standards Implementation at the National, State, District, and Classroom Levels

Abstract

Objectives
A primary challenge in gauging the effects of standards-based reforms is the difficulty in measuring teacher implementation. Previous work has developed tools (among others, the Surveys of Enacted Curriculum (SEC), Porter, 2002) that have been used to report the content of teachers’ instruction and determine alignment with standards. However, this work needed upgrading to ensure compatibility with new CCR standards. This paper reports on three validity studies that provide validity evidence for a revised SEC.

Perspectives
Validity theory emphasizes that validity is a property of inferences or uses of instruments, not a property of the instruments themselves. We sought to make inferences about a) the content of teachers’ instruction and b) the alignment of their instruction with standards. Thus, we present evidence from studies that test whether the instruments we created are up to the task of those inferences.

Methods and Data
We report on three studies based on three data sources and two teacher samples.
The validity of teacher reports of content coverage. We recruited 40 teachers (20 math, 20 ELA) and video observed them for a single lesson. The teachers also completed a log survey to describe their instruction in that lesson. Expert raters viewed the videos and also completed the log survey. We compare the expert raters’ assessments to teachers’ reports using correlations and tests of mean difference.
The validity of year-end surveys as compared to aggregated logs. The next question is whether teachers can provide valid reports over a longer period. Here, we compare reports of content coverage from aggregated biweekly logs to those based on end-of semester surveys for 54 teachers (27 math, 27 ELA) using correlations and alignment indices.
The reliability of content analyses of teacher assignments. Finally, we seek to know the alignment of teachers’ assignments to standards and assessments, which requires reliable content analyses of those documents. We gauge this by content analyzing two weeks’ worth of assignments from each of the 54 teachers from study two. We use four content analysts and a generalizability theory d-study (Porter et al., 2008).

Results
Full results will be presented at the symposium. Results from study 1 indicate that teachers can accurately report on the content of their instruction for a single lesson, with alignment indices between teacher and expert reports averaging .8 in mathematics and slightly lower in ELA. Results from study 2 show that teachers vary considerably in the extent to which their aggregated logs align with their year-end surveys. Teachers report covering more topics and cognitive demand levels on the logs and average alignment indices are moderate and vary by subject. Results from study 3 show high reliability of content analyses (average of .8 or higher in both subjects).

Significance
The work presents new evidence on the validity of tools that can be used to gauge teachers’ instruction in the new CCR-standards era. Based on these results we have revised the instruments and are making them available to interested researchers to use as implementation variables or outcomes in their research.

Authors

• Morgan S. Polikoff, University of Southern California

• Hovanes Gasparian, University of Southern California

• Shira Korn, University of Southern California

• Martin Gamboa, University of Southern California