A Validity Study of the Tripod Survey: Reexamining the Seven C's

• In Event: Promising Scholarship in Education: Dissertation Fellows and Their Research

Abstract

Purpose of study
The goal of this study is to explore the validity of the claims about teacher effectiveness that can be made using the Tripod student survey. Despite its widespread use in teacher evaluation systems, very little in-depth psychometric analysis has been published on the Tripod survey. In theory, the survey measures seven separate dimensions of teacher practice, Care, Confer, Captivate, Clarify, Consolidate, Challenge, and Control, there is little empirical evidence supporting this structure.

Using the multilevel item response theory framework to account for measurement and sampling error, my dissertation investigates the dimensionality, reliability, and validity of student surveys using the MET data. The central research questions are:

1. Is there evidence based on a multilevel item factor model for the Seven C's factor structure? Do the data support a different factor structure instead?

2. Are the teacher-level scores (estimated from the multilevel item factor model) reliable and stable across a teacher’s sections and across school years?

3. Is there a significant relationship between the Tripod teacher practice dimensions and student academic growth in English and Math?

4. Is there sufficient evidence based on the results of the previous three questions to support the current summative and formative uses of the Tripod survey?


Study design and methods
I rely on an interpretive argument framework to investigate the case for using student ratings to measure the effectiveness of teachers. Kane’s (2006) “Validation” chapter will serve as the framework for making an interpretive argument. The first step is scoring, which holds that teacher scores produced from the Tripod survey accurately capture dimensions of teacher practices undistorted by other factors. A set of multilevel item factor models were estimated using flexMIRT to examine for a variety of latent factor structures at both the within and between level of the model (Cai, 2015). The second step is generalization, focusing on the reliability and stability of teacher scores across different groups of students as well as over time. The third step is extrapolation, which focuses on the relationship between the Tripod scores and other measures of teacher effectiveness (e.g., academic growth). The fourth step is implication, which directs attention to how the scores will be used, and specifically the intended benefits/unintended consequences.
My analyses focus on the student responses to 36 Tripod survey items. For the calibration analyses, I focus separately on English and Math classrooms in Grades 6-8 during the 2009-10 school year. The sample is described in Table 1.

Findings and policy implications
Multilevel item factor analyses reveal that the 7C’s classroom factors are very strongly correlated with each other (see Table 2), indicating an underlying “general” teaching practice dimension that explains most of the shared variance between the 7C’s. The “Control” domain, which contains items regarding student behavior in the classroom, consistently remains a separate factor in exploratory factor analyses, implying that a two-dimensional structure (Control & the other 6C’s combined) is sufficient. Final item factor loadings for the multilevel two dimensional model are reported in Table 3. The two classroom factors are correlated 0.64 in the English classroom and 0.61 in Math.
Table 4 presents reliability coefficients for the two Tripod proposed dimensions as well as the overall score. Estimates of the intraclass correlations (ICC(1)) indicate that between 27-35 percent of the variance in the Tripod scores is at the classroom-level, implying there is still substantial within-classroom disagreement in the ratings of teachers. Interestingly, while the ICC is highest for the Control domain, the across-section and across-year correlations for the Six C’s factor is higher than for Control. It is hypothesized that ratings of classroom management and student behavior are more specific to an individual class rather than a stable teacher characteristic.
A hierarchical linear model was estimated predicting student end-of-year state standardized test scores by prior student achievement and class ratings of Control and the other 6 C’s (estimated from a multilevel item factor model). Table 5 presents the fixed effects coefficients from the math and English models. For both subjects, the Control (classroom management) dimensions is significantly predictive of student test scores controlling for student background characteristics, while the aggregated Six C’s dimension is not.

Implications
Seven states now mandate that student surveys are required as a component of teacher evaluations (Doherty & Jacobs, 2015). The Tripod survey is currently used to provide a classroom-aggregated total score for teacher summative feedback, while the scores on the Seven C’s are provided as formative feedback to teachers. The total score was found to be reliable, but it appears that the (Control) classroom management items are most strongly related to student test scores. It is suggested that the Tripod scores are a potentially useful tool for students and teachers, but that providing feedback on seven separate dimensions of teacher practice overstates the information that this measure can provide.

Author

• Megan Kuhfeld, University of California - Los Angeles