Improving the Design of Evaluations That Include Students, Teachers, and Schools

• In Event: Improving the Design of Program Evaluations in Education and Prevention Science

Abstract

Background

The past 15 years have seen a dramatic increase in the number of cluster randomized trials (CRTs) evaluating the impact of educational programs (Spybrook, Shi, Kelcey, 2016). The importance of accurate estimates of design parameters for power analyses for CRTs has been well documented. However, there is a lack of empirical evidence regarding design parameters for three-level CRTs with students nested within teachers nested within schools. explicitly model the teacher level. Estimates of these values are critical for CRT designs that randomize teachers or classrooms to study conditions, or CRTs randomizing at the school level that seek to detect teacher-level moderator effects.

Purpose

We aim to extend the existing evidence base on design parameters for planning CRTs that include the teacher level using data from state longitudinal data systems. Specifically, we seek to determine:
What portion of the variance in student achievement (reading, math, science) can be associated with the student, teacher, and school levels of analysis (measured with intraclass correlations)?
How much of the variance in student achievement (reading, math, science) can be modeled by different sets of covariates such as earlier achievement and demographic indicators (measured with R2 statistics)?

Data Sources and Analysis

The data we use comes from state longitudinal data systems that allow us to link students, teachers, schools, and districts. Specifically, we are using data from multiple school years from Kentucky, North Carolina, and Michigan.

For each dataset we have test scores for state administered tests for grades 3-12. In addition, we have student characteristics such as gender, race/ethnicity, ELL status, and special education status; teacher characteristics such as gender, race/ethnicity, teaching experience and highest degree; and school characteristics such as locale, size, and title 1 status.

For question 1, we estimate three level unconditional HLMs with students nested within teachers nested within schools. We then calculate two ICC parameters: 〖ICC〗_L2 , the proportion of the outcome variance that lies between teachers within schools (i.e., teacher-level ICC) and 〖ICC〗_L3 , the proportion of the outcome variance that lies between schools (i.e., school-level ICC).

To calculate the variance explained by covariates, R2 , the unconditional model is modified to include student, teacher, and school-level covariates. We examine three covariate sets: (1) baseline version of the outcome (i.e., pretest) only; (2) student, teacher, and school demographic characteristics only; and (3) full covariate set including the pretest and all other student, teacher, and school-level baseline characteristics. We calculate separate R2 terms for each of the three covariate sets using the corresponding conditional variance estimates.

Findings/Conclusions

Preliminary findings from North Carolina are provided in Table 1. In general, the proportion of variance at the school level is similar to that at the teacher level. The student, teacher, and school demographic characteristics explain between 60 and 80 percent of the variance at the teacher and school level, and only approximately 30 percent at the student level across outcomes. Full findings will be presented at AERA and will be useful for those planning CRTs in school settings.

Authors

• Jessaca K. Spybrook, Western Michigan University

• Fatih Unlu, Amazon

• Eric C. Hedberg, Abt Associates Inc.

• Dea Mulolli, HumRRO

• Tiffany Tsai, RAND Corporation