Measuring Preschool Classroom Quality: Can we Derive Comparable Scores Across Different Observation Tools?

• In Event: Advancing the Measurement of Early Care and Education Quality to Support Young Children’s Development

Abstract

With research relating early care and education (ECE) quality to positive child development (Burchinal et al. 2016; Zaslow et al. 2010), policy efforts at both the state and federal levels have increasingly emphasized measuring and improving the quality of ECE. Classroom observations are often used to assess quality as part of ECE monitoring and accountability systems. The Classroom Assessment Scoring System™ Pre-K (CLASS™, Pianta et al. 2008) and the Early Childhood Environment Rating Scale-Revised (ECERS-R; Harms et al. 1998) are two widely used observation tools to measure preschool classroom quality, with the CLASS focusing on process quality and the ECERS-R measuring global quality.

The ECERS-R was recently revised to include stronger measurement of interactions that support language and cognitive development. However, the research about the ECERS-3 (Harms et al. 2015) and how well it measures these interactions is limited. The purpose of this study is to explore the alignment of the ECERS-3 indicators with the dimensions on the CLASS Pre-K, and to examine how ECERS-3 indicators might be used to support comparisons of quality over time if monitoring systems change from the CLASS Pre-K to the ECERS-3, as some states are currently considering.

Using data from the ECERS-3 Validation Study (Early et al. 2018), collected during the 2015–2016 academic year from convenience samples of programs in three states, our analysis focused on the 118 classrooms that were observed using both the CLASS Pre-K and the ECERS-3. The ECERS-3 includes 35 items organized into six subscales and a total score. The item scores are based on a set of yes/no indicators under each item. The CLASS Pre-K assesses three domains of teacher-child interactions along 10 dimensions: Instructional Support, Emotional Support, and Classroom Organization.

To inform understanding of these measures, we conducted a crosswalk analysis and confirmed it with iterative Rasch analyses. Drawing on the work of Early and colleagues, three early childhood experts completed a conceptual mapping of the ECERS-3 indicators with the CLASS Pre-K dimensions within each of the domains (Table 1). Analyses included the following steps: (1) implement Rasch model with item “groups” that included ECERS-3 mapped indicators and CLASS Pre-K dimensions for each CLASS domain; (2) exclude the CLASS dimension scores and re-estimate the Rasch model with item difficulties for the mapped ECERS-3 indicators anchored on the estimates from the prior analysis; (3) use Rasch estimates for each dimension and step category from the first model to derive CLASS dimension scores with the same range as the original CLASS scores; (4) estimate domain scores by taking the average of the derived scores across the dimensions within that domain.

Preliminary results show that the derived CLASS scores (based on ECERS-3 indicators) had stronger correlations with the observed CLASS Pre-K scores than the ECERS-3 total scores had (Table 2). Adjacent agreement (the percentage of crosswalk scores that fall within one point of the observed CLASS scores) ranged from 71% to 89% for the domains. We will discuss implications for comparing these widely-used ECE classroom quality assessments using this approach.

Authors

• Yange Xue, Mathematica Policy Research - New Jersey
  Presenting Author

• Sally Atkins-Burnett, Mathematica
  Non-Presenting Author

• Jessica Harding, Mathematica Policy Research - New Jersey
  Non-Presenting Author