Understanding Pre-K CLASS Estimates of Classroom Quality in Head Start Research

• In Event: Advancing the Measurement of Early Care and Education Quality to Support Young Children’s Development

Abstract

The Classroom Assessment Scoring System-Pre-K (Pre-K CLASS; Pianta et al., 2008) is a widely-used observation instrument designed to assess the quality of teacher-child interactions in center-based preschool classrooms. The reliability of estimates may vary based on study procedures in implementing the instrument, including observer training and procedures, and observation factors such as cycle length and number, and score stability (e.g., Buell et al., 2017; Curby et al., 2010; Nguyen, 2020; Derrick-Mills et al., 2016; Mashburn, 2017). As with all measures, validity is related to its use.

Pre-K CLASS is used in the Head Start Family and Child Experiences Survey (FACES), a nationally representative study of Head Start programs and classrooms, the program staff, and the children and families served. FACES data are available to researchers to answer questions about the experiences of low-income children and families. The use of the Pre-K CLASS in large-scale research on Head Start necessitates a deeper understanding of how well the scores characterize a national picture of classroom quality.

This study analyzed potential sources of error using Pre-K CLASS data from the spring 2015 and 2017 rounds of FACES 2014. These data represent 1,284 classroom observations (in 183 programs and 395 centers) by a total of 79 trained observers. We used a generalizability study (Brennan, 2001) to examine the variance associated with different factors and potential sources of error in the estimation of classroom quality scores, and a decision study to estimate what strategies might maximize data reliability. Using the observation data and observer certification results, we also conducted an item response theory (IRT) analysis to assess observers’ leniency and consistency (Fox & Jones, 1998) when rating classrooms, and examine whether controlling for leniency might improve estimates.

Models explained around 70 percent of the variance in the Classroom Organization (CO) domain, and around 80 percent of the variance in the Emotional Support and Instructional Support domains. The greatest proportion of the variance was attributable to the classroom or interactions between the classroom and different facets. Among the domains, Instructional Support (IS) had greatest proportion of the variance (27%) associated with the observer or the interaction of the observer with other facets. Both CO and IS had more than a quarter of the variance associated with the cycle within a classroom. This supports other research about the importance of the classroom content and context in estimating quality. The decisions study results indicate that Pre-K CLASS reliability was within acceptable range with the four cycles as administered in FACES 2014. One limitation of this study is each classroom is nested within one observer. Each observer rated a median of 19 classrooms.

The IRT analyses indicated that the observers were consistent in rating classrooms, and leniency was not related to observers’ certification results. The model suggested that a CLASS total score can be reliably estimated, and the factor structures found in previous studies appear to be driven by the difficulty of reaching higher levels of quality in each of the domains.

Research implications will be discussed.

Authors

• Lizabeth Malone, Mathematica Policy Research - District of Columbia
  Presenting Author

• Cassandra Baxter, Mathematica
  Non-Presenting Author

• Sally Atkins-Burnett, Mathematica
  Non-Presenting Author