Poster #82 - Program Quality Assessment - Revised Validation Study

• In Event: 2-055 - Poster Session 06
  In Poster Session: PS 06 Section - Education, Schooling

Integrative Statement

The PQA-R is a comprehensive program and classroom evaluation tool for preschool programs. The PQA-R measures the quality and use of
• Indoor and outdoor learning environments
• Teaching and learning routines
• Adult-child interactions, including interactions that encourage the development of
o Language (expressive and receptive)
o Vocabulary
o Math
o Literacy
o Social and emotional skills
o Executive function skills
o Conflict resolution skills
• Evidence-based comprehensive curricula
• Planning to differentiate instruction based on student interests
• Ongoing child assessment to monitor learning and adapt/modify learning if needed
• Family engagement activities

Testing the PQA-R
We conducted three sequential studies to test and refine the assessment items: an initial pilot, a revised pilot, and an implementation pilot. We assessed 585 classrooms across the studies. The results, for all three studies, were analyzed using the Rasch. The assessment was iteratively revised using the results of the analyses and assessor feedback. The poster will focus on the iterative development and the Implementation Pilot Phase results.
Implementation Pilot Phase

We conducted an implementation study, during April and May of 2018. The classrooms were located in 5 states throughout the United States and included classrooms located in urban, rural, suburban, and urban fringe areas. The classrooms were comprised of teachers with a mix of experience, education, and training, and children with a mix of socioeconomic status, ethnicity, and race. Assessors completed a survey about the assessment and their use of the assessment at the end of the revised pilot phase. We used their feedback as well as the analysis to add or delete items and to revise the wording of the items for the version we will use in the 2018-2019 implementation study.
Implementation Phase Analysis and Results
After the assessors completed the assessments, we examined whether the levels used to assess preschool classroom quality indeed follow the theorized progression, and we conducted reliability and validity studies. We also examined whether the structure of the instrument is consistent in distinctly capturing the key domains. We found that the items although orderly some were misfitting and we still found some low reliabilities. We also found that the 3- and 5-dimensional models were close, but that domains 2 and 3 were highly correlated and 4 and 5 were highly correlated suggesting a 3-dimensional model fit best.
After removing all of the misfitting items the Scale Reliabilities (Chronbach's Alpha ) for a sample of N=220 preschool classrooms ranged from .850 to .950. The Domain Correlations (RASCH) showed the intercorrelations among the domains of the PQA-R ranged from .279 to .816. The weighted and unweighted Mean-Square fit statistics for the items were all below 1.5 except for 3 items that the SME considered too important to remove.
Summary
The implementation pilot results show that the PQA-R assessment provides reliable and valid psychometric measurement of preschool classroom quality in three key domains. Expected levels of intercorrelations between domains are evident in the data.

Authors

• Jeffrey Allan Beal, HighScope Educational Research Foundation
  Non-Presenting Author

• Fernando Andrade, HighScope
  Non-Presenting Author

• Jill Claxton, Center for Early Education Research and Evaluation (CEERE)/ HighScope
  Presenting Author