Poster #88 - Temporary Assessment Process Insights: Use of Modified Evaluations and Reflective Self-Study in North Carolina

• In Event: Poster Session 13

Abstract

Environment Rating Scale (ERS) assessments are one component of North Carolina’s state regulations for childcare programs seeking higher star ratings for their license. During Covid-19 when lengthy observations were not possible, a Temporary Assessment Process (TAP) was designed to provide an alternative to the typical ERS assessments. The TAP included two parts that represented aspects of classroom and program quality addressed by the ERS. The first part was a Reflective Self-Study (RSS) that addressed content areas for items that could not be observed and scored. The second part was a Modified Assessment (MA) that included scoring selected items on the ERS using photographs from a short visit when children were not present and interviewing the teacher. “Thinking More” worksheets were developed; one for each subscale for the ITERS-R, ECERS-R, SACERS-U, and FCCERS-R. These worksheets include questions about indicator requirements and setting-specific examples. The RSS required three worksheets: Language and Interactions, Personal Care Routines, and Program Structure. Programs were given 4-weeks to complete the RSS worksheets before the MA.

Comparison of scores from the MA with past traditional ERS assessments (Table 1) showed higher scores for the MA when compared to both the full-scale assessment and when just the modified items were matched. The increase in scores may be due to greater impact of the interview to typically observed item scores.

ECE professionals (222 teachers, 58 administrators, and 25 FCCH providers) shared their opinions about the RSS through an online survey. Participants rated each of the three worksheets on a 1-5 scale (1= Not at all helpful, 5= Extremely helpful). In general, the worksheets were considered helpful. Even though the Language and Interactions was the longest and most complex, it was consistently rated as “very helpful (4)”. Nearly every person said the “Thinking More worksheets encouraged me to reflect on my daily practices.” Most teachers and providers reported using the ERS and collaboration with others during the process. Administrators agreed with much of the process (Table 2).

Researchers created a rubric to provide an overall score related to the perceived “completeness” of the Thinking More worksheets. The rubric evaluated difficult-to-define facets relevant to this investigation (e.g., did teachers provide full and complete responses, were responses verbose/elaborate vs. short/brief, did providers simply cut-and-paste answers), and it ranged from 1 to 5 points, with higher scores indicating a “more complete” worksheet. Raters were trained to reliably score the rubric and make judgements about the completeness of the responses. The rubric scores showed the Language and Interaction worksheet was the hardest to complete.

Covid-19 necessitated a new assessment approach but also created a potential future opportunity to include self-assessment in the star rating point structure, which would help incentivize this quality improvement activity. Further work is needed to understand the time and effort required to have an effective self-study process in our system. Greater support is needed to help administrators guide a self-study process with their staff and take a more active role in working with staff to complete the self-assessment with good depth.

Authors

• Linda Hestenes, University of North Carolina at Greensboro
  Presenting Author

• Lia Rucker, University of North Carolina at Greensboro

• Blenda Chor Rodrigues, University of North Carolina at Greensboro

• Stephen Hestenes, University of North Carolina at Greensboro

• Sharon Mims