Engaging Early Childhood Educators in the Process of Setting Standards

• In Event: Communicating Psychometric Information to Diverse Audiences: From Early Childhood Teachers to Policy Makers

Abstract

1. Objectives
Recently, the California Department of Education (CDE) chose to use the DRDP to determine the proportion of children in the state who are Kindergarten-ready, as well as for several points along the age continuum leading up to school readiness. This has involved the selection of cut scores for each of these points, referred to as Criterion Zone Boundaries (CZBs). In this presentation, we will describe the method used for determining cut scores aligned with the state’s early learning and kindergarten standards.

2. Theoretical framework
Although the DRDP was primarily developed as a formative, observational instrument for children birth through Kindergarten, using authentic observational assessment to provide developmental information in eight domains, the CDE desired a method for aggregating these data to provide information about state-level readiness.

We used standard setting methods consistent with the BEAR Center’s assessment development framework, known as the Criterion Mapping method, which has been used successfully with K-12 teacher populations. This method combines item response modeling with professional judgment to set criterion-based standards for school readiness. We use a Wright Map, a graphical representation of an item response model, that shows the person distribution and the item levels on the same scale. In addition, the method uses software that allows committee members to determine the probability of each response to each item, for a child at a given level, and also the “expected response” for each item, the mean response that would be expected on an item for a large number of children at that level.

3. Methods
It was important to include early childhood educators, as well as child development experts, when considering the cut points that would be used for Kindergarten readiness. We provided several modifications to make the Criterion Mapping method appropriate for this group. First, we included an activity to acquaint all participants with California’s early learning foundations and kindergarten standards. We then provided extensive training slides, worksheets, tools, and other support materials, to ensure the understanding of all participants.

4. Data sources
The results were aggregated from children in California’s state-funded early childhood programs, at the end of the 2017-18 school year. These data are weighted toward children from lower-income families, and include large proportions of dual-language learners.

5. Results
As a check on the CZBs selected, we reconvened the groups, with information about what percentages of children were quite likely, somewhat likely, somewhat unlikely, or very unlikely to have passed a given boundary. Even though for several of the domains, a large proportion of the children had not passed the boundary, the committees made few changes in the existing CZBs.

6. Scholarly significance
These CZBs will provide the state with their first look at the proportion of children reaching Kindergarten readiness. They will allow examination of the data by ages, ethnicities, and other subgroups in order to determine where resources can best be placed to increase the proportion of children who are Kindergarten ready. Finally, they will enable monitoring of progress toward California’s readiness goals.

Author

• Karen L. Draney, University of California - Berkeley