Can Local Norms Work Without Data From All Students?

• In Event: Challenges and Considerations for Creating Equitable Identification Systems

Abstract

To lower the costs of gifted student identification, districts often divide the process into universal screening followed by the collection of additional data to make formal identification decisions on a smaller subset of students. Many districts also apply local norms in an effort to increase the diversity of students identified. In this paper we examine whether local norms can be used in two-phase systems where data are not available on all students.
We modeled hypothetical one- and two-phase identification systems using data on achievement, ability, and teacher rating scales (TRS) for approximately n=25,611 students from four large districts (see Table 3). We modeled 36 different two stage systems with three different first stage screening variables (ability, achievement, or TRS), three different phase-one cut scores (top 30%, 20%, or 10%), and local or district norms. We also modeled the second phase as the top 10% of the mean of ability, achievement, and TRS. We allowed phase 2 to vary by local or district norms. We compared the students identified by these systems to who would be identified in a universal consideration system. Our main outcome of interest was percent of students missed, which we define as the proportion of students who would be identified in a single-phase system but are not identified under a two-phase system (holding the actual eligibility criteria constant).
We found that when compared to single-phase systems, two-phase systems miss from 0.7% to 57% of students. See Tables 1 and 2 for more details and equations used to calculate the percent students missed. We also find that raising the phase-one cut-off from the top 30% to the top 10% increases the number of gifted students missed compared to a universal consideration system. Ability and achievement measures used as the universal screener for phase one led to fewer students missed compared to using a TRS as a screener. Further, mixing the type of norms used at either phase is not advised as it often resulted in a large increase in the percentage of students missed (e.g., district norm phase one followed by a school norm at phase two: See Tables 4 & 5; Figures 1 & 2).
The choices of a two-phase identification strategy (stage 1 measure, stage 1 cut-off level, stage 1 norm, and stage 2 norm) and the preferred one-phase system (district or school norm) have a very strong influence on the percent of gifted students missed during the identification process. Our findings suggest that as the number of students allowed to pass through phase one increases (i.e., the lower the cut-off level), the percentage of students missed overall decreases. Our findings suggest that achievement and ability tend to miss fewer students than TRS, likely because of their higher reliability and correlation with phase two. Last, we find that two-phase systems can be compatible with local norms if a district’s vision of the ideal gifted identification strategy also incorporates local norms.

Authors

• Daniel A. Long, NWEA

• D. Betsy Mccoach, University of Connecticut

• Scott Joseph Peters, NWEA

• Anthony J. Gambino, University of Connecticut

• Del Siegle, University of Connecticut