Four Complications for Implementing Local Norms

• In Event: Challenges and Considerations for Creating Equitable Identification Systems

Abstract

Local norms are a recommended strategy for improving the defensibility and (potentially) increasing the representation of students from different demographic backgrounds (Peters et al., 2021). Although local norms are a simple idea in principle, in practice there are several complicating factors that arise during their implementation. Researchers often think of test scores as arising from some idealized continuous distribution, such as the normal distribution. In reality, this is an approximation that is reasonable only under idealized limiting cases (e.g., sample size approaching infinity, non-existence of ties). Implementing local norms with small student populations (e.g., school norms or small school districts) can present challenges. The discrete nature of test scores becomes increasingly important in the small n cases, yet these problems are rarely discussed in the literature. This paper describes four such issues affecting local norms: (1) the remainder problem, which occurs when the target identification percentile does not redound to an integer number of students, (2) the quantile stability problem, in which the statistical instability of extreme quantiles across quantiles can lead to unexpectedly large differences in the ability profiles of identified students from year to year, and (3) the clumpiness problem, in which a pre-set local cutoff may not align with a ‘natural’ breakpoint in the rank-ordered scores, and (4) the quantile algorithm problem, in which the many methods for calculating empirical quantiles (or percentiles) from a sample create ambiguity in determining what score threshold accompanies a desired local percentile. For example, R’s `quantile` function can implement nine different methods for calculating quantiles from data (!!), which differ in how they handle ties and interpolation (Hyndman & Fan, 1996).
Traditionally, identification processes are based on comparing student assessment scores against cutoff specified against national norms; for example, requiring identified students to have a cognitive ability score at the 90th national percentile, which corresponds with a score of 120. The identification criteria are fixed, while the proportion of students identified per school or classroom is free to vary. This leads to large and difficult-to-justify variability in the giftedness rate across schools. For example, McBee (2010) found that the gifted identification rates in Georgia elementary schools, which are based on national norms, varied between 0% to more than 50% of students. Local norms take the opposite approach by fixing the identification rate across contexts (e.g., districts, schools, grade levels, or classrooms) at some desired local percentile, for example, the top 10% of students in each context. The qualifying score that meets this criterion is free to vary across contexts; it could be quite high in some contexts and much lower in others. The four problems described above occur when one attempts to define precisely which students in each context should qualify according to this local norm criteria.
The following paper will use simulation and data visualization to provide an intuitive demonstration of the four issues described above. Also, the paper includes a discussion of the estimation of their potential impact, and (where applicable), practical recommendations for handling them.

Authors

• Matthew McBee, Service Management Group

• Lindsay Ellis Lee, Advanced Learning Research & Consulting, LLC

• D. Betsy Mccoach, University of Connecticut