Do Gaps Grow Before, During, or Between School Years? Effects of Measurement Artifacts on Estimates

• In Event: Summer Learning and Test Score Gap Trends: New Descriptive Research, Methodological Issues, and Policy Implications

Abstract

1. Objectives
The way that tests are designed, administered, and scaled can have profound implications for the apparent magnitude and timing of growth in test score gaps. This study shows that artifacts of measurement have important influence over our understanding of how test score gaps by social group change over time, and by season.

2. Conceptual Framework

Different methods of scaling—i.e., different methods for translating a pattern of right and wrong answers into a test score—impact when and whether score gaps appear to grow. Another potential artifact relates to changes in test form.

3. Data Sources

This study compares three longitudinal datasets: the Beginning School Study, the ECLS-K:99, and the Growth Research Database. The datasets draw from different populations of children and give different tests on different testing schedules.

4. Methods

This study uses the three datasets to estimate the size and growth or shrinkage of various test score gaps through school years and summers from the start of kindergarten (1st grade in the BSS) until the end of 8th grade. Gaps by race/ethnicity and between children in high- and low-income families or high- and low-income schools and are estimated so as to evaluate whether the gap-change patterns are consistent across different datasets with different scaling procedures.

5. Results

Results show that questions about gap growth are vulnerable to measurement artifacts. Specifically, scaling approaches and changes in test forms by season may have exaggerated gap growth in some past studies. Net of artifacts, the most replicable finding is that gaps are already large when children enter school and display less net growth later on. It is less clear whether gaps grow faster during the school years or during summer vacations.

6. Scholarly Significance

Despite measurement artifacts, these results justify some broad conclusions about when test score gaps grow. The sensitivity of seasonal patterns to measurement artifacts demonstrate the caution that researchers must use when examining whether test score gaps grow faster during the summer or the school year.

Authors

• Paul T. Von Hippel, The University of Texas - Austin

• caitlin hamrock, The University of Texas