Sampling Clusters with Unrepresentative Individuals Affecting Stratification: Sampling Considerations, Example, and Policy Implications

• In Event: Science Curriculum and Policy in the Rural South

Abstract

The importance of generalizability of randomized control trials has been emphasized for the past 10-15 years. This has led to numerous methods for study design to ensure that the sample chosen for the randomized control represents the inference population, specifically in cluster randomized trials (Author, 2013; Author et al., 2014).When assigning the treatment to the cluster, using cluster analysis with cluster level variables leads to improvement in the generalizability of the study (Author, 2013); however, this often assumes that the sampled students from the clusters are representative of the clusters themselves or are similar across the strata. However, how is the generalizability effected when the individuals of the clusters are unrepresentative of the clusters and differ across clusters that appear similar? This may happen in particular in high school studies looking at upper-level courses such as advanced science courses where the students in these courses differ from the school level mean characteristics, and more importantly, the make-up of these students differ across schools that may seemingly appear similar to one another, such as other low-income schools in urban areas. This paper evaluates the effects of differing student representation on the generalizability using chemistry and physics students within schools. Using Civil Rights Data (USDE, 2021; CRD) and the Common Core of Data (NCES, 2022; CCD), the sampling strata for a science intervention in the South is established first using only the CCD data with variables on student science achievement, free and reduced lunch, and racial makeup of the student population. Within the strata using the Civil Rights Data on the student populations of chemistry and physics class, the student makeup within these courses is compared within the strata and across the strata. Following understanding how the inclusion of these within cluster differing student populations differ across strata, the strata are redefined to include the within cluster student populations and compared to the original strata proposed to understand how the strata now differ. Finally, a random sample of schools is chosen, and their generalizability is compared assuming an inference population that uses the full CCD and CRD dataset. With a difference in generalizability between these strata, outside of the implications for the randomized control trial, a policy question also arises regarding student access and equitability for these key courses

Authors

• Lydia Bradford, Northwestern University

• Barbara Schneider, Michigan State University

• Elizabeth Tipton, Northwestern University