Designing for Data You Won’t Collect: Lessons Learned from an Early Childhood Planned Missingness Study

• In Event: Survey Data Analysis, Issues in Measurement Error, and Latent Variable Modeling

Abstract

Introduction
Planned missingness designs (PMDs) are used in social policy research to reduce data collection burden, lower costs, and enable broader measurement coverage. In PMDs, data are intentionally not collected from all participants at all time points, with modern missing data techniques used to recover information without biasing estimates. Despite their promise, existing guidance is largely technical and focuses on statistical properties under ideal conditions.

Purpose
This poster asks: How can planned missing designs be effectively implemented and adapted in applied policy research? We present a practice-informed case study documenting the design, execution, and modification of a PMD in a large-scale early childhood evaluation, focusing on design decisions, data structuring, and analytic implications.

Data
We use data from the Pennsylvania Learners Across the Years (PLAY) Study, a multi-site longitudinal evaluation of an early childhood program. The study includes approximately 900 children across treatment and comparison groups and collects repeated measures from parents, families, teachers, and direct child assessments, from baseline and up to three follow-up waves. Measures vary by developmental stage, adding complexity to data collection and structure.

Research Design and Methods
We implemented a multi-form PMD in which all participants completed the full set of measures at baseline, and approximately one-third of measures were omitted at each follow-up wave using rotating cohorts. Missingness was structured across reporters and time to balance coverage while reducing burden. Analyses use longitudinal mixed-effects models with block-specific multiple imputation by reporter type, incorporating both PMD indicators and auxiliary predictors of nonresponse. This approach uses all available data while addressing both planned and unplanned missingness.

Findings
Implementation challenges required mid-study adaptations. Sample composition differed from expectations, with more older children than anticipated, disrupting the intended balance of missingness across age groups. Cohort assignment based on age bands proved misaligned with operational realities, as data collection was organized at the classroom and center levels. High unplanned missingness in the comparison group, combined with planned missingness, resulted in insufficient data coverage, necessitating full data collection for that subgroup. Finally, incomplete alignment between the PMD and age-based measurement transitions meant that some children were missing data when they first became eligible for new instruments. Despite these challenges, the PMD substantially reduced respondent burden and data collection demands while maintaining coverage across key developmental domains. Longitudinal models combined with multiple imputation allowed all participants to contribute data, preserving analytic power and yielding stable, interpretable estimates. The analytic framework remained viable despite deviations from the original design, suggesting PMDs can be robust to real-world implementation challenges.

Implications
The effectiveness of PMDs depends not only on statistical design, but also on alignment with recruitment patterns, measurement systems, and field implementation. We offer practical guidance on structuring PMDs, tracking design-induced missingness, integrating PMD variables into analytic workflows, and adapting designs in response to emerging data realities. This work contributes actionable methods for improving the feasibility and rigor of large-scale policy evaluations.

Author

• Adrienne Danielle Woods, SRI International
  Presenting Author