Taking Selection and Sensitivity (to Confounders) Seriously in Correlational Studies of Child Development

• In Event: 3-238 - New (or Under Appreciated) Statistical Methods for Probing Causal Hypotheses in Developmental Research

Integrative Statement

The causes of individual differences in growth are of great interest to the field of child development. Much of our cumulative knowledge in this area, however, is based on non-experimental studies. A ubiquitous limitation to this work – relative to randomized experiments – is the potential for selection factors (i.e., pre-existing differences between children that influence their chances of experiencing different conditions) to bias results. Put simply, the practical significance of most correlational studies largely rests on the assumption that unobserved selection factors are not serious enough to nullify or alter the practical significance of findings.

While developmental researchers working with correlational data have become increasingly savvy in their use of covariates to adjust estimates or create balanced groups (e.g., propensity scores), it is well established that the value of these methods pales in comparison to the critical issue of whether or not all relevant selection factors have, in fact, been measured (Steiner et al., 2010). Moreover, while there is a mature literature on best practice for examining the robustness of correlational results in light of selection bias, namely sensitivity analysis (e.g., Oster, 2016), child development researchers have very rarely made use of these resources.

In the present paper, we provide a primer on sensitivity analysis and its relevance to the field of child development. Two easy-to-implement approaches to sensitivity analysis are highlighted, both of which provide intuitive interpretations of how much bias must be present to invalidate a result. First, we discuss the coefficient of proportionality approach (Oster, 2015). This method allows researchers to quantify how much selection bias due to unobserved covariates (relative to the degree of selection due to observed covariates) would be necessary to invalidate a result. Next, we discuss a sample replacement approach (Frank, 2013). In effect, this second approach examines how badly the exchangeability assumption (i.e., the comparison group would have identical outcomes to the treatment group, had they too received the treatment and vice versa) must be violated in order to nullify a result.

We illustrate the conceptual issues at stake with two empirical examples, using two previously published correlational studies examining associations between (1) quantity of early childcare and levels of externalizing behavior problems and (2) quality of care and children’s cognitive achievement. In Table 1, we provide summaries of regression models with and without covariates. And, in Figure 1, we provide the coefficients of proportionality for the two empirical examples. Importantly, while a simple comparison of changes in the child care coefficients might lead to the conclusion that quantity of care findings were more robust, the coefficients of proportionality indicate the opposite was true: quality of care findings were more robust to omitted selection factors than quantity of care findings.

Authors

• Eric Dearing, Boston College
  Presenting Author

• Henrik Daae Zachrisson, University of Oslo
  Non-Presenting Author