Poster #124 - Two Methods for Getting Consistent Results from Simple Change Scores and Residualized Change Scores

• In Event: 2-055 - Poster Session 06
  In Poster Session: PS 06 Section - Methods, History, Theory

Integrative Statement

Research for optimizing child development requires valid causal explanations of between-person differences in within-person changes. Causally relevant estimates from non-randomized studies are considered more valid if adjusted for pre-existing differences on the outcome variable. The two major methods to adjust for pretest differences often produce contradictory results, however, as shown by Lord’s (1967) paradox. Historically, ANCOVA-type analyses of residualized change scores have been preferred over analyses of simple change scores, but have been recently shown to be biased in typical applications (Berry & Willoughby, 2017; Hamaker et al., 2015).

The current study features the first known Monte Carlo simulations of Lord’s original paradox. The first simulated dataset is designed to fit the null hypothesis for simple change scores, and the second to fit the null hypothesis for ANCOVA. We replicate Lord’s paradox and show that the discrepancy between the two estimates is a function of the difference between the pretest group means and the pooled within-group stability coefficient. We demonstrate two ways to get consistent results across analyses of both types of change scores: matching on pretest scores and group-mean-centered ANCOVA (adapted from quasi-ANCOVA: Huitema, 2011). Unfortunately, those two sets of consistent results are as discrepant from each other as the original discrepancy in Lord’s paradox.

We then show the same pattern of results for medication and psychological treatments for depression in the Fragile Families longitudinal data. ANCOVA, matching, and 3-wave cross-lagged panel analyses all make both depression treatments appear to be harmful, i.e., predicting more depression severity in the next wave than predicted by preceding depression severity scores. In contrast, analyses of simple gain scores, centered ANCOVA, and 3-wave latent growth models all indicate that both depression treatments are effective in reducing depressive symptoms, from identical data.

These disconcerting results add to recent questions about unrecognized biases in cross-lagged panel analyses and other ANCOVA-based longitudinal analyses (Berry & Willoughby, 2017; Hamaker, Kuiper, & Grasman, 2015; Hoffman, 2015). However, it is not clear when analyses of simple gain scores are less biased. Fifty years after Lord’s paradox, the most causally valid way to analyze change in children is still not clear. Indeed, these results add to prior evidence (LaLonde, 1986) that consistent results across analyses do not guarantee unbiased estimates, even though the expectations of results from analyses of both types of change scores are consistent and unbiased for optimally implemented randomized trials.

Nonetheless, this study supports the following positive contributions: When analyses of simple gain scores are thought to be less biased than ANCOVA-type analyses, centered ANCOVA can estimate that effect size with more statistical power than standard analyses of simple change scores. If pretest matching is used to equate cases selected from two distinct groups, this may lead to biased results, due to regression toward different group-specific means (Campbell & Kenny, 1999).

Child development research needs to more fully understand and solve these problems if it wants to produce the kind of unbiased causal estimates that are necessary for optimizing child development.

Authors

• Hua Lin, Oklahoma State University
  Presenting Author

• Robert E. Larzelere, Oklahoma State University
  Non-Presenting Author