Simulation study supporting the use of IRT ability scores in clinical trials for developmental disabilities

• In Event: Novel measurement and characterization of disability

Abstract

Background: For neurodevelopmental disabilities (NDD) like autism and intellectual disability, domains such as motor ability, thinking and learning, social abilities, and communication are potential intervention targets. Performance on measures of developmental constructs can be expressed using several types of scores. Norm-referenced scores are intended for the diagnostic context, allowing for the identification of impairment relative to age-based expectations. But they are poorly suited to monitoring change in neurodevelopmental disability, as floor effects can dramatically reduce responsivity, a key psychometric property. Further, as both slower-than-expected growth, stability, and actual loss of skills can all lead to decreasing norm-referenced scores, the meaning of change can be indeterminate. Person ability scores, which are derived via Rasch analysis or item response theory, are intended to measure within-person change. They are available on many widely used developmental tests, though they may carry test-specific names (e.g., W-score, change sensitive score). However, ability scores have not been used or evaluated as primary endpoints in NDD clinical trials. The goal of this study is to provide empirical support for the use of ability scores in this context.
Methods: We simulated parallel-arm clinical trials under several chronological age and degree of impairment conditions, to compare empirically evaluate the power (5,000 replications) and Type I error rate (10,000 replications) of operationalizing the same test performance using ability scores (Growth Scale Values, GSV) rather than norm-referenced scores (V-scale) on the Vineland Adaptive Behavior Scales, 3rd Edition. We hypothesized that GSV would outperform V-scale, especially for older and more impaired samples where floor effects tend to be most pronounced.
Results: We observed an advantage of GSV over V-scale that was progressively larger for each degree of impairment in the sample, and this advantage was slightly more pronounced for the older population than the younger. Across all conditions, the median increase in power associated with the use of GSV instead of V-scale ranged from 3% to 77%. This advantage was at least partially driven by floor effects in norm-referenced scores. The Type I error rate closely approximated the nominal type I error rate of 5% for both scores.
Conclusion: The results of this simulation supported our hypotheses, demonstrating a power and interpretative advantage of ability scores over norm-referenced scores for studies of NDD. While the simulation focused on clinical trial endpoints, the reduction in floor effects could confer greater power to natural history and correlations studies as well. These results are expected to generalize to studies of developmental concepts, regardless of the etiology or specific test. However, the relative advantage of ability scores is expected to be even greater for most other developmental tests, which have a higher floor than the Vineland.

Authors

• Cristan Farmer, National Institute of Mental Health
  Presenting Author

• Audrey Thurm, National Institute of Mental Health

• Jesse Troy

• Aaron J Kaat, Feinberg School of Medicine, Northwestern University