Synthesis of Non-Overlap of All Pairs Using Binomial Family Generalized Linear Mixed Models

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Developments

Abstract

Methods for meta-analysis of single-case experimental designs (SCEDs) are needed due to the prevalence of empirical research using SCEDs within special education, school psychology, speech and hearing sciences, and other fields. Available meta-analytic approaches include raw data synthesis methods (Moeyaert et al., 2014; Van den Noortgate & Onghena, 2008) and two-stage methods involving calculation of effect sizes and subsequent meta-analysis (Pustejovsky & Ferron, 2017). The former, raw-data synthesis approach is appropriate when all studies to be synthesized either use the same outcome measure or use outcome measures that can be re-scaled to a common metric. The latter, two-stage approach is appropriate when the studies to be synthesized use a variety of outcome measures, but where an effect size can be estimated for each case within each study. Effect size estimates are then synthesized using a multi-level meta-analytic model that captures variation within and between studies. A previous simulation study found that this approach performed well for some effect size measures but not for others. In particular, a multi-level meta-analysis model with robust variance estimation worked well for log-response ratio effect sizes (LRR; Pustejovsky, 2015, 2018) but not for within-case standardized mean differences (Gingerich, 1984) or non-overlap of all pairs (NAP; Parker & Vannest, 2009) effect size measures. Thus, there is an outstanding need for meta-analytic methods that account for the specific properties of the effect size measure.
NAP is an effect size in the family of non-overlap measures, which quantify effect magnitude in terms of pairwise rank comparisons of outcomes under different treatment conditions. Because it is based on rank (ordinal) comparisons, NAP is a useful metric for outcomes that are not normally distributed and not on a ratio metric. The scale of NAP ranges from 0 to 1, with a value of 0.5 corresponding to no effect. The limited range of NAP, along with the strong association between its magnitude and sampling variance, presents challenges for multi-level meta-analysis with normally distributed random effects. In this study, we consider an alternative approach to meta-analysis of NAP, based on a binomial generalized linear mixed model.
Ryu and Agresti (2008) proposed an approach for combining and comparing NAP values using a binomial generalized linear model with logistic link. We extend their approach to account for the hierarchical structure of NAP effect size estimates by including random effects for each study and for each case. Effect size estimates are modeled as approximately binomially distributed, conditional on the effect size parameter, with a weight function approximated by the unbiased variance estimator proposed by Sen (1967). We estimate the model using approximate maximum likelihood via Laplace approximation, as implemented in the glmmTMB package (Brooks et al., 2017). We demonstrate the approach by re-analyzing data from a meta-analysis of SCEDs on antecedent sensory-based interventions (Zimmerman et al., 2018). We then evaluate the performance of the approach using an extensive simulation study, which focuses on the bias and accuracy of the overall average intervention effect estimator and the variance component estimators at each level of the model.

Authors

• James Eric Pustejovsky, University of Wisconsin - Madison

• Man Chen, University of Wisconsin - Madison