Heterogeneous Level-1 Phase Variances in a Three-Level Meta-Analysis of Single-Subject Research Data

• In Event: Methodological Dilemmas Encountered in Analyses and Meta-Analyses of Single-Case Design Research

Abstract

Multilevel modeling represents a useful framework for meta-analyzing single-subject research data, however, questions remain concerning its methodological properties under the reality of the single-subject data’s characteristics. In this study, a Monte Carlo simulation was conducted to investigate recovery of parameters for a three-level model of repeated measures (level one) within participants (level two) across studies (level three). Participants’ growth trajectories were generated to follow a quadratic growth function. In addition, different level-one error specifications were generated including, different per-phase variance component values and lag-one autoregressive covariance structures with parameter values based on characteristics typical of published single-subject data. Design conditions that were manipulated in the study included the following: the number of data points per phase, the number of subjects per study, the number of studies meta-analyzed, the degree of autocorrelation in residuals, and the homogeneity of level-one variance values across the treatment and baseline phases. Outcome variables examined included the rates of convergence, power for statistical tests of fixed effects, and the relative parameter bias of each fixed effect, random effects’ variance component, and autocorrelation estimates. SAS was used to generate the data and SAS PROC MIXED was used to estimate the multilevel model.
Convergence rates were found to be 100% for all level-one error specifications and design conditions. As expected, power for detecting non-zero fixed effects was associated with the magnitude of the true effects and the numbers of data points per phase, subjects per study, and studies meta-analyzed. The relative parameter bias of the fixed effects estimates was found to have complex associations with the number of data points per phase, levels of autocorrelation, and the continuity/discontinuity of variance across phases. Under the conditions examined here, the random effects’ variance components were found to be consistently substantially biased with, again, complex patterns for the relationship between the degree of bias and design conditions that differed by type of random effect. Last, autocorrelation estimates were also found to be consistently biased across conditions. The final poster will include a fuller description of the study’s results and will include discussion of the implications for the meta-analysis of single subject data and for primary single-subject research.

Authors

• Austin Madison Mulloy, Virginia Commonwealth University

• Susan Natasha Beretvas, The University of Texas - Austin