The Power to Test Moderator Effects in Multilevel Modeling of Single-Case Data

• In Event: 30.075 - The Meta-Analysis of Single-Subject Experimental Design Data: Methodological Challenges and Approaches

Abstract

The increasing number of single-case experimental design (SCED) studies in educational sciences can be used to inform policy, research, and practice decisions. One technique that serves this purpose is multilevel meta-analysis. The focus of this study is on moderator effects to explain variability among effect sizes at the case and study levels. Power calculation of statistical tests to identify true moderator effects in the context of three-level meta-analysis is not investigated yet, but might be problematic given the small number of units at different levels. A simulation study will be presented investigating to which extent power depends on design conditions specific for SCED contexts to identify true moderator effects. A preliminary Monte Carlo simulation study was preformed, containing two predictors at the second level: age (varying between 15 and 25 years), having an effect of 0, 0.2 or 2, and gender (0 for female and 1 for male) set to values of 0, 0.2, and 2. Quality was modeled as a categorical study level predictor (0 = low quality, 1 = medium quality, and 2 = high quality) and also set to values 0, 0.2, and 2. The immediate treatment effect equaled 0 or 2 and the treatment effect on the time trend was 0.2. The number of measurements within a case equaled 20 or 40, the number of cases within a study was 4 or 7, and the number of studies was 10 or 30.
As expected, the magnitude of the parameter value of the moderator effects, the number of cases and the number of studies had a statistically significant effect on power to detect the moderator effects. The power to estimate age (continuous predictor) was too low (ranging from .06 to .56), when the true population value of the moderator effect was set to 0.2 in combination with a small number of cases (4) and a small number of studies (10). When the parameter value was set to 2, the power was larger than .80 across all conditions. The power to detect study quality (ordinal variable) was too low in all conditions with a population value of 0.2. When the population value was set to 2, the power was larger than .80 in the conditions with 30 studies. The same conclusion can be made for gender (dichotomous variable) with the exception that in addition to 30 studies, 7 cases are needed to attain a power of .80. The results of this preliminary study informed the conditions to include in the larger simulation study. In the larger study, the number of predictors at level 1 and level 2 are gradually extended so we can investigate how many cases and how many studies are needed per additional moderator in order to have enough power (>.80) to test moderator effects. The simulation study is currently running; the results are planned to be analyzed by October 2016.

Authors

• Diana Akhmedjanova, University at Albany - SUNY

• David Bogin, University at Albany - SUNY

• Mariola Moeyaert, University at Albany