Confidence Intervals for Fine-Grained Effect Sizes (Poster 4)

• In Event: Methodological Innovations and Complexities in the Analysis and Meta-Analysis of Single-Subject Experimental Design Data

Abstract

A limitation of many commonly used effect sizes for single-case experimental designs is that they index effects at relatively coarse levels, such that there is only one effect size per study (e.g., SMDB), or per case (e.g., TauU). In contrast, fine-grained effect sizes (FGES) provide effect estimates for each treatment phase observation of each case. To estimate FGESs, a model is fit to the baseline observations of a case and used to predict what would have been observed had there been no intervention. The difference between a projected baseline value and the observed treatment value at the same moment in time provides the estimate of the raw score FGES. The projected baseline value is estimated from a regression including only the intercept if temporal stability is assumed or the intercept plus time (or transformations of time) if the expected value is assumed to change over time. To allow visual assessment of how the effect changes as the intervention progresses and how the effect trajectory differs between cases, the FGESs are plotted using line graphs (one line per case). When different cases have different outcome measures, transformations can be used to put the fine-grained effects on a common scale (e.g., converting to proportion change by dividing the raw score effect by the projected baseline value, or proportion of goal obtained by dividing the raw score effect by the difference between the projected baseline value and the goal). The purpose of this poster is to develop confidence intervals for fine-grained effect sizes, along with ways to augment the graphical presentation of fine-grained effects to show the uncertainty in the estimates.
We consider the observed treatment value at time i for case j as known, and thus the error in estimating the FGES at time i for case j comes from the error in predicting what would have been observed at that moment in time had there been no intervention. We use regression (OLS for our examples in this poster, but the approach could be extended to GLS to accommodate assumed autocorrelation), to determine, not only the projected baseline values at each intervention time point, but also the 95% prediction limits of each of these predicted values. We estimate the limits of the CI for a raw-score FGES by finding the differences between the observed treatment observation and the lower (and upper) prediction limits of the projected baseline value. For transformations of the FGES to proportion change or proportion of goal obtained, we use a similar process to create confidence intervals, because again the only quantity subject to sampling error is the prediction of what we would have observed had there been no intervention. We provide examples to illustrate the computation of FGESs and their confidence intervals using extracted data from the graphs of published single-case studies, and show how to integrate the estimates of uncertainty into graphic displays by the use of error bars or the weighting of the trajectory lines.

Authors

• Lodi Lipien, Florida Virtual School

• Megan Kirby, University of South Florida

• John M. Ferron, University of South Florida