Modeling Heterogeneous Variability in Single-Case Designs With Frequency Count Outcomes (Poster 6)

• In Event: Methodological Innovations and Complexities in the Analysis and Meta-Analysis of Single-Subject Experimental Design Data

Abstract

Single-case design (SCDs) studies are used in a variety of settings to evaluate the efficacy of interventions by looking at change over time, with participants serving as their own controls (Ledford, Lane, & Severini, 2018). For primary analysis of SCD data, researchers predominantly rely on visual inspection of graphed data. Additionally, there is growing interest in statistical methods for the analysis and meta-analysis of SCD data (Ledford, Lane, & Severini, 2018; Pustejovsky & Ferron, 2017; Manalov & Moeyaert, 2017). While multilevel models (MLMs) are a well-developed approach for doing so (e.g., Moeyaert et al., 2014; Ugille et al., 2012), typical MLMs specify that outcomes (or more specifically, error terms) follow normal distributions–an assumption that is a poor fit for some common outcomes in SCDs (Pustejovsky, Swan, & English, 2019). Some recent work has focused on fitting generalized linear mixed models (GLMMs) with the poisson distribution for frequency count outcomes (Declerq et al., 2019). However, the poisson and normal distributions are unable to fully capture the under- and over-dispersion observed in frequency count data in SCDs (Pustejovsky, Swan, & English, 2019). Both MLMs and GLMMs allow for inclusion of fixed and random effects on the mean parameter of the data, but cannot include random effects for further features of the distribution such as variance and skewness (Rigby & Stasinopoulos, 2005). Consequently, these available modeling techniques are limited in their capacity to capture the extent of variation in outcome distributions across participants and studies.
This study investigates whether more flexible models provide more realistic descriptions of the commonly assessed outcome in SCDs: frequency count measures of reading performance. Drawing on an exhaustive review of SCDs with academic intervention targets and outcomes (Van Norman & Klingbeil, 2021), we analyze baseline phase data from 103 multiple baseline or multiple probe SCD studies with target outcomes of reading performance measured as words read correct per minute. To model these data, we consider a class of flexible models known as generalized additive models for location, scale, and shape (GAMLSS), which allow researchers to model all parameters of an outcome distribution (Rigby & Stasinopoulos, 2005). We examine GAMLSS models using normal distributions, poisson distributions with observation-level random effects (OLRE; Harrison, 2014), and negative binomial (NB; Linden, 2011) distributions. We compare model fit when using the GAMLSS model to include random effects on both the mean and dispersion parameters for the normal and NB distributions. We estimate all models using the brms package in R (Bürkner, 2017) and assess the utility of different distributional models using leave-one-out cross validation information criteria, diagnostic plots, and posterior predictive checks.
We find that currently used modeling methods do not adequately capture the features of frequency count measures of reading outcomes in SCDs. Results indicate a need to further explore poisson-OLRE models, NB models, and GAMLSS models within the context of SCDs. Findings also suggest that SCD researchers should be wary of utilizing distributions with fixed dispersion and should carefully assess features of their outcome data prior to selecting a model.

Authors

• Paulina Grekov, University of Wisconsin - Madison

• James E. Pustejovsky, University of Wisconsin - Madison