Consequences of Misspecification of Growth Trajectories When Meta-Analyzing Single-Case Data Using a Three-Level Model

• In Event: Methodological Dilemmas Encountered in Analyses and Meta-Analyses of Single-Case Design Research

Abstract

Issues of treatment effectiveness have become increasingly important in education, specifically involving single-subject experimental designs (SSEDs) (Shadish, W. & Rindskopf, 2007). Meta-analytic procedures allow researchers to quantitatively synthesize past research results, and provide evidence for best practices (Hedges & Olkin, 1985). Raudenbush & Byrk (2002) demonstrated that a meta-analysis can be seen as a special case of multilevel analysis with participants (Level 1) nested within studies (Level 2). Raw data from a set of SSED studies have a similar structure. Multilevel models have been suggested as a method for combining single-case data within and across studies (Shadish & Rindskopf, 2007; Van den Noortgate & Onghena, 2008). Van den Noortgate and Onghena (2003b) illustrated the use of a two level model to analyze data in the individual studies in single case designs. If data from several SSEDs are used, now scores can vary at each of three levels: scores may vary over occasions (level 1); across participants from the same study (level 2); and across studies (level 3) (Van den Noortgate & Onghena, 2008).
The multilevel approach allows a large degree of flexibility in modeling the data (Goldstein & Yang, 2000; Hox & de Leeuw, 1997). Researchers can make various methodological decisions when specifying the model to approximate the data. Those decisions are critical since parameters can be biased if the statistical model is not specified correctly. One of those decisions is in the area of growth trajectories. A researcher has to decide if the trajectory should be one of no growth (no change in level over time), linear growth (constant slope over time), or non linear growth (e.g., curvilinear, quadratic, or logistic)? The most common such growth model is that of gradual rise with a leveling off at an asymptote (Shadish & Rindskopf, 2007).
The purpose of this paper is to analyze the degree to which parameter estimates are sensitive to various methodological decisions, specifically regarding the specification of the growth trajectories. A meta-analysis of the SSEDs involving self-monitoring (the independent variable) and academic time on task (dependent variable) using a three level model was conducted. A search of ERIC, PsychINFO, Web of Science, and Google Scholar databases was conducted to identify SSEDs involving self-monitoring and time on task. Raw data were extracted from these studies and transformed when needed to the common metric of percent time on task to be used in the three level models. Three distinct models involving different specifications of growth trajectories (no growth within a phase, constant linear growth, or non linear growth) were analyzed to understand the impact of this methodological decision on the parameter estimates. Results will serve to inform the field regarding the degree to which specification of growth trajectories impact parameter estimates.

Authors

• Merlande Petit-Bois, University of South Florida

• Eun Kyeng Baek, University of South Florida

• John M. Ferron, University of South Florida