Poster #8 - Revisiting a meta-analysis of caregiver training studies: Which factors moderate training effectiveness?

• In Event: 3-001 - Poster Session 2 during Breakfast

Abstract

The present study re-analyzed a published meta-analytic dataset on the effects of specialized training on competencies of caregivers in childcare (Fukkink & Lont, 2007). Noting that the literature was inconclusive about training effectiveness, the original authors conducted an exploratory analysis of the training- and study characteristics associated with the effect size found. They coded 24 moderators, covering the training design and content, methodological characteristics of the study, and the operationalization of the dependent variable. They used multilevel meta-regression to identify the effect of these moderators on the effect size of the intervention. However, due to the low ratio of studies (K = 17, yielding k = 78 effect sizes) to moderators (M = 24), this analysis was severely underpowered, and therefore prone to overfitting (describing noise in the data, rather than true effects; Hastie, Tibshirani, & Friedman, 2009). To overcome this limitation, we re-analyzed the data using MetaForest (Van Lissa, 2018), a method developed to identify relevant moderators in meta-analysis. Based on the random forests algorithm, this technique is robust to overfitting, and retains sufficient power to detect relevant moderators when the ratio of studies to moderators is low (Breiman, 2001). Because of the multilevel nature of the data (effect sizes nested within studies), we used clustered bootstrapping to obtain unbiased estimates of prediction accuracy and variable importance (Janitza, Celik, & Boulesteix, 2016).
Preliminary analysis (ntree = 10.000, mtry = 4) revealed that many (10) of the moderators had negative variable importance metrics, indicating that their presence undermined predictive performance. Because random forests performs poorly when the proportion of irrelevant moderators is high, we removed predictors with the most negative permutation importance recursively, until only variables with positive variable importance remained. We report results averaged across 100 replications of this algorithm (ntree = 2.000, mtry = sqrt[M]).
The original authors concluded that relevant moderators were: Having a fixed curriculum, Location, Alignment of the dependent variable with the training, and domain of training (effects were greater when the training focused on skills). Our analysis yielded slightly different results (Figures 1 and 2). In line with the original authors, we found that Alignment, Location, and Curriculum were important. We did not find an effect of training Skills, however. Instead, we found an effect of training Knowledge, using a pre/post design, and having experimenters blind to condition. The median R_oob^2, which estimates a model’s predictive performance in new data, was .17.

Author

• Caspar J. van Lissa, Utrecht University
  Presenting Author