Lessons from the first ManyBabies project

• In Event: 3-145 - ManyBabies: Bringing large-scale collaborative projects to infant development research

Integrative Statement

ManyBabies emerged as means of addressing questions of reproducibility within the field of experimental infant research (Frank et al., 2017), by supporting large-scale, collaborative efforts across laboratories. ManyBabies1, the first of its kind, collected data from 68 laboratories around the world with over 2,700 infant participants, examining infants’ preference for infant-directed speech (IDS) over adult-directed speech. This phenomenon was selected for the first ManyBabies project because there is a robust theoretical and experimental literature going back decades (see e.g. Soderstrom, 2007 and Golinkoff et al. 2015 for reviews), and meta-analytic support (Dunst et al., 2012; metalab.stanford.edu). The main goals of ManyBabies1 were to a) measure the effect size of this effect, b) examine methodological factors influencing the effect size and c) examine theoretically important infant characteristics influencing preference for IDS (infant age and infant language community). Importantly, this project has also resulted in a rich dataset that can be used for secondary analyses, as well as providing “lessons learned” for conducting future large-scale projects of this type.

ManyBabies1 measured the overall effect size of d = .35, somewhat lower than the meta-analytic estimate from the literature (d = .72; Dunst et al., 2012). The effect size for labs using the Headturn Preference Procedure was larger than for labs using eyetracking or manual coding. As predicted, the preference for IDS was strongest in infants learning North American English. Counter expectations, the preference for IDS increased across the ages tested (3 -15 months).

In this talk, we build on these findings by examining the impact that specific methodological choices have on the effect size of the IDS preference and on trial completion (i.e. ‘fuss-out’ rate). In particular, we examine factors that are sometimes questioned during peer review or offered as explanations for differing findings between labs, including the training or experience of the person administering the study and ‘incidental’ features of the testing environment including room size, wall color, and lighting level.

As the first project of this type, we also discuss challenges, strategies and best practices in the implementation of large-scale, cross-laboratory collaborative experimental research. Given the scale of data collection (close to 50,000 trials), a significant hurdle was collecting accurate, machine-readable data from diverse laboratories around the world. A concern of a different kind involved balancing the need for cross-laboratory methodological consistency with the practical considerations of data collection for each laboratory, further complicated for a project in which methodological factors were themselves under study. A variety of online tools supported the particular needs of this collaborative endeavour. In an exit survey, participating labs reported high satisfaction with the experience and the outcome. However, two central themes emerged in feedback - one related to the choices of visual and auditory stimuli, and the other related to the clarity and organization of the instructional materials. Future projects should pay particular attention to this latter concern, as well as the structure of the data submitted by laboratories.

Authors

• Melanie Soderstrom, University of Manitoba
  Presenting Author

• Melissa Kline, Center for Open Science
  Non-Presenting Author