Planning your pilot study: What to do and what to avoid

• In Event: 2-151 - Step by Step: Improving best practices for research with children

Integrative Statement

Piloting – testing out a study protocol on a small group of participants before collecting data from the whole sample – is an essential part of the research process, particularly for developmental researchers. Here, we will delineate ways that piloting can be used to improve the research process, as well as some current practices in piloting that should be avoided as they may inadvertently inflate Type-I and Type-II error.

A survey of developmental researchers (Eason, Hamlin, & Sommerville, 2017) showed two themes for why researchers pilot their studies. The first is piloting for feasibility, for example, to determine whether (1) the target sample can be recruited in a reasonable timeframe, (2) the stimuli and task are conducive to data collection, (3) the exclusion criteria lead to a higher-than-expected attrition rate, (4) child participants give the type of response that is expected, e.g., pointing, verbal response - regardless of whether the response is “correct”. Behind all of these reasons for pilot lies the question: “Is it possible to execute this study as planned?”. We agree that this type of piloting is very important for developmental research, as it can improve the reliability, validity, and quantity of data obtained.

A second reason researchers cite for piloting is to test whether the results from the pilot conform with their hypothesis (Eason et al., 2017). The preliminary results are then used to decide whether to invest in pursuing a full sample. For example, researchers may be using a “headcount rule” where if e.g. 4 out of 6 pilot participants show an effect in the right direction, the researcher will continue on to run the full study, or if not, tweak the study design (see also Peterson, 2016, for an ethnographic description). The problem with this approach is that estimates from small sample sizes can be unreliable (Lakens & Evers, 2014), and thus researchers using this practice may end up abandoning a true hypothesis because the pilot results were unpromising (thus committing a Type-II error). Another drawback of this type of piloting is that using pilot studies to test for effects can lead researchers to run many underpowered pilot studies instead of running few, high-powered full studies (Schott, Rhemtulla & Byers-Heinlein, under review).

A piloting practice that is more clearly problematic is a preliminary hypothesis test when researchers haven’t decided in advance whether the study is still in the pilot stage or regular data collection has begun (see Eason, Hamlin, & Sommerville, 2017). For example, researchers sometimes keep pilot data as part of their final sample if results were in the expected direction. When simulating this piloting procedure assuming that the null hypothesis is true, Type-I error can increase from 5% to 11% (Schott, Rhemtulla & Byers-Heinlein, under review).

In sum, piloting can be an important approach for developmental researchers to avoid “wasting” precious participant and research resources. However, it is important that piloting efforts focus on questions of feasibility, rather than checking whether the effect of interest is present in an underpowered sample.

Authors

• Esther Schott, Concordia University
  Presenting Author

• Anne Scheel, Eindhoven University of Technology
  Non-Presenting Author

• Mijke Rhemtulla, University of California, Davis
  Non-Presenting Author

• Krista Byers-Heinlein, Concordia University
  Non-Presenting Author