Lessons From Ready to Learn Research: The Molly of Denali Efficacy Study

• In Event: Lessons Learned From Research During COVID-19: Strategies for Incorporating Remote Data Collection Beyond the Pandemic

Abstract

The onset of COVID-19, and the resulting stay-at-home orders across the country, required education researchers either to abandon their studies or to be agile in their adaptations of protocols, instruments, and procedures. We were confronted by this challenge while researching the impact of Molly of Denali, a PBS KIDS series targeting children’s ability to use informational text to answer real-world questions. Public Broadcasting Service (PBS) developed the series in partnership with the Corporation for Public Broadcasting (CPB) and in collaboration with GBH, as part of the Ready To Learn Initiative, funded by the U.S. Department of Education.

We designed a 9-week randomized controlled trial with 500 first graders from low-income communities at six sites across the U.S. After completing pretests with 127 children at two sites, COVID-19 forced us to cancel in-person data collection. We completed posttests with these 127 families remotely, and then conducted an entirely virtual replication study with a new sample of 136 children. Our sample was neurodiverse; about 17% had an IEP or 529 plan.

Our team overcame numerous obstacles as we pivoted to virtual data collection procedures, including research protocols (e.g., consent, randomization), instrumentation, technology, and the virtual environment itself. Our lessons learned include properly informing participants about the study, ensuring eligibility, obtaining consent that was satisfactory to our IRB, randomizing to condition, and screening out less motivated participants. Research instruments presented their own challenges, as we scrambled to find or adapt our selected measures to the virtual format, with an assessor using screen sharing in Zoom. Our testing resulted in the elimination of one baseline measure, the adoption of another in a publisher-provided digital format, and the adaptation of a researcher-developed instrument. However, the potential for virtual data collection to introduce confounds is notable: the virtual format requires skills such as expressive language ability, that are not required during in-person assessment that allows children to rely on pointing or gesture. Ensuring child engagement – and parent restraint from interference – was another set of tasks. In terms of technology, we were fortunate that our study design included providing families with data-enabled tablets, as that ensured a baseline availability, but we still encountered numerous technological difficulties, for both participants and assessors, that required a tech manager to be a part of the data collection team. Finally, there are limitations in the virtual format itself, including noisy backgrounds, setting up breakout rooms to protect privacy, and coordination of multiple participants with limited staff.

Our solutions to these challenges can provide insights for other researchers looking to adopt virtual data collection – a useful process not just for continuing research during a pandemic, but also for reaching participants who might not otherwise be reachable because of rural location or lack of resources to travel to a data collection site. This is particularly important in our work with participants from low-income communities, where limitations in the availability of technology or broadband access might hamper participation and jeopardize the representativeness of the study sample.

Authors

• Joy Lorenzo Kennedy, Education Development Center (EDC)

• Naomi Hupert, Education Development Center, Inc.

• Claire Christensen, SRI International