10 Years of Null Results in a Museum

• In Event: When Nothing Is Something: Null Results in Informal Learning Research

Abstract

Objectives: Null results can sometimes be disheartening. In informal learning environments, a consistent stream of null results can impact the morale of both researchers and practitioners. The goal of this poster is to show researchers how null results can be embraced and used to create meaning. We will describe a wide variety of projects that resulted in null results over a decade of research at [REDACTED] and what we learned from each one.

Perspectives: There are many quantitative techniques for finding meaning in clusters of null results (Harms & Lakens, 2018). But Jacob, et al. (2019) propose a framework for making meaning of null results in education research by looking at potential reasons behind the results. The framework is focused on expected impact size, cost, prior research, methodological factors, program implementation and theory of change.

Data Sources: The Research and Evaluation department in the Education division at [REDACTED] has conducted over 100 studies since its founding in 2012, most of which involve some type of null result. About 5-10 of the projects will be chosen as case studies. They will involve a heterogeneous mix of education research studies about museum guests, after school youth programs, exhibition studies, teacher professional development programs and more.

Methods: A table describing all 10 years of studies, and which ones have null results, will be presented. We then will describe 5-10 studies in more detail using the framework from Jacob, et al. (2019). Then we will compare/contrast to look for trends across the results. Finally, we will describe how our research moving forward will be different due to those results.

Results: One example revolves around our attempts to measure the impact of the Museum on science attitudes. For many years we used a variety of science attitude measures taken from literature. All of them returned null results. Some of it is due to ceiling effects and pre-selection bias. Also, the general nature of the instruments was misaligned with the more focused goals of exhibits and programming. After many years of “failure”, we decided to use an open-ended instrument designed to be culturally relevant to take into consideration cultural differences in what the word “science” means to our guests (Zacharia & Barton, 2004). Another example involves the use of experience sampling to increase temporal resolution in our guest surveys. This largely failed with our longer-term surveys but we found some success when we used them for shorter-term projects (days vs. months). Rather than considering it a methodological failure, it caused us to rethink the type of impact out longer-term programming may have.

Significance: Of the factors from Jacob, et al. (2019), methodological and theory of change were the most influential reasons for finding null results in our work. We took this as meaning more effort needs to be put into those areas in the planning stages of research. We hope by sharing our experiences, other informal learning researchers will find meaning in their own null results, and also not be afraid to take risks since they will see ways to find that meaning.

Author

• C. Aaron Price, Museum of Science and Industry - Chicago