Allowing for Wiggle Room: Cross-Study Patterns of Engagement Emerge From Not Quite Significant Results

• In Event: When Nothing Is Something: Null Results in Informal Learning Research

Abstract

Objectives: Results can fail to reach statistical significance for a number of reasons, including too few subjects (i.e. not enough power) or a p-value that exceeds 0.05. In this poster, I will discuss how a single study addressing engagement with facilitators showed significant results suggesting gender preferences in engagement. This study might have appeared as an isolated instance; however, if we look to other studies this team did with non-significant or not-quite significant results, we see additional occurrences of similar engagement preferences. By acknowledging results outside of statistical significance, we find not just a single instance of gender-related engagement preferences, but a pattern of engagement.

Perspectives: The use of p-values as the sole means of understanding the results of a study “can lead to erroneous beliefs and poor decision making” (Wasserstein, 2016). One issue is the use of binary significant-not significant thinking and reporting. Relationships between two factors do not appear and disappear along the .05 line and by ignoring traditionally non-significant results, we risk ignoring valuable information (Amrhein, V., Greenland, S., & McShane, B., 2019).

Data sources: I discuss three studies on guest engagement under different conditions. The first explores the relationship between guest engagement and facilitation, the second looks at engagement at an interactive, game-based exhibit, and the third studies small group and large group engagement strategies in a collaborative, highly interactive exhibit called Future Energy Chicago (FEC).

Methods: I will present the results of each of the three studies and highlight the overlapping results.

Results: In a study on the effects of facilitation on guest engagement, we conducted analyses relating engagement measures (the number of staff interactions, the number of experiences a guest attended, and the number of facilitated experiences a guest participated in) with measures of guests’ perception of staff. We found that for male guests, participation in larger facilitated experiences significantly related to all three measures of perception of staff (p <.05). For female guests, smaller staff interactions related positively, and significantly to all three perceptions of staff measures.

If we look to non-significant or marginally significant results from other studies, we see similar trends elsewhere, suggesting a larger pattern of interaction. For example, in one study, we saw that male guests expressed higher levels of engagement if they participated in the experience than if they watched the experience. We had not used statistical methods for this result because of the small number of subjects. Less obvious is the possibility that female guests show a smaller interaction preference. Similar to the facilitation study above, we found that female guests performed marginally significantly better on a learning task after facilitators promoted a small-group interaction style at FEC.

Significance: Statistically significant results can be challenging to obtain, especially in informal learning where environments cannot be heavily controlled. For this reason, and many others, it is important that we move beyond the dichotomy of significant or not significant. By focusing only on statistically significant results, we might miss patterns that would illuminate valuable issues in informal learning.

Author

• Lauren R Applebaum, Museum of Science and Industry - Chicago