The Refutation Text Effect and Science Learning: The Moderating Role of Text Characteristics

• In Event: Text Comprehension: What We Know About the Dance Between Reader, Text, and Task in Reading Comprehension

Abstract

Purpose
The purpose of this investigation was to comprehensively synthesize existing research on the role of refutation texts and associated text characteristics in promoting conceptual change (i.e., changing from inaccurate to accurate scientific beliefs) for STEM.

Theoretical Framework
Science is difficult, and learners must overcome inaccurate beliefs (i.e., misconceptions) to align with the current scientific perspective. Refutation texts are a common technique to facilitate conceptual change, have been extensively researched (see Tippett, 2010), and are thought to function in part via increased coherence (Sinatra & Broughton, 2011). However, no systematic review and meta-analysis has been conducted since Guzzetti et al., (1993). Therefore, we synthesized the refutation text literature to understand their average effect compared to expository texts on conceptual change for STEM topics.

Methods
A comprehensive search for research addressing the differential effects of refutational and expository texts on cognitive outcomes was conducted using electronic database searches, citation searches, reference section review, requests to relevant professional organizations, and direct emails to productive scholars. Over 7,500 reports were examined for inclusion. Four trained coders retrieved information from relevant reports with all disagreements resolved resulting in 100% agreement. Standardized mean differences corrected for small-sample bias (Hedges’ g) were retrieved from intervention studies. Inverse variance weighting procedures and random effects meta-regression using robust variance estimation were used to pool effects across studies and examine moderators.

Results
Currently 15 intervention studies have been coded (34 currently being coded) yielding 23 effect sizes for refutation text effects on science knowledge measures and revealed the following key findings.

First, regarding the pooled effects across studies, we documented an average effect of d = .48 indicating a small to medium effect for refutation compared to expository texts. This effect size is larger than that documented by Guzzetti and colleagues (Glass’ delta = .22). Again, while a preliminary finding based on 15 studies, Guzzetti and colleagues analyzed 14 to conclude their effect.
Second, in calculating the pooled refutation text effect size, we documented an I^2 of 76.54% indicating considerable heterogeneity between study effect sizes and provides a rationale for moderator analyses. In addition to participant moderators, we calculated text characteristics using Coh-Metrix (i.e., narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion, and formality) (Graesser et al., 2014; McNamara et al., 2010). Preliminary analyses indicated no significant differences (low power) however the pattern of results indicated that narrativity, concreteness, and referential cohesion were weakly and negatively related to conceptual change (ß’s < -.019) while simplicity, deep cohesion, and formality were weakly and positively related (ß’s < .01).

Discussion and Scholarly Significance
This meta-analysis contributes to the literature on refutation texts and meta-analytic techniques. We conduct the first refutation text meta-analysis since 1993 and provide preliminary evidence for the efficacy of refutation compared to expository texts (d = .48) and as evidence against refutation texts inducing backfire effects. Additionally, we sought to integrate discourse analyses using Coh-Metrix with meta-analytic techniques to provide another technique to contextualize science learning materials.

Authors

• Neil Garrett Jacobson, University of Southern California

• Olusola Olalekan Adesope, Washington State University

• Erika Alisha Patall, University of Southern California

• Gale M. Sinatra, University of Southern California

• Robert William Danielson, Washington State University - Spokane