It's Complicated: Measuring Integrated Understanding in Multiple Document Contexts

• In Event: 45.024 - Integration: A Critical Competency for the Digital Age

Abstract

Objectives and Theoretical Framework
Our purpose is to discuss challenges of measuring integrated understanding in research on multiple document literacy and review efforts to cope with those challenges. Building integrated understanding from the content of multiple documents is the essence of multiple document literacy (Bråten & Strømsø, 2010; Britt et al., 2013). In general, integrated understanding in multiple document contexts can be defined as linking information across different documents in the service of meaning-making (Barzilai et al., in press). More specifically, when multiple documents present conflicting views on the same topic, issue, or phenomenon, building integrated understanding involves drawing bridging inferences across conflicting accounts to achieve conflict resolution or reconciliation (Braasch & Bråten, 2017), and when they present complementary information, it involves combining supplemental information across documents to achieve a more complete understanding (List & Alexander, in press). In any case, measuring integrated understanding as a result of reading multiple documents in reliable and valid ways has proven to be a great challenge for scholars in this area of research.

Mode of Inquiry
We review pertinent efforts to measure integrated understanding in multiple document contexts during the last decade, focusing on integrated understanding as a product resulting from multiple document reading.

Results
Bråten et al. (2009) pioneered the use of an intertextual inference verification task to measure integrated understanding as a product of reading multiple conflicting documents. On this task, each item combines information from at least two documents to form either a valid or an invalid inference, with correct verification depending on readers’ ability to draw bridging inferences across information presented in different documents. While this approach has been used effectively in a number of studies (e.g., Braasch et al., 2014), a more common approach has been to analyze students’ post-reading essays. For example, Anmarkrud et al. (2013) analyzed students’ essays after reading multiple conflicting documents in terms of integrated argumentative reasoning, and Salmerón et al. (2018) analyzed students’ essays in terms of intertextual inferences that reflect coherent mental representations at the level of multiple documents. Following Britt and Sommer (2004), integration across multiple conflicting documents has also been measured in terms of how frequently students merge information from different documents in their post-reading essays (Gil et al., 2010). Finally, several authors (e.g., Barzilai & Ka’adan, 2017; Ferguson & Bråten, 2013) have used open-ended, short-essay questions to measure integrated understanding of multiple conflicting documents. For example, Ferguson and Bråten (2013) asked “indirect integrative questions” that required readers to integrate perspectives across documents and “direct integrative questions” that required readers to pit different perspectives against each other and reason about the issue by considering each perspective’s claims and reasons.

Scholarly Significance
Thus far, less research within multiple document literacy has focused on measuring integrated understanding when students read multiple complementary, rather that conflicting, documents. However, reading multiple documents containing supplemental information to gain a more complete understanding is essential in academic contexts, and developing viable approaches to measuring this competency is therefore an important research agenda for multiple document literacy.

Authors

• Ivar Braten, University of Oslo

• Natalia Latini, University of Oslo