An Integrated Approach to Measuring Multiple Source Use

• In Event: 34.029 - Multiple Text Engagement: Addressing Questions About Methods and Measurement

Abstract

Objectives
To investigate multiple source use through the integration of process, offline, and interview data.
Theoretical Frame
The Multiple-Document Task-based Relevance Assessment and Content Extraction (MD-TRACE) model of Rouet and Britt (2011) describes processes involved in selecting, processing, and using multiple documents to complete a particular task. While most research framed by this model has used only one type of measure to assess these processes, we combined process, offline, and interview data in the current research.
Method
Participants were 101 Spanish undergraduates (M age = 19.8, 81.6% female) majoring in education, psychology, or philology who were tested individually in 90 minute sessions. First, topic knowledge and topic interest were assessed. Then, participants read four documents discussing different perspectives on climate change in order to write an argumentative essay. The documents included (1) a textbook excerpt introducing the greenhouse effect and climate change, (2) a newspaper editorial criticizing the view of ‘climate-skeptics’, (3) a blog entry discussing potential benefits of CO2, and (4) a popular science magazine article arguing that efforts to prevent climate change will have great economic costs. We recorded the eye-movements of half of the participants with an eye-tracking system. After reading, all participants wrote argumentative essays, completed a source memory task, and responded to a structured interview about how they had read the documents.
Results
Interview data indicated that participants had approached the reading task in different ways: 10.3% said they had planned reading the documents in an order that differed from the order in which they were presented; 20.6% said they had stopped reading in the middle of a text to check something previously read in another text; 55.7% said they had looked at source information, and 54.6% said they had considered some documents more important than others. Of these reported strategies, attention to source information predicted memory for sources and intertextual inferences in essays.
However, there were inconsistent findings across the methodologies used to assess multiple source use. Specifically, participants’ reported use of sourcing in the interviews was not consistent with their fixations on source information during reading or their actual inclusion of source information in their essays. For example, a majority of students fixated source information during reading, and some reported in the interviews they had looked at it for various purposes, such as to judge the trustworthiness of the sources or to facilitate comprehension. However, other students reported in the interviews that they had not looked at sources, although their eye-movements revealed that they had actually done so.
Significance
The use of interview methods indicated that objective measures, such as eye-movements, can be difficult to interpret without considering participants’ subjective interpretations, and that interpreting eye movements in isolation actually can be misleading. That participants looked at source information, yet reported that they had not done so, indicates that source information may be processed as peripheral information, that is, without explicit attention. This paradoxical effect resembles a well-established effect in traffic psychology, the ‘look but fail to see’ error (Crundall et al., 2013).

Authors

• Ladislao Salmerón, University of Valencia

• Laura Gil, University of Valencia

• Ivar Braten, University of Oslo