Search Logs, Argumentative Essays, and Performance Profiles: Positioning Multiple Source Use in a Classroom Context

• In Event: Synergy of Reading and Writing Within Multiple Source Tasks: Novel Approaches and Innovative Techniques

Abstract

Objectives
This study investigated the relation between undergraduates’ online search behaviors in a multiple source use (MSU) task and their argumentative essay writing. An innovative search log was employed to record students’ decision-making during source selection, which was examined in relation to essay quality and self-identified performance profiles.

Theoretical Framework
The Integrated Framework for Multiple Texts (List & Alexander, 2019) describes three phases for tasks involving multiple sources: preparation, execution, and production. In the IF-MT, strategies implemented in the execution stage are presumed to influence outcomes in the production stage. The behavioral and (meta)cognitive strategies enacted for searching, selecting, and processing information sources in the execution stage influence the cognitive and affective outcomes externalized in the written products in the production stage. The execution of information search strategies can differ depending on the profile students adopt in the preparation stage (List & Alexander, 2017). However, little research has sought to link students’ decision-making in source selection to their written products, nor have such search processes been examined in relation to students’ adopted profiles. To address this gap, we developed a method to capture students’ processing in both the execution and production phases to investigate interrelations between these phases.

Method
Undergraduates (n = 135) completed an online MSU task as an assignment in an educational psychology course. Participants performed open searches and selected at least four sources to summarize for the purpose of writing argumentative essays on a controversial topic, students’ overreliance on technology. During the search, they logged the search terms they used; numbers of sources scanned, opened, and read; and reasons for including or excluding each read source. Unlike automatically generated log files, these search logs were students’ records of their decision making throughout the search process. Students were also required to write summaries of each source they read in their class journals—a routine activity. They used these summaries to compose their argumentative essays.
Essays were scored based on components of argumentation quality (claim, counterargument, sources, justification, and writing quality) and integration (critical analysis, synthesis, and overall cohesion), each on a 0-2 scale (interrater reliability [ICC = .96], maximum score = 16). In a post-task questionnaire, students’ selected one of four profile descriptions to represent their performance stance (see Table 3).

Results
Multiple regression analyses revealed that search behaviors influenced writing quality. Specifically, number of sources read and summarized significantly predicted total essay scores. Credibility judgments during source selection also significantly predicted essay performance. Interestingly, students profile selections were not related to the number of search terms, sources read, and sources summarized. On average, students used 3.9 search terms, read 7.2 sources, and summarized 4.3 sources.

Significance
Writing was integral to this MSU task in terms of the search log, written summaries, and argumentative essay. This study provides insight into multiple document use within an ecologically valid context, which rarely occurs. Moreover, the links between students’ source selection criteria and essay quality highlight the importance of information search strategies in task execution and in producing quality outcomes.

Authors

• Eric Cornelis Schoute, University of Maryland - College Park

• Julianne Elizabeth van Meerten, University of Maryland - College Park

• Yuting Sun, University of Maryland - College Park

• Anisha Singh, University of Maryland - College Park