Effects of Social Context and Long-Term Scaffolding on Students' Strategies for Evaluating Source Credibility

• In Event: 29.035 - Structured Scenario-Based Assessments as Tools to Support Best Practices in Reading/Writing Instruction

Abstract

Objectives: Evaluating the credibility and accuracy of digital and print sources is a critical skill for college readiness (CCSS, 2010). However, evidence suggests that even adult students have difficulty with various cognitive and metacognitive processes required for reading, writing, and learning from multiple documents (Walraven, Brand-Gruwel, & Boshuizen, 2008), including source credibility evaluation (Graesser et al., 2007). Evidence of students’ source evaluation skills is often obtained through limited laboratory tasks or surveys. In contrast, this project examined students’ use of critical evaluation strategies “in the wild” across contexts, using an online tool designed to assess and support strategy use.
Theoretical Framework: Prior research on learning from multiple documents (Britt & Rouet, 2012), the social psychology of persuasion (Pornpitakpan, 2004), and credibility judgments during information-seeking tasks (Rieh & Hilligoss, 2008) informed the design of the tool, hypotheses about the types of strategies that students might apply, and the categories used to classify the strategies observed. In particular, we examined the context, content, document type, and other important features of students’ information evaluation experiences.
Methods: Participants included 22 undergraduates (13 females) from an instructional technology course at a university in the Midwestern United States. For 30 days, students completed daily diary entries describing their experiences with information they considered credible, not credible, or ambiguous. Students also completed pre- and post-test measures assessing their critical thinking habits and skills.
Data Sources include a corpus of 631 diary entries, coded with respect to the information content, medium, and evaluative strategies used by the students, and responses to pre- and post-test measures.
Instruments: The diary form was designed to scaffold students’ critical analysis and evaluation of the credibility of information obtained from a variety of sources. Students reported various features of the experience, including the context, the information learned, and relevant credibility aspects. The form requested numerical ratings and qualitative descriptions, yielding mixed quantitative and qualitative data.
Results: Students used various strategies to evaluate information credibility, but strategy use varied as a with social context; students were more likely to attend to source position, document type, information quantity, and other aspects of content when evaluating information obtained in academic or personal information seeking contexts, as compared to more informal social settings. Source expertise and trustworthiness were most likely to be used in information-seeking contexts, but use of citations was more likely to be mentioned in academic contexts, suggesting that students are more critical of some contexts than others. Findings suggest that repeated engagement with the diary tool over time influenced students’ use of certain strategies, such as validating information against ones’ prior knowledge.
Significance: Use of a diary activity to assess students’ source evaluation skills makes two important contributions. First, this approach yields insight into considerations of credibility across a variety of situational contexts and source types. Second, the use of such reflective diaries might be used as an instructional scaffold to support students in developing their critical evaluative skills. Examining students’ everyday strategies in this way may suggest instructional opportunities in formative assessment contexts.

Authors

• Jesse R. Sparks, Educational Testing Service

• David Rapp, Northwestern University