Assessing Students' Ability to Engage With Online Information: A Validity Study

• In Event: Telling Fact From Fiction and Other Challenges in Enacting High-Quality Civic Education

Abstract

Objectives and Perspectives
Concerns about students’ abilities to distinguish reliable information from bogus claims have grown in the wake of the 2016 election, and calls for schools to teach students to be savvy consumers of online information have grown louder. If schools are to prepare students to be more discerning consumers of online information, teachers need assessments to determine whether students are learning these skills. Unfortunately, such tools are currently in short supply. Extant assessments tend to rely on self-reports in the form of questionnaires or surveys, but these tend to overestimate what people actually do in practice (Flanagin & Metzger, 2007; Hargittai et al., 2010). Assessments of how young people evaluate information online are needed if we hope to prepare students for citizenship in a world awash in dubious digital content.
We have developed a range of assessments that ask students to evaluate various types of online sources. These tasks are designed to help educators gauge students’ mastery of civic online reasoning, which includes the ability to search for, evaluate, and verify information online. These competencies of civic online reasoning were developed based on research about how professional fact checkers evaluate online information (Authors, 2017). This paper reports a validity study that used think-aloud protocols to investigate whether our assessments tap the online reasoning processes they were designed to measure.

Materials, Methods, and Data
We conducted concurrent and retrospective think-aloud interviews (Ericsson & Simon, 1993; Taylor & Dionne, 2000) with 20 students as they completed three assessments of civic online reasoning. Participants were 11th and 12th grade students at an urban public high school on the West Coast.
Student verbal reports were audio-recorded. To determine whether the tasks tapped the intended constructs, we coded the transcripts based on the cognitive processes students used to complete each task and compared them to the processes the task was designed to measure. To determine whether students’ written responses reflected the processes they engaged in as they completed each task, we coded their written responses based on the cognitive processes used and compared those to the processes coded in the verbal responses.

Results and Significance
This study examined the alignment between the cognitive processes our assessments were designed to measure and the processes students used to answer them, which is a central consideration in a validity argument. Protocol analyses suggested that the assessments did indeed tap the intended cognitive processes. The analyses also revealed alignment between student written responses and the processes students used when evaluating online information. Both of these findings support a validity argument for the use of these assessments to measure student’s grasp of civic online reasoning competencies.
As students increasingly turn to the Internet as their primary source of information, they must learn to separate truth from fiction. Teachers play an important role in helping them do this, and this study suggests that these assessments are a promising resource to support teachers in this work.

Authors

• Sam Wineburg, Stanford University

• Joel Breakstone, Stanford University

• Sarah McGrew

• Mark D. Smith, Stanford University

• Teresa Elena Ortega, Stanford University