Assessing Survey Measures of News Exposure through Digital Footprints

• In Event: New Methods and Frameworks to Study Digital Politics

Abstract

Several features of the online news environment complicate measurement of news and political information exposure in this environment (e.g., Niederdeppe 2016, de Vreese & Neijens, 2016). Although a lot of attention has been placed on the challenges of correctly measuring media exposure on survey research since Althaus & Tewksbury’s report in 2007, fewer efforts have been made on specifically testing survey measures of news exposure online, where the complexity of news consumption is at the highest.

Following previous research, we combine survey and web-tracking data to examine the performance of three types of survey questions: open-ended, check-all, and forced-choice. We ask individuals to report the news outlets they remember having visited in the last 15 days of online navigation by randomly assigning them to one of these three question formats. We rely on passively collected digital trace data to construct direct, unobtrusive measures of exposure, which we then use as a benchmark to test the validity of these measures.

Our study stands on the shoulders of previous research (see especially Guess 2015), but it innovates on several fronts. First, in contrast to previous research, it tests the validity of these measures in two non-US contexts –Spain and the UK—, expanding existing research on the topic by bringing in evidence from two European countries that represent very different media systems.

Secondly, we focus on a period of abnormally high news consumption –the weeks following the outbreak of the COVID-19 pandemic in March 2020. While studying a period of unusually high news consumption can be problematic, it can also be taken as a hard test for the performance of survey-based measures of online news consumption.

Third, validity is assessed not only as a function of accuracy but also of bias. Using two salient characteristics of sources (reach and slant), we examine whether “brand awareness” as measured by the reach and the ideological slant of news outlets (Kalogeropoulos et al., 2019) increases overreporting across different survey measures.

Our analysis suggests that in the complex and fragmented online media environment the check-all question tends to perform slightly better than the open ended and the forced-choice formats, which produce more underreporting and overreporting, respectively. However, performance across questions varies depending on the context. Where the online media environment is more fragmented (Spain), all questions tend to perform poorly in terms of accuracy, and the relative benefits of the check-all question are higher. In contrast, where fragmentation is lower (UK), all questions tend to perform well, and the relative benefits of the check-all format are lower. Our findings also suggest that the check-all and the forced choice questions are less vulnerable to cognitive biases (e.g., availability bias) than the open-ended question. The implications of these findings for improving survey measures to understand news consumption patterns across countries and media systems are discussed in the paper.

Authors

• Ana Sofia Cardenal, Universitat Oberta de Catalunya

• Maria Victoria-Mas

• Sílvia Majó-Vázquez, University of Oxford

• Ivan Lacasa, Universitat Internacional de Catalunya (UIC Barcelona)