IQ Journalism: Predicting the Quality of Politics News in the Post-truth Era

• In Event: Combating Misinformation and Fake News

Abstract

Fake and manipulated information is circulated in all forms and platforms, unverified videos are shared on Facebook, rumors are being forwarded via messaging apps, and conspiracy theories are being shared by Twitter influencers, and these are only a few of the distribution patterns of disinformation. The role of social media platforms is crucial to understand the current state of disinformation globally since Facebook and Twitter changed both the news distribution and the trust of traditional media outlets. In the post-truth era when the role of the information gatekeepers has been transferred to the users as nodes in a wider network of reproduction, shares, and affiliations the reality of what the user encounters in their media flow is what defines the new truth: the user is transformed into the locus of truth construction thus partially undertaking the responsibility of identifying or assess the news item or narrative object.

This work examines the quality of online journalism based on a set of normative criteria by training machine learning algorithms on politics news of various media outlets and Facebook news-related posts. A question central to current examinations in journalism studies is how to measure the quality of digital articles and identify potentially misleading information. Given the dominant ideology of journalism (Deuze 2005), core journalistic principles (Kovach and Rosenstiel 2014), and best practices (Harcup 2015) we blend theoretical principles, results from previous empirical studies, and the output of machine learning algorithms to build a model that can effectively measure quality and identify online disinformation campaigns.

This study employs explainable machine learning models (Lundberg et al. 2020) to predict the quality of news stories based on a theoretical framework originated from previous empirical work on journalism studies. The hypothesis is that the quality question can be examined in great detail if it is treated as a classification problem by using automatic learning approaches to find hidden patterns that confirm or refute the dimensions of the framework, or even provide new insights on high-quality indicators. Sophisticated features like readability, subjectivity, and diversity of sources were coded and a corpus of news stories from legitimate and fake news websites was created. More specifically the corpus consists of 25K articles belonging to the politics category of the following news organizations: New York Times, Guardian, Washington Post, Politico, Business Insider, Buzzfeed, and New Yorker. Additionally, 3 widely acknowledged fake news websites, namely, dailysurge.com, dcgazette.com, and newspunch.com were used.


To test our general assumption that certain characteristics of a news story are associated with quality, we used the corpus as an input for the machine learning model that used content-based features related to language use, and emotions, along with engagement features from Facebook such as likes, shares, and comments for training, to predict the quality of an article and its probability of being fake. The dependent variable (categorical) quantified quality by scoring all the articles belonging to real news as high, and they were given the value of 1 and the fake articles were given the value of 0. To shed light on the hidden decision mechanisms, after the initial modeling follows a series of analyses to interpret the machine learning models in order to examine the importance of each feature in greater detail. For the evaluation of the importance of the features, a binary classification task was applied to build a model that could provide us with the importance of each feature along with an explanation of the system's predictions to improve our understanding of what comprises a high-quality news story. We decided to use tree-based models because according to previous work (Lundberg et al., 2020) these types of classifiers can be explained by providing in-depth interpretations of the model predictions. The findings of the study show that the proposed model is able to predict misleading news stories with a 98% accuracy based on features such as words in all capital letters in the main body, headline length, Facebook likes, the total amount of nouns, and numbers, lexical diversity, and arousal.

This work presents the preliminary results of the project IQ Journalism, an intelligent platform that automatically predicts the quality of news content. So far the results support a number of quality characteristics dictated by the theoretical foundations of the notion of journalistic quality. In the next stage of our research, a study that will involve human annotators will take place to validate the results of the AI model.

Acknowledgment
This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH – CREATE – INNOVATE (project code: T2EDK-04616).

Authors

• Catherine Sotirakou, National and Kapodistrian University of Athens

• Katerina Mandenaki, National and Kapodistrian University of Athens

• Anastasia Karampela, National and Kapodistrian University of Athens

• Costas I. Mourlas, National & Kapodistrian University of Athens