Source-linked AI summary
Predicting Factuality of Reporting and Bias of News Media Sources
Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, Preslav Nakov
TL;DR
The paper addresses the under-studied problem of characterizing entire news media by predicting reporting factuality and political bias, motivated partly by fact-checking needs. It evaluates features from articles, Wikipedia, Twitter, URLs, and web traffic, finding sizable baseline gains and evidence that article content is especially informative.
Problem
Predicting the factuality and bias of entire news media is under-studied, while manual and automatic fact-checking have important timeliness, accuracy, and trust limitations.
Method
The study combines article, Wikipedia, Twitter, URL, and web-traffic features to predict media factuality and bias, evaluating accuracy, macro-averaged F1, MAE, and MAEM.
Results
Article textual features yield the best factuality performance and beat the bias baseline on MAE, while Wikipedia and URL features offer task-specific benefits and Alexa rank does not improve over baseline.
Takeaways & Limitations
The results support analyzing article bodies and multiple media-level feature types, while the released dataset and code are intended to facilitate future research.
Takeaways & Limitations
Alexa rank does not improve over baseline, suggesting that more sophisticated traffic-related features may be needed.
Abstract
from arXiv · showhide
We present a study on predicting the factuality of reporting and bias of news media. While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media. These are under-studied but arguably important research problems, both in their own right and as a prior for fact-checking systems. We experiment with a large list of news websites and with a rich set of features derived from (i) a sample of articles from the target news medium, (ii) its Wikipedia page, (iii) its Twitter account, (iv) the structure of its URL, and (v) information about the Web traffic it attracts. The experimental results show sizable performance gains over the baselines, and confirm the importance of each feature type.
1 Introduction
The paper addresses automatic prediction of news-media factuality and bias, motivated by incomplete source lists and the limits of manual and automatic fact-checking. It introduces a larger annotated dataset, diverse feature sources, and feature ablations.
- Manual fact-checking is slow, while automatic fact-checking remains less accurate and generally untrusted by users.
- Source reliability offers a complementary route because websites that previously published non-factual information are likely to do so again.
- Incomplete and rapidly outdated lists of unreliable sources motivate automatically predicting factuality for each online news medium.
- The study treats medium-level factuality and bias as under-explored research problems relevant to fact-checking systems.
- The dataset contains annotations for both tasks and is 1-2 orders of magnitude larger than datasets used in previous work.
- Features combine articles, Wikipedia, Twitter, URL structure, and Web traffic, with additional ablation experiments for individual feature groups.
2 Related Work
Prior work largely studies veracity at the claim or article level and often projects source labels onto articles. This paper instead uses manually annotated medium-level labels, fine-grained bias classes, more sources, and a substantially larger set of media.
- Veracity research spans claim-, article-, user-, and medium-level analysis, with this paper focused primarily on the medium level.
- Source reliability remains largely under-explored compared with stance detection, especially for Web-source trustworthiness.
- Earlier news-media reliability methods often infer source reliability from stance toward fact-checked claims without gold labels for overall medium factuality.
- Much prior fake-news detection assumes all articles from a website share one reliability class, an assumption the paper finds problematic for testing.
- Unlike article-level studies covering eight sources, this work classifies 1,066 media using manually annotated labels and broader feature types.
- The paper predicts medium-level bias with seven ordinal degrees rather than the binary setups used in some prior work.
3 Method
The method predicts factuality and bias for each news medium from aggregated article features plus Wikipedia, Twitter, URL, and traffic information. These feature families are evaluated individually and jointly.
- A classifier combines information from sampled articles, Wikipedia, Twitter, URL structure, and Web traffic for each target medium.
- Articles: Article analysis uses 141 features covering structure, sentiment, engagement, topic, complexity, bias, and morality.
- Articles: For each medium, article features are computed separately for titles and bodies, then averaged across retrieved articles.
- Wikipedia: Wikipedia features encode page availability and vector representations of page content, infobox, summary, categories, and table of contents.
- Twitter: Twitter features include account presence, verification, creation year, location, URL matching, activity counts, and profile-description representations.
- URL: URL features model character patterns, orthography, and credibility indicators such as HTTPS, hosting platform, and top-level domain.
- Web Traffic: Traffic information is represented only by the reciprocal of Alexa Rank, a global ranking based on website traffic.
- Evaluation: The experiments evaluate feature families individually and in groups to identify useful features for predicting factuality and bias.
4 Experiments and Evaluation
The experiments evaluate factuality and bias prediction using news-source data, article-derived features, and multiple external feature families. Results show that article content performs strongly, feature families are complementary, and traffic information contributes little.
- 4.1 Data: The dataset contains 1,066 websites with factuality labels on a three-point scale and bias labels on a seven-point scale.The labels come from MBFC annotations or straightforward inferences.
- 4.1 Data: The study crawled 10–100 articles per website, totaling 94,814 articles, and averaged article-level feature vectors by website.
- 4.2 Experimental Setup: The models use SVM classifiers trained separately for factuality and bias, with five-fold cross-validation and macro-averaged F1 for hyperparameter tuning.The reported metrics include accuracy, macro-averaged F1, MAE, and MAEM.
- 4.3 Results and Discussion: Article textual features achieve the best factuality performance and are the only feature type that beats the bias baseline on MAE.The results also indicate that article bodies contain information not captured by titles alone.
- 4.3 Results and Discussion: Wikipedia features are less useful for factuality but reasonably effective for bias, while Twitter features perform moderately for factuality and poorly for bias.The combined Twitter features yield the best performance within that family for factuality.
- 4.3 Results and Discussion: URL features help factuality more than bias and provide slight improvements when combined with other features, whereas Alexa rank does not improve over the baseline.The authors suggest that more sophisticated traffic features may be needed.
- 4.4 Ablation Study: The FULL system achieves the best factuality results and best macro-F1 for bias, while removing Wikipedia causes the largest factuality drop and removing traffic has no effect.These ablations suggest that the feature types are largely complementary, with Wikipedia supplying information unavailable from other sources.
5 Conclusion and Future Work
The study predicts news-media factuality and bias using novel combinations of source-derived features, and releases a substantially larger annotated dataset and code. Results show that most feature types affect performance, while future work targets richer task and bias modeling.
- The study addresses the under-explored problems of predicting reporting factuality and media bias at the whole-medium level.
- The released news-media dataset contains annotations for both tasks and is 1-2 orders of magnitude larger than datasets used previously.
- The experiments use novel feature combinations derived from articles, Wikipedia, Twitter, URL structure, and Web traffic.
- Most feature types have a notable performance impact, with target-website articles, Wikipedia, and Twitter features ranked most important in that order.
- Future work will model factuality and bias jointly, treat the task as ordinal regression, and extend factuality analysis to other languages and bias types.