Source-linked AI summary
We Can Detect Your Bias: Predicting the Political Ideology of News Articles
Ramy Baly, Giovanni Da San Martino, James Glass, Preslav Nakov
TL;DR
The paper studies how to predict whether news articles express left, center, or right political ideology without letting models rely on media-source cues. It introduces a manually annotated, balanced dataset and source-debiasing methods, finding sizable improvements over state-of-the-art pre-trained Transformers while showing that media-level information can further help.
Problem
Article-level political ideology prediction is difficult because source-article correlations can cause models to learn media identity instead of article bias.
Method
The paper releases a balanced article-level dataset and combines adversarial media adaptation, triplet-loss pre-training, and media-level representations.
Results
14.12 points absolute in accuracy and 12.73 points in macro-F1: triplet-loss pre-training improves over the no-debiasing baseline under the media-based split.
Takeaways & Limitations
Article-level annotations, unseen-media evaluation, and source-debiasing methods provide a stronger basis for predicting political ideology than relying on source-correlated supervision.
Takeaways & Limitations
Future work is needed to extend prediction beyond left-center-right categories, identify biased fragments, and assess other languages because the division is not universal across cultures.
Abstract
from arXiv · showhide
We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology -left, center, or right-, which is well-balanced across both topics and media. We further use a challenging experimental setup where the test examples come from media that were not seen during training, which prevents the model from learning to detect the source of the target news article instead of predicting its political ideology. From a modeling perspective, we propose an adversarial media adaptation, as well as a specially adapted triplet loss. We further add background information about the source, and we show that it is quite helpful for improving article-level prediction. Our experimental results show very sizable improvements over using state-of-the-art pre-trained Transformers in this challenging setup.
1 Introduction
The paper addresses article-level prediction of left, center, or right political bias, where source-level ideological correlations can mislead models. It introduces a balanced, article-annotated dataset and methods designed to focus prediction on article bias rather than media source.
- Motivation: Media bias can reflect authors’ and organizations’ viewpoints and may contribute to intolerance, ideological segregation, and voting effects.The paper motivates tracking and exposing bias to support healthy public debate.
- Motivation: Political bias may involve omission, selective reporting, emotional appeals, or systematic left-center-right ideological leaning.
- Motivation: Article-level bias prediction supports media quality assurance, balanced news aggregation, viewpoint-oriented exploration, and media-level bias detection.
- Contributions: The paper creates a dataset with article-level annotations, broad topic coverage, and balanced left/center/right perspectives for each topic.
- Contributions: The proposed framework uses adversarial media adaptation and triplet loss to discourage source modeling, with sizable improvements over pre-trained Transformers.
- Contributions: Media-level representations from source information further improve article-level political ideology prediction.
2 Related Work
Prior article-level ideology datasets commonly assign website-level political labels to articles through distant supervision, creating risks of source memorization and limited evaluation validity. Existing alternatives face constraints involving annotation scalability, dataset size, availability, or topic and perspective coverage.
- Distant supervision: Most existing article-level ideology datasets project a known website’s political bias onto all crawled articles using distant supervision.
- Distant supervision: Manual article-level annotation is time-consuming, requires expertise, and can be subjective, motivating distant supervision for scalable datasets.
- Distant supervision: In hyper-partisan news detection, top systems performed best using manually annotated articles while ignoring distantly supervised data.This illustrates a documented risk of relying on distant supervision.
- Evaluation challenges: Source-learning can cause error rates to increase sharply when models are tested on articles from unseen sources.
- Existing datasets: Crowdsourced article-level bias measurement may be risky, does not scale, and may not ensure coverage of topics and events across political perspectives.
- Existing datasets: The 594-article Israeli-Palestinian conflict dataset reduces author-style modeling risk but is too small for modern deep learning approaches.
- Evaluation challenges: Using the same media in training and testing can let models predict an article’s source rather than its bias.
3 Dataset
The dataset uses manually annotated AllSides articles to represent left, center, and right perspectives across topics and media, with evaluation splits designed to test source-independent prediction.
- Dataset construction: AllSides annotations provide article-level political ideology labels and support high-quality training and testing compared with distant supervision.The annotation process is described as rigorous, though the supplied passage truncates before its details.
- Dataset construction: 34,737 articles from 73 news media cover 109 topics, and 1,080 articles (3.11%) receive labels differing from their source’s ideology.Articles include ideology labels, assigned topics, media, authors, and publication dates.
- Dataset construction: The dataset represents multiple political perspectives within frequent topics, requiring models to detect ideological linguistic and semantic nuances rather than topic-label associations.Figure 2 covers elections, immigration, coronavirus, and politics.
- Evaluation splits: The media-based split tests on 1,200 articles from 12 unseen media, while training and validation use the remaining 61 media.All articles from a given medium remain in one split, preventing source overlap across training, development, and testing.
- Evaluation splits: The random split evaluates articles whose sources were observed during training, using the same test set as the media-based split.The split uses stratified random sampling after adding excluded articles to the remaining media.
- Dataset release: The released dataset includes evaluation splits and code that can support future extensions as additional AllSides articles become available.The supplied passage also provides a repository link.
4 Methodology
The methodology combines source-debiased article representations with media information to predict political ideology. Adversarial adaptation removes media-specific signals, while triplet pre-training brings same-ideology articles from different media closer together.
- Task formulation: The task encodes article text into vectors for three-way classification of left, center, and right ideology.The methodology uses deep learning architectures including LSTMs, as introduced in the supplied passage.
- Adversarial adaptation: Media-source correlations can cause models to associate outlet writing patterns or stylistic markers with ideology instead of learning article-level bias.This motivates source-debiased modeling and unseen-media evaluation.
- Adversarial adaptation: Adversarial adaptation replaces domain classification with media classification so the encoder remains ideology-discriminative while becoming invariant across media.The approach uses a media classifier and a trade-off between label-prediction and media-classification losses.
- Triplet loss pre-training: Triplet-loss pre-training uses an anchor, a same-ideology positive from another medium, and a different-ideology negative from the anchor’s medium.This pairing encourages ideological clustering independently of source before fine-tuning with cross-entropy.
- Triplet loss pre-training: The model is fine-tuned on political-ideology prediction after triplet pre-training, jointly updating the encoder and softmax classifier.Fine-tuning minimizes cross-entropy loss.
- Media information: Source information is added as a complementary representation alongside article encoding, using representations derived from follower bios and Wikipedia descriptions.These representations are concatenated before the softmax layer.
5 Experiments and Results
Experiments show that article-bias models struggle when tested on unseen media because they learn source characteristics. Adversarial and triplet-based de-biasing improve performance, while media-level representations provide further gains when combined with better article representations.
- Baseline Results: Both LSTM and BERT perform much better on random splits than on media-based splits with unseen test media.The media-based setup exposes models’ difficulty generalizing beyond sources observed during training.
- Removing the Source Bias: BERT can predict the publication medium in a 73-way classification task substantially better than the majority-class baseline.This supports the concern that article representations contain readily learnable source information.
- Removing the Source Bias: 14.12 points in accuracy and 12.73 points in macro-F1 are the absolute improvements obtained by Triplet Loss Pre-training over the no-debiasing baseline on the media-based split.Both Triplet Loss Pre-training and adversarial adaptation produce sizeable improvements in this challenging setting.
- Impact of Media-Level Representation: Adding article representations without de-biasing produces no or little improvement beyond media-level representations.The authors attribute this to article encoders learning source information rather than bias-related language.
- Impact of Media-Level Representation: Triplet-loss de-biasing makes article representations more accurate and meaningful, allowing media representations to provide complementary information and sizeable gains.Twitter follower bios appear more useful than Wikipedia content for representing media-level political leaning.
- Impact of Media-Level Representation: 30.51 and 28.76 absolute macro-F1 points are the improvements from the best system over the baseline on the challenging media-based split.The best configuration combines triplet-loss source-bias removal with media-level representation from Twitter followers.
6 Conclusion and Future Work
The paper presents article-level political-ideology prediction using a balanced dataset and methods designed to avoid modeling news sources instead of article bias. It reports sizable improvements over state-of-the-art pre-trained Transformers and identifies broader bias categories, topic-level prediction, explainability, and multilingual transfer as future directions.
- Conclusion: The dataset provides large-scale article-level annotations that are well-balanced across topics and media.
- Conclusion: Adversarial media adaptation and a special triplet loss target the pitfall of modeling source-correlated classes instead of political bias.
- Conclusion: Experimental results show very sizable improvements over state-of-the-art pre-trained Transformers.
- Future Work: Future work includes topic-level prediction, bias categories beyond left-center-right, fragment-level bias detection for explainability, and experiments across languages.The authors also plan to study cross-language transfer because left-center-right divisions are not universal across countries and cultures.