Source-linked AI summary
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, Gerhard Weikum
TL;DR
Misinformation and the time required for manual fact-checking motivate automated credibility assessment that uses more than labeled claim text. DeClarE is an end-to-end neural model that combines external evidence, article language, source trustworthiness, and interpretable attention-based explanations. Experiments on four datasets and ablation studies demonstrate effectiveness over state-of-the-art baselines, including a 17% MSE decrease on NewsTrust and a 7-9% AUC margin on PolitiFact.
Problem
Manual fact-checking is time-consuming, while prior automated methods either omit external evidence or require feature engineering and lexicons.
Method
DeClarE automatically assesses natural-language claim credibility by aggregating claim, external-article, attention, and source-trustworthiness signals without hand-crafted features or lexicons.
Results
DeClarE outperforms baselines, with a 17% decrease in MSE on NewsTrust and a 7-9% AUC margin over baselines on PolitiFact.
Takeaways & Limitations
Attention-based evidence and source representations support transparent credibility verdicts while contributing to performance across the evaluated datasets.
Takeaways & Limitations
The paper notes that fully objective and unarguable truth can be elusive or ill-defined, so it assesses credibility rather than truth.
Abstract
from arXiv · showhide
Misinformation such as fake news is one of the big challenges of our society. Research on automated fact-checking has proposed methods based on supervised learning, but these approaches do not consider external evidence apart from labeled training instances. Recent approaches counter this deficit by considering external sources related to a claim. However, these methods require substantial feature modeling and rich lexicons. This paper overcomes these limitations of prior work with an end-to-end model for evidence-aware credibility assessment of arbitrary textual claims, without any human intervention. It presents a neural network model that judiciously aggregates signals from external evidence articles, the language of these articles and the trustworthiness of their sources. It also derives informative features for generating user-comprehensible explanations that makes the neural network predictions transparent to the end-user. Experiments with four datasets and ablation studies show the strength of our method.
1 Introduction
DeClarE addresses the need to automate time-consuming credibility assessment for natural-language claims by combining external evidence, source trustworthiness, and claim-specific article signals. Its end-to-end model also produces interpretable explanations and is evaluated across four datasets with ablation studies.
- Motivation: Manual fact-checking cannot keep pace with the scale and speed of misinformation, motivating automated debunking tools.Fact-checking websites manually research claims, assess credibility, and present verdicts with evidence.
- Prior work: Prior neural approaches assess claims from their text alone, omitting external evidence and explanations for their verdicts.These methods provide limited context for credibility analysis because they use only questionable-claim text.
- DeClarE: DeClarE combines web evidence, article language, source trustworthiness, and joint interactions in an end-to-end model without hand-crafted features or lexicons.The model automatically assesses natural-language claim credibility without human intervention or manual feature engineering.
- Interpretability: Claim-specific attention focuses on salient article words and generates user-comprehensible evidence for transparent credibility verdicts.Attention weights and salient words are used as interpretable features alongside source embeddings that capture trustworthiness.
- Experiments: Experiments on four datasets and ablation studies demonstrate DeClarE's effectiveness over state-of-the-art baselines.The evaluation examines the model's overall effectiveness and the contribution of its components.
2 End-to-end Framework for Credibility Analysis
DeClarE represents claims, reporting articles, and their sources, then uses claim-specific attention and biLSTM article representations to assess credibility. It combines attention-focused article information with source embeddings and aggregates per-article scores into a claim-level score.
- Input representations: Each training instance pairs a claim and its origin with reporting articles, their sources, and the claim’s credibility label.
- Input representations: Claims and reporting articles are represented with word embeddings, while claim and article origins use source embedding vectors.
- Article representation: A bidirectional LSTM captures article representations, including information from both past and future token contexts.
- Claim-specific attention: Claim-specific attention combines the overall claim representation with each article term to assign relevance scores and focus on salient evidence words.
- Credibility prediction: The model combines claim-source, attention-focused article, and article-source representations through nonlinear dense layers to predict credibility.
- Credibility prediction: Per-article credibility scores are averaged after training to produce the overall credibility score for each claim.
3 Datasets
The paper evaluates DeClarE across four datasets covering fact-checking, political claims, news reviews, and Twitter rumors, using distinct credibility objectives and preprocessing choices.
- Experiments span Snopes, PolitiFact, NewsTrust, and SemEval, covering general rumors, political claims, reviewed news articles, and Twitter rumors.
- 3.1 Snopes: Snopes provides rumors with binary credibility labels, reporting articles, and their web sources.
- 3.2 PolitiFact: PolitiFact provides claims, political speakers, and editor-assigned credibility ratings, which are converted into binary true or false labels.
- 3.3 NewsTrust: NewsTrust supports regression by mapping article titles and sources to claims and claim sources, and reviews and reviewers to reporting articles and article sources.Its overall credibility score ranges from 1 to 5.
- 3.4 SemEval: SemEval evaluates credibility prediction for questionable tweets in closed and open variants, with labels true, false, or unverified and a confidence score.
- 3.5 Data Processing: For Snopes and PolitiFact, 100-word reporting-article snippets are selected using lexical and semantic relevance, with δ=0.5 chosen on withheld data.Articles from the Snopes and PolitiFact websites are discarded to reduce dataset bias.
4 Experiments
Experiments evaluate DeClarE across four datasets using classification and regression metrics, comparing it with state-of-the-art baselines and model ablations.
- Experimental Setup: The evaluation covers four datasets, with binary credibility classification for Snopes, PolitiFact, and SemEval, and regression for NewsTrust.Validation and cross-validation procedures are used for three datasets, while SemEval uses organizer-provided splits.
- Experimental Setup: The comparisons include text-only neural models, distant supervision, CCRF+SVR, and DeClarE variants differing in attention and source embeddings.The ablations isolate biLSTM, attention, and source-embedding components.
- NewsTrust: 17% decrease in MSE: DeClarE (Full) outperforms four baselines on NewsTrust compared with the best-performing baselines.MSE is evaluated for credibility-rating prediction, where lower is better.
- SemEval: DeClarE (Full) outperforms the other approaches on both SemEval task variants using Macro F1 and RMSE over confidence scores.The evaluation follows the measures proposed by the task organizers.
5 Discussion
The discussion examines representations and attention weights learned by DeClarE, showing separable article and source embeddings and attention focused on claim-relevant words.
- Article Representations: PCA projections show clear separation between credible and non-credible article representations in the Snopes dataset.The projections reduce high-dimensional article representations to a two-dimensional plane.
- Source Embeddings: PCA projections separate fake-news sources from mainstream sources and place politicians with similar ideologies and opinions close together.The source analyses use sampled news sources from Snopes and claim sources from PolitiFact.
- Attention Weights: Attention weights identify salient words in reporting articles that are relevant to the input claim, making model decisions more transparent.Table 6 uses darker shades to indicate higher attention weights.
6 Related Work
Related work spans credibility analysis of natural-language Web claims, source reliability and claim correctness, truth discovery, and language-bias detection.
- Credibility Analysis of Web Claims: Prior credibility-analysis methods address open-domain Web claims, while some estimate source reliability and claim correctness jointly.The cited approaches include stylistic, probabilistic, and iterative methods.
- Truth Discovery: Truth-discovery approaches primarily focus on structured data rather than the deeper semantic aspects of natural-language credibility analysis.The discussion contrasts structured-data truth discovery with language-focused credibility work.
- Language Bias: Language-bias research studies biased language but does not address credibility, leaving a distinction between bias detection and claim assessment.The cited studies concern detecting bias in language.
7 Conclusion
DeClarE is a completely automated, end-to-end neural network for evidence-aware credibility assessment of natural-language claims without hand-crafted features or lexicons. Experiments on real-world datasets demonstrate effectiveness over state-of-the-art baselines.
- DeClarE assesses natural-language claim credibility using external evidence while modeling interactions among claim context, article language, and source trustworthiness.