Source-linked AI summary
That is a Known Lie: Detecting Previously Fact-Checked Claims
Shaden Shaar, Giovanni Da San Martino, Nikolay Babulkov, Preslav Nakov
TL;DR
The paper targets the underexplored problem of quickly identifying whether a new claim has already been fact-checked, which can save manual checking effort. It formulates the task, releases specialized datasets, and develops learning-to-rank and matching models. The reported experiments show sizable improvements over state-of-the-art retrieval and textual-similarity approaches, while the study remains limited to textual claims and its evaluated datasets.
Problem
The paper addresses the largely ignored task of detecting whether a new claim has already been fact-checked, motivated by repeated claims, rapid spread, and costly manual checking.
Method
The authors formulate the ranking task, create specialized datasets, and combine retrieval with BERT-based semantic matching and learning-to-rank models.
Results
The learning-to-rank experiments demonstrate sizable improvements over state-of-the-art retrieval and textual-similarity approaches.
Takeaways & Limitations
Previously fact-checked claim detection can serve as a distinct component of automated fact-checking and as a tool for human fact-checkers or journalists.
Takeaways & Limitations
The study plans extension to more datasets and languages and beyond textual claims to claim–image and claim–video pairs.
Abstract
from arXiv · showhide
The recent proliferation of "fake news" has triggered a number of responses, most notably the emergence of several manual fact-checking initiatives. As a result and over time, a large number of fact-checked claims have been accumulated, which increases the likelihood that a new claim in social media or a new statement by a politician might have already been fact-checked by some trusted fact-checking organization, as viral claims often come back after a while in social media, and politicians like to repeat their favorite statements, true or false, over and over again. As manual fact-checking is very time-consuming (and fully automatic fact-checking has credibility issues), it is important to try to save this effort and to avoid wasting time on claims that have already been fact-checked. Interestingly, despite the importance of the task, it has been largely ignored by the research community so far. Here, we aim to bridge this gap. In particular, we formulate the task and we discuss how it relates to, but also differs from, previous work. We further create a specialized dataset, which we release to the research community. Finally, we present learning-to-rank experiments that demonstrate sizable improvements over state-of-the-art retrieval and textual similarity approaches.
1 Introduction
The paper addresses the underexplored task of detecting whether a new claim has already been fact-checked, motivated by rapid misinformation spread and repeated claims. It formulates the task, releases a specialized dataset of naturally occurring claims, and evaluates learning-to-rank models that outperform retrieval and textual-similarity approaches.
- Motivation: Rapidly spreading misinformation and repeated claims make timely detection of previously fact-checked claims valuable.Some viral claims spread within minutes, while manual fact-checking is time-consuming.
- Research gap: The task had received little research attention despite the growth of manual fact-checking resources.The authors position their work as addressing this research gap.
- Contributions: The paper formulates previously fact-checked claim detection and relates it to, while distinguishing it from, prior work.The task is intended to support automated fact-checking pipelines and human fact-checkers.
- Contributions: The authors create and release a specialized dataset containing naturally occurring claims from debates and social media.This differs from prior fact-checking datasets based on normalized claims.
- Contributions: The proposed learning-to-rank model achieves sizable improvements over state-of-the-art retrieval and textual-similarity models.The experiments target matching new claims to claims that have already been fact-checked.
2 Related Work
Prior research largely focused on automatic fact-checking, claim exploration, or general document retrieval rather than detecting whether an input claim was previously fact-checked. The paper adapts retrieval and semantic-matching methods to this task while distinguishing it from related language-understanding problems.
- Existing fact-checking research: Previous work used fact-checked claim collections for automatic fact-checking and check-worthiness detection, not primarily for prior-check detection.The paper identifies prior-check detection as an insufficiently addressed task.
- Industrial systems: Fact Check Explorer searches fact-checking websites but was described as unable to handle complex long claims as semantic matches.The paper presents this as an industrially related but distinct tool.
- Related datasets and systems: ClaimsKG supports exploration of claims in a knowledge graph, whereas this paper detects whether an input claim was previously fact-checked.The distinction is between data exploration and matching an input claim to a prior fact-check.
- Retrieval and matching: The approach combines BM25 retrieval with BERT-based similarity matching against claims, titles, and full article text.The paper also combines scores and reciprocal ranks and uses sentence-BERT representations.
- Task distinction: The task differs from paraphrase detection, natural language inference, recognizing textual entailment, and semantic textual similarity.Input claims may contain sub-claims and may be useful to verify without being paraphrases or entailments.
3 Task Definition
The task ranks previously verified claims so that those useful for verifying a check-worthy input claim or one of its sub-claims appear first. Examples show that useful matches may differ lexically, involve multiple sub-claims, and require more than paraphrase or textual similarity.
- Task definition: Given a check-worthy input claim and verified claims, the task ranks claims that help verify the input above unhelpful claims.The ranking target includes claims that verify the whole input or a sub-claim.
- Examples: The dataset examples pair sentences from the 2016 US Presidential debates with corresponding fact-checked claims from PolitiFact.Table 1 illustrates the input–verified claim pairing used to define the task.
- Examples: Some useful verified claims are lexically different from the input claim, rather than identical matches.The paper describes these as harder matching cases.
- Examples: A complex input claim may contain two sub-claims, each verified by corresponding claims in PolitiFact.This structure complicates direct entailment reasoning.
- Annotation basis: The task relies on expert annotations rather than a formal definition of what makes a verified claim a good match.Fact-checking experts identify correspondences when commenting on political debates and speeches.
4 Datasets
The paper constructs PolitiFact and Snopes datasets for ranking previously fact-checked claims against naturally occurring inputs, and analyzes why matching can be difficult.
- Dataset construction: The datasets contain naturally occurring claims from political debates, speeches, and tweets paired with corresponding verified claims.PolitiFact covers debates and speeches, while Snopes focuses on tweets.
- PolitiFact dataset: PolitiFact provides 16,636 verified claims with normalized claim text, truth values, article titles, and article bodies.The truth-value labels were collected for possible future use but were not used in the experiments.
- PolitiFact dataset: The PolitiFact dataset contains 768 Input–VerClaim pairs collected from 78 public events between 2012 and 2019.An input may map to multiple verified claims, and multiple inputs may map to the same verified claim.
- Snopes dataset: The Snopes dataset contains 1,000 tweet–claim pairs and 10,396 verified claims, with tweets manually filtered for claim relevance.The tweets are generally more self-contained and context-independent than the PolitiFact inputs.
- Task analysis: 48% of a manually annotated sample of 100 PolitiFact pairs were Type-2 cases that could not be matched easily by approximate string matching.TF.IDF-weighted cosine similarity exceeded 0.25 for 27% of PolitiFact pairs and 50% of Snopes pairs, suggesting Snopes is easier.
5 Evaluation Measures
The task is evaluated as ranking, using reciprocal-rank, average-precision, and top-k retrieval measures tailored to real-time and non-real-time fact-checking.
- Ranking measures: The evaluation uses MRR, MAP, MAP@k, and HasPositive@k, where HasPositive@k indicates whether a true positive appears among the top-k results.The reported values cover both truncated and untruncated ranking performance.
- Use cases: MAP@k and HasPositive@k for k ∈{1, 3, 5} model a journalist quickly reviewing a short list of candidate claims in real time.The intended shortlist contains approximately 3–5 claims for rapid checking.
- Use cases: MAP@k and HasPositive@k for k ∈{10, 20}, together with untruncated MAP, are intended for non-real-time settings where recall matters more.These measures evaluate retrieval beyond the short lists used for immediate verification.
6 Models
The models progress from lexical BM25 retrieval to BERT-based semantic similarity and then combine complementary signals with pairwise learning-to-rank.
- Information retrieval models: BM25 ranks verified claims using exact word matches between the input claim and representations such as titles, verified claims, and article bodies.The experiments also test combinations of these representations.
- BERT-based models: Because input and verified claims can use different words, the paper evaluates semantic matching with BERT sentence representations and cosine similarity.The representation is obtained by max-pooling the penultimate layer, rather than fine-tuning BERT.
- BERT-based models: The semantic models include BERT:base,uncased, RoBERTa:base, sentence-BERT:base, and sentence-BERT:large.Sentence-BERT is trained specifically to produce sentence representations, unlike the BERT and RoBERTa variants used here.
- Full-article matching: The full-article model encodes each body sentence with sentence-BERT and compares the input claim with every sentence to produce multiple claim-document scores.This extends claim-level matching to article-level evidence representations.
- Learning to rank: RankSVM with an RBF kernel combines BM25 rankings and embedding-based similarity scores using a pairwise learning-to-rank loss.The combination exploits the different information captured by lexical retrieval and semantic similarity.
7 Experiments
Experiments on PolitiFact and Snopes compare information-retrieval baselines, BERT-based similarity models, and pairwise learning-to-rank models.
- Experimental setup: The experiments proceed from IR-based models to semantic similarity on claims and articles, followed by pairwise learning-to-rank models.The reported datasets are PolitiFact and Snopes.
7.1 Politifact Experiments
Politifact experiments compare lexical, semantic, full-article, and reranking approaches for matching input claims to previously verified claims. Body representations and a RankSVM reranker perform best, while combining exact and semantic signals improves retrieval.
- Full-article matching: The article Body outperformed VerClaim by 0.12-0.14 in MAP@k and MAP, and by 0.09 in MRR.Title performed substantially worse, while matching Title+VerClaim+Body was slightly worse than Body alone.
- Semantic matching: Sentence-BERT base and large beat BERT and RoBERTa by at least 13% absolute across all measures.BERT:uncased also outperformed RoBERTa on all measures.
- Full-article matching: Top-4 Body sentence scores combined with VerClaim and Title scores yielded the best full-article BERT performance, though it still lagged BM25.The model used sentence-BERT:large similarity scores for the article body, verified claim, and title.
- Reranking: The RankSVM reranker improved over previous models by 0.04-0.05 points absolute across evaluation measures.It combines exact BM25 matching with semantic similarity and reciprocal-rank features.
- Reranking: Reranking performance improved as the reranked list grew to 100 results, then degraded for longer lists.
7.2 Experiments on Snopes
Snopes experiments evaluate BM25, sentence-BERT, full-article matching, and reranking under a dataset where input tweets were extracted from article bodies. VerClaim matching is strong, while reranking delivers the largest gains.
- BM25-based baselines: VerClaim outperformed the article Title on every evaluation measure, although the margin was smaller than for Politifact.Combining Title and VerClaim produced small improvements on all MAP@k measures, and BM25 was strong because of high word overlap.
- Semantic matching: Sentence-BERT scored at least 0.10 absolute below BM25 on almost every reported measure.VerClaim still outperformed Title by a sizable margin for semantic matching.
- Full-article matching: Article-body matching was excluded because the input tweet appeared verbatim in the target article, and filtering it still produced unrealistically high results.The authors therefore abandoned the body-matching experiments.
- Reranking: The best RankSVM reranker improved over the best individual model by 0.09-0.18 points absolute on all evaluation measures.The reranker used Title, VerClaim, Title+VerClaim, and sentence-BERT similarity scores and reciprocal ranks.
8 Conclusions and Future Work
The paper establishes previously fact-checked claim detection as a distinct task and releases specialized datasets and code. It reports learning-to-rank improvements and identifies broader datasets, languages, and multimodal claims as future directions.
- Conclusions: Previously fact-checked claim detection is proposed as a task in its own right and as a possible component of automated fact-checking or support tool for fact-checkers and journalists.
- Conclusions: Specialized datasets and code were released to the research community to enable further research.
- Conclusions: Learning-to-rank experiments demonstrated sizable improvements over state-of-the-art retrieval and textual similarity approaches.
- Future work: Future work will extend the approach to more datasets and languages and to claim-image and claim-video pairs.