PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Fact-Checking: A Survey

Xia Zeng, Amani S. Abumansour, Arkaitz Zubiaga

arXiv:2109.11427v1cs.CL

TL;DR

Online false information raises the risk of misleading people, while manual fact-checking cannot keep pace. This survey reviews automated fact-checking research on claim detection and validation, finding substantial progress alongside unresolved challenges in both areas.

  • Problem

    Online false information can mislead people and cause harm, while manual fact-checking cannot keep pace with the volume of online information.

    Source: §1 · p. 1

  • Method

    The paper surveys automated fact-checking research, focusing on claim detection and claim validation.

    Source: §1 · p. 2

  • Results

    Substantial progress has come from applying large pre-trained language models through designed pipelines, but open challenges remain.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Further research is needed on claim-detection definitions, domains, annotation, and datasets, as well as claim-validation dataset quality, system integrity, and interpretability.

    Source: §6 · p. 10

    Because claim-detection methods use different settings and datasets, it is difficult to establish the current state-of-the-art model.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

As online false information continues to grow, automated fact-checking has gained an increasing amount of attention in recent years. Researchers in the field of Natural Language Processing (NLP) have contributed to the task by building fact-checking datasets, devising automated fact-checking pipelines and proposing NLP methods to further research in the development of different components. This paper reviews relevant research on automated fact-checking covering both the claim detection and claim validation components.

1 Introduction

The spread of online false information increases the risk of misleading people and causing harm, while manual fact-checking cannot keep pace with online information. This survey focuses on claim detection and claim validation within automated fact-checking.

1 Introduction

  • Manual fact-checking involves identifying claims, aggregating evidence, checking source credibility and reasoning, and verifying claims, but cannot keep pace with online information.Automation of mechanical tasks can leave human effort for more labor-intensive work.Source: §1 · p. 1
  • Research has produced fact-checking datasets, pipelines, and models, with shared tasks including RumourEval, CheckThat!, ClaimBuster, FEVER, and SCIVER.The cited passage also lists the Fake News Challenge and HeroX fact-checking challenge.Source: §1 · p. 1
  • A comprehensive pipeline detects claims needing checks and matches related claims, then retrieves evidence and produces a verification verdict.Some pipelines include claim detection, while FEVER and SCIVER assume check-worthy claims are already available.Source: §1 · pp. 1–2
  • The survey reviews automated fact-checking with special focus on claim detection and claim validation.It also covers claim matching, evidence retrieval, claim verification, pipeline challenges, and related NLP tasks.Source: §1 · p. 2

2 Claim Detection

Claim detection selects claims for verification and can prioritize check-worthy statements; research covers classification and ranking, datasets, detection methods, and matching against prior fact-checks.

2 Claim Detection

  • Claim detection selects claims needing verification for the rest of the pipeline, while claim matching distinguishes check-worthy from non-check-worthy claims.The task is framed as binary classification or ranking sentences by check-worthiness.Source: §2 · p. 3, §2.1 · p. 3

2.1 Approaches

  • Approaches distinguish claims from opinions and may separately rank claims by whether they merit fact-checking.For example, a factual statement can be a claim while an opinion is not; a claim can still be not check-worthy.Source: §2.1 · p. 3

2.2 Datasets

  • Check-worthiness datasets vary in domain, size, annotation, and language, with most focused on politics and publicly available data spanning English, Arabic, Dutch, and Turkish.Some datasets draw claims from fact-checking sites, while others use manual annotations or crowdsourcing; English datasets predominate and many Arabic datasets are translations.Source: §2.2 · pp. 3–4, Table 1 · p. 3

2.3 Check-Worthy Claim Detection

  • ClaimBuster’s claim spotter classifies sentences into three categories, while CNC combines InferSent embeddings with linguistic features and improves recall and F1.Neural systems and surrounding context or discourse features have also proved effective; work includes Arabic and Dutch detection, but varied settings and datasets make state-of-the-art comparisons difficult.Source: §2.3 · p. 4

2.4 Claim Matching

  • Claim matching ranks previously fact-checked claims against a new check-worthy claim to identify matches that may avoid another validation run.Released datasets use PolitiFact and Snopes; approaches include BM25, BERT-based models, SVM reranking, and fine-tuned transformer models.Source: §2.4 · pp. 4–5

3 Claim Validation

This section surveys claim validation through textual references or knowledge bases, then reviews datasets, evidence retrieval, and claim verification. It describes retrieval as selecting documents and rationales, while verification draws on RTE/NLI approaches and increasingly large pre-trained models.

3.1 Approaches

  • Text-based verification uses references such as Wikipedia, while knowledge-base verification faces coverage limits; the survey focuses on the former for its maturity and reliability.Automatically populating knowledge bases may introduce unreliable noise and make them harder to maintain.Source: §3.1 · p. 5

3.2 Datasets

  • Validation datasets remain commonly synthetic and imbalanced, although newer datasets increasingly include natural sentences, context, metadata, varied domains, and evidence chains.Table 2 presents claim-validation datasets.Source: §3.2 · p. 5, Table 2 · p. 6

3.3 Evidence Retrieval

  • Document retrieval ranks relevant documents, using claim-derived queries or similarity measures, and supervised reranking can improve on unsupervised retrieval.When search engines are unavailable, TF-IDF is a common baseline, BM25 has been effective, and in-domain embeddings are a promising option for domain-specific tasks.Source: §3.3 · pp. 5–6
  • Rationale selection ranks sentences using keyword matching, similarity, or supervised methods, while pipeline retrieval overlooks valuable information across sentences.Systems commonly select the top k sentences, with k chosen manually.Source: §3.3 · p. 6

3.4 Claim Verification

  • Claim verification is commonly framed as text classification, with labels ranging from SUPPORT, CONTRADICTION, and NOT ENOUGH INFORMATION to finer-grained truth values or support labels.Truth-value labels express a claim’s veracity, whereas support labels express alignment with a reference without explicitly asserting truthfulness.Source: §3.4 · pp. 6–7
  • RTE moved from binary entailment decisions to three-way entailment, contradiction, and unknown labels, while NLI datasets offer larger, more balanced, and broader-domain training data.Large pre-trained language models fine-tuned on NLI datasets have become the dominant approach, improving representations and generalisability over traditional approaches.Source: §3.4 · pp. 7–8
  • Earlier RTE work explored semantic logic, NatLog, rewrite rules, and hybrid inference, but reported systems remained imperfect and were seen as having potential with better representations and neural models.A model using deep semantic features did not outperform shallow semantic features, with the authors pointing to rigid representations, limited background knowledge, and dataset issues as likely reasons.Source: §3.4 · p. 7

4 Discussion and Challenges

The discussion identifies unresolved challenges in claim detection and claim validation. Detection is constrained by unclear check-worthiness, narrow domains, annotation choices, and imbalance; validation faces dataset, system-integrity, scalability, and interpretability concerns.

4.1 Claim detection

  • Claim check-worthiness lacks a sufficiently decisive definition, and not every factual claim is verified by fact-checkers.Full Fact describes a claim as an assertion about the world that can be checked, but researchers note this does not decide whether it merits checking.Source: §4.1 · p. 8
  • Claim-detection research has concentrated on politics, while methods have struggled to perform consistently across domains; more recent work also addresses health claims.The cross-domain example considered Twitter rumours, Wikipedia citations, and political speeches.Source: §4.1 · p. 8
  • Claim labels are often assigned manually by non-experts, while fact-checking-site data lack non-claim examples and can require positive-unlabelled learning.Most datasets are imbalanced, with non-check-worthy claims outnumbering check-worthy ones, and this imbalance can encourage overfitting.Source: §4.1 · p. 8

4.2 Claim Validation

  • Claim-validation datasets used to train large language models are expensive to create and may be unrealistic for specific domains; existing datasets also tend to be imbalanced and synthetic.The survey anticipates that higher-quality datasets will continue to help the field progress.Source: §4.2 · pp. 8–9
  • Most validation systems handle subtasks separately, and pipeline errors can accumulate when verification receives low-quality evidence or claims that are not checkable.Large pre-trained language models also require substantial training and inference resources, and the scalability and accessibility of proposed systems remain inferior.Source: §4.2 · p. 9
  • Neural models can make correct predictions for the wrong evidence, while symbolic systems are interpretable but fragile and inflexible, motivating interest in neural-symbolic methods.The survey also suggests multitask learning as a possible direction for a more systematic and compact validation system.Source: §4.2 · p. 9

5 Related Tasks

The survey distinguishes automated fact-checking from related NLP tasks concerned with information credibility. These tasks differ in what they classify, what evidence they use, and whether they require reasoning beyond explicit text.

Fake News Detection

  • Fake news detection classifies whether a web news article is accurate, but shallow features may capture writing style rather than distinguish reliable from unreliable articles.Some approaches also use article metadata.Source: §5 · p. 9

Rumour Detection

  • Rumour detection identifies unverified reports circulating on social media, typically using subjectivity and social-media metadata that do not directly determine a claim’s truth value..Source: §5 · p. 9

Clickbait Detection

  • Clickbait detection does not require external evidence and has achieved reasonable performance with relatively shallow linguistic features.Source: §5 · p. 10

Commonsense Reasoning

  • Commonsense reasoning goes beyond information explicit in sentence pairs, a capability valued in fact-checking, but the area remains under investigation and calls for major breakthroughs.Research includes learned knowledge in pre-trained language models, external knowledge bases, and symbolic knowledge.Source: §5 · p. 10

6 Conclusions

The survey reviews automated fact-checking with particular attention to claim detection and claim validation. It reports progress using large pre-trained language models in designed pipelines, alongside open challenges in both components.

6 Conclusions

  • The survey focuses on claim detection and claim validation within automated fact-checking.Source: §6 · p. 10
  • Large pre-trained language models applied through designed pipelines have supported substantial progress in automated fact-checking.Source: §6 · p. 10
  • Further work is needed on claim-detection challenges and on validation dataset quality, system integrity, and model interpretability.Detection challenges include conceptual definition, narrow domains, annotation issues, and imbalanced datasets.Source: §6 · p. 10

Related papers

Loading 2109.11427v1…