PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Fact Verification by Claim Generation

Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang

arXiv:2105.14682v1cs.CLcs.AI

TL;DR

Creating evidence-linked training claims for each new fact-verification domain is costly. QACG generates question-answer pairs from evidence and converts them into supported, refuted, or unverifiable claims; on FEVER, generated claims reach over 70% of fully supervised F1 without human-labeled training examples.

  • Problem

    Creating training data by manually writing claims and linking them to evidence is costly, while large-scale data is not available for every domain requiring fact verification.

    Source: §1 · p. 1

  • Method

    QACG generates question-answer pairs from evidence and converts them into supported, refuted, or Not Enough Information claims for training fact-verification models.

    Source: §1 · p. 1

  • Results

    Over 70% of fully supervised F1 performance is achieved without human-labeled training examples, rising to 89.1% with 100 labeled examples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Generated-claim training reduces the demand for in-domain human annotation and supports zero-shot and few-shot fact verification.

    Source: §1 · p. 2, §4 · p. 5

    Generated claims mostly lack deep reasoning over evidence, which the authors attribute to the prevalence of shallow factoid questions in SQuAD.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Neural models for automated fact verification have achieved promising results thanks to the availability of large, human-annotated datasets. However, for each new domain that requires fact verification, creating a dataset by manually writing claims and linking them to their supporting evidence is expensive. We develop QACG, a framework for training a robust fact verification model by using automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. QACG generates question-answer pairs from the evidence and then converts them into different types of claims. Experiments on the FEVER dataset show that our QACG framework significantly reduces the demand for human-annotated training data. In a zero-shot scenario, QACG improves a RoBERTa model's F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. Our QACG code is publicly available.

1 Introduction

Fact verification relies on human-labeled data, but creating evidence-linked claims for every new domain is costly. QACG generates supported, refuted, and unverifiable claims from evidence, and its generated data supports zero-shot and few-shot verification.

1 Introduction

  • Creating training data by having people write claims and find supporting or refuting evidence is costly, and large-scale data is not available for every domain.Fact verification is needed for domains including news, social media, and scientific documents.Source: §1 · p. 1
  • QACG generates supported, refuted, and Not Enough Information claims from evidence to train fact-verification models.The framework uses question generation, which the authors say they are the first to employ for fact verification.Source: §1 · p. 1
  • The authors generate 100K evidence-claim pairs for each claim type and evaluate on three FEVER-based test sets.The provided passage does not specify the names of the test sets.Source: §1 · p. 1
  • Over 70% of fully supervised F1 performance is achieved without human-labeled training examples, rising to 89.1% with 100 labeled examples.The authors report that generated-claim pretraining reduces in-domain annotation demand.Source: §1 · p. 2
  • The authors report this as the first investigation of zero-shot fact verification and propose generating claims through question generation.Source: §1 · pp. 1–2

2 Methodology

QACG trains fact-verification models using synthetic evidence-claim pairs rather than human-annotated examples. It generates questions from evidence and converts question-answer pairs into claims, with separate procedures for supported, refuted, and NEI claims.

2 Methodology

  • In the zero-shot setting, the model is trained on a synthetic dataset because no human-annotated training examples are assumed available.Source: §2 · p. 2

2.1 Question Generator and QA-to-Claim

  • The Question Generator forms questions from an evidence passage and an answer span, while QA-to-Claim converts question-answer pairs into declarative claims.The models are BART fine-tuned on SQuAD and QA2D, respectively.Source: §2.1 · p. 2

2.2 Claim Generation

  • For supported claims, named entities in the evidence are used in turn as answers, and the generated question-answer pairs are converted into claims.Source: §2.2 · pp. 2–3
  • For refuted claims, the answer is replaced with a same-type entity sampled from similar phrases, then the altered pair is converted into a claim.Lexical-overlap rules reduce the chance that the replacement remains correct, but the authors report two such cases among 100 sampled claims.Source: §2.2 · p. 3
  • For NEI claims, five sentences retrieved from the same Wikipedia article but absent from the evidence are added to generate a question-answer pair.An answer drawn from the extra context yields a claim relevant to, but unverifiable from, the original evidence.Source: §2.2 · p. 3

3 Experiments

Experiments test QACG-generated training data across FEVER evaluation settings and few-shot learning. The reported results cover zero-shot performance, robustness, and the scope of generated claims.

3 Experiments

  • 176,370 supported, 360,924 refuted, and 258,452 NEI claims were generated from 18,541 articles, roughly five times FEVER's human-annotated claims.Evaluation spans FEVER-S/R and FEVER-Symmetric with supported/refuted claims, plus three-class FEVER-S/R/N; the supervised models are BERT and RoBERTa.Source: §3 · p. 3

3.1 Main Results

  • 78.1 F1 on FEVER-S/R and 62.6 F1 on FEVER-S/R/N for zero-shot RoBERTa-large (U6), with gaps of 17.0 and 15.2 F1 to fully supervised RoBERTa-large (S2).Table 1 summarizes supervised and zero-shot performance across three settings.Source: §3.1 · p. 4, Table 1 · p. 4
  • U6 achieves the best results among zero-shot baselines across all three test sets; MNLI transfer beats random guessing by 9.30 F1, while the authors cite domain and task gaps.The authors say generating pseudo-training data from Wikipedia avoids the domain-gap issue because FEVER is also from Wikipedia.Source: §3.1 · p. 4
  • On FEVER-Symmetric, supervised BERT and RoBERTa lose 11.4 and 9.6 F1, versus 1.2 and 1.0 F1 drops for models trained with generated data.The authors suggest the varied generated claims help eliminate FEVER annotation artifacts, leading to a more robust model.Source: §3.1 · p. 4

3.2 Few-shot Fact Verification

  • 78.6 F1 with 50 labeled claims per class versus 52.9 without pretraining (+25.7), leaving a 7.9 F1 gap to the fully supervised 86.5 F1.The figure tracks FEVER-Symmetric F1 as training-set size increases; the reported comparison illustrates the larger gain in a data-poor setting.Source: §3.2 · p. 4, Figure 2 · p. 4

SUPPORTED claims

  • Generated claims offer a warm start for new domains with limited training data; Table 2 illustrates examples categorized by claim label.The table highlights answer spans in evidence and claims, and marks replaced wrong answers in refuted claims.Source: SUPPORTED claims · p. 5, Table 2 · p. 5

3.3 Analysis of Generated Claims

  • Generated claims are mostly shallow and lack deep reasoning, which the authors attribute to SQuAD question generation, where over 80% of questions are shallow factoids.The analysis identifies this as a limitation of the generated claims.Source: §3.3 · p. 5

4 Conclusion and Future Work

QACG uses question generation to create varied claims and trains a well-performing fact-verification model in zero-shot and few-shot settings. Future work includes more complex claims, domains beyond Wikipedia, and robustness.

  • QACG uses question generation to ask different questions about evidence and convert question–answer pairs into claims with different labels.Source: §4 · p. 5
  • The generated claims train a well-performing fact-verification model in both zero-shot and few-shot settings.Source: §4 · p. 5
  • Future directions include generating claims requiring deep reasoning, extending beyond Wikipedia, and using generated claims to improve system robustness.Source: §4 · p. 5

Ethical Considerations

The authors focus claim generation on factoid claims from Wikipedia, treated as a source of objective fact. They caution about misuse of refuted claims and advise against releasing generated claims for public use.

  • The work focuses on factoid claims from Wikipedia, which it treats as a source of objective fact.Source: Ethical Considerations · p. 6
  • Generating refuted claims could spread misinformation, so practitioners should handle generated claims carefully.Source: Ethical Considerations · p. 6
  • The authors use generated claims only to optimize fact verification and advise against releasing them publicly, where they may be crawled and used for inference.Source: Ethical Considerations · p. 6

A Evaluation of Question Generation

The authors fine-tune pretrained BART on SQuAD for question generation and evaluate question quality and its impact on zero-shot fact verification.

A.1 Automatic Evaluation

BART achieves a BLEU-4 score of 21.32, outperforming three non-pretrained Seq2Seq baselines; against UniLM, its results are comparable, with lower BLEU-4 but higher METEOR.

A.1 Automatic Evaluation

  • BART is compared with question-generation baselines using BLEU-4, METEOR, and ROUGE-L.The reported baseline results use the Zhou split of SQuAD.Source: §A.1 · p. 7, Table 4 · p. 7
  • 21.32 BLEU-4: BART outperforms NQG++, S2ga-mp-gsa, and CGC-QG, while achieving comparable results to UniLM with lower BLEU-4 but higher METEOR.The comparison evaluates question-generator implementations; the reported baseline gap is attributed to those models lacking language-model pretraining.Source: §A.1 · p. 7, Table 4 · p. 7

A.2 Impact of Answerability

The section evaluates whether question answerability matters for the validity of generated claims and fact-verification performance. BART-generated questions were more often answerable than NQG++ questions, and switching to NQG++ coincided with lower verification F1.

A.2 Impact of Answerability

  • 89.5% of BART-generated questions were answerable versus 63.5% for NQG++, while switching to NQG++ lowered zero-shot F1 to 62.3 (−22.1% compared with BART).The table covers answerable rates and zero-shot fact-verification performance on FEVER-Symmetric.Source: Table 5 · p. 8, §A.2 · p. 8
  • Answerability is tested because a supported claim depends on the evidence answering the generated question with the generated answer as correct.The study samples 100 question-answer pairs per generator and asks two workers to judge answerability.Source: §A.2 · pp. 7–8
  • BART's higher answerable-question rate and NQG++'s lower verification F1 are reported as evidence that answerability affects generated-claim validity and verification performance.The passage describes answerability as playing an important role and says it has a huge impact on performance.Source: §A.2 · p. 8

B Dataset Statistics

QACG-Filtered is a balanced sample from QACG-Full used to train the fact-verification model in zero- and few-shot settings. Its class balance and multiple claim types per evidence passage are intended to make prediction depend on the evidence-claim pair.

B Dataset Statistics

  • QACG-Filtered is sampled from QACG-Full and used to train the fact-verification model in zero- and few-shot settings.Source: B Dataset Statistics · p. 8
  • Unlike the original FEVER dataset, QACG-Filtered has a balanced number of claims for each class.Source: B Dataset Statistics · p. 8
  • Because QACG generates supported, refuted, and NEI claims from the same evidence, the model must use the evidence-claim pair rather than evidence alone to infer the label.Source: B Dataset Statistics · p. 8

C Model Implementation Details

The implementation covers BERT and RoBERTa fact-verification models and three unsupervised baselines. The baselines use GPT-2 perplexity, MNLI entailment labels, or masked-entity prediction with an entailment model.

C Model Implementation Details

  • Table 6 gives basic statistics for FEVER and the dataset generated by QACG.Source: Table 6 · p. 8
  • The implementation uses BERT-base and RoBERTa-large, with 110M and 355M parameters respectively, and fine-tunes models for five epochs using batch size 16 and learning rate 1e-5.The epoch with the best performance is saved.Source: C Model Implementation Details · p. 8
  • The GPT-2 perplexity baseline treats high-perplexity claims as misinformation, assigning the top third as refuted, bottom third as supported, and the rest as NEI.The threshold is difficult to set manually; equal thirds are used because FEVER test classes are balanced.Source: C Model Implementation Details · p. 8
  • MNLI-Transfer applies a BERT-base model fine-tuned on 433K MNLI sentence pairs, mapping entailment, contradiction, and neutral to supported, refuted, and NEI.Source: C Model Implementation Details · p. 8
  • The LM-as-Fact-Checker baseline predicts masked named entities with BERT and uses the MNLI model as its entailment model.The authors implement it following Lee et al. (2020b).Source: C Model Implementation Details · p. 8

Related papers

Loading 2105.14682v1…