PaperlayerView exact revision on arXiv

Source-linked AI summary

Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

Tal Schuster, Adam Fisch, Regina Barzilay

arXiv:2103.08541v1cs.CLcs.IRcs.LG

TL;DR

Fact verification must handle evidence that changes over time, but existing systems are not necessarily sensitive to subtle factual revisions. VitaminC constructs contrastive examples from Wikipedia revisions and shows improved robustness, while enabling revision flagging, word-level rationale supervision, and factually consistent generation.

  • Problem

    Fact verification systems need to adapt when written evidence changes, requiring sensitivity to subtle differences between supporting and non-supporting contexts.

  • Method

    VitaminC uses factual Wikipedia revisions to create contrastive claim-evidence pairs and supports additional tasks including revision flagging and word-level rationale modeling.

  • Results

    Training with VitaminC improves classifier sensitivity to subtle evidence changes and increases robustness to adversarial examples.

  • Takeaways & Limitations

    VitaminC’s before-and-after structure supports fact verification, factual revision detection, finer-grained explanations, and factually consistent text generation.

  • Takeaways & Limitations

    The setup assumes access to an oracle retriever and focuses on edits intended to introduce a factual modification expressible through one sentence but not the other.

Abstract

from arXiv · show

Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness -- improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

1 Introduction

VitaminC frames fact verification around contrastive evidence from changing Wikipedia revisions, requiring models to respond to subtle factual changes. Its dataset and training approach improve sensitivity while supporting additional fact-checking tasks.

  • Motivation: Dynamic written resources motivate fact verification systems that adjust to changing evidence rather than relying on static knowledge.The paper highlights facts that are established, changed, or added over time and seeks robustness in practical settings.
  • Contrastive paradigm: VitaminC pairs nearly identical contexts from factual revisions, with one supporting and the other refuting the same claim.This contrastive structure targets sensitivity to factual changes in presented evidence.
  • Results: 56% of contrastive cases triggered prediction flips for a FEVER-trained classifier, compared with 86% when VitaminC examples were added during training.The comparison illustrates increased sensitivity to contrastive evidence after VitaminC training.
  • Additional tasks: The dataset supports factual revision identification, evidence-word relevance tagging, and factually consistent text generation.Its before-and-after structure provides supervised and distantly supervised data for these additional questions.
  • Contributions: VitaminC introduces a large-scale dataset supporting contrastive fact verification and training on it improves performance on standard tasks.These are presented among the paper’s key contributions.

2 Related Work

Prior fact-verification resources often assume static ground truths and can contain annotation artifacts or weakly enforced explanations. VitaminC addresses these issues through dynamic evidence, contrastive structure, and revision-based supervision.

  • Annotation Bias: Crowd-sourced claim construction can introduce biases that enable evidence-agnostic models to achieve unexpectedly high performance.The related-work discussion identifies this as a problem in FEVER-style resources.
  • Fact Verification: Earlier fact-verification datasets commonly assume static ground truths, whereas VitaminC compares claims against a dynamic source of truth.The contrastive setup requires models to change verdicts when evidence changes.
  • Annotation Bias: Contrastive examples are used to study annotation artifacts and model behavior on adversarial or contrastive inputs.The paper situates its approach within broader work on dataset bias in NLP.
  • Explainability: VitaminC conditions verdicts on retrieved evidence and uses revision history as distant supervision for word-level rationales.This provides finer-grained explanations than sentence-level rationale resources alone.
  • Factually Consistent Generation: VitaminC provides supervised sequence-to-sequence data and automatic evaluation through a fact-verification classifier for factually consistent generation.This addresses the difficulty and often manual nature of evaluating factual consistency.

3 The VITAMINC Dataset

VITAMINC is built from factual revisions to English Wikipedia, pairing closely related sentence versions with claims whose support changes across the revision. Real and synthetic revisions yield a large contrastive claim-evidence resource.

  • Dataset construction: VITAMINC uses revisions to English Wikipedia, selected as a trusted and actively maintained evidence source.The corpus draws on popular, FEVER-linked, and COVID-19-related articles and their revisions.
  • Contrastive examples: Factual revisions were converted into symmetric claims supported by one sentence version and refuted by the other when possible.When explicit contradiction was unavailable, annotators used the NEI relation; claims were written to be short and self-contained.
  • Motivation: VITAMINC targets facts that change over time or are prone to correction, supporting contrastive verification and consistency with external sources.The authors connect sensitivity to changing contexts with model faithfulness to specific evidence.
  • Dataset construction: 304,671 revised sentences were annotated, with 107,056 (35%) identified as factual modifications.Factual edits exclude grammatical corrections, paraphrasing, and formatting changes.
  • Dataset construction: 325,724 claim-evidence pairs came from real revisions, supplemented by 163,180 pairs created through synthetic revisions.The synthetic process edits FEVER evidence so originally false claims become supported and creates opposing claims across sentence versions.

4 VITAMINC Tasks

VITAMINC derives four fact-verification tasks from contrastive Wikipedia revisions: detecting factual edits, classifying claim-evidence relations, identifying responsible evidence words, and generating consistent revisions or claims.

  • Task overview: The four tasks are factual revision flagging, fact verification, word-level rationales, and factually consistent generation.They are derived from VITAMINC’s revision-based structure and illustrated through Wikipedia examples.
  • 4.2 Fact Verification: Fact verification predicts whether evidence supports, refutes, or provides insufficient information for a claim.VITAMINC pairs claims with revisions whose relations differ across the two sentence versions.
  • 4.1 Factual Revision Flagging: Factual revision flagging identifies edits that change a claim’s relation from SUP or REF to another verdict.The task is posed as binary classification over sentence revisions.
  • 4.3 Word-level Rationales: Word-level rationales mask a sparse subset of evidence tokens so the claim-evidence relation changes to NEI.The selected tokens are intended to identify the words most responsible for resolving the claim, with F1 evaluated against edit-based and manual references.
  • 4.4 Factually Consistent Generation: Factually consistent generation includes revising outdated evidence to agree with a claim and extracting a claim that expresses a revision.Outputs should support the claim in the updated context and refute it in the outdated context.

5 Experiments

The experiments evaluate VITAMINC across factual revision flagging, fact verification, rationale identification, and factually consistent generation. Training with contrastive VITAMINC examples improves sensitivity to factual changes and robustness on adversarial evaluations.

  • Factual Revision Flagging: 93.42 F1 on PAWS does not transfer well to factual revision flagging, performing on par with a simple edit-distance heuristic.The authors attribute this difference to PAWS’s synthetic entity-scrambling edits versus real factual Wikipedia revisions.
  • Factual Revision Flagging: 10 AUC points separate ALBERT (diff) from a BOW model with similar input, while richer full-sentence inputs and advanced models improve revision flagging further.The best model still reaches only 83 macro-F1, indicating task difficulty.
  • Fact Verification: 89% accuracy is achieved by the best fact-verification model on real revisions, lower than performance on VITAMINC synthetic cases or FEVER.Adding VITAMINC preserves in-domain accuracy while increasing sensitivity to contrastive examples.
  • Robustness: VITAMINC training improves performance on challenging adversarial datasets, whereas augmenting FEVER with MNLI has limited adversarial impact.Increasing the VITAMINC proportion continuously improves adversarial and symmetric-set performance.
  • Rationales and Generation: Distant token-level supervision improves both edit prediction and word-level rationale prediction, while the revision generator succeeds in agreeing with the claim 76% of the time.Its SARI KEEP F1 is 75, and generated outputs were rated on par with human-written ones.

6 Conclusion

The conclusion presents VITAMINC as a large-scale contrastive-context dataset built from factual Wikipedia revisions. It reports improved sensitivity to subtle evidence changes and robustness to adversarial examples, while extending evaluation to revision flagging, rationales, and consistent generation.

  • Conclusion: VITAMINC uses factual Wikipedia revisions to create lexically similar but factually opposing contexts for fact-verification training and evaluation.The dataset is designed around contrastive contexts.
  • Conclusion: Training on VITAMINC improves classifier sensitivity to subtle evidence changes and increases robustness to adversarial examples.This is the paper’s central reported modeling result.
  • Conclusion: VITAMINC supports factual revision flagging, word-level rationale supervision, and factually consistent text generation.These tasks derive from the dataset’s before-and-after revision structure.

A.1 Claim Statistics

The dataset combines real and synthetic claims with varied topical coverage, while manual annotation shows substantial agreement. Real claims emphasize frequently changing or corrected facts, whereas synthetic claims focus on entity attributes.

  • Topic and Category Distribution: 15% of VITAMINC-real claims concern COVID-19, and the synthetic subset contains many claims about specific human entities.These distributions reflect the source and collection period of the subsets.
  • Topic and Category Distribution: VITAMINC combines real and synthetic claims covering diverse topics from Wikipedia articles.Topic labels were derived using DBpedia parent classes, with about 25% of article labels missing.
  • Topic and Category Distribution: About half of VITAMINC-real claims describe changes in numerical values, while VITAMINC-synthetic claims mostly concern entity facts such as birth dates and occupations.The difference follows from the real revision-based and synthetic FEVER-based construction processes.
  • Annotation Quality: 0.7065 Fleiss κ indicates substantial agreement among four annotations of 2,000 claim-evidence pairs.The agreement is comparable to reported fact-verification and NLI annotation scores.

A.3 Claim-only Classification

VITAMINC’s contrastive pairing is designed to prevent claim-only and simple word-overlap shortcuts. Results show claim-only performance at chance and only slight overlap differences between supporting and refuting pairs.

  • Claim-only Bias: 61.7% claim-only accuracy in prior work illustrates annotation artifacts that VITAMINC’s contrastive contexts are designed to avoid.Each claim is paired with evidence contexts that differ in factual relation.
  • Claim-only Bias: 50% claim-only accuracy on VITAMINC-real matches always predicting SUP, providing no useful lexical-cue advantage.VITAMINC-synthetic pairing makes claim-only performance no better than random.
  • Overlap Bias: Supporting and refuting pairs in VITAMINC have only slightly different claim-evidence bigram overlap ratios.Annotators were asked to avoid copying exact evidence phrases into claims when possible.

B Experimental Setting

The experiments use Hugging Face Transformers models, controlled training schedules, and development-set checkpoint selection before test evaluation.

  • All models are implemented with the Hugging Face Transformers library.
  • Models trained on datasets of different sizes receive the same number of update steps, with smaller datasets upsampled.
  • The checkpoint with the highest training-task development accuracy is selected, and performance is reported on the test set.

C GPT-3 Evaluation

The GPT-3 evaluation examines few-shot fact verification and claim extraction, while noting exploratory limitations from restricted API access. GPT-3 follows task-specific prompting but can produce unsupported or incomplete outputs.

  • Evaluation Scope: Because of limited API quota, the GPT-3 results are qualitative evaluations on several VITAMINC test examples and should be viewed as exploratory.
  • GPT-3 for Claim Extraction: GPT-3 is also tested in a two-shot claim-extraction setting that converts Wikipedia revisions into short claims expressing the post-edit fact.
  • GPT-3 for Claim Extraction: The GPT-3 experiments use temperatures of 0 or 0.7 for claim extraction, reflecting a trade-off between factual consistency and fluency.
  • GPT-3 for Claim Extraction: GPT-3 can follow the requested claim format with two examples, but some generations describe the previous state or add unsupported facts.
  • GPT-3 for Claim Extraction: For one example, fverdict labels a GPT-3 output as Not Enough Information because it reports about 20 deaths absent from the input.
  • GPT-3 for Fact Verification: GPT-3 is evaluated for fact verification using six examples from the VITAMINC training set, with numerical-only prompts for numerical claims and mixed prompts otherwise.

D Complementary Experiments

Complementary experiments compare fine-tuned BERT-family models on fact verification datasets, finding stronger performance for ALBERT variants in most reported settings.

  • The complementary fact verification results are reported with a fine-tuned BERT-base model in Table D.1.
  • ALBERT-base outperforms BERT-base on most of the evaluated fact verification datasets.
  • ALBERT-xlarge performs better than both base models on every dataset except Triggers.Triggers contains 186 examples and unnaturally looking claims, which may contribute to high variance across models.

E Example Outputs

The appendix provides prompts, rationale masks, generation examples, and supplementary fact-verification outputs for the paper’s models and GPT-3 evaluations.

  • Rationale Examples: Table E.1 gives example word-level rationale masks identifying evidence words responsible for classifier verdicts.
  • Generation Examples: Table E.2 shows BART-base outputs for generating factually consistent revisions from an old version and an updated claim, alongside the ground-truth revision.
  • Claim-Extraction Examples: Table E.3 provides additional claim-extraction outputs from BART-base and two-shot GPT-3, with human-written claims for reference.
  • Prompt Examples: Figures C.1–C.3 show GPT-3 prompts for few-shot claim extraction and fact verification on numerical and non-numerical claims.
  • GPT-3 Evaluation: Table C.1 presents GPT-3 fact-verification predictions on VITAMINC test examples and compares them with VITAMINC-trained ALBERT classifier predictions.
  • Complementary Experiments: Table D.1 reports complementary fact-verification results using a BERT-base model.
Loading 2103.08541v1…