PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiVerS: Improving scientific claim verification with weak supervision and full-document context

David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, Hannaneh Hajishirzi

arXiv:2112.01640v2cs.CLcs.AI

TL;DR

Scientific claim verification must label abstracts and select evidentiary sentences, yet rationale extraction can discard context and requires costly annotations. MULTIVERS jointly predicts labels and rationales from full-document context, enabling weakly supervised adaptation; it outperforms two baselines across three datasets, especially in zero- and few-shot settings.

  • Problem

    Scientific claim verification needs document labels and evidentiary rationales, but extract-then-label systems can lose context and require costly sentence-level annotations.

  • Method

    MULTIVERS uses a shared Longformer encoding of the full claim and abstract to predict labels and rationales jointly, including training instances with only document-level weak labels.

  • Results

    MULTIVERS outperforms two baselines across three datasets, with average improvements of 11% fully supervised, 14% few-shot, and 26% zero-shot.

  • Takeaways & Limitations

    The approach supports scientific claim verification and domain adaptation when labeled data are limited or rationale annotations are unavailable.

  • Takeaways & Limitations

    Automated scientific fact-checking systems are not mature enough to inform real-world medical decisions.

Abstract

from arXiv · show

The scientific claim verification task requires an NLP system to label scientific documents which Support or Refute an input claim, and to select evidentiary sentences (or rationales) justifying each predicted label. In this work, we present MultiVerS, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. This approach accomplishes two key modeling goals. First, it ensures that all relevant contextual information is incorporated into each labeling decision. Second, it enables the model to learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales. This allows MultiVerS to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics. Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments. Our code and data are available at https://github.com/dwadden/multivers.

1 Introduction

Scientific claim verification labels abstracts as SUPPORT or REFUTE and selects evidentiary rationales, but rationale-only decisions can lose essential context. MULTIVERS addresses this with full-context multitask prediction and weakly supervised training.

  • Scientific claim verification labels research abstracts as SUPPORT or REFUTE and selects sentences reporting evidence for each label.
  • Extract-then-label systems can lose acronyms, coreferences, or scope qualifiers when rationales are separated from document context.They also require rationale annotations for every training instance, although document-level labels can be generated cheaply with high-precision heuristics.
  • MULTIVERS encodes the entire claim and abstract with Longformer, then predicts labels and rationales jointly while enforcing consistency during decoding.This lets label decisions use all available context and supports training when only document-level labels are available.
  • MULTIVERS outperforms two state-of-the-art baselines on three scientific claim verification datasets, including one baseline with more than 10x its parameters.
  • Average improvements are 11% in fully supervised, 14% in few-shot, and 26% in zero-shot settings across three datasets.The authors also report that weakly labeled in-domain data substantially improves zero- and few-shot domain adaptation.

2 Background

The task evaluates whether candidate abstracts support, refute, or fail to provide enough information for a claim, while also requiring rationale selection. Prior systems generally extract rationales before labeling, limiting context or supervision flexibility.

  • The scientific claim verification task: Scientific claim verification predicts SUPPORTS, REFUTES, or NEI for each candidate abstract and identifies rationales for SUPPORTS or REFUTES cases.NEI means Not Enough Info.
  • The scientific claim verification task: Rationales may require surrounding abstract context to resolve coreference, acronyms, or qualifiers about experimental settings and populations.
  • The scientific claim verification task: Abstract evaluation measures label-only F1, whereas sentence evaluation multiplies rationale-selection F1 by abstract-level label performance.Sentence evaluation rewards identifying exactly the sentences containing evidence for the label.
  • The scientific claim verification task: Open verification requires retrieving candidate abstracts, while abstract-provided verification supplies candidates as input.
  • The scientific claim verification task: Zero-shot adaptation excludes in-domain fact-checking training, few-shot adaptation permits 45 target claims, and fully supervised training uses all target claims.
  • Prior models: Existing extract-then-label models first predict rationales and then label the claim from their representations, which can restrict access to necessary context or require rationale supervision.

3 The MULTIVERS model

MULTIVERS uses a full-context Longformer representation to predict document labels and sentence rationales jointly. Its multitask design also permits weakly labeled training without rationale annotations.

  • Full-context claim verification: MULTIVERS predicts the fact-checking label directly from the entire claim and abstract while enforcing rationale consistency during decoding.
  • Long-document encoding: Longformer handles claim-abstract inputs exceeding common 512-token limits, with global attention assigned to the claim and sentence-separating tokens.The <s> token receives global attention for document-level prediction.
  • Multitask rationale selection and label prediction: A binary head predicts whether each sentence is a rationale from its globally contextualized separator-token representation.
  • Multitask rationale selection and label prediction: A three-way head predicts the overall label from the globally attended <s> representation of the full claim and abstract.
  • Training: Training minimizes the sum of label loss and rationale loss weighted by λ_rationale.The rationale weight is tuned on the development set.
  • Weak supervision: When no rationale annotations exist, MULTIVERS sets λ_rationale to zero and trains using document-level labels.This supports pretraining on general-domain and weakly labeled in-domain data before target-dataset fine-tuning.

4 Datasets

Experiments use three scientific claim verification datasets covering COVID-19 and biomedical claims, alongside pretraining datasets converted into claim-evidence examples. Weak rationale labels are estimated with sentence similarity but are not used to train MULTIVERS.

  • Scientific claim verification datasets: The experiments use HealthVer, COVIDFact, and SCIFACT, converting some NLI-style resources into claim-abstract verification examples.
  • Scientific claim verification datasets: SCIFACT contains citation-derived atomic claims verified against cited abstracts, with manually negated REFUTED claims and NEI abstracts lacking evidence.The task also requires candidate-abstract retrieval.
  • Scientific claim verification datasets: HealthVer contains COVID-related claims verified against CORD-19 abstracts, and its claims may be complex.
  • Scientific claim verification datasets: COVIDFact contains COVID-19 claims from a subreddit and linked or retrieved scientific papers, with automatically generated negations that make labels somewhat noisy.
  • Pretraining datasets: EvidenceInference prompts are converted into claims describing intervention effects relative to comparators using rule-based heuristics.
  • Pretraining datasets: PUBMEDQA's PQA-A subset supplies claim-like titles as claims and matching abstracts as evidence sources.
  • Weak rationale annotations: Sentence-BERT similarity selects weak rationale candidates, whose Precision@1 is 49.4 for SCIFACT, 48.8 for COVIDFact, and 43.4 for HealthVer.These weak rationale annotations are not used to train MULTIVERS.

5 Experimental setup

The experiments compare MULTIVERS with baseline systems across supervision settings and ablate pretraining, encoder, and modeling choices.

  • Training procedure: Training uses pretraining followed by target-dataset finetuning, with zero-shot, few-shot, and fully-supervised settings defined by available target labels.Few-shot finetuning uses 45 target examples; fully supervised training uses all target data.
  • Experimental design: Experiments with combined target-dataset training did not improve performance.
  • Baseline systems: MULTIVERS is compared with PARAGRAPHJOINT and VERT5ERINI, whose parameter counts range from 360M to 5.6B.
  • Evaluation setup: MULTIVERS is evaluated against baselines in fully supervised experiments, while zero- and few-shot comparisons exclude VERT5ERINI because pretraining it is expensive.
  • Ablations: The ablations vary pretraining data, base encoder, and modeling approach to characterize three key components of MULTIVERS.Pretraining compares FEVERSCI, FEVER, and No-Pretrain; encoder comparisons swap Longformer for RoBERTa; modeling comparisons include Multitask, Pipeline, and MT / PI.

6 Experimental results

MULTIVERS outperforms the baselines across supervision settings, while ablations show that in-domain pretraining and multitask modeling are especially valuable with limited target data.

  • Main results: 26%, 14%, and 11% relative improvements over baselines occur in zero-shot, few-shot, and fully-supervised settings, respectively.These averages span three datasets and two evaluation methods.
  • Main results: HealthVer is the most challenging dataset, with an approximately 20-F1 few-shot-to-fully-supervised gap versus roughly 10 F1 for COVIDFact and SCIFACT.
  • Pretraining: In-domain scientific pretraining produces a 65% zero-shot relative performance decrease when removed, driven primarily by very low recall.
  • Pretraining: 39% lower few-shot performance results without pretraining, while FEVER pretraining remains within 4% of FEVERSCI.
  • Encoder: Longformer consistently helps on SCIFACT but not the other datasets, where fewer than 15% of instances exceed the RoBERTa token limit versus 27% for SCIFACT.
  • Modeling approach: Multitask modeling performs best in zero- and few-shot settings, while Pipeline and MT / PI show performance drops exceeding 50% and roughly 10%, respectively.The multitask approach predicts rationales conditioned on the predicted label, improving recall.
  • Modeling approach: MT / PI outperforms Pipeline in zero-shot experiments, whereas weakly supervised sentence-level rationale training leads to worse performance than omitting that rationale training.
  • Modeling approach: In fully supervised experiments, Multitask performs best on SCIFACT, while Pipeline slightly outperforms it on HealthVer and COVIDFact.MT / PI performs substantially worse than the other approaches on all datasets.

7 Analysis

The analysis examines how rationale context affects verification and compares model performance with human agreement. Multitask modeling is most robust to context-dependent evidence, while abstract-level agreement remains harder than sentence-level agreement.

  • Context dependence: Multitask suffers the smallest performance loss on context-dependent rationales, while MT / PI suffers the largest.Table 4 compares Multitask, Pipeline, and MT / PI on self-contained versus context-dependent instances; sentence-level results are similar.
  • Context dependence: Context-dependent rationales are sufficient only when interpreted within the abstract, unlike self-contained rationales that justify labels in isolation.The analysis defines the distinction using annotations for 128 claim / evidence instances from the SCIFACT test set.
  • Context dependence: 22.8% worse performance for Pipeline and 66.4% worse performance for MT / PI occur on context-dependent instances relative to self-contained ones.The MT / PI drop is driven predominantly by low recall, because it predicts that context-dependent rationales are insufficient for label decisions.
  • Human agreement: Models exceed human agreement on sentence-level F1 but not abstract-level evaluation.Experts tend to agree on the overall claim–abstract relationship more than on which sentences provide the best evidence.
  • Human agreement: Abstract-level label prediction may be nearly solved when claims are atomic and roughly 1,000 in-domain labeled claims are available.This conclusion concerns evidence-containing abstracts and the abstract-level SUPPORTS / REFUTES decision.

8 Related work

Related work frames MultiVerS against extract-then-label pipelines, multitask rationale prediction, latent-rationale methods, long-document verification, and domain adaptation for scientific text.

  • Document context and adaptation: Document-level context is often required to properly interpret scientific findings.This observation was previously reported by Nye et al. (2020).
  • Rationale-based verification: Extract-then-label pipelines first select rationales and then condition label predictions on those selected sentences.This approach was previously used for the EVIDENCEINFERENCE task.
  • Rationale-based verification: Multitask rationale selection and label prediction has been applied to sentiment analysis and propaganda detection, including conditioning rationale prediction on the predicted label.These precedents provide related multitask formulations rather than scientific claim verification results.
  • Rationale-based verification: Alternative approaches model rationales as latent variables instead of using supervised rationale selection.The cited latent-rationale methods include Lei et al. (2016) and Paranjape et al. (2020).
  • Document context and adaptation: Prior work has explored long-document encodings for fact verification and domain adaptation methods for scientific text.Stammbach (2021) used Big Bird for full-document evidence extraction from FEVER, while several works studied scientific-text adaptation.

9 Conclusion

The conclusion presents MultiVerS as addressing context and limited-supervision challenges in scientific claim verification. It reports stronger performance across datasets and points toward specialized-domain and longer-context extensions.

  • Future directions: Future work includes adapting the approach to new scientific sub-domains with handfuls of labeled examples and extending verification to full papers or larger corpora.The paper also proposes studying rationale decontextualization when models may rely on insufficient evidence.
  • Conclusion: MultiVerS addresses rationale-boundary limitations by modeling full-document context and supports zero- and few-shot adaptation through weak supervision.Its multitask design enables training with document-level labels when sentence-level rationales are unavailable.
  • Conclusion: MultiVerS outperforms existing systems across several scientific claim verification datasets.The conclusion summarizes the experimental result without selecting a single dataset or metric.
  • Future directions: The authors hope the task, data, and modeling resources will encourage further progress toward identifying and addressing scientific mis- and disinformation.This is stated as a broader goal for the released resources.

10 Ethical considerations and broader impact

The paper situates scientific claim verification within the broader problem of online scientific mis- and disinformation. It also identifies near-term risks involving medical use and misuse by disinformation generators.

  • Broader impact: Automatically identifying scientific mis- and disinformation is described as a long-term socially beneficial goal because such content is prevalent online.The benefit is framed as a long-term research goal rather than a current system capability.
  • Risks: Automated scientific fact-checking systems are not mature enough to inform real-world medical decisions.The authors state that released software will include a disclaimer about this limitation.
  • Risks: Bad actors could potentially use the work to develop disinformation generators trained to fool automated fact-checkers.The paper says this misuse risk cannot be ruled out, while weighing publication benefits against it.

A.1 Data preprocessing

The preprocessing pipeline reconstructs evidence-bearing abstracts, creates rationale labels and claim variants, and prepares negatives and annotations for model training. It also documents dataset-specific filtering and evaluation-related preprocessing choices.

  • HealthVer: HealthVer evidence sentences are matched to CORD-19 abstracts, and abstract sentences with over 50% string overlap are marked as rationales.Claims with both supporting and refuting evidence in one abstract are removed.
  • COVIDFact: COVIDFact evidence is matched to CORD-19 abstracts, while evidence from unavailable non-CORD-19 sources is discarded.The release includes web-scraped sources such as news articles that are not provided with the data.
  • COVIDFact: COVIDFact refuted claims are generated by replacing words, but manual inspection found truly contradictory claims only roughly a third of the time.Most other generated claims were ungrammatical or had irrelevant evidence.
  • EVIDENCEINFERENCE: EVIDENCEINFERENCE prompts are converted into claims with templates, and refuted claims are generated by swapping increase and decrease templates.The templates express increase, no-change, and decrease relations between interventions, outcomes, and comparators.
  • PUBMEDQA: PUBMEDQA negations are generated by removing negation phrases and inflecting the relevant verbs, while pseudo-rationales are selected by claim-sentence cosine similarity.The top-k sentences are labeled as rationales, with k sampled from {1, 2, 3} in a 4:2:1 ratio.
  • Model preparation: Pretraining uses FEVER, EVIDENCEINFERENCE, and PUBMEDQA, while finetuning uses target datasets with sampled hard negative abstracts retrieved from S2ORC.The implementation initializes from a Longformer-large checkpoint pretrained on S2ORC.

C.4 Cross-dataset generalization

The cross-dataset experiment trains MultiVerS on one or all scientific claim-verification datasets and evaluates transfer across the others. Abstract-level transfer is uneven but possible, whereas sentence-level rationale transfer is consistently weak.

  • Combined training: MultiVerS achieves reasonable performance across all three datasets when trained on combined data, but remains below dataset-specific models.
  • Sentence-level evaluation: None of the datasets generalizes well to another in sentence-level rationale identification.HealthVer transfers better to SCIFACT than SCIFACT transfers to HealthVer.
  • Abstract-level evaluation: SCIFACT and HealthVer generalize reasonably well to each other for abstract labeling but not to COVIDFact.COVIDFact generalizes well to SCIFACT but not to HealthVer, and SCIFACT is generally the easiest dataset to generalize to.
  • Experimental setup: The evaluation compares training and evaluation datasets by reporting performance loss when the model is tested on a different dataset.The combined row represents training on all datasets together.
Loading 2112.01640v2…