PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Scientific Claims for Zero-Shot Scientific Fact Checking

Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, Lucy Lu Wang

arXiv:2203.12990v1cs.CL

TL;DR

Scientific fact checking is challenging because scientific language is complex and expert annotation is costly. The paper generates supported biomedical claims with supervised and unsupervised methods and creates refuted variants with KBIN. In zero-shot fact checking, generated claims achieve within 90% of the performance of a model trained on human-written claims, and the authors report quality improvements over prior work.

  • Problem

    Complex scientific language makes automated claim extraction and fact checking difficult, while creating manually annotated training labels is time-consuming and expensive because it requires domain expertise.

    Source: §1 · p. 1

  • Method

    The paper proposes CLAIMGEN-ENTITY and CLAIMGEN-BART for generating supported biomedical claims, and KBIN for generating refuted claims through knowledge-base-informed entity replacement.

    Source: §1 · p. 2, §3 · pp. 2–3

  • Results

    Generated claims used with KBIN achieve within 90% of the performance of a model trained on human-written claims, and CLAIMGEN-BART and KBIN yield higher claim and negation quality than prior work.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Claim generation enables zero-shot biomedical fact checking and can expand existing datasets without additional manual annotation.

    Source: §1 · p. 1

    The authors note that dataset artifacts cannot be fully separated from model performance in this setting, and pretrained language models may have encountered claim content during pretraining.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data, as annotation requires domain expertise. To address this challenge, we propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its usefulness in zero-shot fact checking for biomedical claims. We propose CLAIMGEN-BART, a new supervised method for generating claims supported by the literature, as well as KBIN, a novel method for generating claim negations. Additionally, we adapt an existing unsupervised entity-centric method of claim generation to biomedical claims, which we call CLAIMGEN-ENTITY. Experiments on zero-shot fact checking demonstrate that both CLAIMGEN-ENTITY and CLAIMGEN-BART, coupled with KBIN, achieve up to 90% performance of fully supervised models trained on manually annotated claims and evidence. A rigorous evaluation study demonstrates significant improvement in generated claim and negation quality over existing baselines

1 Introduction

The paper addresses the difficulty of extracting and checking scientific claims when scientific language is complex and expert annotation is costly. It proposes automatic claim generation for biomedical zero-shot fact checking, including methods for supported claims and refutations.

1 Introduction

  • The work targets the difficulty of automating claim extraction and fact checking from complex scientific assertions, where expert annotation is time-consuming and expensive.The authors motivate methods requiring less manual annotation for scientific data.Source: §1 · p. 1
  • Scientific claim generation is presented as a way to support biomedical zero-shot fact checking and expand datasets without additional manual annotation.The introduction also notes that claim generation can reduce reliance on expert-labeled data.Source: §1 · p. 1
  • The work compares unsupervised CLAIMGEN-ENTITY and supervised CLAIMGEN-BART, and introduces KBIN to generate refuted scientific claims.The paper also reports that CLAIMGEN-BART and KBIN produce higher-quality claims and more convincing negations than prior work.Source: §1 · p. 2
  • Generated claims coupled with KBIN achieve 90% of the performance of a model trained on in-domain manually written claims in zero-shot fact checking.The paper reports a rigorous evaluation of claim and negation quality against existing baselines.Source: §1 · p. 2

2 Preliminaries

The preliminaries define the qualities of a valid claim and specify the claim-generation task. In fact-checking settings, generated claims must also include refuted variants so they can support training data.

Valid Claims

  • A valid claim is fluent, atomic, de-contextualized, and accurately reflects the original sentence's meaning.Atomicity means a verifiable finding about one aspect of a scientific entity or process that can be checked from a single source.Source: §2 · p. 2

Scientific Claim Generation

  • Scientific claim generation distills one or more valid claims entailed by a scientific sentence and, optionally, its context sentences.The claims may be paired with evidence documents as fact-checking training data.Source: §2 · p. 2
  • Fact-checking data also requires claims refuted by the source, alongside supported claims and claims whose veracity cannot be judged from available information.The task therefore includes generating negations of claims entailed by the source sentence.Source: §2 · p. 2

3 Generating Supported Claims

The paper presents unsupervised and supervised approaches for generating claims supported by scientific source sentences. CLAIMGEN-ENTITY uses an entity-centric pipeline, while CLAIMGEN-BART generates claims directly from sentence–claim examples.

3.1 CLAIMGEN-ENTITY

  • CLAIMGEN-ENTITY adapts an unsupervised entity-centric method to biomedical text without requiring sentence–claim training pairs.The pipeline identifies entities, generates answerable questions about them, and converts the questions into declarative claims.Source: §3 · p. 2, §3.1 · p. 3

3 Generating Supported Claims

  • KBIN replaces a source-claim entity with related same-type UMLS concepts, then selects a fluent replacement that maximizes contradiction with the original.The described pipeline uses GPT-2 to rank replacement fluency and an external NLI model to select for contradiction.Source: Figure 2 · p. 3

3.2 CLAIMGEN-BART

  • CLAIMGEN-BART uses BART trained on a small set of sentence–claim pairs to generate claims directly.It encodes the citance with neighboring context to help de-contextualize claims and generates multiple claims using top-k sampling.Source: §3 · p. 3, §3.2 · pp. 3–4

4 Knowledge Base Informed Negations

KBIN generates claim variants by replacing entities with related biomedical concepts. It ranks the resulting sentences for fluency, then selects the replacement most contradictory to the original claim.

4 Knowledge Base Informed Negations

  • KBIN addresses the need for refuted claim variants as negative training instances when building fact-checking datasets.It uses UMLS as a source of replacement terms for biomedical claims.Source: §4 · p. 4
  • For each linked claim entity, KBIN considers sibling UMLS concepts of the same semantic type and retains the 20 closest concepts by cui2vec vector distance.The approach first applies named entity recognition and links entities to UMLS concepts.Source: §4 · p. 4
  • KBIN replaces entity text with related concepts' canonical names and aliases, ranks variants by GPT-2 perplexity, then chooses the fluent variant with greatest NLI contradiction.The contradiction model is RoBERTa pretrained on MNLI.Source: §4 · p. 4

5 Experiments

The experiments test generated claims in zero-shot biomedical fact checking and evaluate claim and negation quality. Generated-claim models approach the in-domain supervised upper bound, while evaluations identify a yield–acceptability trade-off and stronger negations from KBIN.

5.1 RQ1: Fact Checking Performance

  • 77.70 F1 is the in-domain upper bound; models trained on automatically generated claims achieve within 90% of manually trained performance on SciFact.The reported comparison is invariant to whether CLAIMGEN-ENTITY or CLAIMGEN-BART generated the claims.Source: Table 1 · p. 5, §5.1 · p. 5
  • CLAIMGEN-ENTITY and CLAIMGEN-BART perform similarly, and both significantly improve over FEVER-only pretraining, especially without an NEI threshold.Source: §5.1 · p. 5

5.2 RQ2: Claim Quality Evaluation

  • CLAIMGEN-ENTITY generates more claims per citance, but its acceptable-claim precision is much lower than CLAIMGEN-BART’s.The authors describe a trade-off between claim yield and acceptability; whether higher yield improves coverage is left for future work.Source: §5.2 · p. 6, Table 3 · p. 7
  • Generated claims from both methods have high overlap with references, with CLAIMGEN-BART significantly closer than CLAIMGEN-ENTITY.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 RQ3: Negation Evaluation

  • KBIN produces more fluent and more convincing negations on average than both baselines, and fewer directly entailed claims than Saakyan et al. (2021).It can replace UMLS entity types including verb phrases that encode relations.Source: §5.3 · p. 8

5.4 Further Analysis

  • Original SciFact claims show performance well above majority and random baselines, which the authors attribute likely in part to dataset artifacts.Source: §5.4 · p. 8
  • Claim-only models trained on generated claims perform closer to random on SciFact than those trained on original claims, indicating fewer label-associated artifacts.The authors note that artifacts and model performance cannot be fully separated, and pretrained-model exposure may affect this setting.Source: §5.4 · pp. 8–9, Figure 3 · p. 9

6 Related Work

Prior scientific fact-checking work developed retrieval and stance systems, while general-domain research explored adversarial and label-free approaches. This work extends zero-shot fact checking to science and uses biomedical structured knowledge to improve generated negations.

6 Related Work

  • Scientific fact checking retrieves literature documents and evidence sentences, then classifies claims as supported, refuted, or lacking enough evidence [Wadden et al., 2020].Related systems use neural retrieval or joint rationale and stance prediction.Source: §6 · p. 9
  • The proposed methods extend zero-shot fact checking to science, reaching 90% of state-of-the-art inference performance without domain-specific labeled data.Source: §6 · p. 9
  • KBIN improves on COVID-Fact's masked-language-model negations by leveraging in-domain structured knowledge from UMLS.COVID-Fact combines Reddit claim extraction and human verification with automatically generated negations.Source: §6 · p. 9

7 Conclusion

The paper introduces scientific claim generation methods and reports strong zero-shot fact-checking and human-evaluation results. It identifies quality and broader-domain evaluation as remaining work.

7 Conclusion

  • The paper introduces scientific claim generation, CLAIMGEN-BART, CLAIMGEN-ENTITY, and KBIN.Source: §7 · p. 9
  • Generated claims support zero-shot scientific fact checking within 90% of a model trained on human-written claims.A user study found CLAIMGEN-BART claims higher quality than CLAIMGEN-ENTITY claims, and KBIN negations more fluent and convincing than previous work.Source: §7 · p. 9
  • Future work should improve claim-generation quality and assess other scientific domains and use in evidence retrieval.The authors hope the methods will speed training-dataset creation and improve scientific fact-checking models.Source: §7 · p. 9

Ethical Considerations

The authors aim to improve science fact-checking generalizability where training resources are limited, while enabling study of trustworthiness questions around generated data and inferred labels.

Ethical Considerations

  • Scientific fact checking may help address scientific misinformation, and the authors aim to improve its generalizability in domains with fewer training resources.Source: Ethical Considerations · p. 10
  • Generated or synthetic training data raises questions about data veracity and whether models trained on inferred labels produce trustworthy judgments.Source: Ethical Considerations · p. 10
  • The authors hope their methods will support research into these questions and reduce the manual effort and cost of scientific data curation.Source: Ethical Considerations · p. 10

A.1 Computing Infrastructure

Experiments ran on a single AWS instance with a Tesla V100 GPU.

A.1 Computing Infrastructure

  • All experiments used an AWS p3.2xlarge instance with a Tesla V100 GPU and 16GB RAM.Source: §A.1 · p. 12

A.2 Number of Parameters per Model

The appendix reports model-size and training details, then describes the datasets used across fact checking and claim-generation components. These passages identify dataset roles but do not give the model-size values.

A.2 Number of Parameters per Model

  • Table 8 lists the sizes of the models used in the work, but the supplied passage gives no values.The passage points to the table for the parameter counts.Source: §A.2 · p. 12

A.3.1 Fact Checking

  • SciFact training used 809 claims, while CLAIMGEN-BART and CLAIMGEN-ENTITY used 1,561 and 8,592 training claims, respectively.The stated configurations also specify learning rates, epochs, gradient accumulation, batch size, and precision.Source: §A.3.1 · p. 12
  • The NEI threshold was tuned on SciFact’s training set, with 5e-5 reported as the best value.The passage lists the tested threshold range.Source: §A.3.1 · p. 12

A.3.2 CLAIMGEN-BART

  • CLAIMGEN-BART training used a 2e-5 learning rate, three epochs, 200-step linear warmup, linear decay, weight decay of 0.01, and batch size 8.Source: §A.3.2 · p. 13

A.4 Description of Datasets

  • SciFact provides 585 citances with one to two rewritten claims each; its rewritten claims train CLAIMGEN-BART, while its labeled dev set tests fact checking.The SciFact test set has no labels, and the train set is used for supervised fact-checking experiments.Source: §A.4 · p. 13
  • FEVER supplies Wikipedia-based claims, evidence, and labels for pretraining fact-checking models before zero-shot transfer to biomedical claims.Source: §A.4 · p. 13
  • MedMentions trains ScispaCy’s entity models; UMLS supplies KBIN’s normalization and candidate-selection knowledge base and trains cui2vec for candidate concept selection.Source: §A.4 · p. 13
  • SQuAD trains CLAIMGEN-ENTITY’s question-generation module, QA2D trains its question-to-declarative stage, and MNLI trains KBIN’s entailment model for selecting negations.MNLI contains 433k sentence pairs annotated for textual entailment.Source: §A.4 · p. 13
Loading 2203.12990v1…