PaperlayerView exact revision on arXiv

Source-linked AI summary

SciFact-Open: Towards open-domain scientific claim verification

David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, Hannaneh Hajishirzi

arXiv:2210.13777v1cs.CLcs.AI

TL;DR

Scientific claim verification systems have largely been evaluated on limited contexts, while open-domain evaluation against large literature corpora faces infeasible exhaustive evidence annotation. SCIFACT-OPEN uses pooled model predictions to collect evidence for a 500K-abstract test collection. Models developed on SCIFACT perform at least 15 F1 worse in this setting, and the collection exposes phenomena such as conflicting evidence and specificity mismatches.

  • Problem

    Prior scientific claim verification work focused on limited contexts, while realistic open-domain evaluation is difficult because exhaustive evidence annotation is infeasible.

    Source: §1 · p. 1

  • Method

    The authors construct SCIFACT-OPEN by combining existing evidence with pooled model predictions for annotation against a corpus of 500K abstracts.

    Source: §1 · pp. 1–2

  • Results

    Models developed for SCIFACT perform at least 15 F1 worse in the open-domain setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The collection supports analysis of open-domain modeling challenges, including conflicting evidence and mismatches between claim and evidence specificity.

    Source: §1 · p. 2, §7 · p. 9

    Pooling substantially fewer systems than typical TREC competitions may increase uncertainty in the test collection.

    Source: §8 · p. 9

Abstract

from arXiv · show

While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.

1 Introduction

SCIFACT-OPEN evaluates scientific claim verification against a 500K-abstract corpus, where exhaustive evidence annotation is infeasible. The paper reports substantial generalization drops and claim–evidence phenomena such as conflicting evidence and specificity mismatches.

1 Introduction

  • Open-domain evaluation asks systems to identify and categorize supporting or refuting evidence across large corpora, but exhaustive annotation is infeasible.The paper frames this evaluation challenge as analogous to information retrieval.Source: §1 · p. 1
  • SCIFACT-OPEN combines SCIFACT evidence with additional evidence from 500K abstracts, using pooled model predictions to select candidates for human annotation.This pooling strategy avoids exhaustive annotation.Source: §1 · p. 2
  • Systems developed for SCIFACT perform at least 15 F1 worse in the open-domain setting.The collection also exposes conflicting evidence and mismatches between claim and evidence specificity.Source: §1 · p. 2
  • The collection provides a more realistic open-domain test set for studying model generalizability and claim–evidence phenomena.Its evidence includes cases where a specific result supports only a special case of a broader claim.Source: Figure 1 · p. 1, §1 · p. 2

2 Background and Task Overview

The section introduces the scientific claim verification task and the SCIFACT-ORIG dataset that this work builds upon. SCIFACT-ORIG uses a small abstract corpus and models that retrieve candidate documents before labeling them.

2.1 Task definition

  • Given a claim and a corpus, systems identify relevant abstracts and label evidentiary pairs SUPPORTS or REFUTES; other pairs are NEI.A claim/abstract pair with supporting or refuting evidence is an ECAP, evaluated through precision, recall, and F1.Source: §2.1 · p. 2

2.2 SCIFACT-ORIG

  • SCIFACT-ORIG uses diverse claims rewritten from citation sentences and requires evidence retrieval from a roughly 5K-abstract corpus.Its claims span subject matter and levels of specificity.Source: §2.2 · p. 2
  • SCIFACT-ORIG models retrieve candidate abstracts, then classify each as SUPPORTS, REFUTES, or NEI; negative sampling ratio is a key determinant of generalization.The ratio counts irrelevant CAPs per relevant ECAP during training.Source: §2.2 · p. 2

3 The SCIFACT-OPEN dataset

SCIFACT-OPEN extends the SCIFACT test claims to a 500K-abstract corpus, using pooled model predictions to gather evidence without exhaustive annotation. Its evidence includes specificity mismatches and an unequal distribution across claims.

3 The SCIFACT-OPEN dataset

  • The evaluation omits rationale prediction because collecting rationale annotations was considered too expensive, and instead uses abstract-level label-only F1.The original SCIFACT task also required rationales justifying each label.Source: §3 · p. 2
  • The test collection uses SCIFACT-ORIG test claims and adds evidence from a randomly sampled corpus of 500K medicine- and biology-related abstracts.The corpus retains the original 5K abstracts and does not impose additional article-quality filters.Source: §3 · p. 3

3.1 Pooling for evidence collection

  • Pooling combines each of four systems’ 250 highest-confidence claim–abstract predictions, which experts annotate and a second annotator double-checks.The pool is the union of systems’ top predictions; unannotated pairs are assumed to have the NEI label.Source: §3.1 · pp. 3–4

3.2 Dataset statistics

  • Roughly a third of pooled predictions contain evidence, and 732 pairs were annotated rather than 1,000 because systems’ predictions overlap.Most annotated pairs were selected by only one system, indicating diversity in their predictions; the procedure also included the great majority of original SCIFACT evidence pairs.Source: §3.2 · p. 4

3.3 Evidence phenomena in SCIFACT-OPEN

  • Half of all evidence-containing pairs are allocated to 34 highly studied claims, or 12% of the dataset’s claims.Highly studied claims tend to be short and mention a few common scientific entities; their entities return, on average, four times as many PubMed search results as entities in claims with no evidence.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
  • 91 of 206 examined pairs (44%) show a claim–evidence specificity mismatch; among claims with at least two evidence pairs, 16 of 81 (20%) have conflicting evidence.Conflicts were often associated with specificity mismatches.Source: §3.3 · pp. 5–6

4 Model performance on SCIFACT-OPEN

SCIFACT-OPEN tests claim verification on a large, unseen corpus, where existing models perform substantially worse and agree less on newly added abstracts. The results also indicate that negative sampling affects generalization more than model architecture.

SCIFACT-OPEN is challenging

  • 15 to 30 F1 points: models lose this much on SCIFACT-OPEN relative to SCIFACT-ORIG, indicating difficulty generalizing to unseen large corpora.PARAGRAPHJOINT, MULTIVERS, and MULTIVERS10 perform similarly; VERT5ERINI performs worse due to low precision.Source: §4 · p. 6

Negative sampling affects generalization

  • Negative sampling rate affects open-setting precision and recall more than model architecture, with MULTIVERS10 behaving more like PARAGRAPHJOINT than MULTIVERS.VERT5ERINI has recall more than double its precision, while MULTIVERS shows the reverse pattern.Source: §4 · p. 6

Models have low agreement on SCIFACT-OPEN

  • Jaccard similarity is at least 0.5 for predictions on SCIFACT-ORIG abstracts but at most 0.2 for newly added SCIFACT-OPEN abstracts.The lower agreement on new abstracts diversifies the annotation pool and indicates less agreement when models generalize to novel corpora.Source: Figure 4 · p. 6, §4 · p. 7

5 Dataset reliability

Reliability checks examine how annotation depth and the number of pooling systems affect evidence discovery and evaluation. System inclusion also affects measured performance, so comparisons with pool-building models should account for their excluded performance.

5 Dataset reliability

  • Reliability checks vary annotations per system and system count to test whether added annotations reveal many ECAPs or alter evaluation metrics.The checks follow Zobel (1998).Source: §5 · p. 7

5.1 Annotations per system

  • 24 new ECAPs: increasing pool depth from 200 to 250 discovers this many, versus 83 from annotating the 50 most-confident CAPs per system.From d = 225 to d = 250, each model’s F1 changes by less than 2%; further increases are unlikely to affect evaluation metrics.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 System count

  • Adding a fourth system still changes VERT5ERINI and MULTIVERS F1 by 10%, despite diminishing new ECAP discoveries as system count increases.The collection could benefit from data identified by new models, but system count is limited by the available systems.Source: §5.2 · pp. 7–8, Figure 6 · p. 8

5.3 System inclusion

  • Roughly 15%: all systems except MULTIVERS lose about this much F1 when excluded from data collection, so new models should be compared with pool-builders’ excluded performance.MULTIVERS does not benefit from inclusion because it was over-fit to SCIFACT-ORIG and struggles to identify new evidence.Source: §5.3 · p. 8

6 Related work

Related work includes pooled information-retrieval evaluation, open-domain verification with different targets, and claim revision tasks. SCIFACT-OPEN differs from TREC-style pooling in how annotations are allocated across claims.

TREC and pooled data collection

  • Unlike TREC datasets, which often use around 50 annotations per topic, SCIFACT-OPEN varies annotations per claim based on model confidence to maximize annotation yield.TREC datasets typically have fewer queries and more models available for annotation.Source: §6 · p. 8

Claim verification and revision

7 Discussion & Conclusion

The paper presents SCIFACT-OPEN, built with pooling to evaluate open-domain scientific claim verification on a 500K-document corpus. Its analyses identify generalization challenges and evidence that can conflict or differ in specificity from claims.

Discussion & Conclusion

  • SCIFACT-OPEN adapts information-retrieval pooling to identify and annotate evidence from a 500K-document corpus for open-domain claim-verification evaluation.The authors hope the methodology will also be used for other NLP tasks where exhaustive annotation is infeasible.Source: §7 · pp. 8–9
  • Some claims have much conflicting evidence, and evidence may not match the specificity of claims as written.The paper releases 91 claim revisions and proposes generating revisions matched to each evidence abstract’s specificity; using evidence for summaries may be concise but risks hallucination.Source: §7 · p. 9
  • Existing models’ performance declines substantially on SCIFACT-OPEN, suggesting current claim-verification systems are not yet ready for deployment at scale.The authors expect the dataset and analyses to facilitate future modeling improvements and new understanding of the task.Source: §7 · p. 9

8 Limitations

The collection relies on pooled annotations rather than exhaustive evidence labeling, leaving some uncertainty. Its claims come from the original SciFact test set, and candidate abstracts were identified with one retrieval system.

8 Limitations

  • Pooling substantially fewer systems than typical TREC competitions may leave greater uncertainty in the test collection.The authors analyze sensitivity to annotation depth and system count and find the collection remains useful for distinguishing substantive performance differences.Source: §8 · p. 9
  • All claims derive from the original SciFact test collection; the study does not add new claims.The authors note that highly-studied claims may merit separate treatment or annotation pipelines, but do not treat them differently here.Source: §8 · p. 9
  • A single retrieval system identifies candidate abstracts, although analysis indicates it finds the great majority of relevant abstracts.The authors suggest that future work could extend the dataset using a wider variety of information-retrieval approaches.Source: §8 · p. 9

A Dataset construction

The authors build SCIFACT-OPEN from a sampled scientific-abstract corpus, SciFact test claims, and pooled evidence annotations. They describe corpus and claim filtering, annotation procedures, retrieval depth, and the systems used to create the pool.

A.1 Corpus

  • The corpus samples 500K documents from roughly 6.5 million filtered S2ORC documents, including the 5K documents from SCIFACT-ORIG.Documents were filtered for Medicine or Biology field-of-study metadata and at least one inbound and one outbound citation.Source: §A.1 · p. 11

A.2 Claims and evidence

  • The test set retains 279 claims after removing 21 claims whose source-citation metadata was unavailable in S2ORC.Three biology-related professional annotators labeled pooled instances, with a second annotator checking labels when an instance might contain evidence.Source: §A.2 · p. 11

A.3 Retrieval

  • Most evidentiary abstracts ranked among the top 20 retrievals per claim, supporting annotation of the top-ranked predictions from 50 retrieved abstracts.Very few evidentiary abstracts had retrieval ranks worse than 40.Source: §A.3 · p. 11, Figure 7 · p. 12

A.4 Models

  • The annotation pool combined predictions from VERT5ERINI, PARAGRAPHJOINT, MULTIVERS, and MULTIVERS10 to diversify model precision and recall behavior.ARSJOINT was released later and was evaluated, but was not part of the original annotation pool.Source: §A.4 · p. 12
  • Models shared VERT5ERINI retrieval, while their rationale and label prediction architectures differed across T5-3B, RoBERTa, and Longformer systems.The authors also describe negative sampling in training, where additional claim-abstract pairs may be randomly selected or chosen as hard negatives.Source: §A.4 · p. 12

B.1 Unequal allocation of evidence

The analysis finds that pooled evidence is unevenly distributed across claims and that claim-evidence specificity can mismatch. It also reports examples of conflicting evidence in the collection.

B.1 Unequal allocation of evidence

  • 14 claims account for 50% of the evidence documents discovered through pooling.Pooling found no new evidence for most claims, while discovering substantial evidence for a small handful.Source: §B.1 · pp. 12–13
  • Claims with at least four newly pooled evidence documents mention short claims and well-studied entities, whose entities return four times as many PubMed documents on average as entities in claims with no new evidence.The table caption reports that all listed differences are significant at the 0.05 level.Source: §B.1 · p. 13, Table 7 · p. 13

B.2 Claim / evidence specificity mismatch

  • Specificity mismatch labels depend on direction: evidence supporting a claim-specific case is SUPPORTS, while evidence supporting a generalization of the claim is NEI.The corresponding rules assign NEI when evidence refutes a special case and REFUTES when it refutes a generalization.Source: §B.2 · p. 13
  • Specificity mismatches occur for both well-studied and less-studied claims, while more-general evidence occurs more often for less-studied claims.The authors suggest this may be because less-studied claims tend to be specific and cover narrower topics.Source: §B.2 · p. 13, Table 8 · p. 13

B.3 Conflicting evidence

  • The paper provides two claims from SCIFACT-OPEN as examples of conflicting evidence.Table 10 presents the examples.Source: §B.3 · p. 13

C Model performance

The performance discussion describes uncertainty estimation, performance differences by claim study level, and common model errors. Higher-recall models tend to do better on well-studied claims, while higher-precision models tend to do better on less-studied claims.

C Model performance

  • Performance uncertainty estimates are standard deviations over 1,000 bootstrap samples that resample claims with replacement.Evidence is weighted by how many times each claim is sampled in a bootstrap iteration.Source: C Model performance · pp. 13–14
  • Higher-recall models tend to perform better on well-studied claims, while higher-precision models perform better on less-studied claims.The authors relate the first pattern to evidence availability in the corpus for well-studied claims.Source: C Model performance · p. 14
  • Models rarely confuse SUPPORTS with REFUTES; more commonly, they label irrelevant abstracts as evidence or miss relevant abstracts.Source: C Model performance · p. 14

D Dataset reliability: Additional experiments

Additional analyses find that the pool-depth and system-count conclusions are consistent across F1 and average precision, and that recalibrating classification thresholds would not change them. Performance patterns also differ between well-studied and less-studied claims.

D Dataset reliability: Additional experiments

  • F1 and average precision yield the same qualitative conclusions, indicating that recalibrating classification thresholds for training-time negative sampling would not change the results.The average-precision analysis excludes ARSJOINT because it requires model confidence scores.Source: D Dataset reliability: Additional… · pp. 14–15, Figure 12 · p. 17, Figure 13 · p. 17
  • Table 10 presents examples of two claims with conflicting evidence.Source: Table 10 · p. 15
  • Higher-recall models generally perform better on well-studied claims, while high-precision models perform better on less-studied claims.ARSJOINT results are not comparable with the other systems because ARSJOINT was not used for data collection.Source: Table 11 · p. 15
  • In the SCIFACT-OPEN confusion matrices, the {NEI, NEI} cell is zero because the task retrieves documents, making agreement on unlabeled and unpredicted corpus documents uninformative.Source: Figure 9 · p. 16
  • Figures 10 and 11 show percentage changes in F1 as pool depth and system count vary.The accompanying analysis examines the effects of pool depth and system count on F1.Source: D Dataset reliability: Additional… · p. 14, Figure 10 · p. 17, Figure 11 · p. 17
Loading 2210.13777v1…