PaperlayerView exact revision on arXiv

Source-linked AI summary

WiCE: Real-World Entailment for Claims in Wikipedia

Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, Greg Durrett

arXiv:2303.01432v2cs.CL

TL;DR

Existing entailment datasets offer limited coverage of real-world verification needs, even as entailment models are used for tasks such as question answering and summary evaluation. WiCE pairs Wikipedia claims with fine-grained evidence annotations and claim decomposition; its evaluation finds that claims pose challenges beyond existing NLI datasets, while systems remain below human performance and retrieval is poor.

  • Problem

    Existing entailment datasets offer limited coverage of real-world, document-level verification needs, despite entailment systems’ use in question answering, summary evaluation, and dialog.

    Source: §1 · p. 1

  • Method

    WiCE pairs real-world Wikipedia claims with cited-article evidence, annotating entailment, supporting sentences, and unsupported claim tokens, and uses GPT-3.5 Claim-Split to decompose claims into subclaims.

    Source: §1 · p. 2

  • Results

    WiCE claims pose verification and retrieval challenges beyond current NLI datasets; evaluated systems remain below human performance, and retrieval performance is poor.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Decomposing complex claims into subclaims can be a valuable preprocessing step for annotation and entailment prediction.

    Source: §7 · p. 9

    Because nearly all recent language models are pretrained on Wikipedia articles, WiCE cannot evaluate truly zero-shot performance.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models' performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.

1 Introduction

WiCE targets the gap between conventional NLI datasets and real-world claim verification with fine-grained claim and evidence annotations. Claim-Split decomposes claims into subclaims, while evaluated systems remain below human performance on WiCE.

1 Introduction

  • Existing entailment datasets often use short premises, lack ecologically valid negative examples, and do not indicate which claim parts are supported.The introduction notes that synthetic negatives can encourage spurious correlations and that document-level systems often aggregate local scores or use retrieval.Source: §1 · p. 1
  • WICE annotates Wikipedia claims with entailment labels, supporting sentences from cited articles, and unsupported claim tokens, addressing real-world verification and retrieval challenges.These annotations cover claims that may be only partially supported.Source: §1 · pp. 1–2
  • Claim-Split uses few-shot GPT-3.5 prompting to decompose hypotheses into subclaims, simplifying annotation and prediction while improving off-the-shelf entailment classification on long claims.The method avoids relying on annotated data.Source: §1 · p. 2
  • Chunk-level processing and retrieval are strong starting points, but current systems remain below human performance on WiCE and retrieval performance is poor.The evaluation includes short-paragraph entailment models adapted to make document-level judgments.Source: §1 · p. 2

2 The WICE Dataset

WiCE uses naturally occurring Wikipedia claims and cited documents in context, decomposing complex claims into subclaims for fine-grained entailment judgments. It also provides evidence-sentence and unsupported-token annotations.

2.2 Tasks in WICE

  • WICE supports three subclaim-level tasks: three-way entailment classification, retrieval of supporting evidence sentences, and detection of unsupported tokens.Claim-level labels are projected from subclaim annotations using rules described in the dataset collection process.Source: §2.2 · p. 3

2.3 Dataset Collection

  • The dataset starts with SIDE Wikipedia claims and re-retrieves their cited web articles from Common Crawl, segmenting those articles into evidence sentences.Re-retrieval allows the evidence set to include multiple cited web articles for a Wikipedia claim.Source: §2.3 · p. 3, Claim-Split · p. 3

Claim-Split

  • After filtering, WICE retains 16.3% of claims; filtering removes claims with one or more than six subclaims and examples whose subclaims are all classified as entailed by a RoBERTa-Large model.For the development set, the subclaim-count filter removes 19.1% of examples; the model filter targets trivial entailments while avoiding an adversarially difficult dataset.Source: Claim-Split · p. 3

Deriving claim labels from subclaim labels

  • Claim-level labels are derived by assigning SUPPORTED or NOT-SUPPORTED when all subclaims share that label, and PARTIALLY-SUPPORTED otherwise.Claim-level supporting sentences are formed from the union of subclaim-level supporting sentences.Source: Deriving claim labels from subclaim… · p. 4

2.4 Dataset Statistics

  • Claims average 3.0 subclaims, and WICE includes approximately 5.9K subclaim-level (or 2K claim-level) entailment examples; supported or partially supported examples average 1.9 (or 3.1) evidence sentences.Roughly 56% of subclaims are SUPPORTED, compared with 33% of claims; 25.2% of tokens in PARTIALLY-SUPPORTED subclaims were marked unsupported.Source: §2.4 · p. 4, Table 1 · p. 4, Table 2 · p. 4

2.1 Claim-Split using LLMs

  • Claim-Split decomposed claims into an average of 3.0 subclaims, and the final WICE dataset contains approximately 5.9K subclaim-level (or 2K claim-level) entailment-classification examples.The authors manually fixed errors detected in the annotation to provide a high-quality evaluation dataset.Source: §2.1 · p. 3, §2.4 · p. 4

2.5 Analysis of Phenomena

  • WICE claims often pose difficult entailment problems requiring inference at the subclaim level, whereas VitaminC contains relatively few inference cases and more calculation.The analysis estimates verification-problem distributions from annotations of 50 claims per dataset, including 127 WICE subclaims.Source: Table 3 · p. 4, §2.5 · p. 5

3 Experiments on WICE

Experiments test off-the-shelf and WICE-fine-tuned entailment models, evidence retrieval, and retrieve-then-predict. Results show that performance improves with fine-tuning, context, and oracle evidence, while important gaps remain.

Stretching NLI for document-level entailment

  • MAX stretching partitions evidence into sentences or chunks of at most 256 tokens, scores each claim-partition pair, and takes the highest local entailment score.Applying stretching to large models such as GPT-3.5 and GPT-4 is expensive because it requires a model call for every document-chunk and subclaim pair.Source: Stretching NLI for document-level… · p. 5
  • Chunk-level T5-3B fine-tuned on ANLI then WICE performs best at both granularities, but fine-tuned models remain substantially below human performance.The authors report that WICE fine-tuning improves over off-the-shelf results; the cited results do not give a numeric human comparison for the fine-tuned model.Source: Table 5 · p. 6, Stretching NLI for document-level… · p. 6
  • Chunk-level MAX stretching outperforms sentence-level scoring; off-the-shelf T5-3B trained on ANLI is best, at 64.3 vs 83.3 claim-level F1 for humans.The performance gap indicates that WICE’s realistic claims and document-level setting challenge models trained on previous NLI datasets.Source: Stretching NLI for document-level… · p. 6

3.3 Entailment Classification using Retrieval

  • In retrieve-then-predict, retrieval without evidence context performs poorly, while context and oracle evidence improve entailment classification.The authors attribute a possible performance drop in ANLI+WICE to a mismatch between context-free ANLI pretraining and WICE fine-tuning with evidence context.Source: Table 7 · p. 7, §3.3 · p. 7

3.2 Evidence Retrieval on WICE

  • T5-3B fine-tuned on WICE with evidence context achieves the best retrieval performance, and including context improves retrieval overall.Including context did not improve retrieval for claims, which the authors suggest may be because the evidence already relates to the claim’s entity or event.Source: §3.2 · pp. 6–7, Table 6 · p. 6

3.4 Entailment Classification by GPT

  • With oracle retrieval, GPT-4 outperforms the best fine-tuned subclaim-level model, but few-shot claim-level classification remains challenging.The GPT evaluation uses the first 100 claims/subclaims because GPT-4 evaluation is costly.Source: Table 8 · p. 7, §3.4 · p. 7

4 Is Claim-Split useful for entailment classification?

Claim-Split tests whether decomposing claims into simpler subclaims eases entailment classification; AUROC results show improvements over standard inference for most model–dataset pairs.

Setup

  • The method predicts entailment scores for each subclaim and combines them using a harmonic mean, which was less sensitive to errors than minimum aggregation.Human evaluation found Claim-Split made mistakes less than 10% of the time.Source: §4 · p. 8

Test Data and Models

  • Evaluation covered WiCE, VitaminC, PAWS, and FRANK-XSum, using T5 models fine-tuned on ANLI or WiCE.VitaminC and PAWS evaluations used their 500 longest claims; FRANK was restricted to the XSum subset’s one-sentence claims, and both inference configurations used the same trained models.Source: §4 · p. 8

Results

  • Claim-Split outperformed standard inference for most model–dataset pairs, with statistically significant gains for T5-Large on three datasets.The comparison uses AUROC; the table caption identifies the standard and Claim-Split configurations and the paired-bootstrap significance test.Source: §4 · pp. 8–9, Table 9 · p. 8

5 Discussion: Outstanding Challenges

The discussion identifies evidence retrieval, unsupported-token localization, and contextualization as unresolved challenges for explainable document-level entailment.

Explainable entailment

  • An explainable document-level entailment system must localize non-factuality within claims, but surfacing the right evidence remains difficult.Subclaims and token-level annotations make localization possible, while evidence selection remains a hard problem.Source: §5 · p. 9

Unsupported token detection

  • Unsupported-token localization is an important, difficult problem that warrants study with decomposition techniques like Claim-Split (Kamoi et al., 2023).The authors did not conduct experiments on this problem.Source: §5 · p. 9

Better understanding of contextualization

  • Contextualization remains unsolved: some claims resist succinct decontextualization, and the authors propose subclaims-in-context as a unit for further study (Choi et al., 2021).A fresco claim may require substantial identifying information, reducing some benefits of splitting; the proposal needs experimental substantiation.Source: §5 · p. 9

6 Related Work

Related work spans entailment and fact-verification datasets, claim decomposition, and factuality evaluation in summarization. The section distinguishes WiCE’s approach from prior work by the scope of its data and the granularity and supervision of its methods.

Short-paragraph entailment

  • Most NLI datasets use single-sentence or short-paragraph pairs and involve less multi-step reasoning; ContractNLI is contract-only, while DocNLI uses synthetic negatives (Koreeda and Manning, 2021; Yin et al., 2021).The cited dataset examples include word-replacement negatives in DocNLI.Source: §6 · p. 9

Fact-verification datasets

  • Fact-verification datasets often rarely require multiple evidence sentences or skew toward quantity claims, while Wikipedia-citation attribution has focused on finding a better supporting document (Thorne et al., 2018; Schuster et al., 2021; Petroni et al., 2023).The passage associates FEVER with the first limitation and VitaminC with the second.Source: §6 · p. 9

Hypothesis Decomposition

  • Our proposition decomposition resembles prior proposition-splitting methods (Stanovsky et al., 2018; Ernst et al., 2021; Chen et al., 2022, 2023), but needs no supervised judgments and is not limited to token extraction.Pyramid-based methods target content coverage rather than reliability; purely extractive methods are unsuitable for off-the-shelf entailment models compared with Claim-Split (Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al., 2023).Source: §6 · p. 9

Factuality in summarization

7 Conclusion

WICE is a new NLI dataset constructed from Wikipedia by comparing its sentences with cited evidence documents. The paper identifies real-world entailment phenomena distinct from prior NLI datasets and finds that claim decomposition can aid annotation and entailment prediction.

7 Conclusion

  • WICE is a new NLI dataset built by comparing Wikipedia sentences with cited evidence documents.Source: §7 · p. 9
  • The dataset reveals real-world entailment phenomena distinct from those in prior NLI datasets.Source: §7 · p. 9
  • Decomposing complex claims into subclaims can benefit both annotation and entailment prediction.Source: §7 · p. 9

Limitations

WiCE is limited by its Wikipedia-based scope and prior model exposure to Wikipedia, and the experiments leave open questions about larger language models and the role of claim context. Its results may not transfer to more diverse claims or open-domain verification.

Scope and Diversity of the Dataset

  • WICE covers English Wikipedia claims and evidence from cited websites, not diverse real-world claims such as political claims or social media posts.The authors note that claims outside Wikipedia may differ in style and content.Source: Limitations · pp. 9–10
  • Because recent language models are generally pretrained on Wikipedia, WICE cannot measure truly zero-shot performance, motivating datasets built from brand-new texts.The authors note that Wikipedia pretraining does not establish whether a fact is supported by a particular evidence document; models pretrained on Wikipedia still perform unevenly on WICE.Source: Limitations · p. 10

Baseline Models

  • Experiments mainly use T5-3B, and GPT-3.5 and GPT-4 are evaluated only with oracle retrieval, leaving full-pipeline LLM performance unevaluated.The authors propose WICE as a realistic testbed for evaluating LLMs on long documents.Source: Limitations · p. 10

Context for Claims

  • Evidence-sentence context helps supporting-sentence retrieval, but experiments do not show that claim context improves entailment evaluation on WICE.The authors hypothesize that cited evidence documents already concern the same entities as the claims, and suggest exploring open-domain settings where document mismatches may make context more important.Source: Limitations · p. 10

A Human Evaluation of Claim-Split

A human evaluation and manual error analysis found both prompt-addressable omissions and harder Claim-Split failures; the authors manually corrected errors in the WICE development and test sets.

A Human Evaluation of Claim-Split

  • Each HIT asked annotators to judge whether decomposed sentences were correct and complete, with free-text explanations collected but excluded from the analysis.Each HIT covered four claims and their decompositions.Source: A Human Evaluation of Claim-Split · p. 14
  • 30% of observed mistakes came from dropped first or intermediate clauses, while 6.7% involved removed parentheses; the 30-error analysis left 33.0% uncategorized.The authors expected prompt examples to help address these omission errors.Source: A Human Evaluation of Claim-Split · p. 14, Table 10 · p. 14
  • Some errors were harder to prevent: Claim-Split sometimes omitted one of multiple facts that should be generated from a single span, such as Howard living with his wife.The decomposition retained that his wife was a doctor but omitted that he lived with her.Source: A Human Evaluation of Claim-Split · pp. 14–15
  • Over-splitting could erase temporal relations, as when decomposing the Shine Limited example lost the fact that it was set up after Elisabeth Murdoch quit broadcasting.The passage identifies a correct decomposition that keeps the latter facts together.Source: A Human Evaluation of Claim-Split · p. 15
  • The authors manually fixed Claim-Split mistakes in the WICE development and test sets.Source: A Human Evaluation of Claim-Split · p. 15

B Additional Data Collection Details

This appendix details how WiCE’s source data were retrieved and filtered, how subclaims were annotated, and how worker judgments were aggregated. It also reports agreement and development-set filtering statistics.

Base Dataset

  • WiCE reuses SIDE’s Wikipedia article-claim pairs (Petroni et al., 2023), but re-retrieves citations and extracts text from August 2019 Wikipedia and Common Crawl pages.Noisy HTML artifacts were retained, and claims were limited to one or two citations at the sentence end.Source: B Additional Data Collection Details · p. 15

Additional Filtering

  • For NLI filtering, DocNLI-finetuned RoBERTa-Large scores subclaims against evidence chunks under 256 tokens; any entailing chunk makes a subclaim entailed.This filtering removes claims whose subclaims are all predicted as entailed.Source: B Additional Data Collection Details · p. 15

Task Interface

  • Each annotation task presents evidence split into numbered sentences, the claim and context, and its 2–6 Claim-Split subclaims for annotation.A HIT took a median of about five minutes at $9/hour; the interface was based on FALTE.Source: B Additional Data Collection Details · p. 15

Filtering Steps

  • Development-set filtering occurred during Common Crawl retrieval, base-data cleaning, and NLI filtering; Table 11 reports 4,545 initial items and 739 final items across the process.The table notes that the development set contains 349 claims and that not all listed data were annotated.Source: B Additional Data Collection Details · p. 16, Table 11 · p. 16

Aggregation of subclaim level labels from different workers

  • Majority votes determine entailment labels; unresolved disagreements remove the subclaim and claim, filtering out 12.5% of development-set claims.For development and test cases with a split between PARTIALLY-SUPPORTED and NOT-SUPPORTED, PARTIALLY-SUPPORTED is selected; other no-majority cases are removed.Source: B Additional Data Collection Details · p. 16
  • Workers selected identical supporting-sentence sets in 56.1% of SUPPORTED cases and 34.4% of PARTIALLY-SUPPORTED cases, indicating high inter-annotator agreement.Source: B Additional Data Collection Details · p. 16
  • Unsupported-token labels use a union of PARTIALLY-SUPPORTED workers’ tokens, with removal if any annotator differs by more than three tokens.The development-set table reports an average of 3.3 non-supported tokens per subclaim, or 25.2% of tokens.Source: B Additional Data Collection Details · p. 16, Table 12 · p. 17

C Additional Dataset Statistics and Analysis … F Effect of Training Data Size on Entailment Classification Performance

The appendices report Claim-Split improvements and suggest that WICE fine-tuning performance may be saturating, alongside further dataset analyses and implementation details.

C Additional Dataset Statistics and Analysis

  • WICE has competitively low claim-evidence word overlap, and entailment labels have similar overlap, especially at the subclaim level.The authors suggest this indicates WICE does not suffer from spurious biases. Supported and partially supported subclaims average 1.9 supporting sentences; supported claims average 3.4 versus 2.9 for partially supported claims. Partially supported subclaims average 3.3 non-supported tokens (25.2%); the training set has 374 such subclaims averaging 12.8 tokens.Source: C Additional Dataset Statistics and… · p. 17
  • WICE subclaims are characterized by semantic roles including events, properties, location, time, reason, manner, and evidentials, drawing on Davidsonian event semantics (Truswell, 2019).Claims may receive multiple labels, and subclaims may contain overlapping subsets; the appendix also gives examples of entailment categories, including FEVER’s simple hypernymy example.Source: C Additional Dataset Statistics and… · p. 17, Table 14 · p. 18, Table 13 · p. 18

D Claim-Split / D.1 Claim-Split Prompts

  • Claim-Split prompts ask GPT-3.5 to segment claims into individual facts; evaluation prompts instead request decontextualized sentences with retained information and minimal wording changes.The appendix provides prompt templates and says experiments on VitaminC, PAWS, and FRANK used three or four examples for entailment-model evaluation.Source: §D.1 · p. 18

D.2 Claim-Split Aggregation Performance

  • Claim-Split improves entailment classification in almost all models and metrics, suggesting it reduces claim complexity.Table 15 reports claim-level performance from aggregating scores for generated subclaims and marks paired-bootstrap significance with an asterisk.Source: §D.2 · p. 19, Table 15 · p. 19

E Implementation Details / E.1 Baseline Models

  • The implementation uses PyTorch and Hugging Face Transformers, and draws baseline models from the Hugging Face Hub when available.T5 models were fine-tuned on VitaminC plus MNLI, ANLI, or DocNLI; BM25 evaluation used rank_bm25.Source: E Implementation Details · p. 19, §E.1 · p. 19

E.2 Training Data of WICE

  • For WICE training, evidence partitions without any supporting sentence are NOT-SUPPORTED; partitions containing all support for a SUPPORTED claim are SUPPORTED, and other cases are PARTIALLY-SUPPORTED.Documents are partitioned into individual sentences or concatenations of at most 256 tokens, with labels derived from WICE’s supporting-sentence annotations.Source: §E.2 · p. 19

E.3 Fine-tuning T5 Models

  • T5 models generate single-character entailment labels from normalized target-label probabilities and were fine-tuned for 25K steps with learning rate 10^-4 and batch size 32.Training inputs were capped at 512 tokens, while inference used all tokens; the authors note this limit may help explain DocNLI’s lower performance than ANLI because many DocNLI premises exceed 512 tokens.Source: §E.3 · pp. 19–20

F Effect of Training Data Size on Entailment Classification Performance

  • WICE fine-tuning performance appears to saturate with the tested procedure and models, unless substantially more training data is needed.The passage presents these as alternative interpretations of the T5-3B results across different training-set sizes.Source: F Effect of Training Data Size on… · p. 20

G Oracle Retrieval Dataset … I.5 Dataset Distribution

The supplementary sections define an oracle-retrieval setup to isolate entailment classification and document WICE’s motivation, construction, preprocessing, and distribution. They describe its fine-grained annotations and the procedures used to balance oracle chunks and collect dataset labels.

G Oracle Retrieval Dataset

  • The oracle-retrieval dataset supplies gold supporting sentences to NLI models to simulate ideal retrieval and isolate entailment classification.Oracle chunks may reflect different annotators’ sufficient evidence sets; chunks are padded with random evidence to at most 256 tokens, and three chunks are sampled per claim or subclaim to equalize chunk counts. Non-supported cases use MAX-setting chunks.Source: G Oracle Retrieval Dataset · p. 20

H Entailment Classification by GPT

  • GPT-3.5 and GPT-4 entailment classification used the first 100 oracle-dataset claims or subclaims and prompts requesting explanations and XML-formatted labels.The prompt included three WICE development-set examples and offered supported, partially_supported, and not_supported as conclusions.Source: H Entailment Classification by GPT · p. 21

I Datasheet for WICE

  • The paper provides a WICE datasheet; creator and funding information was to be added in the camera-ready version.The datasheet is framed as a dataset documentation resource.Source: I Datasheet for WICE · p. 21, §I.1 · p. 21

For what purpose was the dataset created?

  • WICE addresses limitations of existing NLI datasets: short claims and evidence, often artificial negative examples, and under-studied fine-grained labels.The dataset was created to support real-world attribution and factuality-consistency evaluation.Source: For what purpose was the dataset… · p. 21

I.2 Dataset Composition

  • Each WICE instance groups subclaims derived from a Wikipedia claim with cited-website evidence; annotations include entailment labels, supporting sentences, and unsupported tokens.Train, development, and test splits contain 1,260, 349, and 358 claims, respectively, decomposed into 3,470, 949, and 958 subclaims.Source: §I.2 · p. 21

I.3 Data Collection Process

  • Claims came from Wikipedia and evidence from the August 2019 Common Crawl; five Mechanical Turk workers annotated each instance for $0.75 per claim.The claims and evidence were collected from August 2019 sources, and worker annotation took place in November and December 2022.Source: §I.3 · pp. 21–22, Over what timeframe was the data… · p. 22

I.4 Data Preprocessing

  • Preprocessing extracted article text from cited-page HTML and used Claim-Split to decompose Wikipedia claim sentences into subclaims.Beautiful Soup 4 extracted sentences, and text-davinci-002 via the OpenAI API was used for claim decomposition in November 2022.Source: §I.4 · p. 22

I.5 Dataset Distribution

  • WICE is available in a GitHub repository and will be maintained by the paper’s authors.Source: §I.5 · p. 22
Loading 2303.01432v2…