Source-linked AI summary
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
Wenpeng Yin, Dragomir Radev, Caiming Xiong
TL;DR
Most NLI research focuses on sentence-level inference, although some downstream tasks require reasoning over documents. DOCNLI reformats NLP tasks into document-level NLI, with document-granularity premises and hypotheses ranging from sentences to long passages. Its pretrained model performs promisingly on sentence-level benchmarks and generalizes to out-of-domain document-inference tasks.
Problem
Most NLI datasets and studies focus on sentence-level inference, while some downstream NLP tasks require reasoning over both document-scale premises and hypotheses.
Source: §1 · p. 1Method
DOCNLI reformats mainstream NLP tasks, including question answering and document summarization, and integrates existing NLI into a document-level dataset.
Source: §1 · p. 2Results
A RoBERTa model pretrained on DOCNLI performs promisingly on sentence-level benchmarks and generalizes to out-of-domain tasks requiring document-level inference.
Source: §1 · p. 2Takeaways & Limitations
DOCNLI covers multiple genres and premise and hypothesis lengths, and the authors expect it to help with document-level QA, summarization, and fact-checking.
Source: §5 · p. 9Minor grammar issues inevitably remain in the data.
Source: §3.3 · p. 6
Abstract
from arXiv · showhide
Natural language inference (NLI) is formulated as a unified framework for solving various NLP problems such as relation extraction, question answering, summarization, etc. It has been studied intensively in the past few years thanks to the availability of large-scale labeled datasets. However, most existing studies focus on merely sentence-level inference, which limits the scope of NLI's application in downstream NLP problems. This work presents DocNLI -- a newly-constructed large-scale dataset for document-level NLI. DocNLI is transformed from a broad range of NLP problems and covers multiple genres of text. The premises always stay in the document granularity, whereas the hypotheses vary in length from single sentences to passages with hundreds of words. Additionally, DocNLI has pretty limited artifacts which unfortunately widely exist in some popular sentence-level NLI datasets. Our experiments demonstrate that, even without fine-tuning, a model pretrained on DocNLI shows promising performance on popular sentence-level benchmarks, and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. Task-specific fine-tuning can bring further improvements. Data, code, and pretrained models can be found at https://github.com/salesforce/DocNLI.
1 Introduction
Most NLI datasets focus on sentence-level inference, while some downstream tasks require reasoning over document-scale premises and hypotheses. DOCNLI reformats NLP tasks into document-level NLI and reports promising transfer to sentence-level benchmarks and document-inference tasks.
1 Introduction
- Some downstream NLP tasks require NLI reasoning beyond sentence granularity for both premises and hypotheses.Sentence-level inference may be unsuitable for tasks such as verifying document summaries or claims against Wikipedia articles.Source: §1 · pp. 1–2
- DOCNLI reformats mainstream NLP tasks, including question answering and document summarization, and integrates existing NLI with longer premises.The dataset is presented as highly related to end NLP tasks.Source: §1 · p. 2
- Premises contain more than one sentence, while hypotheses range from a single sentence to documents hundreds of words long.The setting targets truth-value inference regardless of the text's length.Source: §1 · p. 2
- DOCNLI has limited artifacts, and a “hypothesis-only” baseline has difficulty discovering some spurious correlations.The authors describe an approach to disconnect potential artifacts from the NLI task itself.Source: §1 · p. 2
- A RoBERTa system pretrained on DOCNLI shows promising performance on sentence-level benchmarks and generalizes to out-of-domain tasks requiring document-level inference.Task-specific fine-tuning can further improve performance and reach state of the art for some end tasks.Source: §1 · p. 2
2 Related Work
Prior NLI datasets include end-task-driven resources and work using longer premises, but many retain single-sentence hypotheses or narrow task scope. DOCNLI is presented as the first dataset with longer hypotheses closely tied to end tasks.
2 Related Work
- MCTest and SciTail derive from question-answering tasks, but MCTest uses paragraph premises whereas SciTail uses retrieved single-sentence premises.Casting an end NLP task as NLI can allow a good NLI recognizer to be used for that task.Source: §2 · p. 2
- Prior document-level NLI work had limits: MCTest was small, FactCC was tied to summarization and single-sentence hypotheses, and ANLI used single-sentence hypotheses.ANLI also had relatively limited premise lengths.Source: §2 · p. 2
- DOCNLI is presented as the first dataset with hypotheses longer than single sentences while staying closely connected to end NLP tasks.Earlier work included document-level premises, but the hypotheses in FactCC and ANLI remained single sentences.Source: §2 · p. 2
3 Data Creation
DOCNLI reformats existing NLI, question-answering, and summarization resources into document-level inference data with varied text lengths and genres. Its construction also addresses summary-based label artifacts and includes human checks of generated summaries.
3 Data Creation
- DOCNLI combines ANLI, SQuAD, and three summarization benchmarks, with document premises and hypotheses spanning single sentences to longer passages.The preferred dataset also calls for diverse domains and no severe artifacts.Source: §3 · pp. 2–3, §3.1 · p. 3
3.1 Data Preprocessing
- Summarization pairs use human-written reference summaries as entail examples and word, entity, or sentence replacements to create fake summaries.DOCNLI also uses existing QA2D data for SQuAD and merges neutral and contradict into not entail for ANLI.Source: §3.1 · pp. 3–4
3.2 Mitigating Artifacts in DOCNLI
- A hypothesis-only RoBERTa baseline achieves non-trivial performance on raw-DOCNLI, so the authors add pairs to separate real-versus-fake from entail-versus-not-entail.Added examples make fake summaries entailment hypotheses when supported by their premises and real summaries not-entail hypotheses when they mismatch the premise.Source: §3.2 · p. 5
- Training data are roughly balanced, while about 12% of dev and test examples are entailment; most premises are under 150 words, but many span 150–900 words.The authors report that most hypotheses have fewer than 150 words.Source: §3.2 · p. 5
3.3 Human Verification
- In 200 randomly checked fake summaries, none was true of the source document, though minor grammar issues remained.The authors attribute the absence of true examples mainly to replacing relatively much of the original summaries.Source: §3.3 · p. 6
- Entity substitutions can produce awkward grammar, including date swaps that make a generated summary grammatically incorrect.The authors describe minor grammar issues as inevitable.Source: §3.3 · p. 6
4 Experiments
The experiments assess DOCNLI’s challenge level, transfer to out-of-domain document-level tasks, and transfer to sentence-level NLI. Results show stronger performance than comparison datasets on the evaluated end tasks and promising sentence-level transfer, with task-specific fine-tuning improving SciTail performance.
4 Experiments
- The experiments test DOCNLI’s challenge level, out-of-domain generalization, and transfer to sentence-level NLI; sentence-level transformer systems face a 512-token input limit.The main experiments also report Longformer-base, which can handle up to 4096 tokens; only that version is reported because of memory constraints.Source: §4 · p. 6, §4.1 · p. 6
4.1 The DOCNLI task is challenging
- 61.52 test F1 for RoBERTa-large using premises and hypotheses exceeds the hypothesis-only system’s 22.02; Longformer performs lower despite covering more tokens.The authors suggest limited computing resources may have prevented adequate exploration of better Longformer settings.Source: §4.1 · p. 7
- Longformer’s performance generally improves with more tokens, but performance for premise-hypothesis pairs longer than 450 does not change clearly, possibly because of model truncation.The task becomes increasingly challenging as hypotheses grow longer.Source: §4.1 · p. 7, Figure 4 · p. 7
- Overall, DOCNLI is described as a very challenging task requiring stronger representation-learning capability.The authors frame this conclusion after reporting the system comparisons and length-based results.Source: §4.1 · p. 7
4.2 Applying DOCNLI to end NLP tasks
- DOCNLI generalizes better than ANLI and MNLI on FEVER-binary and MCTest; both DOCNLI settings surpass MCTest’s prior state of the art by large margins.The tested settings use DOCNLI pretraining with or without task-specific fine-tuning.Source: §4.2 · p. 8, Table 6 · p. 8
4.3 Applying DOCNLI to sentence-level NLI
- 78.17 SciTail accuracy exceeds that of some fully supervised task-specific models, while DOCNLI pretraining yields performance comparable to BERT, Longformer, and RoBERTa on binary-MNLI.Task-specific fine-tuning further improves performance and brings SciTail close to its state of the art.Source: §4.3 · pp. 8–9, Table 7 · p. 8
5 Summary
The paper releases DOCNLI, a large-scale document-level NLI dataset spanning multiple genres and premise and hypothesis lengths. Experiments show that models trained on it generalize to downstream NLP tasks and popular sentence-level NLI tasks.
5 Summary
- DOCNLI is a released large-scale document-level NLI dataset covering multiple genres and a range of premise and hypothesis lengths.The authors expect it to help with document-level reasoning tasks such as question answering, summarization, and fact-checking; experiments show generalization to downstream and popular sentence-level NLI tasks.Source: §5 · p. 9