PaperlayerView exact revision on arXiv

Source-linked AI summary

ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts

Yuta Koreeda, Christopher D. Manning

arXiv:2110.01799v1cs.CLcs.AIcs.LG

TL;DR

Contract review is expensive and inaccessible to some users, motivating document-level natural language inference over contracts. The paper releases 607 annotated contracts and introduces Span NLI BERT, which significantly outperforms existing models while leaving rare labels and exception-based negations difficult.

  • Problem

    Contract review is time-consuming and costly, while prior contract datasets identify term types rather than what those terms state.

  • Method

    The paper formulates contract review as document-level NLI with evidence spans and introduces Span NLI BERT for multi-label span classification and long-document context segmentation.

  • Results

    607 annotated contracts were released, and Span NLI BERT significantly outperformed existing Transformer-based models on evidence identification and NLI.

  • Takeaways & Limitations

    Contract-specific linguistic characteristics, particularly negations by exceptions, make document-level contract NLI difficult even for Span NLI BERT.

  • Takeaways & Limitations

    Span NLI BERT still performs poorly on rare labels and is easily impacted by negations by exceptions.

Abstract

from arXiv · show

Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose "document-level natural language inference (NLI) for contracts", a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as "Some obligations of Agreement may survive termination.") and a contract, and it is asked to classify whether each hypothesis is "entailed by", "contradicting to" or "not mentioned by" (neutral to) the contract as well as identifying "evidence" for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (1) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (2) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.

1 Introduction

The paper frames contract review as document-level natural language inference, requiring judgments about hypotheses and supporting evidence across entire contracts. It introduces ContractNLI to address costly manual review and limitations of prior term-extraction datasets.

  • Motivation: Contract review is costly and time-consuming, and some smaller companies or individuals lack access to professional services.A cited study reports that Fortune 1000 companies maintain 20,000 to 40,000 active contracts.
  • Motivation: Prior contract datasets extract which terms appear but not what those terms state, limiting direct answers to policy-relevant questions.The paper gives employee-solicitation restrictions as an example of a question that requires interpreting a clause rather than merely locating it.
  • Task: ContractNLI classifies each hypothesis as entailed, contradicted, or not mentioned by a whole contract and identifies supporting evidence spans.This document-level formulation differs from NLI against a short passage and requires evidence for the judgment.
  • Contributions: The authors release 607 annotated contracts and introduce Span NLI BERT for joint NLI and evidence identification.The model uses multi-label span classification and more sophisticated context segmentation for long documents.
  • Contributions: Span NLI BERT significantly outperforms existing models, while contract-specific linguistic characteristics make the task challenging.The paper identifies these characteristics as an additional source of difficulty beyond the document-level task formulation.

2 ContractNLI Dataset

ContractNLI formalizes contract understanding as three-class document-level NLI with evidence identification, using annotated NDAs and fixed, fine-grained hypotheses. Its dataset contains long documents, imbalanced labels, and varied evidence-span structures that motivate specialized modeling.

  • 2.1 Task Formulation: The task assigns each hypothesis one of ENTAILMENT, CONTRADICTION, or NOTMENTIONED at the document level.Evidence identification is defined for entailment and contradiction cases.
  • 2.1 Task Formulation: Evidence identification uses multi-label classification over sentence or inline-list-item spans rather than token boundaries.Whole sentences provide comprehensible context for lawyers, while spans should remain concise, self-contained, and potentially discontinuous.
  • 2.1 Task Formulation: The dataset targets non-disclosure agreements and uses 17 fine-grained hypotheses fixed across contracts, including the test set.The single contract type supports more fine-grained hypotheses, while scaling to another contract type requires a similar dataset.
  • 2.2 Data Collection: 607 documents were collected, manually screened, and split into training, development, and testing data at a 70:10:20 ratio.Documents came from Internet searches and EDGAR, with plain-text extraction followed by manual correction and sentence segmentation.
  • 2.3 Data Statistics: Documents average 2,254.0 tokens and 77.8 candidate spans, and 86% exceed BERT’s 512-token context limit.These statistics establish the long-document setting addressed by the model architecture.
  • 2.3 Data Statistics: Most entailed or contradicted hypotheses have one or two evidence spans, but some require up to nine spans.The label distribution includes substantial ENTAILMENT and NOTMENTIONED proportions, while around half of hypotheses contain both ENTAILMENT and CONTRADICTION.

3 Span NLI BERT for ContractNLI

Span NLI BERT jointly identifies evidence and classifies document-level NLI by modeling predefined spans directly and using dynamic context segmentation for long contracts.

  • Model: Span NLI BERT replaces start/end-token prediction with multi-label binary classification over special [SPAN] tokens representing predefined spans.This separates evidence identification from span-boundary detection.
  • Context segmentation: Dynamic context segmentation uses overlapping contexts so each span appears intact with sufficient surrounding context in at least one context.Contexts are built from a maximum length l and minimum surrounding-token requirement n.
  • Prediction: The model concatenates contract and hypothesis tokens, then predicts span probabilities from [SPAN] tokens and NLI probabilities from [CLS].Evidence uses sigmoid activation, while NLI uses a softmax over ENTAILMENT, CONTRADICTION, and NOTMENTIONED.
  • Training: Evidence identification uses cross-entropy over binary span labels, including negative labels for NOTMENTIONED examples.NOTMENTIONED pairs have no evidence spans but are incorporated with all span labels set to zero.
  • Training: NLI loss ignores contexts without evidence spans, avoiding inconsistency between the model input and its training signal.The two losses are balanced by a hyperparameter λ.
  • Aggregation: Document-level evidence probabilities are averaged across contexts, while NLI probabilities are weighted by the summed span probabilities.The aggregation gives greater influence to contexts containing evidence spans.

4 Experiments

The experiments compare Span NLI BERT with several baselines and model configurations, finding stronger performance overall but weaker contradiction prediction and mixed benefits from legal-domain pretraining.

  • Baselines: Five baselines evaluate majority-label prediction, document TF-IDF classification, span similarity, span SVM classification, and SQuAD-style BERT evidence identification.The baselines differ in whether they address NLI, evidence identification, or both.
  • Metrics: Evidence identification is evaluated with mAP and P@R80, while NLI uses accuracy, contradiction F1, and entailment F1.P@R80 measures precision at a required recall of 0.8; NLI scores are macro averaged over labels.
  • Model comparison: DeBERTa v2xlarge fine-tuned on span identification obtains mAP .936, P@R80 .860, accuracy .892, F1 (C) .405, and F1 (E) .859.These are the reported values for that backbone and fine-tuning configuration.
  • Main results: Span NLI BERT significantly outperforms the baselines on both evidence identification and NLI, but contradiction performance is much worse than entailment performance.The authors attribute the contradiction gap to imbalanced label distribution.
  • Main results: SQuAD BERT’s evidence-identification mAP is no better than Span TF-IDF+SVM’s, underscoring the importance of explicitly incorporating span boundaries.The comparison removes arbitrary-boundary effects by scoring predefined spans.
  • Model comparison: Larger backbone models improve both evidence identification and NLI, while legal-corpus pretraining produces mixed results.Case-law pretraining does not help, NDA fine-tuning slightly helps BERTbase, and the benefit disappears for BERTlarge.
  • Hypothesis information: Using hypothesis surface tokens improves performance over replacing each hypothesis with a randomly initialized special token.The controlled experiment finds a consistent decrease after removing the surface tokens.
  • Model comparison: CUAD pretraining gives marginal NLI improvement and yields the best-performing model on ContractNLI.The statement concerns the model comparison reported in the experiments.

5 Discussion

Controlled experiments show that evidence identification and NLI depend on each other, while discontinuous spans and negation by exception create important challenges. Hypothesis wording helps, but models still have substantial difficulty with contradiction labels and scarce, imbalanced data.

  • Controlled Experiments: Hypothesis surface tokens improve both evidence identification and NLI, although the narrow differences suggest better use of hypothesis text is possible.Removing hypothesis surface tokens consistently decreases performance relative to using the original hypothesis tokens.
  • Controlled Experiments: Oracle evidence spans substantially improve NLI performance, notably contradiction F1, suggesting evidence identification limits NLI.The oracle model concatenates each hypothesis with ground-truth evidence spans before predicting entailment or contradiction.
  • Challenges of ContractNLI: ContractNLI is difficult because labels are imbalanced, training data are scarce, and the task is naturally multi-task.The paper also identifies multiple contract-specific linguistic characteristics that contribute to the challenge.
  • Challenges of ContractNLI: Discontinuous evidence spans occur in 28% of document-hypothesis pairs and 81% of documents containing at least one such hypothesis.Their overall evidence-identification mAP effect is not negative, but finding all spans is harder when the minimum context is small.
  • Challenges of ContractNLI: Span NLI BERT performs better with more surrounding tokens when discontinuous evidence spans cannot fit within one context window.The gap between discontinuous spans correlates positively with the number of spans read before finding all spans (ρ = 0.205, p = 0.015).
  • Challenges of ContractNLI: References to definitions do not make examples more difficult, despite requiring models to connect multiple spans for correct NLI.The authors report that examples with references are no more difficult than those without them.

6 Related Works

ContractNLI extends legal NLI beyond cases and statutes by addressing contracts, where hypotheses and documents have many-to-many relationships. It also differs from contract information-extraction datasets by requiring reasoning over what terms state.

  • Legal NLI: Case and statute NLI typically extracts relevant paragraphs and classifies entailment, whereas ContractNLI evaluates hypotheses against entire contracts.Prior legal NLI datasets focus on court decisions, relevant cases, or statute excerpts.
  • Legal NLI: Contracts differ from statutes and cases in vocabulary, style, and answer structure: a hypothesis may be entailed or contradicted by each contract.Case and statute NLI generally has one-to-one hypothesis-document relationships, while ContractNLI has many-to-many relationships.
  • Contract Information Extraction: Unlike contract information extraction, ContractNLI requires reasoning over what identified terms exactly state, not merely which term types appear.Prior datasets provide consistent legends or extract spans for particular contract questions without determining the statement’s relation to the contract.
  • Related Tasks: ContractNLI requires reasoning over a more diverse set of hypotheses than tasks classifying clauses as fair or unfair.Its input and output format resembles fairness annotation, but its hypothesis space is broader.

7 Conclusion

The paper introduces document-level NLI for contract review, releases a 607-contract dataset, and proposes Span NLI BERT. Its results show improved performance but persistent difficulty with rare labels and exception-based negation.

  • Contributions: 607 contracts form the annotated dataset for document-level NLI applied to contract review.The paper presents this as the first dataset using NLI for contracts and the largest corpus of annotated contracts.
  • Contributions: Span NLI BERT models evidence identification as multi-label classification over spans rather than predicting start and end tokens.This design is presented as a more natural solution for the task.
  • Results: Span NLI BERT performs significantly better than existing Transformer-based models.The reported improvement covers the paper’s proposed baseline relative to existing Transformer-based systems.
  • Limitations: Span NLI BERT still performs poorly on rare labels and is easily impacted by negation by exception.The conclusion states that substantial room for improvement remains despite the performance gain.
  • Future Work: Future work will study generalization across contract and hypothesis types, including how hypothesis phrasing affects performance.The authors identify better use of hypothesis text as potentially important for this goal.
  • Impact: The dataset and Span NLI BERT are intended as a starting point for addressing ContractNLI’s challenges.This is the paper’s stated forward-looking role for its released resources and baseline.

Ethical Consideration

The authors describe data-access, annotation-labor, energy-use, and legal-usage considerations surrounding the dataset and models.

  • Data and annotation: Contracts were collected from public EDGAR filings and publicly accessible documents obtained through Internet searches.The authors state that EDGAR documents may be redistributed and that academic use of the other documents falls within fair use.
  • Data and annotation: Annotators could participate voluntarily and withdraw at any time, received at least the US federal wage, and averaged $18.31 per hour excluding platform fees.
  • Computational cost: The authors ran fine-tuning 156 times to support validity and reproducibility on a modest-sized dataset.They state that this energy consumption could be justified by resources potentially saved through contract-review assistance.
  • Legal use: The authors addressed concerns that releasing annotations or models could constitute unauthorized legal practice by consulting an attorney and adding safeguards.They added a disclaimer advising users not to rely on the information as legal advice and to seek an attorney, and restricted crawler indexing of annotations.

A.1 Details on Data Collection

The dataset consists of screened non-disclosure agreements collected from Internet searches and EDGAR, with exclusions based on agreement type and document preprocessability.

  • Dataset scope: The dataset contains unilateral or bilateral NDAs and confidentiality agreements between two parties, while excluding employer-employee and nested agreements.These exclusions were intended to remove agreements considered substantially different from the rest of the NDA collection.
  • Internet collection: Internet searches used a non-disclosure agreement PDF query across seven domains from English-speaking countries.The authors note that different Google domains return different results.
  • Internet collection: 557 Internet-search PDFs were manually screened, and documents without embedded text or with multiple columns were removed as difficult to preprocess.
  • EDGAR collection: EDGAR archives from 1996 to 2020 yielded 12,851,835 filings, which were deserialized and filtered using exhibit metadata, file types, and content patterns.
  • EDGAR collection: Manual screening of 28,780 EDGAR NDA candidates produced 236 NDAs, all in HTML or plain-text format.

A.1.1 Details on Contract Annotation

The authors developed and reviewed fixed hypotheses, used span-and-label annotation with multiple annotators, and tuned and evaluated several baseline implementations.

  • Hypothesis design: The authors developed 17 hypotheses by comparing NDAs and had paralegals review them.The complete hypothesis list appears in Table 10.
  • Annotation procedure: Because the hypotheses were fixed, each received an example-oriented guideline, and the interface let annotators select evidence spans before assigning an NLI label.
  • Annotation procedure: Two consistently performing Mechanical Turk workers redundantly annotated each document for coverage, after which the primary annotator merged and reviewed their spans.For much of the test split, the primary annotator also reviewed the full contracts to improve coverage.
  • Model evaluation: The SQuAD BERT baseline created one example per span and split each example into fixed-window contexts, while Span NLI BERT was evaluated across ten hyperparameter settings.
  • Model evaluation: The experiments averaged three models with the best development scores, using macro-average NLI accuracy as the selection criterion.
  • Model evaluation: Weighted NLI probabilities achieved average macro-average accuracies of 0.782 for BERTbase and 0.803 for BERTlarge, versus 0.458 and 0.454 unweighted.The authors conclude that incorporating the weighted probabilities is critical.
  • Model evaluation: Pilot experiments found that NLI overfit faster than span detection, motivating a search over loss weights λ < 1.The authors hypothesize that contexts from one hypothesis-document pair provide less diverse NLI supervision than span labels.
Loading 2110.01799v1…