Source-linked AI summary
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Dan Hendrycks, Collin Burns, Anya Chen, Spencer Ball
TL;DR
Specialized legal NLP lacks large expert-labeled datasets, although contract review is costly and requires identifying sparse, salient clauses. The paper introduces CUAD, a 510-contract benchmark with over 13,000 annotations across 41 categories, and evaluates pretrained Transformers on clause extraction. Performance is promising but nascent: it varies with model design and improves substantially with more labeled data, leaving substantial room for improvement.
Problem
Specialized domains lack large labeled datasets because expert annotation is costly, while contract review imposes substantial time and financial burdens.
Method
The paper constructs CUAD from expert-annotated contracts and evaluates pretrained Transformer models on extracting clauses across 41 label categories.
Results
Transformer performance is nascent and strongly influenced by model design and training-data size; DeBERTa-xlarge achieves 44.0% Precision at 80% Recall.
Takeaways & Limitations
CUAD provides a challenging expert-annotated benchmark for measuring contract-review progress and language-model generalization to specialized domains.
Takeaways & Limitations
Performance remains substantially below ceiling in some label categories, with category-level AUPR ranging from nearly 100% to around 20%.
Abstract
from arXiv · showhide
Many specialized domains remain untouched by deep learning, as large labeled datasets require expensive expert annotators. We address this bottleneck within the legal domain by introducing the Contract Understanding Atticus Dataset (CUAD), a new dataset for legal contract review. CUAD was created with dozens of legal experts from The Atticus Project and consists of over 13,000 annotations. The task is to highlight salient portions of a contract that are important for a human to review. We find that Transformer models have nascent performance, but that this performance is strongly influenced by model design and training dataset size. Despite these promising results, there is still substantial room for improvement. As one of the only large, specialized NLP benchmarks annotated by experts, CUAD can serve as a challenging research benchmark for the broader NLP community.
1 Introduction
Contract review is costly, expert-intensive, and difficult to automate because important clauses are sparse within very long documents. CUAD addresses this gap with expert annotations and provides evidence that Transformer performance depends strongly on model design and training-data scale.
- Motivation: Specialized document analysis remains underdeveloped because expert annotations are expensive and models’ ability to transfer to specialized domains is unresolved.Specialized datasets require thousands of annotations from trained experts who are costly and time-constrained.
- Motivation: Contract review consumes substantial legal time and money, with large U.S. law firms typically billing $500-$900 per hour for lawyers.Many law firms spend approximately 50% of their time reviewing contracts.
- Motivation: High review costs leave many small companies and individuals signing contracts unread, potentially exposing consumers to predatory behavior.The authors present openly released data and fine-tuned models as a way to increase access to legal support.
- Contribution: CUAD contains more than 500 contracts, over 13,000 expert annotations, and 41 label categories for highlighting salient contract clauses.The dataset was created through a year-long effort involving law students, lawyers, and machine learning researchers.
- Results: Precision @ 80% Recall rises from 8.2% for BERT to 44.0% for DeBERTa, while labeled training volume also strongly influences performance.These results show both rapid model-related gains and the importance of dataset scale.
- Contribution: CUAD enables progress measurement for legal contract review and provides an indicator of language-model learning in highly specialized domains.The dataset is positioned as a specialized expert-annotated benchmark for broader NLP research.
2 Related Work
Legal NLP includes several established tasks, but contract review has received comparatively little attention. Existing contract datasets focus on narrower information extraction, leaving expert-curated large-scale review benchmarks scarce.
- Existing legal NLP: Legal NLP research spans legal judgment prediction, entity recognition, classification, question answering, and summarization.Prior datasets include work in both English and Chinese legal settings.
- Contract review gap: Contract review remains relatively underexplored despite being one of lawyers’ most time-consuming and tedious tasks.The passage contrasts contract review with broader legal NLP coverage.
- Contract review gap: Prior contract work extracted a limited number of basic entities, which the authors characterize as simpler than their clause-review task.The cited work used named entity recognition rather than identifying salient review clauses across categories.
- Expert curation: CUAD is described as one of the few large-scale NLP datasets explicitly curated for machine learning by domain experts.The authors state that freely available contract-review annotations were unavailable, motivating expert curation.
- Expert curation: Specialized-domain benchmarks such as contract review differ from machine translation because comparable supervised data are not generally scraped from freely available sources.The contrast highlights why expert annotation is especially important for this task.
3 CUAD: A Contract Review Dataset
CUAD targets the low-level retrieval work of contract review: locating clauses relevant to 41 lawyer-selected categories in diverse, lengthy contracts. Its annotations support clause extraction and additional practice-oriented contract questions.
- Task scope: Contract review involves reading agreements to understand rights, obligations, and their impact, and is viewed as repetitive, tedious, and expensive.The dataset targets a task considered amenable to automation.
- Task scope: The project aims to automate contract review and the low-level portion of contract analysis, rather than contextual counseling and risk assessment.Counseling depends on industry, business model, risk tolerance, and company priorities.
- Labels: CUAD labels 41 categories of clauses that lawyers particularly attend to, including general information, restrictive covenants, and revenue risks.The categories cover operational terms, business restrictions, and additional costs or remedial measures.
- Task definition: Models extract each relevant clause by outputting the start and end tokens of the span associated with a label category.This operationalizes contract review as highlighting text lawyers should attend to.
- Dataset statistics: CUAD contains 510 contracts and 13,101 labeled clauses across 25 contract types, with labeled clauses making up about 10% of each contract on average.Contracts range from a few pages to over one hundred pages, and only about 0.25% is highlighted per category on average.
- Supplementary annotations: The dataset also releases supplementary annotations, such as yes/no answers derivable from extracted clauses, while evaluation focuses on the more challenging extraction task.These additional annotations can support practical contract-review applications.
4 Experiments
CUAD experiments evaluate extractive Transformer models using imbalance-aware precision–recall metrics and span-overlap matching. DeBERTa-xlarge performs best but remains limited overall, while results show that training-data scale and model design substantially affect performance.
- Task and evaluation: Models predict the start and end tokens of contract substrings relevant to each label category, framing CUAD as an extractive question-answering task.Each label category is treated as a question, and the model highlights the contract spans lawyers should review.
- Task and evaluation: Precision–recall metrics are emphasized because most contract clauses are irrelevant, creating severe class imbalance.Reported measures include AUPR and precision at fixed recall levels.
- Task and evaluation: A highlighted span matches a reference when its Jaccard similarity reaches at least 0.5, using word-set overlap after lowercasing and punctuation removal.The threshold is intended to require sufficiently high overlap for a valid match.
- Overall results: 44.0% Precision @ 80% Recall is achieved by DeBERTa-xlarge, compared with 8.2% for BERT-base, although DeBERTa-xlarge’s overall AUPR is 47.8%.At 90% recall, DeBERTa-xlarge reaches 17.8% precision, illustrating the precision decline at higher recall.
- Data and model effects: Performance varies substantially across label categories, ranging from nearly 100% AUPR for some labels to around 20% for others.Model size is inconsistent as a benefit: BERT-base and BERT-large have AUPR values of 32.4% and 32.3%, while RoBERTa-base and RoBERTa-large reach 42.6% and 48.2%.
5 Conclusion
CUAD provides a high-quality expert-annotated contract-review dataset for specialized-domain NLP research, while results show that performance remains promising but substantially improvable.
- CUAD contains over 13,000 annotations by legal experts across 41 labels.
- The authors evaluated ten pretrained language models on CUAD and found that model performance is promising but has large room for improvement.
- Decreasing the training data by an order of magnitude cuts performance dramatically, identifying data as a major bottleneck.
- Performance is markedly influenced by model design, suggesting that algorithmic improvements can help address contract review.
- CUAD can accelerate contract-review research and benchmark NLP models on specialized domains more broadly.
A Appendix
The appendix illustrates CUAD's contract coverage with a representative transfer and servicing agreement and describes a dataset spanning more than 500 contracts and 25 contract types.
- A representative contract is a Transfer and Servicing Agreement dated January 29, 2020, involving Verizon Owner Trust 2020-A, Verizon ABS LLC, and Cellco Partnership.
- CUAD contains over 500 contracts covering 25 different contract types.
- Manual review of long contracts can be tedious and expensive for legal professionals.
A.1 Special Cases
The appendix documents special evaluation handling for the Parties label and excludes Price Restrictions because the test set contains no examples.
- DeBERTa-xlarge's Precision @ 80% Recall varies across label categories, with substantial room for improvement in some categories.
- The Parties label counts a prediction as matching when the ground-truth segment is a substring of the predicted extraction.
- The relaxed Parties matching rule is justified because predicted extractions are bounded to about a paragraph in length.
- Price Restrictions performance is ignored because randomization left that class without test-set examples.
A.2 Dataset Details
CUAD's dataset-creation process combines extensive law-student training, manual and tool-assisted labeling, attorney review, and documentation of contract coverage and redactions.
- Labeling Process Details: Law-student training included category summaries, attorney-led videos, quizzes, workshops, and required 70–100 hours of initial training.
- Labeling Process Details: Students manually labeled contracts, used keyword searches to find missed categories, and reviewed reports category by category.
- Labeling Process Details: Experienced attorneys reviewed student comments, answered questions, and reached consensus with students when applicable.
- Labeling Process Details: Attorneys and students repeatedly reviewed eBrevia-generated extras until all or substantially all remaining extras were incorrect labels.
- Labeling Process Details: The final report was exported to CSV, with volunteers adding Yes/No answers for categories without an answer.
- Dataset Details: Redacted clauses remain represented in the dataset and answers, including dates shown with blank or placeholder content.
- Dataset Details: The dataset includes confidential legends, irrelevant text such as footers or page numbers, category mismatches, and sentences requiring only sub-part annotations.
- Dataset Details: CUAD includes 25 contract types, whose counts are reported in Table 3.
A.3 Conversion to SQuAD 2.0 Format
The dataset is converted into a SQuAD 2.0-style question-answering format by pairing contract paragraphs with label-specific highlighting questions and span answers. Each answer records either the relevant text span and its starting character position or an empty string when nothing is relevant.
- The conversion adopts the SQuAD 2.0 format, where answers are spans of the provided passage.
- Contracts are first segmented into paragraphs typically ranging from one to five sentences.
- For each label category and paragraph, the question asks the system to highlight any clause parts related to that category and provides its description.
- The answer consists of the passage span that should be highlighted, or an empty string if no relevant text exists.
- Each non-empty answer also includes the character position where the highlighted span begins.