CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

• Contract review is costly and specialized: many law firms spend approximately 50% of their time on it, while large-firm lawyers typically bill $500–$900 per hour in the US.High costs mean small companies and individuals often sign contracts without reading them; openly released data and finetuned models could widen access to legal support.Source: §1 · p. 1
• CUAD contains more than 500 contracts and 13,000 expert annotations across 41 labels, with models tasked to highlight contract portions salient to each label.The annotations identify important clauses, including salient obligations and red-flag clauses.Source: §1 · p. 2, Figure 1 · p. 2
• Each student annotator completed 70–100 hours of contract-review training, followed more than 100 pages of standards, and each annotation was verified by three additional annotators.The authors conservatively estimate CUAD’s pecuniary value at over $2 million.Source: §1 · p. 2
• Precision @ 80% Recall rises from 8.2% for a 2018 BERT model to 44.0% for a 2021 DeBERTa model, and labeled training annotation volume greatly influences performance.The authors report that this metric improves quickly as models improve.Source: §1 · p. 2
• CUAD provides a way to assess progress on legal contract review and language-model learning in specialized domains, while inviting broader NLP research on specialized domains.The authors describe CUAD as one of the only large specialized NLP benchmarks annotated by experts.Source: §1 · p. 2

2.1 Legal NLP

• Prior legal NLP spans prediction, entity recognition, classification, question answering, and summarization, with datasets also covering tax law, EU legislation, and statute and case law.Some judgment-prediction and judicial-reading datasets are in Chinese, limiting their applicability to English speakers.Source: §2.1 · pp. 2–3
• Contract review has little prior NLP work; earlier contract datasets target limited entity extraction or lease-only review with fewer labels and over tenfold fewer annotations than CUAD.The related work describes contract review as time-consuming and tedious for lawyers.Source: §2.1 · p. 3

2.2 NLP Models for Specialized Domains

• CUAD is among the few large-scale NLP datasets explicitly curated for machine learning by domain experts, because freely available contract-review annotations cannot be scraped.Unlike many specialized domains, contract review lacks freely available annotated data.Source: §2.2 · p. 3
• Specialized-domain benchmarks include legal multiple-choice QA, math, coding, and medical diagnosis, but these tasks do not directly perform lawyers’ contract-review work.The paper argues that specialized tasks remain unsolved by current systems, placing them at the research forefront.Source: §2.2 · p. 3

Contract Review

• CUAD targets repetitive, costly contract review and the low-level contract analysis of finding relevant clauses, not contextual risk counseling.The authors describe counseling as highly contextual work done by experienced lawyers and identify automation of contract review and low-level analysis as their aim.Source: §3 · p. 3

Labels and Task Definition

• For each of 41 lawyer-prioritized categories, CUAD identifies relevant clauses and trains models to extract their start and end tokens as text spans.The categories cover general information, restrictive covenants, and revenue risks.Source: §3 · p. 4

Dataset Statistics

• 510 contracts and 13,101 labeled clauses span 25 contract types; labeled clauses average about 10% of each contract, or about 0.25% per category.Contracts range from a few pages to over one hundred pages.Source: §3 · p. 4

Supplementary Annotations

• CUAD also releases supplementary annotations, such as yes/no answers derivable from extracted clauses, while the paper focuses on the harder clause-extraction task.These additional annotations can further support applying models to contract review in practice.Source: §3 · p. 4

Contract Sources and Labeling Process

• EDGAR contracts provide complex, heavily negotiated examples rich in difficult-to-find clauses; law students label them after 70–100 hours of lawyer-designed training, with experienced lawyers checking quality.The labeling process also used extensive documentation to promote consistency.Source: §3 · p. 5

4.1 Setup

• The experiments cast each contract-label pair as extractive span prediction, using label descriptions, sliding windows, and a Jaccard overlap threshold of 0.5.They evaluate BERT, RoBERTa, ALBERT, and DeBERTa; because more than 99% of windowed features lack relevant labels, they downweight empty features during training.Source: §4.1 · pp. 5–6

4.2 Results

• DeBERTa-xlarge led CUAD with AUPR 47.8%, Precision at 80% Recall 44.0%, and Precision at 90% Recall 17.8%, though performance remained nascent.The authors argue that 80% recall may be useful to some lawyers and could save substantial review time compared with reading an entire contract.Source: Table 2 · p. 6, §4.2 · p. 7
• Pretraining RoBERTa-base on approximately 8GB of unlabeled contracts increased AUPR by only about 3%, less than the value of CUAD’s annotated data.The authors suggest that masked language modeling may not effectively leverage the contract corpus, so improved pretraining algorithms may be needed.Source: §4.2 · p. 7
• DeBERTa-xlarge AUPR varied from nearly 100% to around 20% across the 41 label categories, indicating substantial category-specific room for improvement.The figure compares AUPR across categories; some labels approach the ceiling while others remain much lower.Source: §4.2 · p. 7, Figure 4 · p. 7
• Increasing ALBERT size by more than 20 times yielded only around 3% better performance, while BERT-base and BERT-large had AUPR 32.4% and 32.3%, respectively.By contrast, RoBERTa-base had AUPR 42.6% versus 48.2% for RoBERTa-large, and DeBERTa performed far better than BERT; size helps inconsistently, while design can matter.Source: §4.2 · p. 8
• Increasing RoBERTa-base training data by an order of magnitude raised AUPR from 27.6% to 42.6%, a 15% absolute difference.These gains were comparable to the 15.4% absolute AUPR difference between the best and worst models, suggesting data is a major bottleneck in this regime.Source: §4.2 · p. 8, Figure 6 · p. 8

• CUAD contains over 13,000 legal-expert annotations across 41 labels, supporting research on contract review.Source: §5 · p. 8
• Performance on CUAD is promising but has substantial room for improvement, and depends markedly on model design.Source: §5 · p. 8
• Reducing the training data by an order of magnitude cuts performance dramatically, highlighting data quantity as a major bottleneck.Source: §5 · p. 8
• CUAD can accelerate work on contract review and benchmark NLP models in specialized domains more broadly.Source: §5 · p. 8

A Appendix

• CUAD contains over 500 expert-labeled contracts across 25 contract types, including examples longer than 100 pages where manually locating clauses can be tedious and expensive.The labels identify important clauses for models to learn to extract.Source: Figure 7 · p. 10

A.1 Special Cases

• Parties counted as a match when the ground-truth segment was a substring of a predicted extraction, accommodating multiple small segments and paragraph-bounded predictions.This exception applied only to the Parties label.Source: §A.1 · pp. 10–11
• Price Restrictions performance was ignored because randomization in the train-test split left that class without test examples.Source: §A.1 · p. 11

Labeling Process Details

• Law students received 70-100 hours of category training before manually reviewing and labeling contracts in eBrevia.Training included category summaries, attorney video instructions, quizzes, workshops, and sample-contract labeling.Source: §A.2 · p. 12
• Students used keyword searches to find missed clauses, reviewed labeled clauses category by category, and consulted attorneys to resolve questions and mislabeled clauses.Where applicable, attorneys discussed results with students and reached consensus before students made corrections in eBrevia.Source: §A.2 · p. 12
• Attorneys and students reviewed eBrevia-identified “extras,” repeating the process until all or substantially all were incorrect labels, then exported a final report to CSV.Volunteers added Yes/No answers for categories without an answer column.Source: §A.2 · p. 12

Redacted Information

• Redactions appear in the dataset and answers as submitted, including placeholders or blank spaces; for example, “January __ 2020” maps to “1/[]/2020”.Redactions may appear as asterisks, underscores, or blank space.Source: §A.2 · p. 12
• Contract text may include confidential legends, irrelevant footers or page numbers, category-mismatched sentences, and clauses where only a sub-part is responsive.Source: §A.2 · p. 13

Contract Types

• CUAD covers 25 contract types, with the number of contracts of each type listed in Table 3.Source: Table 3 · p. 12, §A.2 · p. 13

A.3 Conversion to SQuAD 2.0 Format

• SQuAD 2.0 is a notable precedent because, like CUAD, it uses questions whose answers are spans of the input passage.Source: §A.3 · p. 13
• Contracts were segmented into paragraphs typically one to five sentences long, then paired with a question for each label category asking which parts of the clause relate to that category.The question also includes the label-category description.Source: §A.3 · p. 13
• Answers were the relevant passage span or an empty string when nothing should be highlighted, with the span’s starting character position recorded.Source: §A.3 · p. 13