PaperlayerView exact revision on arXiv

Source-linked AI summary

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

arXiv:2308.11462v1cs.CLcs.AIcs.CY

TL;DR

LegalBench addresses the question of what types of legal reasoning LLMs can perform by providing a collaboratively constructed benchmark. It contains 162 tasks designed with legal professionals, evaluates 20 open-source and commercial LLMs, and supports comparisons across legal-reasoning forms while remaining incomplete and unsuitable as a substitute for context-specific evaluation.

  • Problem

    The paper addresses limited ability to rigorously evaluate the diverse legal reasoning capabilities of large language models.

  • Method

    The authors construct LegalBench as an interdisciplinary, open-source benchmark of 162 tasks from 36 data sources, with tasks designed and hand-crafted by legal professionals.

  • Results

    The paper evaluates 20 open-source and commercial LLMs across LegalBench tasks and reports that GPT-4 outperforms GPT-3.5 and Claude-1 on rule-conclusion tasks.

  • Takeaways & Limitations

    LegalBench provides a common vocabulary and technical artifact for stakeholders to study LLM legal reasoning capabilities.

  • Takeaways & Limitations

    LegalBench is necessarily incomplete because important legal tasks, document types, and reasoning types are not included.

Abstract

from arXiv · show

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.

G Full results

The full-results material is organized into three subsections covering models, prompts, and results.

  • G.1 Models: G.1 covers the models used in the evaluation.
  • G.2 Prompts: G.2 covers the prompts used in the evaluation.
  • G.3 Results: G.3 presents the evaluation results.

1 Introduction

LegalBench addresses the need for better evidence about the distinct legal reasoning capabilities of LLMs by combining legal expertise with benchmark construction. It introduces a collaborative benchmark, a legal reasoning typology, and evaluations intended to support safer and more informed study of LLMs in law.

  • Legal applications create safety concerns because misleading or incorrect LLM outputs could cause substantial harm, motivating legal-context benchmarking.
  • Existing legal benchmarks often emphasize supervised learning or certification exams and blur the distinct reasoning skills involved in legal work.
  • LEGALBENCH assembles 162 tasks from 36 data sources, with legal professionals actively contributing tasks that measure interesting or practically useful reasoning skills.
  • The benchmark organizes tasks using a typology based on legal frameworks, giving legal professionals and AI researchers shared terminology for discussing model performance.
  • The authors evaluate 20 LLMs from 11 families across multiple size points and study prompt-engineering strategies as directions for future research.
  • LEGALBENCH is intended to support empirical understanding of LLM legal-task capabilities rather than determine whether computational systems should replace lawyers.

2 Related work

LEGALBENCH extends prior legal NLP benchmarks by standardizing 162 few-shot evaluation tasks and organizing them by fine-grained reasoning types. Its collaborative design supports comparisons across task groupings and broader participation by legal professionals.

  • The benchmark is currently skewed toward American law, and future contributions are expected to incorporate other jurisdictions.
  • Prior legal benchmarks commonly target supervised or fine-tuned models, while certification exams do not always represent LLM use cases.
  • LEGALBENCH provides 162 standardized evaluation tasks with candidate prompts, in-context demonstrations, and metrics.
  • Its framework groups tasks so researchers can compare performance trends and distinguish areas of strength from task types needing further work.
  • Few-shot prompting reduces the need for large labeled training sets, allowing benchmark effort to support development of more diverse legal reasoning tasks.
  • The project models open-source interdisciplinary collaboration, recognizing legal professionals’ role in supervising LLMs and selecting legal use cases.

3 The LEGALBENCH typology

LEGALBENCH defines six categories of legal reasoning and uses them to organize tasks that isolate particular capabilities. The typology connects legal frameworks such as IRAC to concrete few-shot evaluation tasks while remaining non-exhaustive.

  • LEGALBENCH evaluates issue-spotting, rule-recall, rule-application, rule-conclusion, interpretation, and rhetorical-understanding.
  • The six-category framework can characterize non-American legal tasks but is explicitly non-exhaustive.
  • Frameworks for legal reasoning: IRAC frames legal reasoning as sequential issue identification, rule identification, rule application, and legal conclusion.
  • Frameworks for legal reasoning: The BusinessMart example applies the IRAC steps to determining a corporation’s principal place of business under the Hertz rule.
  • Evaluating legal reasoning in large language models: Issue-spotting tasks ask whether facts raise specified legal questions, implicate an area of law, or concern a particular party.
  • Evaluating legal reasoning in large language models: The learned_hands_benefits task classifies whether a legal-aid forum post raises welfare-law issues using a Yes/No response.
  • Evaluating legal reasoning in large language models: Rule-recall tasks require generating or classifying jurisdiction-specific legal rules, making them useful for measuring hallucinations.
  • Frameworks for legal reasoning: LEGALBENCH tasks generally evaluate one reasoning type at a time rather than requiring a complete IRAC-style answer.

4 LEGALBENCH tasks

LEGALBENCH is an overviewed benchmark of legal reasoning tasks built through collaborative contributions, dataset restructuring, and supporting documentation. Its tasks span multiple formats, reasoning categories, and legal domains, while emphasizing objectively answerable questions.

  • Task sources: LEGALBENCH draws tasks from existing datasets and corpora, often significantly reformatting data and restructuring prediction objectives for LLM evaluation.For example, CUAD’s long-document span-extraction data was adapted into a different task format.
  • Collaborative component: Contributors from academic, practitioner, and computational legal backgrounds submitted tasks that were vetted for legal correctness and task validity.The collaborative process began with a public call for tasks in August 2022.
  • Infrastructure: Each task includes documentation covering its procedure, legal significance, and construction to help non-law AI researchers interpret benchmark performance.The documentation is intended to explain the mechanical processes behind tasks.
  • Task structure: LEGALBENCH tasks contain at least 50 samples, average 563 samples, and use multiple-choice, open-generation, binary-classification, and multi-class or multi-label formats.The benchmark includes 35 multiple-choice, 7 open-generation, 112 binary-classification, and 8 multi-class or multi-label tasks.
  • Reasoning types and legal domains: The benchmark covers rule-recall, issue-spotting, rule-application, rule-conclusion, interpretation, and rhetorical-understanding reasoning categories.Its task distribution is concentrated in interpretation, with 119 tasks, and includes predominantly civil-law domains such as contracts and corporate law.
  • Reasoning task examples: Issue-spotting tasks identify whether facts implicate legal questions or domains, while rule-recall tasks generate or classify jurisdiction-specific legal rules.The issue-spotting set includes 17 tasks, and the rule-recall set includes 5 tasks.
  • Reasoning task examples: Rule-application and rule-conclusion tasks include diversity-jurisdiction reasoning that combines arithmetic and logical reasoning across increasingly complex party and claim configurations.The diversity tasks range from one plaintiff, one defendant, and one claim to two plaintiffs, two defendants, and two claims against each defendant.
  • Reasoning task examples: Interpretation tasks include contractual question answering, ContractNLI assertions, and CUAD clause classification, while other tasks cover hearsay, personal jurisdiction, successor liability, telemarketing rules, and UCC classification.The supplied examples show that tasks evaluate both contract interpretation and application of specific legal doctrines.

5 Results

LEGALBENCH reveals substantial task- and model-dependent variation in legal reasoning performance, enabling fine-grained comparisons across reasoning categories, model families, sizes, and prompting conditions. The results also show that current capabilities are uneven: some tasks appear within reach, while others remain challenging.

  • Overall performance: Performance varies significantly across tasks, indicating that LEGALBENCH spans a diverse spectrum of difficulty.The authors therefore argue that legal LLM assessments should be made task by task and informed by each task’s specific nuances.
  • Parameter count: Larger models usually outperform smaller models within the same LLM family, although the size gap differs across families and reasoning categories.Flan-T5-XXL outperforms Flan-T5-XL across all five categories, while LLaMA-2-13B outperforms LLaMA-2-7B across four categories.
  • Variation across families: LLMs of similar size can perform very differently across families, with pretraining data, instruction tuning, and architecture associated with substantial performance gaps.Flan-T5-XXL significantly outperforms Vicuna-13B-16k across all reasoning categories, and family-specific patterns persist across multiple size points.
  • Open-source and commercial models: Open-source models can match or exceed some commercial models: Flan-T5-XXL outperforms GPT-3.5 and Claude-1 on issue-spotting and rhetorical-understanding.The largest open-versus-closed gap occurs for rule-conclusion tasks.
  • Issue-spotting: GPT-4 outperforms GPT-3.5 and Claude-1 on issue-spotting, with an absolute margin exceeding 20 points over the other closed-API models.GPT-3.5 and Claude-1 differ by only two average points, while Incite-Base collapses to a single predicted class on nine issue-spotting tasks.
  • Rule-recall: Rule-recall is the only category in which Claude-1 performs comparably to GPT-4 and exceeds GPT-4 on rule_qa, international_citizenship_questions, and nys_judicial_ethics.Both GPT-4 and Claude-1 outperform GPT-3.5 on average, with p < 0.05.

6 Conclusion

LEGALBENCH is a collaboratively constructed benchmark of 162 tasks for measuring the legal reasoning capabilities of LLMs, with planned expansion through continued contributions from the legal community.

  • LEGALBENCH provides 162 tasks for measuring the legal reasoning capabilities of LLMs.

B Limitations and social impact

LEGALBENCH is necessarily incomplete: it omits important document types and reasoning abilities, has domain, jurisdictional, and language skew, and excludes subjective or ambiguous tasks. The benchmark is intended to support evaluation, not replace context-specific assessment or justify automation on its own.

  • Limitations: LEGALBENCH is necessarily incomplete because important legal tasks, document types, and reasoning types are not included.
  • Limitations: LEGALBENCH excludes long-document tasks, although contracts, filings, statutes, and judicial opinions may span hundreds of pages.
  • Limitations: LEGALBENCH evaluates only a subset of legal reasoning abilities and does not currently include analogical reasoning grounded in case law.
  • Limitations: LEGALBENCH tasks skew toward contracts, civil procedure, and US Federal law, limiting representativeness for other domains, jurisdictions, and international law.
  • Limitations: LEGALBENCH does not support multilingual or non-English legal-task evaluation and excludes more subjective or ambiguous tasks.
  • Future work: The authors intend to expand LEGALBENCH to other jurisdictions and a broader cross-section of legal domains.
  • Social impact: LEGALBENCH is not a substitute for in-depth, context-specific evaluation; legal AI deployments require in-domain data and ethical and legal compliance assessments.
  • Social impact: The consequences of LLM mistakes depend significantly on the task and broader social context, such as dataset annotation versus legal-aid answers.

C Datasheet

LEGALBENCH is a self-contained benchmark of text-based legal tasks assembled from novel and adapted datasets, with small few-shot training splits. Its instances are generally samples or synthetic data, while legal-data availability, annotation cost, OCR artifacts, and possible source-document identification constrain coverage and privacy.

  • Purpose: LEGALBENCH was created to evaluate LLMs on legal tasks and understand their legal reasoning capabilities, including adaptation needs and safe, reliable use.
  • Construction: The dataset combines novel author-created datasets with transformed or adapted datasets from prior research, with task origins documented in Section F.
  • Data: All instances are text, including sentences, paragraphs, and documents from real-world sources or synthetic construction, and labels include classes, extracted entities, and open-ended generation.
  • Composition: Nearly every task samples a larger population or uses synthetic data; unpublished documents and expensive annotation limit complete or representative coverage.The CUAD project estimated that annotating 500 contracts cost $2 million US dollars.
  • Splits: Train splits contain 2-8 randomly sampled instances, preserving a few-shot setting in which practitioners have only a handful of labeled examples.
  • Limitations: Legal data may contain scanning and OCR artifacts, and public-source provenance can make original documents identifiable.

D Task overview

LEGALBENCH organizes tasks by publication status and by legal or NLP reasoning type, while documenting licenses and task statistics. The benchmark includes previously published, original-but-available, and original-unavailable tasks, with inputs ranging from sentences to two pages.

  • Publication status: Tasks are categorized as previously published, original but available, or original and unavailable, reflecting their online publication status.
  • Publication status: Table 9 summarizes the availability status of each task, while Table 8 summarizes task licenses.
  • Task organization: Table 10 organizes tasks by LEGALBENCH reasoning type, and Table 11 organizes them by NLP reasoning type with comparable general-domain benchmarks.

LEGALBENCH reasoning type Tasks

LEGALBENCH spans six legal reasoning types through a large and varied collection of tasks, with substantial variation in dataset size and input length. The task inventory includes contract, privacy, securities, judicial, statutory, and other legal domains.

  • Reasoning types: LEGALBENCH organizes its benchmark around distinct legal reasoning types, with Table 10 listing tasks by reasoning type.
  • Task scale: Tasks vary from at least 50 instances to datasets with thousands of examples, while inputs range from short clauses to hundreds of words.
  • Task formats: The benchmark includes classification, extraction, question answering, entailment, and open-ended legal reasoning tasks across these domains.
  • Task families: The inventory includes contract NLI, CUAD contract clauses, privacy-policy, securities-complaint, judicial, statutory, and legal-reasoning tasks.
  • Task families: Additional tasks cover MAUD merger agreements, Learned Hands legal forums, textualism, personal jurisdiction, tax, consumer contracts, and UCC versus common law.

E Evaluation

LEGALBENCH evaluates classification, extraction, generation, and rule-application tasks with task-specific metrics and expert-informed criteria. Its task examples include trademark classification, tax-court outcomes, citation prediction, and legal rule application.

  • Rule application: Rule-application correctness measures errors in facts, rules, outcomes, logic, or arithmetic, whereas analysis measures necessary inferences connecting facts to legal conclusions.
  • Rule application: Rule-application evaluations use task-specific answer guides containing required inferences and common error modes.
  • Task-specific evaluation: Specialized tasks use tailored evaluation procedures, including human assessment for rule QA, 10% tolerance for sara_numeric, and case-name matching for citation_open.
  • Task examples: The benchmark also includes legal text tasks such as identifying parties in securities complaints and predicting Tax Court appeal outcomes.
  • Metrics: Classification uses exact-match with class-balanced accuracy, while extraction tasks use normalized outputs and F1 when multiple entities or terms are required.
  • Task examples: The Abercrombie task classifies a candidate mark and product or service into one of five distinctiveness categories, using approximately 20 samples per category and a 95-sample test set.

F.4 Clause Classification Tasks

LEGALBENCH includes clause-classification tasks that ask LLMs to determine whether contractual provisions belong to specified categories. These tasks cover common contract clauses, loan-agreement provisions, and potentially unfair terms of service.

  • Task scope: LEGALBENCH classifies clauses from contracts, loan agreements, and terms-of-service agreements into legally meaningful categories.The categories include common contractual clause types, J-Crew blockers, and eight terms-of-service categories.
  • Task scope: 38 CUAD-derived tasks use binary classification to identify whether EDGAR contract clauses belong to specified categories.The categories include audit rights clauses and other clause types.
  • Significance and value: Clause review is practically useful because lawyers inspect agreements to identify potentially problematic terms, including after changing events such as the COVID-19 pandemic.The benchmark spans categories and document types because classification complexity depends on both.

F.4.1 CUAD Tasks

The CUAD portion of LEGALBENCH adapts contract-clause annotations into balanced binary classification tasks covering 38 clause categories. It standardizes questions and examples for evaluating whether LLMs recognize legally defined provisions.

  • Relation to CUAD: CUAD was originally built from 500 contracts annotated with up to 41 clause types, while LEGALBENCH restructures the data for few-shot LLM evaluation.The original work studied BERT-based span identification; LEGALBENCH instead creates balanced classification tasks.
  • Construction: 38 of CUAD’s 41 clause categories are selected, each paired with an equal number of randomly sampled clauses from other categories.This produces a balanced binary classification task for each selected category.
  • Construction: Each CUAD task asks whether a clause satisfies the category description, assigning “Yes” for affirmative answers and “No” otherwise.The descriptions are presented as the questions used during data labeling.
  • Task coverage: The 38 tasks cover provisions such as affiliate licensing, anti-assignment, audit rights, liability caps, exclusivity, insurance, and intellectual-property ownership.The task descriptions operationalize these categories through clause-level questions and examples.
  • Significance and value: Clause classification is among LEGALBENCH’s most practically useful task groups because it captures a current-day use case for LLMs.The tasks span different clause categories and source documents.

F.6 Contract NLI Tasks

Contract NLI tasks test whether an LLM can determine if a contract excerpt supports or fails to support a legal assertion. LEGALBENCH transforms annotated contract data into separate short-context tasks covering confidentiality rights and obligations.

  • Task definition: Contract NLI asks whether an assertion about a contract excerpt is supported or unsupported by that excerpt.The tasks evaluate reasoning over rights and obligations created by contracts.
  • Construction: The source dataset contains 607 contracts and 17 assertions labeled as supporting, negating, or not mentioning each assertion.LEGALBENCH transforms these annotations into task-specific examples.
  • Construction: Each assertion becomes a separate short-context task using supportive test-set excerpts as positives and sampled alternatives to generate negatives.This restructuring targets a short-context LLM setting.
  • Task coverage: The task family covers confidentiality provisions involving retention, verbal information, third-party sharing, copying, compelled disclosure, similar information, employees, and limited use.These questions concern permissions and restrictions governing confidential information.

F.16 MAUD Tasks

MAUD-based LEGALBENCH tasks evaluate legal reading comprehension over merger-agreement clauses by asking LLMs to select the best description from multiple-choice options. The benchmark uses abridged examples to focus on comprehension rather than extracting deal points from complete agreements.

  • Source data: MAUD originates from over 37,000 expert-annotated examples based on the ABA’s 2021 Public Target Deal Point Study.Lawyers identified deal points and answered predetermined questions about their legal characteristics.
  • Task definition: MAUD tasks present a key legal clause and descriptions, requiring the LLM to choose the option that best describes the clause.LEGALBENCH converts the questions into human-readable multiple-choice prompts.
  • Construction: The adapted MAUD data contains 14,928 examples from 94 merger agreements and is narrowed from 92 to 57 questions.Questions with fewer than 50 examples are filtered out.
  • Scope: Abridged examples are used to assess legal reading comprehension rather than extracting relevant text from complete deal points.Complete deal-point texts can be too long for many open-source LLM context windows.
  • Task coverage: The tasks cover questions such as the required accuracy standard for fundamental representations and warranties under a bringdown provision.Examples distinguish standards including all material respects and all respects.
  • Significance and value: MAUD tasks evaluate an LLM’s ability to understand and categorize a wide spectrum of legal clauses in merger agreements.The section characterizes contract reading comprehension as challenging for both humans and machines.

F.30 Supply Chain Disclosure Tasks

The Supply Chain Disclosure task evaluates whether LLMs can assess corporate compliance with disclosure requirements across varied, complex supply-chain statements. It converts human coding of required topics and best-practice consistency into binary classification tasks.

  • Background: The task concerns corporate disclosures intended to address supply-chain risks such as human trafficking and human rights violations.Applicable disclosures cover corporations doing business in specified jurisdictions.
  • Background: The California requirements address verification, audits, certification, accountability, and training.The Attorney General’s office also provides best-practice guidance for these topics.
  • Significance and value: Prior research found that companies do not always cover every required topic or follow recommended best practices.This motivates evaluating automated assessment of disclosure compliance.
  • Construction process: Researchers collected disclosures from roughly 400 of roughly 500 identified firms and had assistants code five topics and their best-practice consistency.The dataset was based on firms identified by KnowTheChain as subject to the California law.
  • Construction process: The 10 coded variables became 10 distinct binary classification tasks.Each task has a precise coding question listed in Table 53.

G.3 Results

The results section reports model performance across LegalBench task families, organizing models into size or type groups and presenting detailed task-level tables.

  • Organization: Results are reported for each LLM, with models divided into commercial, 13B, 7B, and 3B groups.
  • Evaluation setup: The appendix tables also specify the number of in-context demonstrations used for tasks and report correctness/analysis for rule-application evaluations.
  • Task families: Tables cover rule-application, rule-recall, issue-spotting, rule-conclusion, rhetorical-understanding, and interpretation tasks.
  • Task-level results: Task-level entries include CUAD, MAUD, insurance policy interpretation, J.Crew blocker, privacy-policy, supply-chain disclosure, and other LegalBench tasks.
Loading 2308.11462v1…