PaperlayerView exact revision on arXiv

Source-linked AI summary

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

arXiv:2308.11462v1cs.CLcs.AIcs.CY

TL;DR

What kinds of legal reasoning LLMs can perform remains a question requiring rigorous, legally informed benchmarking. LegalBench supplies 162 tasks across six reasoning types, and GPT-4 outperformed GPT-3.5 and Claude-1 on rule-conclusion tasks (p < 0.01 for each).

  • Problem

    The paper addresses the need for rigorous evaluation of LLM legal reasoning through benchmarking in which the legal community takes a proactive role.

    Source: §1 · p. 4

  • Method

    LegalBench is an interdisciplinary benchmark of 162 tasks, organized around six legal-reasoning types and built through collaboration involving legal professionals.

    Source: §1 · p. 4, §3 · p. 6

  • Results

    On rule-conclusion tasks, GPT-4 outperformed GPT-3.5 and Claude-1 (p < 0.01 for each), while GPT-3.5 and Claude-1 performed approximately level.

    Source: §5.3.4 · p. 16

  • Takeaways & Limitations

    The authors position LegalBench as a technical artifact to help stakeholders conduct rigorous evaluations, not as a substitute for context-specific evaluation or evidence about replacing legal professionals.

    Source: B Limitations and social impact · p. 28

    LegalBench is necessarily incomplete: important legal tasks, document types, and reasoning types are not included.

    Source: B Limitations and social impact · p. 28

Abstract

from arXiv · show

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.

G Full results

The full-results section is organized into subsections on models, prompts, and results.

G Full results

  • The section lists separate subsections for models, prompts, and results.The contents listing places all three on page 120.Source: G Full results · p. 4

1 Introduction

LegalBench addresses gaps in legal LLM evaluation by assembling an interdisciplinary, open-source benchmark of 162 tasks that measure distinct types of legal reasoning. Its task typology and evaluation resources are intended to help stakeholders assess LLM capabilities and support further research, not to evaluate replacing lawyers.

Introduction

  • Legal LLM evaluation is urgent because harmful or misleading outputs could cause substantial legal harms, while existing benchmarks miss few-shot task diversity and lawyers’ distinctions among reasoning types.The potential burden could fall disproportionately on marginalized and under-resourced populations.Source: §1 · p. 4
  • LegalBench assembles 162 tasks from 36 sources, combining existing datasets adapted for few-shot prompting with tasks crafted by legal professionals for interesting skills or practical applications.The authors describe it as, to their knowledge, the first open-source legal benchmarking effort.Source: §1 · pp. 4–5
  • Its typology organizes tasks by required reasoning using frameworks familiar to lawyers, giving legal professionals shared terminology for discussing LLM performance.The paper also presents a typology based on frameworks lawyers use to describe legal reasoning.Source: §1 · p. 5
  • The paper describes task construction, heterogeneity, and limitations, with full task descriptions provided in the Appendix.Source: §1 · p. 5
  • The authors evaluate 20 LLMs from 11 families across size points and explore prompt-engineering strategies to identify research directions enabled by LegalBench.The benchmark also offers prompting and evaluation support for researchers without legal expertise and is intended to grow through further contributions from legal professionals.Source: §1 · p. 5
  • The work aims to help stakeholders empirically understand LLMs’ capacity for different legal tasks and support safe, ethical use, not assess whether computational systems should replace legal professionals.Source: §1 · p. 5

2 Related work

LegalBench builds on prior legal NLP and LLM evaluations by standardizing 162 tasks for few-shot assessment and organizing them to compare capabilities across task types. It also draws on fine-grained benchmarking and collaborative AI efforts to connect legal and AI communities.

2.1 Legal reasoning benchmarks

  • Prior legal NLP benchmarks covered lawyer tasks, legal inference, pretraining, multilingual work, judgment prediction, role labeling, and retrieval; newer LLM studies often assess exams, prompting, applications, and human interaction.Earlier benchmarks predominantly evaluated supervised or fine-tuned models, while LLMs are evaluated mainly through in-context prompting.Source: §2.1 · pp. 5–6
  • LEGALBENCH provides 162 standardized evaluation tasks, potential prompts, in-context demonstrations, metrics, and a framework for comparing performance across task groupings.The framework can help distinguish task types where current LLMs perform well from those needing further work.Source: §2.1 · p. 6
  • Few-shot prompting requires only a few labeled demonstrations, allowing effort otherwise spent building large training sets to support new tasks and a more diverse benchmark.Traditional NLP benchmarks have relied on costly legal annotations or naturally available annotations.Source: §2.1 · p. 6

2.2 Connections to other LLM benchmarking efforts

  • Fine-grained measures inspired by GLUE [141], [100], HELM [80], and BigBench [123] enable precise capability comparisons and help assess effects of model or training changes.Such measures are especially relevant as legal NLP researchers debate how to specialize language models for the domain [56].Source: §2.2 · p. 6
  • LEGALBENCH’s open-source, interdisciplinary design involves legal professionals in task construction so evaluations better reflect their interests and information needs.The paper notes the benchmark’s current skew toward American law and hopes future contributions add other jurisdictions.Source: §2.1 · p. 5, §2.2 · p. 6

3 The LEGALBENCH typology

LEGALBENCH organizes legal reasoning into six types—issue-spotting, rule-recall, rule-application, rule-conclusion, interpretation, and rhetorical-understanding—and uses task datasets to evaluate them. Its framework draws heavily on American legal thought but can characterize tasks involving non-American law; the types are non-exhaustive.

3.1 Frameworks for legal reasoning

  • IRAC frames legal reasoning as sequential issue-spotting, rule-recall, rule-application, and rule-conclusion, while lawyers also use interpretive and rhetorical skills.Relevant rules vary by jurisdiction, and application can involve weighing facts and comparing precedents; the conclusion determines the legal outcome. IRAC is a framework for grouping tasks, not a requirement that each benchmark task evaluate all steps.Source: §3.1 · pp. 6–7
  • LEGALBENCH treats tasks as having objectively correct answers, focusing on clear law and unambiguous facts rather than disputed conclusions.When facts are unclear or contested, reasonable minds may differ about the conclusion; the benchmark focuses on the former, less ambiguous setting.Source: §3.1 · p. 7

3.2 Evaluating legal reasoning in large language models

  • LEGALBENCH tests issue-spotting by identifying legal questions or implicated law, and rule-recall by generating or classifying jurisdiction-specific rules.Rule-recall tasks can measure hallucinations; examples include identifying welfare-law issues in legal-aid posts and answering doctrinal questions in rule_qa.Source: §3.2 · pp. 7–8
  • Rule-conclusion tasks test outcomes under a supplied rule, while rule-application tasks assess explanations for correctness and analysis.Correctness checks for rule, fact, outcome, logic, or arithmetic errors; analysis requires relevant inferences showing how the conclusion follows. Answer guides specify sample-level inferences and common errors.Source: §3.2 · p. 8
  • Interpretation tasks parse legal texts, often for practical use cases, while rhetorical-understanding tasks classify functions or properties of legal arguments.Examples include detecting an audit right in a contract clause and determining whether a judicial-opinion sentence defines a term.Source: §3.2 · pp. 8–9
  • Each LEGALBENCH task evaluates a single reasoning type rather than jointly testing IRAC steps; the authors hope to add multi-step tasks in future work.For example, learned_hands_benefits evaluates issue-spotting, not rule-recall.Source: §3.2 · p. 9

4 LEGALBENCH tasks

The section outlines how LEGALBENCH was collaboratively constructed and supported, the formats and legal-language domains its tasks cover, and examples across reasoning categories. It also notes limits on the benchmark’s current scope.

4.1 Construction process

  • LEGALBENCH combines existing corpora, unreleased datasets created by legal professionals, and author-developed tasks, with contributors vetting submissions for legal correctness and validity.The tasks draw on 36 corpora; some source data was substantially reformatted, including CUAD [62], and each task comes with documentation and a base prompt to support understanding and replicability.Source: §4.1 · pp. 9–10

4.2 Dimensions of variation

  • The benchmark’s tasks have at least 50 samples, average 563 samples, and use multiple-choice, open-generation, binary-classification, or multi-class/multi-label formats.The formats cover 35 multiple-choice tasks, 7 open-generation tasks, 112 binary-classification tasks, and 8 multi-class/multi-label tasks.Source: §4.2 · p. 10
  • Tasks span six reasoning categories and multiple civil-law domains, with interpretation and contract-law tasks especially common.The listed categories include rule-recall, issue-spotting, rule-application, rule-conclusion, interpretation, and rhetorical-understanding; contracts and corporate law each account for 58 tasks, while civil procedure has 8 and evidence law 1.Source: §4.2 · p. 10
  • LEGALBENCH draws on varied legal-language types, including plain English, legal opinions, merger agreements, contracts, and statutory text.The listed task counts are 32 plain-English, 11 legal-opinion, 34 merger-agreement, 55 contract, and 3 statutory-text tasks, alongside other sources.Source: §4.2 · p. 10

4.3 Tasks

  • Issue-spotting includes 17 tasks, mostly classifying whether legal-advice posts implicate a legal domain, plus a task identifying bills’ implications for companies.The 16 Learned Hands tasks use posts from /r/legaladvice; the corporate_lobbying task assesses a legislative bill’s legal implications for a described company.Source: §4.3 · p. 10
  • Five rule-recall tasks test citation generation or verification, retrieval of legal tests, citizenship requirements, and judicial-ethics rules.The tasks include yes/no questions about citizenship requirements in different countries and about ethical rules under New York State guidance.Source: §4.3 · pp. 10–11
  • Twelve tasks serve both rule-application and rule-conclusion, including diversity-jurisdiction calculations and logical reasoning, and determining whether evidence is hearsay for a litigated issue.Diversity tasks range from one plaintiff, one defendant, and one claim to two plaintiffs, two defendants, and two claims against each defendant.Source: §4.3 · p. 11
  • Interpretation comprises 118 tasks, including identifying contractual provisions, testing contract excerpts against assertions, and answering questions about merger-agreement excerpts.Other examples include CUAD clause-type classification [62] and tax-code interpretation and application using SARA [65].Source: §4.3 · p. 11

5 Results

The results examine model performance across tasks, model families, sizes, and reasoning categories, alongside prompting strategies. They show task-dependent variation in both performance and the effects of prompt wording, rule descriptions, and in-context demonstrations.

5.1.1 Models

  • The study evaluates 20 LLMs from 11 families, including 17 open-source models at 3B, 7B, and 13B sizes, but excludes models larger than 13B and law-customized models.The three commercial models are GPT-3.5, GPT-4, and Claude-1; their architectures and training processes are unknown, and some benchmark tasks may have appeared in pretraining data.Source: §5 · p. 12, §5.1.1 · pp. 12–13

5.1.2 Prompts; 5.1.3 Evaluation

  • Prompts used manually written task instructions with zero to eight demonstrations; classification and extraction used task-specific metrics, while law-trained evaluators graded rule-application explanations for three commercial models.Claude prompts followed Anthropic formatting guidance, and the authors describe their prompts as a lower bound because refinement could improve performance.Source: §5.1.2 · p. 13, §5.1.3 · p. 13

5.2 Performance trends

  • Performance varies widely by task; larger models usually outperform smaller ones within families, but gaps differ by family and reasoning category.Flan-T5 models lead at both the 3B and 13B scales, while Vicuna models underperform competitors at 7B and 13B; open-source Flan-T5-XXL also exceeds GPT-3.5 and Claude-1 on issue-spotting and rhetorical-understanding.Source: §5.2 · pp. 13–15, Table 2 · p. 14

5.3.1 Issue-spotting; 5.3.2 Rule-recall

  • GPT-4 leads GPT-3.5 and Claude-1 in issue-spotting, while Claude-1 nearly matches GPT-4 and beats GPT-3.5 on rule-recall.Issue-spotting averages are skewed because 16 of 17 tasks come from Learned Hands; Claude-1 beats GPT-3.5 on the non-Learned Hands corporate_lobbying task and outperforms GPT-4 on three rule-recall tasks.Source: §5.3.1 · p. 15, §5.3.2 · p. 15

5.3.3 Rule-application

  • GPT-4 outperforms GPT-3.5 and Claude-1 on rule-application correctness and analysis, with errors including faulty arithmetic, misapplied rule citations, predictions without reasoning, and fact-rule restatements.Performance varies with task difficulty: diversity_jurisdiction_1 is easier than successor_liability, which requires identifying multiple exceptions in a complex transaction.Source: §5.3.3 · pp. 15–16

5.3.4 Rule-conclusion

  • GPT-4 outperforms GPT-3.5 on rule-conclusion, while Claude-1 and GPT-3.5 are approximately level; slice analyses locate GPT-4 gains in particular hearsay and trademark categories.On hearsay, gains appear concentrated in non-verbal hearsay and statements made in court; on abercrombie, gains appear strongest for suggestive or arbitrary marks, while descriptive-mark performance is consistent. The small slices offer intuition, not rigorous statistical evidence.Source: §5.3.4 · pp. 16–17, Table 6 · p. 17, Table 7 · p. 18

5.3.5 Interpretation

  • On interpretation, GPT-4 outperforms GPT-3.5, which outperforms Claude-1; all three reach balanced-accuracy ≥88% across 38 CUAD tasks, but GPT-4 records 47.8% accuracy on MAUD.On proa, GPT-4 and GPT-3.5 each reach balanced-accuracy ≥90%; on long Supply Chain Disclosure texts, large commercial models average balanced-accuracy between 74-75%.Source: §5.3.5 · p. 17

5.3.6 Rhetorical-analysis

  • GPT-4 outperforms GPT-3.5 and Claude-1 on rhetorical-understanding, while Incite-Base nearly matches GPT-4 on definition_extraction at 80.6% versus 81.8%.Nearly all evaluated models struggle with two tasks that classify legal roles into six or more categories.Source: §5.3.6 · p. 18

5.4 Prompt engineering strategies

  • Description prompts appeared to improve performance significantly on abercrombie, ucc_v_common_law, diversity_2, and diversity_4, but results were approximately unchanged or worse on other tasks.The authors suggest that difficult fact patterns may exceed the guidance provided by descriptions, or that improvements may occur on rules less frequent in pretraining data.Source: §5.4 · p. 18, Figure 1 · p. 19
  • For GPT-3.5, plain-language prompts significantly outperformed technical prompts on four of five tasks, by up to 21 balanced-accuracy points; contract_nli_permissible_post-agreement_possession showed the opposite pattern.The experiments compared plain-language prompts with prompts using technical legal language.Source: Figure 2 · p. 19, §5.4 · p. 19
  • Across eight binary classification tasks, best–worst prompt differences were significant (p < 0.01) for Incite-Instruct on all tasks and GPT-3.5 except opp115_third_party_sharing_collection and overruling; on overruling, Incite-Instruct improved by over 20 points (balanced-accuracy).Both models were evaluated with five prompts containing randomly selected, class-balanced demonstrations.Source: §5.4 · pp. 19–20, Figure 3 · p. 20

6 Conclusion

The paper presents LEGALBENCH as a collaboratively constructed benchmark of 162 tasks for measuring LLM legal reasoning and hopes to expand it by collecting further tasks from the legal community.

6 Conclusion

  • LEGALBENCH comprises 162 collaboratively constructed tasks for measuring LLM legal reasoning, with future work aimed at gathering more useful and interesting tasks from the legal community.Source: §6 · p. 20

B Limitations and social impact … C.4 Preprocessing, cleaning, labeling

The authors describe LegalBench as an incomplete benchmark with limits in coverage and warn against treating its scores as sufficient grounds for legal automation. The datasheet summarizes the dataset and directs readers to task-specific documentation for further detail.

B Limitations and social impact

  • LegalBench omits long-document, analogical, subjective or ambiguous, multilingual, and non-US-Federal tasks, while overrepresenting some domains and leaving others unrepresented.The authors hope to broaden its jurisdictional and domain coverage in future work.Source: B Limitations and social impact · p. 28
  • The authors caution that benchmark performance alone does not endorse legal automation or justify deployment; legal AI requires in-domain evaluation and ethical and legal compliance assessments.LegalBench is intended to help stakeholders evaluate systems, not replace context-specific assessment; potential harms also depend on the task and social context.Source: B Limitations and social impact · p. 28

C.3 Collection process

  • People represented in source documents did not separately consent to inclusion, and the authors report no data-protection impact analysis; deployments still require in-domain evaluation and ethical and legal compliance assessments.The authors note that the documents are generally public and accessible through judicial data services.Source: B Limitations and social impact · p. 28, §C.3 · p. 31

C Datasheet

  • The datasheet presents general answers about LegalBench, while Appendix F provides more detailed information for individual tasks.LegalBench was created to evaluate LLMs on legal tasks and better understand their legal reasoning capabilities.Source: C Datasheet · p. 29, §C.1 · p. 29

C.1 Motivation

  • LegalBench addresses whether technical adaptations can support legal tasks involving long sequences, jargon, and multistep reasoning, and which tasks LLMs can perform safely and reliably.The benchmark’s task specifications and expected outputs are provided in the full task descriptions in Section F.Source: §C.1 · p. 29

C.2 Composition

  • Instances are text ranging from sentences to documents, drawn from real-world sources or synthetically crafted, and targets include classes, extracted entities, and open-ended generation.Tasks largely use unprocessed text.Source: §C.2 · pp. 29–30
  • Representative coverage is constrained because some legal documents are unavailable and expert annotation is costly; scanned legal data can also contain OCR artifacts.The paper notes that annotating 500 contracts was estimated to cost $2 million US dollars.Source: §C.2 · pp. 29–30
  • Training splits contain only 2–8 examples, reflecting a few-shot setting in which practitioners have access to a handful of labeled instances.Tasks are split into training and test sets.Source: §C.2 · p. 30

C.5 Use · C.6 Distribution · C.7 Maintenance

LEGALBENCH is intended for LLM evaluation and fine-tuning, but its results provide only a preliminary understanding and should not guide real-world legal predictions or advice. Distribution follows the source licenses for transformed tasks, and maintenance plans include community contributions and error correction.

C.5 Use

  • The constructed datasets have been used to evaluate several LLMs, and proposed further uses include evaluation and fine-tuning, including self-instruct-style generations.Source: §C.5 · p. 32
  • LEGALBENCH offers only a preliminary understanding of LLM performance, so practitioners should collect and validate data for their own use case.Its tasks do not generalize to all legal reasoning tasks or legal documents.Source: §C.5 · p. 32
  • LEGALBENCH should not be used to predict real-world legality or lawsuit outcomes, or as legal advice.Source: §C.5 · p. 32

C.6 Distribution

  • Adapted or transformed tasks are released under the original dataset’s license, with task-specific licenses listed in Table 8 and source references in Section F.Source: §C.6 · p. 32

C.7 Maintenance

  • Neel Guha will support the dataset, older versions will be available by email request, and any errata found so far will be made available on the website.The authors report that they have not currently found any errata.Source: §C.7 · pp. 32–33
  • LEGALBENCH will be updated for new legal-community contributions and identified data errors, with updates publicized as soon as possible under Neel Guha’s management.Source: §C.7 · p. 33
  • The legal community is encouraged to contribute tasks, and procedures for reviewing, validating, and incorporating submissions are being formalized.Many tasks use permissive licenses, allowing other researchers to modify them.Source: §C.7 · p. 33

D Task overview … Publication status Tasks

The task overview documents dataset licensing, publication availability, reasoning-type mappings, and task-level size statistics. It also classifies task availability to address concerns that web-trained LLMs may have encountered benchmark data during training.

D.1 Licenses

  • LEGALBENCH tasks have licenses that vary with contributor choices or the original data’s license, and the authors accept responsibility for rights violations and confirm dataset licenses.Source: §D.1 · p. 34

License Tasks

D.2 Public availability status

  • Because many commercial LLMs may be trained on web content and their training data is often unclear, the paper groups tasks as previously published, original but available, or original and unavailable.The categories distinguish tasks derived from other works and available online, project-created tasks previously available online, and project-created tasks not released online.Source: §D.2 · p. 34

D.3 Reasoning type

  • Tables 10 and 11 organize tasks by LEGALBENCH and NLP reasoning types, respectively, and give similar general-domain NLP benchmarks as examples.Individual task descriptions in Appendix F provide more information about the reasoning each task requires.Source: §D.3 · p. 34

D.4 Task statistics

  • Task statistics report sample counts and average input lengths, with inputs ranging from a single sentence to two pages.Source: §D.4 · p. 34

Publication status Tasks

LEGALBENCH reasoning type Tasks · NLP reasoning type Tasks

The section organizes LegalBench tasks by legal and NLP reasoning types, and describes dataset sizes, input-text lengths, and related general-domain NLP benchmarks.

LEGALBENCH reasoning type Tasks

NLP reasoning type Tasks

  • Table 11 pairs each reasoning type with examples of similar general-domain NLP benchmarks.Source: Table 11 · p. 41
  • Appendix F provides individual task descriptions for more information about the reasoning each task requires.Source: Table 11 · p. 41
  • Figure 4 presents LegalBench task sizes and input-text lengths.Source: Figure 4 · p. 41
  • Table 12 reports the number of LegalBench tasks at different dataset sizes.Source: Table 12 · p. 42

E Evaluation … F.3 Citation Prediction Tasks

The sections define task-specific evaluation protocols and describe benchmark tasks, their construction, and their intended value. The task descriptions span rule application, classification, extraction, outcome identification, and citation prediction.

E Evaluation

  • Rule-application generations are scored for correctness and sufficient factual inference, with law-trained expert review and task-specific answer guides standardizing evaluation.Incorrect explanations automatically fail the analysis measure.Source: E Evaluation · p. 43
  • Generation tasks use task-specific scoring: rule_qa is checked against an answer key, party extraction and successor liability use F1, and tax estimates count as accurate within 10%.Definition extraction uses accuracy and permissible-term rates, comparing stemmed answers to account for tense.Source: E Evaluation · p. 43
  • Citation-name prediction checks for the correct case name, while classification tasks use exact-match class-balanced accuracy to address class imbalance.The Securities Complaint Extraction task also uses F1, as noted above.Source: E Evaluation · p. 43

F Task descriptions

  • The task-description section provides detailed accounts of each family of tasks in LEGALBENCH.The following descriptions cover Abercrombie, Canada Tax Court Outcomes, and Citation Prediction tasks.Source: F Task descriptions · p. 44

F.1 Abercrombie

  • Abercrombie asks models to classify a mark into one of five distinctiveness categories for a product or service, using a manually constructed, unambiguous dataset of 95 test samples.The task measures reasoning progress rather than a capability expected to change lawyers’ IP practice; one sample per category is used for training.Source: §F.1 · pp. 44–45

F.2 Canada Tax Court Outcomes

  • Canada Tax Court Outcomes classifies excerpts as allowed, dismissed, or other, distinguishing appeal outcomes from partial issues and avoiding inference when the excerpt omits the outcome.The dataset uses English decisions from 2001–2022 and manually reviewed 250 excerpts; outcome extraction could support research and judicial analytics where human review is costly.Source: §F.2 · p. 46

F.3 Citation Prediction Tasks

  • Citation Prediction tests whether a case supports a legal proposition or asks a model to name a supporting case, using quotation-containing sentences with a single citation.The authors collected 55 sentences from post-2023 circuit opinions; classification creates positive and randomly paired negative samples, while accurate prediction could improve access to law and support legal statements with authority.Source: §F.3 · pp. 48–49

F.4 Clause Classification Tasks

LegalBench’s clause-classification tasks cover varied clause categories and document types, capturing a practically useful current-day LLM use case. They address the costly, consequential work of reviewing legal agreements for potentially problematic terms.

F.4 Clause Classification Tasks

  • Tasks range from Contract QA examples and J.Crew-blocker detection to 38 CUAD binary categories in EDGAR contracts [62] and eight ToS types, seven potentially unfair under European law [82].Contract QA asks whether a clause exemplifies a named type; J.Crew blocker classification uses loan-agreement clauses.Source: §F.4 · p. 50
  • Legal document review helps parties identify problematic terms, but its expense [62] and individuals’ limited access to lawyers raise access-to-justice concerns.Firms may inspect signed agreements after changing events; the COVID-19 pandemic prompted searches for force majeure clauses, while agreements can contain terms a court would invalidate.Source: §F.4 · p. 50
  • Clause classification is among LegalBench’s most practically useful tasks, spanning clause categories and source documents because complexity depends on both.The tasks capture a current-day use case for LLMs.Source: §F.4 · p. 50

F.4.1 CUAD Tasks … F.5 Consumer Contracts QA

This section describes clause-classification and contract-question-answering tasks, including CUAD clause categories and a J.Crew blocker task on restrictions on intellectual-property transfers. It covers their dataset designs, legal scope, and potential relevance to consumers and market participants.

F.4.1 CUAD Tasks

  • The CUAD adaptation selects 38 of 41 clause categories and creates a balanced binary classification task for each using category clauses and an equal number sampled from other categories.The source dataset contains 500 contracts; prior work identified contract length and limited training data as principal difficulties [62].Source: §F.4.1 · p. 50

F.4.2 J.Crew Blocker

  • The J.Crew blocker arose after J.Crew moved valuable IP into an unrestricted subsidiary in 2016, which then used the assets as collateral for new financing.The task is denoted jcrew-blocker in LEGALBENCH.Source: §F.4.2 · pp. 54–55
  • The J.Crew blocker task classifies loan provisions as blockers if they prohibit transfers of IP assets to unrestricted subsidiaries, require lender consent for IP transfers to subsidiaries, or both.The provision aims to keep material assets within lenders’ collateral pools and protect lenders from unexpected IP transfers.Source: §F.4.2 · p. 55
  • Legal experts labeled 60 clauses from public loan agreements as J.Crew blockers or not, testing interpretation of secured-lending and intellectual-property provisions.Identifying these provisions can help lenders assess protections and borrowers understand restrictions on their activities.Source: §F.4.2 · p. 55

F.4.3 Unfair Terms of Service

  • Unfair Terms of Service recasts the source task’s fair/unfair judgment as 8-way classification across the categories identified in [82], using data from [21].The task tests multiclass clause classification on a highly imbalanced dataset.Source: §F.4.3 · p. 57

F.4.4 Contract QA

  • Contract QA tests whether an LLM recognizes 22 provision types not described in the prompt, using two example and two non-example clauses per type.The manually extracted clauses come from sample agreements contributed by a LegalTech vendor and public sources.Source: §F.4.4 · pp. 58–59

F.5 Consumer Contracts QA

  • Consumer Contracts QA evaluates understanding of website terms through 200 yes/no questions first introduced in and alternate, less-readable wordings designed to probe brittleness.An attorney drafted the questions for terms of service from the 20 most-visited U.S. websites; the task is a preliminary evaluation, and errors could hinder consumers’ understanding and exercise of contractual rights.Source: §F.5 · pp. 60–61

F.6 Contract NLI Tasks … F.15 Legal Reasoning Causality

This section describes benchmark tasks spanning contract and statutory reasoning, judicial opinions and evidence, legal rules, and issue spotting in informal narratives. It covers their construction and potential uses in legal services, litigation, and research on judicial decision-making.

F.6 Contract NLI Tasks

  • Contract NLI tests whether contract excerpts support assertions about legal effects, assessing reasoning over contractual rights and obligations.The tasks transform a dataset of 607 contracts and 17 assertions: each assertion becomes a separate Yes/No task, with supportive excerpts as positive instances and contradicted or randomly sampled excerpts as unsupportive instances.Source: §F.6 · pp. 62, 64

F.7 Corporate Lobbying

  • Corporate Lobbying asks whether a proposed Congressional bill may be relevant to a company based on its SEC 10K self-description.The task requires identifying the bill’s legal consequences and whether they matter to the company’s business model, structure, or activities; the data was manually labeled.Source: §F.7 · p. 65

F.8 Definition Tasks

  • Definition tasks classify whether Supreme Court excerpts define a term or extract the defined term, despite imperfect surface cues such as quotation marks or dictionary citations.The classification dataset contains 691 defining and 646 non-defining sentences; extraction is open-ended, with hundreds of possible answers across items.Source: §F.8 · pp. 67–68

F.9 Diversity Jurisdiction

  • Six diversity-jurisdiction tasks test different fact patterns involving complete diversity, the amount-in-controversy threshold, and claim aggregation.The dataset is programmatically generated with randomized party names, claims, and amounts; the task requires identifying party relationships, permissible aggregation, and whether the threshold is met.Source: §F.9 · p. 69

F.10 Function of Decision Section

  • Function of Decision Section concerns extracting legal principles from judicial decisions and applying them to other cases, reflecting decisions’ roles in resolving disputes and creating binding law.The paper describes this reading-and-digesting skill as one lawyers in common-law systems develop through time and practice.Source: §F.10 · p. 71
  • The Function of Decision Sections task classifies a decision paragraph into one of seven functions, using sequential Federal Reporter excerpts and measuring a foundational reasoning skill rather than a task expected in legal practice.The dataset spans civil and criminal decisions within federal appellate jurisdiction.Source: §F.10 · p. 71

Conclusion

  • Table 34 provides examples for the function_of_decision_section task.Source: Table 34 · p. 72

F.11 Hearsay

  • The hearsay task asks whether prospective evidence meets the hearsay test; its 95 test samples span five rule dimensions, exclude hearsay exceptions, and probe multi-step reasoning.The dimensions include in-court statements, non-assertive conduct, standard and non-verbal hearsay, and statements not introduced to prove their truth.Source: §F.11 · p. 73

F.12 Insurance Policy Interpretation

  • Insurance Policy Interpretation classifies policy-claim pairs as clearly covered, clearly not covered, or unclear, assigning labels from crowd responses using a 95% multinomial confidence interval.A label is selected when its interval lower bound is at least .5; otherwise the pair is labeled ambiguous. The crowdsourced ground truth represents lay legal interpretations.Source: §F.12 · p. 75

F.13 International Citizenship Questions

  • International Citizenship Questions converts the GLOBALCIT dataset into 9,300 yes/no question-answer pairs about citizenship laws in 190 countries as of 2020, testing rule retrieval and reasoning.A script adds country and survey-time information to questions generated from the tabular data.Source: §F.13 · p. 76

F.14 Learned Hand Tasks

  • Legal Issue-spotting requires identifying one or more applicable legal categories from a person’s informal narrative, reflecting how professionals triage potential clients.The task uses narrative descriptions rather than concise, legally structured accounts.Source: §F.14 · p. 78
  • Learned Hands crowdsources legal-issue labels for r/legaladvice narratives, supporting issue-spotting and triage to relevant services, resources, and procedures.Law students and lawyers answer yes-no-skip questions; labels are finalized after sufficient consistent votes, and narratives may receive multiple labels. The task could support online legal-help tools or intake by paralegals and justice workers.Source: §F.14 · p. 78

F.15 Legal Reasoning Causality

  • The causality task classifies whether a court relied on statistical or direct-probative evidence to link a plaintiff’s protected trait to an allegedly discriminatory decision.It requires distinguishing statistical terminology, such as regression and correlation, that substantiates causality from language related to other legal components. The dataset contains 59 excerpts from US federal district court labor-discrimination decisions, coded by evidence type; two random samples from each class were used for training.Source: §F.15 · p. 79
  • Identifying the evidence courts use to find causation could help lawyers assess judges’ reasoning and enable large-scale study of judicial opinions.The passage notes that legal databases typically treat reasoning as a black box, and that understanding judges’ evidence preferences can inform litigation strategy.Source: §F.15 · p. 79

F.16 MAUD Tasks … F.29 Successor Liability

This section describes LegalBench tasks that probe legal reasoning across merger-agreement clauses, judicial and lawyer ethics, privacy policies, oral advocacy, and core legal doctrines. The tasks cover clause interpretation, rule recall and statutory reasoning, case-holding relevance, complaint extraction, and successor-liability issue spotting.

F.16 MAUD Tasks

  • MAUD contributes over 37,000 expert-annotated legal reading-comprehension examples based on the ABA’s 2021 Public Target Deal Point Study, and its multiple-choice format assesses interpretation of specialized legal language.Lawyers identify deal points in merger agreements and answer questions on topics such as closing conditions, material adverse effect, and breach remedies.Source: §F.16 · p. 81
  • Each task presents a key legal clause and descriptions, requiring the model to choose the option that best describes the clause.Source: §F.16 · p. 81
  • The tasks transform an abridged release of 14,928 examples from 94 merger agreements, filtering 92 multiple-choice questions down to 57 with at least 50 examples each.Each retained example includes a deal-point text, question, options, and answer key; questions are translated into human-readable prompts.Source: §F.16 · p. 81
  • The authors use abridged examples to assess legal reading comprehension rather than text extraction, since complete deal-point inputs can exceed average open-source LLM context limits.Source: §F.16 · p. 81
  • MAUD tasks assess LLMs’ ability to understand and categorize a wide range of legal clauses in merger agreements.The paper identifies contract reading comprehension as challenging for both humans and machines.Source: §F.16 · p. 92

F.17 New York State Judicial Ethics

  • The judicial-ethics task reformulates 300 real-world scenarios and fact patterns into yes-or-no questions about ethical rules and their application.The task uses rulings from the New York State Advisory Committee on Judicial Ethics.Source: §F.17 · p. 93
  • The task collects ethics digest statements from 2010, 2021, 2022, and 2023 and uses ChatGPT to reformulate them as yes-or-no questions.The authors leave out 2010 and 2021 data to study potential leakage from opinions available online; they say 2022 and 2023 data should have been unseen by most models trained before those years.Source: §F.17 · p. 93

F.18 OPP-115 Tasks

  • OPP-115 tasks classify privacy-policy text into nine specific data-practice categories, using a separate binary task for each category and excluding the broad “other” category.The source corpus contains 115 policies annotated by law graduate students, and texts shorter than 10 words are excluded.Source: §F.18 · p. 94

F.19 Purpose of Oral Argument Questions

  • Oral Argument Questions asks models to classify a judge’s transcript question by its purpose, using Supreme Court questions sampled to include rarer types rather than at random.The seven functions include background, clarification, implications, support, criticism, communication, and humor; the task provides only text, not audible tone or full case context.Source: §F.19 · p. 96

F.20 Overruling

  • The Overruling task identifies whether judicial text overrules a prior decision, a capability that could help lawyers check whether cited cases remain good law.The task is taken from prior work on fine-tuned BERT models.Source: §F.20 · p. 98

F.21 Personal Jurisdiction

  • The personal-jurisdiction task asks whether a court has jurisdiction under a simplified rule, while limiting its fact patterns to individuals and excluding complex doctrines such as stream-of-commerce problems.Its manually constructed dataset uses five base patterns and four slices covering domicile, insufficient contacts, contacts without nexus, and contacts with nexus.Source: §F.21 · p. 99

F.22 Privacy Policy Entailment

  • Privacy Policy Entailment tests whether a policy clause performs a described practice, framing the clause as a premise and the practice description as a hypothesis.Derived from APP-350, the dataset represents instances as clause-description-label triples and covers 57 unique policy descriptions.Source: §F.22 · p. 101

F.23 Privacy Policy QA

  • Privacy Policy QA tests whether policy excerpts are relevant to questions; its 10,923 pairs are split nearly evenly between relevant and irrelevant, supporting evaluation of long-document filtering.The task removes snippets under 10 words and can measure the segment-filtering step in systems that process long policies in chunks.Source: §F.23 · p. 102

F.24 Private Right of Action (PROA)

  • The PROA task classifies whether a statutory clause creates a private right of action, requiring the model to determine both whether a claim can be brought and whether the claimant is private.The hand-built dataset has 50 positive and 50 negative clauses, with 5 training and 95 test examples; accurate statutory annotation could enable larger empirical studies.Source: §F.24 · p. 103

F.25 Rule QA

  • Rule QA is an open-generation task that evaluates answers to questions about legal rules drawn from subjects typically studied in the first year of law school.The rules may come from cases, statutes, or other sources.Source: §F.25 · p. 104
  • Rule QA comprises 50 manually written questions asking models to restate legal rules, identify where rules are codified, or list rule factors, sometimes with a specified jurisdiction.The task is an initial probe of legal hallucination using basic questions that law students in the relevant courses are expected to answer nearly perfectly.Source: §F.25 · p. 104

F.26 SARA Tasks

  • SARA tests statutory reasoning through binary entailment and tax-calculation tasks based on nine simplified federal tax-law sections and hand-crafted cases.The entailment prompt supplies the statute specified by the case; numeric cases use the full statutes, which may exceed the context window. The dataset was introduced by [65], building on statutory reasoning described by.Source: §F.26 · p. 105

F.27 SCALR

  • SCALR uses 571 multiple-choice questions to test whether a model can match a Supreme Court question presented with the responsive holding parenthetical.It pairs questions with plausible alternatives selected by TF-IDF similarity; the task is intended to test legal relevance beyond word- or synonym-level matching.Source: §F.27 · pp. 106–107

F.28 Securities Complaint Extraction

  • Securities Complaint Extraction asks models to identify named plaintiffs, individual defendants, and corporate defendants in complaint text.Some excerpts omit entity names or use shortened names, so the task may require supplying an official corporate name or returning “Not named”; the source text is limited to 2,000 characters or less and can omit entities named earlier in a complaint.Source: §F.28 · pp. 108–110

F.29 Successor Liability

  • The Successor Liability task asks models to spot the issue and identify each applicable exception to the general rule that asset purchasers are not liable for sellers’ debts.The exceptions are express assumption, fraudulent conveyance, de facto merger, and mere continuation; the task excludes the continuity-of-enterprise and product-line variations of mere continuation.Source: §F.29 · p. 111

F.30 Supply Chain Disclosure Tasks … G.2 Prompts

The sections describe four legal-reasoning tasks, including their scope and construction, and provide links to studied open-source models and prompts for the experiments.

F.30 Supply Chain Disclosure Tasks

  • The CTSCA tasks assess whether qualifying retailers and manufacturers disclose supply-chain practices across five required topics.The topics cover verification, audits, supplier certification, accountability standards, and training.Source: §F.30 · p. 113
  • Researchers collected disclosures for roughly 400 of 500 firms and converted coding of five topics and best-practice consistency into 10 binary classification tasks.The tasks measure whether LLMs can parse disclosures that may be differently formatted, complex, or designed to obscure relevant information.Source: §F.30 · p. 113

F.31 Telemarketing Sales Rule

  • The TSR task classifies clear fact patterns as violations or nonviolations, using 50 manually written examples across relevant provisions.The task omits 16 C.F.R. § 310.3(a)(2)(vi) and (viii), given their complexity or references to other laws.Source: §F.31 · p. 115

F.32 Textualism Tasks

  • Textualism tasks detect plain-meaning or dictionary tools in appellate paragraphs, requiring evidence that the court actually relied on the tool rather than merely discussing it.Annotations distinguish use in a rule of decision or application to facts from criticism of the interpretive tool.Source: §F.32 · p. 116

F.33 UCC vs Common Law

  • The UCC-versus-common-law task classifies contracts using 100 simple descriptions and excludes mixed-purpose contracts.The task is framed as a measure of legal reasoning relative to lawyers and as a potential aid to contract lifecycle management.Source: §F.33 · p. 119

G.1 Models; G.2 Prompts

G.3 Results · 2 Citation Prediction Tasks · 9 Unfair Terms of Service

This section organizes LegalBench results by model type and task, reports in-context demonstration counts, and presents performance tables. It includes 3B-model results on interpretation tasks across contract, MAUD, privacy, supply-chain, and unfair_tos benchmarks.

G.3 Results

  • Results are reported for each LLM on each task, with models grouped as commercial, 13B, 7B, or 3B and names abbreviated to family names.Source: §G.3 · p. 120

2 Citation Prediction Tasks

  • The listing groups tasks by number of in-context demonstrations, with entries shown for 0, 3, 4, 5, and 6 demonstrations; the task sets differ across these listed counts.Source: §G.3 · p. 121, §2 · p. 121

9 Unfair Terms of Service

  • Tables 59–63 report commercial-model rule-application performance and rule-recall results for commercial, 13B, 7B, and 3B models.Source: Table 59 · p. 121, Table 60 · p. 122, Table 61 · p. 122
  • Tables 64–67 report issue-spotting task results for commercial, 13B, 7B, and 3B models, while Tables 68–69 report rule-conclusion task results for commercial and 13B models.Source: Table 64 · p. 123, Table 65 · p. 123, Table 66 · p. 124
  • Tables 72–75 report rhetorical-understanding results for commercial, 13B, 7B, and 3B models, while Tables 76–77 report interpretation results for commercial and 13B models.Source: Table 72 · p. 127, Table 73 · p. 128, Table 74 · p. 128
  • The unfair_tos row reports scores of 9.0, 8.2, 4.0, and 15.3 across four 3B-model columns.The supplied passages do not identify the columns’ model names.Source: Table 79 · p. 140, §9 · p. 143
  • MAUD results include 6.3, 0.5, 12.9, and 13.3 for the fiduciary-exception board-determination standard, and 13.5, 0.0, 18.0, and 17.0 for the initial matching-rights period (COR).The table also reports 18.9, 0.0, 19.6, and 19.4 for the initial matching-rights period (FTR).Source: Table 79 · p. 140, §9 · p. 142
  • For Opp115, first-party collection/use scores are 60.9, 52.3, 69.5, and 56.1, while user access, edit, and deletion scores are 49.2, 82.3, 66.4, and 53.5.International and specific audiences scores are 59.9, 70.5, 51.8, and 57.0; third-party sharing/collection scores are 55.6, 63.3, 64.8, and 47.0.Source: Table 79 · p. 140, §9 · p. 143
  • Supply-chain disclosure scores for training are 47.9, 87.2, 58.8, and 16.9, and for verification are 44.1, 66.1, 55.3, and 0.2.Best-practice training scores are 44.9, 83.4, 64.2, and 41.0; best-practice verification scores are 47.7, 54.9, 53.1, and 21.8.Source: Table 79 · p. 140, §9 · p. 143

Related papers

Loading 2308.11462v1…