PaperlayerView exact revision on arXiv

Source-linked AI summary

LawBench: Benchmarking Legal Knowledge of Large Language Models

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, Jidong Ge

arXiv:2309.16289v1cs.CLcs.AIcs.LG

TL;DR

Legal LLMs lack comprehensive evaluation for specialized legal tasks, including those under Chinese civil law. LawBench assesses 51 LLMs across 20 tasks organized by legal skill and task type; results place GPT-4 ahead, while legal fine-tuning offers some gains but models remain far from reliable legal assistance.

  • Problem

    Comprehensive evidence about LLM legal knowledge is lacking, and Chinese civil law’s emphasis on statutory law calls for separate evaluation tasks.

    Source: §1 · pp. 2–3

  • Method

    LawBench evaluates 20 legal tasks under Chinese civil law, organized by memorization, understanding, and applying, across five task types.

    Source: §1 · p. 3

  • Results

    Legal-specific fine-tuning brings some advances, but those models still lag far behind GPT-4, and current LLMs remain unable to provide meaningful judicial aid.

    Source: §1 · p. 3, §5 · p. 16

  • Takeaways & Limitations

    The authors present LawBench as a foundation for future research and call for institutional collaboration to develop high-quality legal-task LLMs.

    Source: §5 · p. 16

    Generative tasks are evaluated with Rouge-L for convenience, which cannot fully reflect human judgments of answer quality, and extraction rules can fail to match outputs.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in https://github.com/open-compass/LawBench/. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain.

1 Introduction

LawBench addresses the need to evaluate legal-task abilities with a benchmark tailored to Chinese civil law. Its evaluation finds that legal-specific fine-tuning usually helps, but general LLMs lead the zero-shot results.

1 Introduction

  • Legal tasks require specialist interpretation, and legal expertise could improve officers’ efficiency and public access to justice.The paper notes that legal work is primarily conducted by experts with years of specialized training.Source: §1 · p. 2
  • Chinese civil law emphasizes applying statutory articles rather than studying precedents, motivating a benchmark designed for that legal system.The paper contrasts this focus with common-law systems such as those in the U.S. and U.K.Source: §1 · p. 3
  • LawBench organizes 20 tasks into five task types and three levels: legal knowledge memorization, understanding, and applying.The benchmark assesses legal-related tasks under the Chinese civil-law system.Source: §1 · p. 3
  • Legal-specific fine-tuning usually improves base models, yet general LLMs occupy the top six spots in averaged zero-shot performance.Figure 1 presents the zero-shot results across 20 legal tests and three cognitive dimensions.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior LLM benchmarks cover general capabilities and some legal tasks, but a comprehensive evaluation of legal knowledge remains lacking, especially for Chinese law.

2 Related Work

  • A comprehensive comparison of instruction-following LLMs’ legal knowledge is lacking; the paper excludes small pretrained models that require task-specific fine-tuning.The authors focus on models that can perform corresponding tasks based on instructions.Source: §2 · p. 3
  • General benchmarks cover broad capabilities, while existing legal benchmarks address EU, American, South Korean, or American-law tasks rather than Chinese law.Some earlier legal benchmarks did not use instruction-following formats for LLMs.Source: §2 · p. 4

3 LawBench Construction

LawBench uses a Bloom-inspired taxonomy to select legal tasks, build datasets, and evaluate model answers. Its tasks span memorization, understanding, and application, with evaluation tailored to task outputs.

3 LawBench Construction

  • The construction section lays out LawBench’s design principles and test-task selection.It introduces the principles behind the benchmark before describing the taxonomy and tasks.Source: §3 · p. 4

3.1 The Hierarchical Ability Taxonomy of LawBench

  • LawBench simplifies Bloom’s hierarchy to memorization, understanding, and applying, assigning tasks by primary capability even when they involve others.The tasks also span five formats: generation, single-label classification, multi-label classification, regression, and extraction.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Data Source and Selected Tasks

  • The benchmark selects recent dataset versions and formats prompts to resemble realistic task instructions, while excluding long-document case retrieval for now.It includes 20 tasks and omits case retrieval because documents can exceed many LLMs’ length limits.Source: §3.2 · p. 5

Legal Knowledge Memorization Tasks

  • Memorization tasks test reciting law articles and answering basic legal-knowledge questions; article data include updated laws and regulations.The article task uses 152 sub-laws under five core laws; the question task uses single-label items from JEC-QA.Source: Legal Knowledge Memorization Tasks · p. 6

Legal Knowledge Understanding Tasks

  • Ten understanding tasks probe legal-text comprehension through proofreading, dispute identification, classification, reading comprehension, entity extraction, summarization, argument mining, and event or trigger-word extraction.The tasks draw on legal judgments, inquiries, and public legal-news reports, among other sources.Source: Legal Knowledge Understanding Tasks · pp. 6–7

Legal Knowledge Applying Tasks

  • Applying tasks simulate legal reasoning, including judgment prediction, case analysis, consultation, and criminal-damages calculation.Prison-term prediction compares providing an article number with providing the article’s content; the damages task tests rule understanding and numerical calculation.Source: Legal Knowledge Applying Tasks · pp. 7–9

3.3 Evaluation

  • Evaluation extracts answers from model outputs before computing task-specific metrics, with rules and metrics adapted to different output forms.Examples include soft-F1 for phrase extraction, nLog-distance for prison terms, and Rouge-L for several generation tasks.Source: §3.3 · pp. 9–10

4 Experiment

The experiments evaluate multilingual, Chinese-oriented, and legal-specific models under zero-shot and one-shot settings, finding strong performance from GPT-4 but substantial variation across tasks. Analyses examine model size, article-content use, and fine-tuning effects.

4.1 Models

  • The evaluated models span multilingual, Chinese-oriented, and legal-specific groups, with models classified by their main training corpora.The multilingual group includes 18 open-source models and two commercial models; the Chinese-oriented group includes 21 open-source models.Source: §4.1 · pp. 10–11, Table 2 · p. 11

4.2 Experiment Setting

  • Models were tested in zero-shot and one-shot settings, with one example appended to the instruction and query in the one-shot setting.Open-source models use greedy decoding; inputs over 2048 tokens are right-truncated, and output length is limited to 1024 tokens.Source: §4.2 · p. 12

4.3 Main Results

  • 52.35 (zero-shot) / 53.85 (one-shot): GPT-4’s average score; GPT-4 and ChatGPT lead the zero-shot benchmark, while one-shot performance surpasses zero-shot.The table reports top-performing systems by category and colors cells by abstention rate.Source: §4.3 · pp. 12–13, Table 3 · p. 13
  • Task scores vary substantially: top models exceed 60 on tasks 3-4 and 3-5, while none exceed 30 on tasks 1-1 and 2-1.Fuzi-Mingcha scores 97.59 on task 2-4, which the authors suspect may reflect data contamination.Source: §4.3 · p. 13

4.4 Analysis

  • Larger models typically improve performance and reduce abstention in one-shot settings, but zero-shot size effects are mixed and ChatLaw is an outlier.The authors note that simply increasing model size does not automatically improve zero-shot performance.Source: §4.4 · pp. 13–14, Figure 4 · p. 14
  • Adding related article content for tasks 3-4 and 3-5 yields no progress for most models on jail-sentence prediction and often lowers performance, including GPT-4.The authors identify effective use of retrieval information as an open problem.Source: Figure 5 · p. 14, §4.4 · p. 14
  • Supervised fine-tuning usually improves legal-task performance, whereas RLHF-trained models can refuse questions, increasing abstention and lowering scores.The authors suggest that stacking RLHF on SFT can be counterproductive for legal tasks.Source: §4.4 · pp. 14–15, Figure 6 · p. 15
  • Legal-specific fine-tuning improves scores and reduces abstention; instruction fine-tuning on a stronger foundation model can outperform continued pre-training followed by instruction tuning.The comparison finds LawGPT version 1.1, tuned on 350k instruction examples, outperformed version 1.0, which used 500k judgment documents followed by 300k instructions.Source: §4.4 · p. 15

5 Conclusion

LawBench evaluates LLMs on legal tasks under the Chinese civil-law system across 20 tasks and three cognitive dimensions. The results indicate that legal-specific fine-tuning helps, but current models remain far from reliable legal assistance and trail GPT-4.

5 Conclusion

  • LawBench organizes 20 Chinese civil-law legal tasks across memorization, understanding, and application dimensions.Its structured taxonomy addresses limited task coverage and benchmarks based on American laws in English.Source: §5 · p. 15
  • Legal-specific fine-tuning improves model scores and reduces abstention rates, yet open-source models still lag far behind GPT-4.The conclusion also reports poor scores on most tasks and says current LLMs remain unable to provide meaningful judicial aid.Source: §5 · p. 16, Figure 7 · p. 16
  • Developing high-quality legal LLMs requires collaboration among institutions because practical legal data are often confidential.The authors hope LawBench can provide a foundation for future research and encourage that cooperation.Source: §5 · p. 16

Limitations

The benchmark has two stated evaluation limitations: possible contamination from public internet data and imperfect answer extraction and evaluation for generative tasks.

Limitations

  • Internet-sampled datasets may create test-data leakage because LLMs have been extensively trained on internet data.Models trained on the task formats or exact test data could consequently receive exceptionally high scores.Source: Limitations · p. 16
  • Task-specific extraction rules can fail to match answers, while Rouge-L does not fully reflect human judgments of generative-answer quality.The authors note a lack of automated methods for evaluating legal aspects of model predictions.Source: Limitations · p. 16

A Details of Task Instruction

The appendix provides task instructions and examples for LawBench’s memorization, understanding, and application tasks, plus dataset statistics.

A.1 Legal Knowledge Memorization Tasks

  • Table 4 gives an instruction and example for Article Recitation, task 1-1.It appears under Legal Knowledge Memorization Tasks.Source: Table 4 · p. 23
  • Table 5 provides the instruction and an example for Task 1-2 Knowledge Question Answering, identifying the task as knowledge question answering numbered 1-2.Source: Table 5 · p. 23

A.2 Legal Knowledge Understanding Tasks

  • Table 6 gives the instruction and an example for Task 2-1, Document Proofread, while Table 7 gives the instruction and an example for Task 2-2, Dispute Focus Identification.Source: Table 6 · p. 24, Table 7 · p. 24
  • Tables 8–10 cover Marital Disputes Identification, Issue Topic Identification, and Reading Comprehension.Each table provides an instruction and an example for its task.Source: Table 8 · p. 25, Table 9 · p. 25, Table 10 · p. 25
  • Tables 11–13 provide task instructions and examples for Name Entity Recognition, Opinion Summarization, and Argument Mining.These tables are part of Legal Knowledge Understanding Tasks.Source: Table 11 · p. 26, Table 12 · p. 26, Table 13 · p. 26

A.3 Legal Knowledge Applying Tasks

  • Tables 14–17 provide instructions and examples for Task 3-1 Fact-based Article Prediction, Task 3-2 Scene-based Article Prediction, Task 3-3 Charge Prediction, and Task 3-4 Prison Term Prediction w.o Article.Source: Table 14 · p. 27, Table 15 · p. 27, Table 16 · p. 27
  • Tables 18–21 present instructions and examples for four tasks: Task 3-5 Prison Term Prediction w. Article, Task 3-6 Case Analysis, Task 3-7 Crime Amount Calculation, and Task 3-8 Consultation.Source: Table 18 · p. 28, Table 19 · p. 28, Table 20 · p. 29

A.4 Statics Information of Our Datasets

  • Table 22 reports dataset statistics, including instruction, input-question, and answer lengths.The table is listed in the appendix’s dataset statistics subsection.Source: Table 22 · p. 30

B Details of Results

The appendix lists results tables for legal-specific models, cognitive task groups, and overall performance in zero-shot and one-shot settings.

B Details of Results

Loading 2309.16289v1…