Source-linked AI summary
DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services
Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, Zhongyu Wei
TL;DR
Domain-specific legal systems need both specialized reasoning and access to precise, current legal knowledge. DISC-LawLLM addresses these needs through legal-syllogism fine-tuning, retrieval augmentation, and a benchmark spanning objective and subjective evaluation, outperforming existing legal LLMs in reported tests.
Problem
Legal assistance requires intricate legal-syllogism reasoning and precise external legal knowledge, while fine-tuned LLMs can still hallucinate or use outdated knowledge.
Method
DISC-LawLLM uses legal-syllogism prompting to construct Chinese Judicial supervised fine-tuning data, adds retrieval of legal evidence, and evaluates systems with DISC-Law-Eval.
Results
DISC-LawLLM outperforms existing legal LLMs on DISC-Law-Eval and improves objective-evaluation accuracy by an average of 7% over GPT-3.5-Turbo.
Takeaways & Limitations
The system provides legal reasoning and knowledge-retrieval capabilities for diverse legal-service scenarios, with performance assessed objectively and subjectively.
Takeaways & Limitations
The reported implementation uses Baichuan as its base model, although the strategy is stated to apply to decoder-only foundation models.
Abstract
from arXiv · showhide
We propose DISC-LawLLM, an intelligent legal system utilizing large language models (LLMs) to provide a wide range of legal services. We adopt legal syllogism prompting strategies to construct supervised fine-tuning datasets in the Chinese Judicial domain and fine-tune LLMs with legal reasoning capability. We augment LLMs with a retrieval module to enhance models' ability to access and utilize external legal knowledge. A comprehensive legal benchmark, DISC-Law-Eval, is presented to evaluate intelligent legal systems from both objective and subjective dimensions. Quantitative and qualitative results on DISC-Law-Eval demonstrate the effectiveness of our system in serving various users across diverse legal scenarios. The detailed resources are available at https://github.com/FudanDISC/DISC-LawLLM.
1 Introduction
DISC-LawLLM targets diverse legal users and scenarios by combining legal-syllogism reasoning, external legal-knowledge retrieval, and comprehensive evaluation.
- Motivation: LegalAI supports professionals, the general public, and students across tasks including information extraction, case retrieval, judgment prediction, and legal question answering.These applications can reduce paperwork, improve access to legal services, and support legal learning.
- Challenges: Legal assistance requires legal-syllogism reasoning that connects a legal proposition, relevant facts, and a judgment.The system must also retrieve precise external legal knowledge to support interpretation and alleviate hallucination.
- Approach: DISC-LawLLM uses legal-syllogism prompting to construct Chinese Judicial supervised fine-tuning data and train legal reasoning capability.The approach builds on a general-domain Chinese LLM with 13B parameters.
- Approach: A retrieval module sources up-to-date and precise legal evidence for the model to use alongside user input.Retrieved statutes function as the major premise, while question facts provide the minor premise for inference.
- Evaluation: DISC-Law-Eval assesses intelligent legal systems through objective and subjective dimensions.It combines graded multiple-choice legal questions with GPT-3.5 arbitration of legal answers using Accuracy, Completeness, and Clarity.
- Results: DISC-LawLLM outperforms existing legal large language models and exceeds GPT-3.5-turbo on a majority of tested objective-evaluation subjects.The reported comparison highlights more extensive Chinese legal knowledge and legal reasoning.
2 Related Work
Earlier LegalAI work mainly addressed specific legal tasks with dedicated datasets and algorithms, while newer legal LLM efforts expanded dialogue capabilities.
- Task-specific LegalAI: Prior LegalAI research focused on specific tasks by constructing datasets and presenting machine-learning algorithms.This task-specific orientation limited broader intelligent legal-system applications.
- Legal LLMs: Existing legal LLMs were developed from Chinese foundation models using methods such as LoRA, P-tuning, fine-tuning, and self-instruction.Examples include LaWGPT, iLaw, LawGPT_zh, and ChatLaw.
- Positioning: The cited previous work largely emphasized dialogue competence, whereas DISC-LawLLM proposes an intelligent legal system with broader legal reasoning and retrieval capabilities.The distinction is framed against prior focus on dialogue as one intelligent-justice task.
3 DISC-Law-SFT Datasets
DISC-Law-SFT combines diverse Chinese legal data with pair and triplet instruction formats, using legal-syllogism-oriented reconstruction to teach reasoning and external-knowledge use.
- Dataset Structure: DISC-Law-SFT contains Pair and Triplet subsets, targeting legal reasoning and the use of external knowledge respectively.Triplets add a reference to the input-output format for retrieval-augmented training.
- Data Sources: Training examples come from public legal NLP datasets, Chinese legal raw text, and open-source instruction datasets.Raw text includes laws, regulations, cases, verdicts, consultations, and law-related examinations.
- Instruction Reconstruction: Rule-based cleaning first converts source material into input-output pairs, after which general LLMs reconstruct instructions to reduce rigidity, noise, and stylistic inconsistency.The reconstruction uses behavior shaping, knowledge expansion, and thinking development.
- Instruction Reconstruction: Law-specific chain-of-thought, termed LCoT, transforms inputs so models conduct legal syllogism when deriving answers.The legal syllogism maps law articles to the major premise, case facts to the minor premise, and judgment to the conclusion.
- Retrieval-Oriented Data: DISC-Law-SFT-Triplet is created by processing original data into inputs and outputs, then extracting reference information with heuristic rules.The resulting triplets support retrieval-augmented DISC-LawLLM training.
- Task Coverage: The dataset covers more than 10 legal tasks, including element extraction, case matching, judgment prediction, summarization, and question answering.Over 100k general instruction samples are also included to preserve foundational capability during legal-domain fine-tuning.
4 DISC-LawLLM
DISC-LawLLM is built through supervised fine-tuning and retrieval augmentation, pairing a Chinese base model with a dynamically updateable legal knowledge base.
- System Construction: DISC-LawLLM is formed through two steps: supervised fine-tuning and retrieval augmentation.The SFT stage equips the model with legal reasoning and judicial behavioral patterns.
- Retrieval Augmentation: Retrieval augmentation addresses hallucinations and outdated knowledge by supplying legal evidence relevant to user input.A knowledge base contains over 50 categories of Chinese laws, which are encoded as vectors for similarity-based retrieval.
- Retrieval Augmentation: Retrieved candidate documents and the user input are formatted together and fed into the model to support legal inference.The retrieval-oriented SFT triplets train the model to infer reliable results from retrieved references.
- Retrieval Augmentation: The legal knowledge base is designed for dynamic updates, allowing the system to use newly retrieved laws.This design supports deductions based on updated external knowledge.
5 DISC-Law-Eval Benchmark
DISC-Law-Eval is a comprehensive benchmark for intelligent legal systems, measuring legal knowledge and reasoning through objective and subjective evaluations.
- DISC-Law-Eval addresses the absence of an established comprehensive benchmark by evaluating intelligent legal systems from objective and subjective perspectives.
- Objective Evaluation: The objective evaluation uses single-answer and multiple-answer multiple-choice questions to quantitatively assess legal knowledge and reasoning.Accuracy is calculated by comparing extracted model answers with ground truth.
- Objective Evaluation: Objective questions are collected from Chinese legal examinations and contests and categorized as Easy, Normal, or Hard by content complexity and deduction difficulty.
- Subjective Evaluation: The subjective evaluation contains 300 manually constructed question-answer examples covering legal tools, consultations, and judgment prediction.
- Subjective Evaluation: GPT-3.5 rates subjective answers from 1 to 5 on accuracy, completeness, and clarity, using ground truth to reduce referee self-bias.
6 Experiments
DISC-LawLLM outperforms general and legal language models across most DISC-Law-Eval tests, with especially strong gains on difficult legal reasoning questions.
- Objective Evaluation: DISC-LawLLM surpasses nearly all competing LLMs across subjects and difficulty levels in objective evaluation.
- Objective Evaluation: 7% average accuracy improvement over GPT-3.5-Turbo is reported across most objective-evaluation subjects.
- Objective Evaluation: Over 50% improvement over top-performing GPT-3.5-Turbo is reported for multi-answer NJE and PAE questions.These higher-difficulty questions require more discerning judgment and reasoning.
- Subjective Evaluation: DISC-LawLLM achieves the best performance on most subjective-evaluation metrics and a 6% increase in average performance compared with ChatLaw.The evaluation concerns short-answer responses scored against ground truth.
7 Applications
DISC-LawLLM is demonstrated in legal professional tools, legal consultation, and examination assistance, spanning practitioners, the general public, and law students.
- Legal Professional Tools: The system supports legal event detection and judicial summarization for legal professionals.It extracts event triggers and types and generates judicial case summaries.
- Legal Professional Tools: These professional tools streamline judicial event monitoring, accelerate decision-making, and facilitate other intelligent legal tasks.
- Legal Consultation: DISC-LawLLM provides legal consultation for dispute resolution, including debt apportionment and agreement-drafting suggestions.
- Examination Assistant: For law students, the system predicts examination answers while explaining their rationale and provides detailed legal explanations.
- Retrieval results are demonstrated as an extension to additional application scenarios, positioning the system across diverse judicial use cases.
8 Conclusion
The paper introduces DISC-LawLLM as a legal service system combining legal reasoning, retrieval, and comprehensive evaluation, and reports superior benchmark performance to existing legal LLMs.
- DISC-LawLLM is trained with legal-syllogism responses reconstructed by ChatGPT from legal datasets, raw text, and open-source instruction data.
- An external retrieval module is incorporated to enhance the reliability of output responses.
- DISC-LawLLM outperforms existing legal LLMs on a benchmark assessing intelligent legal systems from subjective and objective perspectives.
- The paper releases its constructed datasets and model weights to support further research.
A Appendix: Supplementary Figures
The appendix presents DISC-LawLLM cases as legal professional tools, a legal consultation system, an examination assistant, and a retrieval-augmented system. It also includes legal case-analysis, multiple-choice, and employment-law consultation examples, including a response grounded in labor protections.
- A case-analysis example asks the system to identify event trigger words and their corresponding event types in a judicial case.
- DISC-LawLLM is illustrated as a legal professional tool, legal consultation system, and examination assistant.
- A multiple-choice example asks which attached civil lawsuit a court can accept among four legal scenarios.
- In an employment consultation example, the response states that an employer may not dismiss a pregnant employee for the stated reason and recommends labor arbitration or lawful compensation.
- A separate case presents Retrieval Augmented DISC-LawLLM.