DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services

Motivation

• Prior LegalAI work often focuses on individual tasks, while fine-tuning general LLMs for simple legal question answering does not address the complexity of broader legal services.The paper presents LLM instruction-following ability across domains as potential for the legal sector.Source: §1 · p. 1

Application scope

• The system targets practitioners’ legal tools, public-facing consultation and dispute resolution, and tutoring and exam support for law students.The paper frames intelligent legal systems as assisting professionals, general populations, and students in distinct legal scenarios.Source: §1 · p. 1

Challenges

• Legal assistance requires syllogistic reasoning from legal propositions and facts to judgments, plus precise retrieval and inference from evolving external legal knowledge to support interpretation and alleviate hallucination.The paper identifies the major premise as the legal proposition, the minor premise as the factual proposition, and the conclusion as the judgment.Source: §1 · p. 2

System

• DISC-LawLLM uses legal-syllogism prompting to build Chinese Judicial DISC-Law-SFT data, fine-tunes a 13B-parameter Chinese LLM, and adds retrieval for up-to-date, precise legal evidence.The described version uses Baichuan as its base model; the authors state the strategy can apply to decoder-only foundation models.Source: §1 · p. 2

Evaluation

• DISC-Law-Eval assesses systems objectively with law-exam multiple-choice questions across three difficulty levels and subjectively with legal Q&A judged by GPT-3.5 on accuracy, completeness, and clarity.Providing the arbitrator with ground truth is intended to reduce potential biases.Source: §1 · p. 2

Results

• DISC-LawLLM excels in a majority of tested objective-evaluation subjects even versus GPT-3.5-turbo (165B parameters), and significantly outperforms existing legal large language models.The system also consistently generates more reliable responses, with more extensive Chinese legal knowledge and legal reasoning.Source: §1 · p. 3

2 Related Work

• LaWGPT used Chinese legal datasets and corpora, while Lawyer LLaMa combined continued pre-training with instruction tuning to improve legal advice (Huang et al., 2023).LaWGPT was built on Chinese-LLaMA-7B, ChatGLM, and Chinese-alpaca-plus-7B; Lawyer LLaMa continued pre-training Chinese-LLaMA-13B.Source: §2 · p. 3
• Generic LLMs have proved unsuitable for some domain-specific tasks such as law, and prior work focused on dialogue competence, whereas this paper proposes a system for diverse legal services.The paper characterizes dialogue as one of the intelligent-justice tasks addressed by prior work.Source: §2 · p. 3
• iLaw explored LoRA, P-tuning, and fine-tuning; LawGPT_zh used self-instruct Q&A data with LoRA, and Chatlaw trained on Ziya-LLaMA-13B-v1 and Anima-33B.The passage identifies ChatGLM-6B as iLaw’s base model and ChatGLM-6B for LawGPT_zh.Source: §2 · p. 3

3 DISC-Law-SFT Datasets

• DISC-Law-SFT pairs target legal reasoning, while triplets add references to support external-knowledge use.The formats are <input, output> and <input, output, reference>, respectively.Source: §3 · pp. 3–4

3.1 Data Sources

• The dataset draws on annotated legal NLP task datasets, crawled legal texts, and open-source legal instruction datasets.The legal texts include consultations, laws and regulations, typical cases, verdicts, and law-related examinations.Source: §3.1 · p. 4

3.2 Pair Instruction Generation

• Pair instructions are reconstructed through behavior shaping, knowledge expansion, and law-specific chains of thought (LCoT).The process starts by cleaning source data into input-output pairs; behavior shaping aligns responses with legal syllogism, while knowledge expansion adds legal reasoning to multiple-choice answers.Source: §3.2 · pp. 4–5

3.3 Triplet Instruction Generation

• Triplet instructions use the pair-generation strategies to produce inputs and outputs, then apply heuristic rules to extract references.These <input, output, reference> examples are intended for retrieval-augmented DISC-LawLLM.Source: §3.3 · p. 5

3.4 Dataset Overview

• The dataset covers more than 10 legal tasks and adds over 100k general instruction samples to help preserve foundational capabilities during legal-domain fine-tuning.Examples of legal tasks include element extraction, case matching, judgment prediction, document summarization, and question answering.Source: §3.4 · p. 5

4.1 Supervised Fine-Tuning

• The study performs supervised fine-tuning using the DISC-Law-SFT dataset and reports that this refined SFT data enabled the model to acquire legal reasoning and judicial behavioral patterns.Source: §4.1 · p. 6

4.2 Retrieval Augmentation

• Retrieval augmentation addresses hallucinations and outdated knowledge by retrieving relevant Chinese-law documents and feeding them with the user input to DISC-LawLLM.The knowledge base covers over 50 categories of laws; the retriever encodes them as vectors and returns the Top-K documents most similar to the input, supporting answers with references.Source: §4.2 · p. 6
• DISC-Law-SFT-Triplet trains the model to reason from retrieved references, while a dynamically updated knowledge base keeps laws current.The model can deduce answers based on newly retrieved knowledge.Source: §4.2 · p. 6

5 DISC-Law-Eval Benchmark

• The benchmark evaluates systems from both objective and subjective perspectives.Its design is inspired by the composition of the bar exam.Source: §5 · p. 6

5.1 Objective Evaluation

• The objective evaluation uses single- or multiple-answer questions to assess legal knowledge and reasoning, scoring performance by accuracy.The questions are intended to provide a challenging and reliable measure of reasoning toward correct answers.Source: §5.1 · pp. 6–7
• Objective questions come from Chinese legal examinations and contests, span three difficulty levels, and include NJE questions from 2018–2022 for fair evaluation.Sources include NJE, PAE, CPA, UNGEE, PFE, and LBK; the newer NJE questions address overlap with JEC-QA training data based on 2007–2017 exams.Source: §5.1 · p. 7

5.2 Subjective Evaluation

• The subjective test uses 300 manually constructed question-answer examples covering legal tools, consultations, and judgment prediction.Examples were drawn from legal consultations, online postings, justice-related publications, and legal documents.Source: §5.2 · p. 7
• GPT-3.5 acts as referee, rating each answer from 1 to 5 for accuracy, completeness, and clarity; the ground truth is also provided to reduce referee-model self-bias.Completeness checks whether reference-answer details are missing without allowing answer length to affect judgment; clarity assesses rigorous, clear juridical reasoning and well-organized sentences.Source: §5.2 · pp. 7–8

6 Experiments

• The objective comparison evaluates DISC-LawLLM without retrieval augmentation against instruction-aligned general LLMs and existing legal LLMs on DISC-Law-Eval.The general-model set includes GPT-3.5-Turbo, Chatglm-6B, Baichuan-13B-Chat, and Chinese-Alpaca2-13B; the legal-model set includes LaWGPT, Lawyer-LLaMa, ChatLaw, and LexiLaw.Source: §6 · p. 8

6.1 Results in Objective Evaluation

• 7% average accuracy gain over 175B-parameter GPT-3.5-Turbo, with DISC-LawLLM surpassing nearly all competitors across subjects and difficulty levels.The comparison is on DISC-Law-Eval objective evaluation.Source: §6.1 · p. 8
• On difficult NJE and PAE questions, DISC-LawLLM surpasses all LLMs by a large margin and improves over top-performing GPT-3.5-Turbo by over 50% on multi-answer questions.The authors interpret this as evidence of strong jurisprudential reasoning; they suggest generic LLMs may sometimes outperform legal LLMs because of weaker few-shot instruction following during training.Source: §6.1 · p. 8

6.2 Results in Subjective Evaluation

• DISC-LawLLM achieves the best performance on most subjective metrics and a 6% increase in average performance over Chatlaw.The evaluation uses Chat-GPT’s comprehension to assess short answers against the Ground Truth; the authors attribute ACC and CPL reliability to DISC-Law-SFT and stronger jurisprudential logic to cultivated juridical thinking.Source: §6.2 · p. 8

Legal Consultation

• DISC-LawLLM offers public legal consultation, using relevant legal bases to answer debt-apportionment questions and providing precise agreement-drafting suggestions.The examples are presented as evidence of its legal knowledge and reasoning proficiency.Source: §7 · p. 9

Examination Assistant

• DISC-LawLLM assists law students by predicting exam answers with rationale and providing detailed explanations of legal knowledge questions.The paper presents this as support for deeper understanding of challenging questions and stronger command of legal subjects.Source: §7 · p. 9

Applications

• Retrieval-result examples illustrate applications that can extend to additional scenarios, while the system is presented as bridging LLMs and diverse judicial settings.The paper describes the system as serving broad populations and having significant application value.Source: §7 · p. 9

8 Conclusion

• DISC-LawLLM outperforms existing legal LLMs on the constructed legal benchmark, which assesses legal systems from objective and subjective perspectives.The benchmark is designed as a comprehensive assessment of intelligent legal systems.Source: §8 · p. 9
• The system uses ChatGPT to reconstruct legal responses following legal syllogism for supervised fine-tuning, drawing on public legal NLP datasets, legal raw text, and open-source instruction datasets.Source: §8 · p. 9
• DISC-LawLLM incorporates an external retrieval module to enhance the reliability of its responses and develops legal reasoning and knowledge-retrieval capabilities.Source: §8 · p. 9
• The authors release the constructed datasets and model weights to foster further research.Source: §8 · p. 9

• Figure 5 presents a case of DISC-LawLLM as legal professional tools.Source: Figure 5 · p. 12
• Figure 6 presents a case of DISC-LawLLM in legal consultation.Source: Figure 6 · p. 12
• Figure 7 presents a case of DISC-LawLLM as an examination assistant.Source: Figure 7 · p. 13
• Figure 8 presents cases of retrieval-augmented DISC-LawLLM.Source: Figure 8 · p. 13