PaperlayerView exact revision on arXiv

Source-linked AI summary

LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultation

Jingyun Sun, Chengxiao Dai, Zhongze Luo, Yangbo Chang, Yang Li

arXiv:2407.16252v3cs.CLcs.AIcs.CV

TL;DR

Existing Chinese legal consultation models largely rely on single-agent systems, while real consultations involve coordinated work among multiple professionals. LawLuo simulates a law firm with four collaborating agents, specialized fine-tuning datasets, and case graph-based retrieval, and it reports stronger personalization, ambiguity handling, and multi-turn instruction following than baselines.

  • Problem

    Existing Chinese legal consultation models focus on individual legal LLMs rather than the collaborative workflows used in real law firms.

    Source: §2.1 · p. 2, §3 · p. 3

  • Method

    LawLuo uses receptionist, lawyer, secretary, and boss agents, trained with distinct datasets and supported by case graph-based RAG for ambiguous queries.

    Source: §1 · p. 1, §3.2 · pp. 4–5

  • Results

    LawLuo outperforms traditional single-agent models in personalized and professional advice, ambiguous inquiries, and multi-turn legal-instruction following.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Multi-agent collaboration can support legal consultation that more closely simulates real law-firm interactions within the evaluated scope.

    Source: §7 · p. 8

    The framework still requires improvement in inter-agent communication, coordination, and model interpretability.

    Source: §7 · p. 8, Limitation and Future Work · p. 8

Abstract

from arXiv · show

Legal Large Language Models (LLMs) have shown promise in providing legal consultations to non-experts. However, most existing Chinese legal consultation models are based on single-agent systems, which differ from real-world legal consultations, where multiple professionals collaborate to offer more tailored responses. To better simulate real consultations, we propose LawLuo, a multi-agent framework for multi-turn Chinese legal consultations. LawLuo includes four agents: the receptionist agent, which assesses user intent and selects a lawyer agent; the lawyer agent, which interacts with the user; the secretary agent, which organizes conversation records and generates consultation reports; and the boss agent, which evaluates the performance of the lawyer and secretary agents to ensure optimal results. These agents' interactions mimic the operations of real law firms. To train them to follow different legal instructions, we developed distinct fine-tuning datasets. We also introduce a case graph-based RAG to help the lawyer agent address vague user inputs. Experimental results show that LawLuo outperforms baselines in generating more personalized and professional responses, handling ambiguous queries, and following legal instructions in multi-turn conversations. Our full code and constructed datasets will be open-sourced upon paper acceptance.

1 Introduction

LawLuo addresses the limits of single-agent Chinese legal consultation systems by simulating collaborative law-firm workflows with specialized agents, datasets, and case graph-based retrieval. Experiments report more personalized and professional advice, better handling of ambiguous queries, and stronger multi-turn instruction following.

Context

Motivation

  • Existing Chinese legal consultation models mainly use single-agent systems that do not replicate collaborative law-firm workflows or reliably personalize responses.These limitations are associated with superficial, generalized responses that may not capture user intent and case details.Source: §1 · p. 1, Figure 1 · p. 2

Framework

  • LawLuo uses receptionist, lawyer, secretary, and boss agents to simulate law-firm operations and deliver personalized, professional legal consultation.The receptionist assesses intent and assigns a specialist; the lawyer responds across rounds; the secretary organizes records and reports; and the boss monitors performance.Source: §1 · p. 1

Contributions

  • LawLuo introduces three fine-tuning datasets for the receptionist, lawyer, and secretary agents and a case graph-based RAG module for ambiguous queries.The datasets target intent-to-lawyer matching, multi-round legal dialogue, and consultation-report generation.Source: §1 · pp. 1–2

Results

  • LawLuo outperforms baselines in personalized and professional advice, ambiguous-query handling, and multi-round legal-instruction following.When users provide vague questions, LawLuo guides them to elaborate case details instead of giving only broad answers.Source: §1 · p. 2

2 Related work

Prior Chinese legal LLM research improves individual models through legal data, continued pre-training, instruction tuning, or external knowledge. LawLuo instead draws on multi-agent collaboration to decompose legal consultation into coordinated subtasks resembling real law-firm practice.

2.1 LLMs for Legal Consultation

  • Existing legal LLMs are primarily improved through continued pre-training, instruction fine-tuning, legal datasets, training from scratch, or external legal knowledge.Training sources include legal documents, judicial-exam data, and legal question-answering datasets.Source: §2.1 · p. 2
  • Prior efforts focus on improving individual legal LLMs, whereas real legal consultations involve collaboration among multiple professionals.This gap motivates a multi-agent framework intended to provide more personalized and professional consultation.Source: §2.1 · pp. 2–3

2.2 Multi-Agent Collaboration

  • Multi-agent systems define agents as autonomous entities that perceive, reason, learn, decide, and interact with other agents.Research uses diverse agent roles to divide complex tasks into simpler subtasks and support collaborative problem solving.Source: §2.2 · p. 3
  • LLM-based multi-agent systems can simulate social environments, supporting the decomposition of legal consultation into subtasks handled collaboratively.The paper connects this capability to improved personalization and professionalism in legal responses.Source: §2.2 · p. 3

3 Framework

LawLuo models Chinese legal consultation as a multi-agent workflow spanning intake, lawyer-led dialogue, report generation, and performance evaluation. Its specialized datasets and case graph-based retrieval support domain assignment, multi-turn guidance, consultation summarization, and agent optimization.

3 Framework

  • LawLuo coordinates receptionist, lawyer, secretary, and boss agents to simulate collaborative law-firm consultation workflows.The receptionist assigns a suitable lawyer, the lawyer analyzes the case through dialogue, the secretary produces a report, and the boss monitors agent performance.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Receptionist

  • The receptionist evaluates the initial inquiry against lawyer specialties and selects a suitable lawyer from a candidate pool.The framework defines 16 lawyer descriptions covering consultation areas such as contract, labor, criminal, family, and real-estate law.Source: §3.1 · p. 3

3.2 Lawyer

  • The lawyer agent uses multi-turn dialogue to acquire case details, addresses vague queries with case graph-based RAG, and is trained on MURLED’s 16,734 multi-turn conversations across 16 consultation domains.The case graph uses a collection of 4,320 criminal cases and 12,345 civil cases sourced from China Judgments Online.Source: §3.2 · pp. 3–5

3.3 Secretary

  • The secretary agent converts lawyer-user conversation records into a final consultation report using LCRG, which contains 420 dialogues paired with reports written by professional lawyers.The dataset fine-tunes BaiChuan to generate reports from legal consultation dialogues.Source: §3.3 · p. 5

3.4 Boss

  • The boss agent evaluates lawyer and secretary outputs as better or worse and uses PPO-based reinforcement learning to optimize both agents.Its binary reward model assigns labels where 0 denotes worse and 1 denotes better.Source: §3.4 · p. 5

4 Experimental Setup

The experiments use a single 40G A100 GPU with specified PyTorch and Transformers versions and LoRA fine-tuning settings. Training uses a small batch, three epochs, and fixed LoRA hyperparameters.

4 Experimental Setup

  • Experiments ran on one 40G A100 GPU using PyTorch 2.3.0 and Hugging-Face Transformers 4.40.0.The reported hardware and software define the experimental computing environment.Source: §4 · p. 5
  • LoRA fine-tuning used learning rate 0.00005, batch size 2, three epochs, rank 16, alpha 32, and dropout 0.05.Model weights were saved every 1,000 steps.Source: §4 · pp. 5–6

5 Results and Analysis

LawLuo outperforms baselines in personalized legal consultation, guides users through ambiguous inquiries, maintains instruction compliance across multiple turns, and remains effective on routine legal tasks.

5.1 Pairwise Comparison Evaluation

  • LawLuo outperforms widely used Chinese base LLMs and all legal LLM baselines in personalized and professional response generation.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 Case Study on Ambiguous Inquiry

  • LawLuo produces more guiding first-round responses that help users without legal backgrounds elaborate case details during ambiguous consultations.Source: §5.2 · p. 6

5.3 Multi-Turn Dialogue

  • After five dialogue rounds, LawLuo maintains a high level of legal instruction compliance across evolving and increasingly complex tasks.Source: §5.3 · p. 6, Figure 6 · p. 7

5.4 Ablation Study

  • Ablating the receptionist agent reduces LawLuo’s win rate over GPT-3.5 by 2%, while removing case graph-based RAG causes a significant performance decline.Source: §5.4 · pp. 6–7, Figure 7 · p. 7
  • The ablations support domain-specific agent roles, boss-agent response optimization, case clarification through RAG, and instruction fine-tuning as contributors to performance.Source: §5.4 · pp. 6–7

5.5 Legal Knowledge Probing Experiment

  • LawLuo performs well across five routine legal NLP tasks and remains highly competitive despite not achieving the best performance against all baselines.Source: §5.5 · p. 7

6 System Implementation

The implemented system operationalizes LawLuo as a web-based legal consultation platform, while its evaluation includes routine legal NLP tasks that probe acquired legal knowledge.

6 System Implementation

  • Table 1 evaluates LawLuo and baselines on five routine legal NLP tasks as a measure of legal knowledge understanding.Source: Table 1 · p. 8
  • The web system routes users from a receptionist to a relevant lawyer, then generates a consultation report while the boss monitors service quality.Source: §6 · pp. 7–8, Figure 8 · p. 8

7 Conclusion

LawLuo simulates law-firm collaboration through multiple agents and improves personalized consultation, ambiguous-query handling, and multi-turn instruction following. The authors identify inter-agent communication and interpretability as future improvement targets.

7 Conclusion

  • LawLuo simulates multi-party law-firm interactions and outperforms single-agent models in personalized advice, ambiguous inquiries, and multi-turn instruction following.Source: §7 · p. 8
  • Future work will optimize inter-agent communication and enhance model interpretability.Source: §7 · p. 8

Limitation and Future Work

Multi-agent collaboration can improve legal consultation, but effective coordination requires domain expertise and agents able to understand one another’s decisions and feedback.

Limitation and Future Work

  • Multi-agent systems introduce communication and coordination challenges that require domain-specific expertise and mutual understanding among agents.Source: Limitation and Future Work · p. 8

A A sample from Legal Consultation Report Generation

The legal consultation summary report is presented as a structured document for organizing consultation outcomes. Its sample contains nine sections covering case information, analysis, advice, and risks.

A A sample from Legal Consultation Report Generation

  • A legal consultation summary report contains nine sections, including identifying information, consultation purpose, factual background, legal analysis, advice, and risk warnings.The listed sections also include report number, consultation date, client, and subject of consultation.Source: A A sample from Legal Consultation… · p. 11
  • Figure 9 provides a sample legal consultation summary report for in-context learning.Source: Figure 9 · p. 12

B Case Analysis Presentation

The case-analysis presentation includes responses to a vague legal inquiry about loan repayment. Table 2 presents these responses for comparison.

B Case Analysis Presentation

C Prompt Template for Guiding GPT-4o in Scoring

The scoring procedure uses a prompt template to guide GPT-4o. Figure 10 presents that template, alongside the loan-repayment response comparison in Table 2.

C Prompt Template for Guiding GPT-4o in Scoring

Related papers

Loading 2407.16252v3…