Source-linked AI summary
Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, Li Yuan
TL;DR
Limited access to legal services and LLM hallucinations create challenges for legal assistance, particularly in China. Chatlaw addresses these challenges with a multi-agent workflow aligned to legal-firm procedures and a Role-Aligned Mixture-of-Experts architecture, outperforming existing models including GPT-4 in reported evaluations.
Problem
Limited legal-service availability and high costs restrict access, while LLM hallucinations and incomplete or outdated knowledge create risks for legal applications.
Method
Chatlaw combines a multi-agent legal workflow with Role-Aligned Mixture-of-Experts, matching specialized experts to agents’ task-specific roles and legal-service procedures.
Results
Chatlaw surpasses GPT-4 by 7.73% in LawBench accuracy and 11 points on the Unified Qualification Exam for Legal Professionals, while receiving the highest real-case expert scores across four dimensions.
Takeaways & Limitations
The reported evaluations indicate that the multi-agent, role-aligned design supports legal consultation performance across benchmark, examination, and real-case expert assessments.
Takeaways & Limitations
Deployment remains constrained by privacy and data-retention demands, as users differ over whether sensitive consultation data should be stored.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) holds great potential in legal services, yet Large Language Models (LLMs) face two major challenges: limited knowledge of the Chinese legal system and vulnerability to hallucinations. To address these issues, we present Chatlaw, a multi-agent legal assistant. Chatlaw's framework is designed to emulate the Standard Operating Procedures (SOP) of real law firms, where different roles (e.g., assistant, researcher, senior lawyer) collaborate on a case. To computationally mirror this collaborative structure, we developed a novel Role-Aligned Mixture-of-Experts (RA-MoE) architecture. In this system, the internal "experts" are specifically trained to align with the distinct tasks of each agent role (e.g., inquiry, analysis, drafting). These specialized agents (Legal Assistant, Researcher, etc.) then form the collaborative framework. When they interact with users, retrieve legal knowledge, analyze case details, or generate reliable consultations, the RA-MoE architecture intelligently routes their computations to the corresponding dedicated expert, ensuring each step is handled by the most qualified parameters. In evaluations, Chatlaw surpasses general-purpose AI models, including GPT-4, achieving a 7.73% improvement in accuracy on the LawBench benchmark and an 11-point higher score on the Unified Qualification Exam for Legal Professionals. Real-case studies and expert assessments further confirm its robustness. Chatlaw enhances the accessibility and reliability of legal services, advancing the provision of legal support to the public.
1. Introduction
Legal services are difficult to access in China because professionals are scarce and costly, while LLMs offer assistance but remain vulnerable to incomplete, outdated, and hallucinated legal information. Chatlaw addresses these challenges by aligning specialized computational experts with law-firm workflows and supporting agents with legal data resources.
- Limited legal professionals and high service costs restrict access to legal services, particularly in China.
- LLMs can provide legal recommendations, but hallucinations, incomplete knowledge, and outdated information create risks in legal applications.
- Chatlaw uses a Role-Aligned Mixture-of-Experts architecture that specializes experts according to legal-service workflow tasks.
- The system aligns agents with task-specific models for information gathering, research, legal advice, and report writing.
- A legal dataset and knowledge graph support the agents’ information provision.
- 7.73% higher accuracy on LawBench and 11 points on the Unified Qualification Exam for Legal Professionals were reported against GPT-4.
2. Related Work
Mixture-of-Experts and multi-agent systems improve scalable or specialized task execution, but conventional designs often lack alignment between experts, agent roles, and high-level tasks. Chatlaw’s RA-MoE addresses this gap by coordinating role-specific experts within a multi-agent legal workflow.
- 2.1. Mixture-of-Experts Architecture: Mixture-of-Experts models activate sparse subsets of expert layers during inference to reduce computational cost and scale large models.
- 2.1. Mixture-of-Experts Architecture: Traditional MoE routers assign tokens according to data distributions rather than high-level tasks such as reasoning or drafting.
- 2.1. Mixture-of-Experts Architecture: RA-MoE explicitly trains routing to align expert selection with high-level agent roles, creating functionally specialized experts.
- 2.2. Multi-Agent Systems: Multi-agent systems coordinate agents with specialized functions to solve complex problems.
- 2.2. Multi-Agent Systems: Existing multi-agent systems commonly use the same general-purpose LLM API for every agent, causing computational redundancy.
- 2.2. Multi-Agent Systems: Chatlaw combines macro-level agent SOPs with the micro-level RA-MoE architecture to align experts with agent roles and reduce redundancy.
3. Chatlaw Framework
Chatlaw combines role-conditioned, knowledge-aware expert routing with a multi-agent SOP framework that mirrors legal-firm workflows. Its agents divide consultation into specialized stages, supported by agent–expert co-training and a large processed legal dataset.
- 3.1. Role-Aligned Mixture-of-Experts: RA-MoE extends MoE with role-conditioned routing and knowledge-aware gating for legal workflows.The router uses task, agent-role, and legal-knowledge information to select specialized experts.
- 3.1.1. Role-Conditioned Routing: The routing distribution selects top experts and aggregates their outputs as a weighted sum, with temperature controlling distribution sharpness.The routing inputs include token embeddings, role embeddings, and knowledge-graph embeddings.
- 3.1.3. Agent–Expert Co-Training: Agent–expert co-training shares gradients between task-specific expert clusters while a routing loss encourages the intended experts for each role-specific task.The routing loss penalizes deviations between predicted routing probabilities and the intended expert selection.
- 3.2. Chatlaw Agent Design: The SOP framework uses four coordinated agents—Legal Assistant, Legal Researcher, Senior Lawyer, and Legal Editor—to transform inquiries into reviewed legal outputs.The workflow includes structured inquiry processing, legal retrieval, professional reasoning, and document generation or editing.
- 3.4. Comprehensive Legal Data Pipeline and Preprocessing for Multi-Task Training: The data pipeline collects and preprocesses multi-source legal materials into task-oriented training data, producing a dataset with over 3.4 million samples.Sources include statutes, case documents, legal competition datasets, public consultations, and legal examinations; preprocessing includes deduplication, denoising, and standardization.
4.1. Performance Across Legal Benchmarks
Chatlaw-MoE is evaluated on LawBench and China’s Unified Qualification Exam, covering legal knowledge and its application across benchmark formats. It outperforms GPT-4 on both evaluations and leads across LawBench’s cognitive categories.
- Evaluation setup: Chatlaw-MoE is evaluated on two benchmarks assessing legal-system knowledge, regulations, case law, procedures, and their application to specific situations.The evaluations include LawBench and China’s Unified Qualification Exam for Legal Professionals.
- LawBench: 60.08 versus 52.35: Chatlaw-MoE achieves a higher LawBench average score than GPT-4.LawBench evaluates legal knowledge memory, understanding, and application.
- LawBench: Chatlaw-MoE leads GPT-4 across LawBench memory, understanding, and application, scoring 43.86 versus 35.29, 62.11 versus 54.41, and 61.60 versus 54.05, respectively.These categories assess remembering legal concepts, interpreting legal texts, and applying legal knowledge to real-world tasks.
- Unified Qualification Exam: 115 versus 104: Chatlaw-MoE records the higher average score than GPT-4 on the Unified Qualification Exam from 2018 to 2022.Chatlaw-MoE scores 113, 124, 143, 115, and 78 across the five years, while GPT-4 scores 102, 108, 82, 82, and 118.
4.2. Hallucination Detection
This section presents semantic entropy as a hallucination-detection method that groups outputs by meaning rather than raw form, and reports Chatlaw’s strong performance on legal benchmarks.
- Semantic entropy: Semantic entropy groups generated outputs by meaning before calculating uncertainty, unlike naive entropy over raw outputs.The method forms semantic-equivalence classes and estimates their entropy, using Monte Carlo integration when all meaning-classes are inaccessible.
- Hallucination target: Semantic entropy targets confabulation, where uncertain models generate plausible-looking answers containing detailed factual errors.The approach distinguishes this phenomenon from consistently wrong answers caused by erroneous training data or reward-seeking behavior.
- Benchmark results: The semantic-entropy evaluation indicates that Chatlaw combines high accuracy with stable performance across question types and option formats.The reported comparison also states that Chatlaw outperforms state-of-the-art models on open-ended questions.
- Benchmark results: Chatlaw achieved the lowest Average Entropy (0.4), highest AUROC (0.96), and highest AURAC (0.97) on the Unified Qualification Exam.GPT-4 recorded entropy 0.9, AUROC 0.92, and AURAC 0.90, while InternLM2 recorded entropy 1.5, AUROC 0.84, and AURAC 0.85.
4.3. Real-world Legal Consultation
This section evaluates Chatlaw on real-world legal consultations using expert ratings of response quality across four dimensions. Chatlaw ranked above the other tested models, including GPT-4.
- Evaluation design: Real-world consultation quality was assessed by Completeness, Correctness, Guidance, and Authority using expert ratings on a six-level scale.The levels map Strongly Agree, Agree, Neutral, Slightly Disagree, Disagree, and Strongly Disagree to scores of 10, 8, 6, 4, 2, and 0.
- Evaluation design: The evaluation used 250 open-ended questions, with three legal experts independently assessing AI-generated answers across four dimensions.The questions were based on online law-firm service records covering legal consultations and advice.
- Results: Chatlaw’s answers ranked higher on average than those of all other tested models, including GPT-4.The results concern legal experts’ multidimensional preferences regarding responses to real-world consultation questions.
- Results: Chatlaw’s average score was 8.71, exceeding GPT-4 by 0.60 points.Chatlaw also exceeded InternLM2 by 0.81 points and Fuzi-Mingcha by 1.47 points.
5. Discussion
Chatlaw combines a law-firm-inspired multi-agent workflow with role-aligned computation to provide end-to-end legal services. The discussion also identifies privacy, data retention, inference-resource, and environmental challenges that motivate future deployment work.
- Principal Contributions and Implications: Chatlaw integrates a legal data pipeline, RA-MoE model, and multi-agent SOP framework to provide end-to-end legal services as a virtual law firm.The framework is based on real-world law firm practices.
- Principal Contributions and Implications: The collaboration workflow assigns fact gathering to a Legal Assistant, legal retrieval to a Legal Researcher, and advice synthesis to a Senior Lawyer.The illustrated workflow concerns a divorce-case consultation.
- Limitations and In-built Mitigation Strategies: A dedicated Legal Researcher retrieves and verifies current legal provisions and cases, while the Legal Assistant uses knowledge graphs to elicit necessary user information.These roles are designed to fact-check outputs and ground queries before consultation.
- Deployment Challenges and Future Work: Privacy and data retention remain deployment challenges because users differ on whether sensitive consultation histories should be stored.The authors plan stronger privacy protection, communication security, and data-storage security.
- Deployment Challenges and Future Work: Model compression and on-device deployment are proposed to reduce computational demands while keeping sensitive data on users’ devices.Proposed techniques include knowledge distillation and quantization.
- Computational Overhead and Environmental Impact: High-concurrency inference can pressure computational resources and cause response delays, motivating future model-compression work.The current training run used eight 80GB A100 GPUs for approximately 23.14 hours and generated an estimated 42.81 kilograms of additional CO2 emissions.
6. Supplementary Material
The supplementary material details Chatlaw-MoE’s sparse model scale, high-concurrency deployment, semantic-entropy evaluation, and comparisons with recent models on LawBench. These results report strong throughput, consistency, and benchmark performance across the examined comparisons.
- Model Specifications and Expert Scale: Chatlaw-MoE contains approximately 28B total parameters as four 7B expert replicas, while sparse activation limits parameters used at each inference step.Each individual expert layer maintains a 7B scale.
- High-Concurrency Performance: At 1,000 concurrent users, combining Expert Parallelism with Data Parallelism (DP=4) achieves 6,983.583 tokens/s average throughput.The reported configuration is presented as supporting intensive, multi-stage consultations without throughput bottlenecks.
- Open-ended Questions: Chatlaw-MoE records the lowest average semantic entropy, 0.61, compared with Qwen3-8B at 0.69.The passage interprets this as greater consistency and less confabulation during long-form generation.
- Comparison with Latest Models: Chatlaw-MoE scores 60.08 on updated LawBench, exceeding Claude 3.7 at 58.66 and Hunyuan2.0-400B at 45.88.The passage attributes this performance across understanding and application tasks primarily to role-aligned expert activation.
- Comparison with Latest Models: LawBench evaluates average score, legal-provision memorization, legal-case understanding, and legal-judge application across multiple tasks.The comparison includes representative open-source and closed-source models such as Hunyuan 2.0, Qwen-3, GPT-4.2, and Claude 3.7.