Source-linked AI summary
AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation
Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
TL;DR
Existing legal AI methods often address individual judicial stages rather than the connected process required for complex decisions. The paper proposes AgentsCourt, a multi-agent framework combining court debate simulation, legal-resource retrieval, and decision refinement, alongside SimuCourt and Legal-KB. It reports stronger performance than existing advanced methods, including 8.6% and 9.1% F1 improvements for legal-article generation in first- and second-instance settings.
Problem
Existing legal research often focuses on individual judicial stages, while complex judicial decisions require connected analysis across multiple stages.
Source: §1 · p. 1Method
AgentsCourt simulates court debate, retrieves legal resources, and refines decisions, supported by the SimuCourt benchmark and Legal-KB.
Source: §1 · p. 3, §2.1 · pp. 3–4Results
8.6% and 9.1% F1 improvements were achieved for legal-article generation in first- and second-instance settings, respectively.
Source: §1 · p. 3Takeaways & Limitations
The framework provides an integrated multi-agent approach for judicial analysis and decision-making across criminal, civil, and administrative cases.
Source: §1 · p. 3, §6 · p. 8The paper identifies further exploration of Legal-KB's potential as future work.
Source: §7 · p. 9
Abstract
from arXiv · showhide
With the development of deep learning, natural language processing technology has effectively improved the efficiency of various aspects of the traditional judicial industry. However, most current efforts focus on tasks within individual judicial stages, making it difficult to handle complex tasks that span multiple stages. As the autonomous agents powered by large language models are becoming increasingly smart and able to make complex decisions in real-world settings, offering new insights for judicial intelligence. In this paper, (1) we propose a novel multi-agent framework, AgentsCourt, for judicial decision-making. Our framework follows the classic court trial process, consisting of court debate simulation, legal resources retrieval and decision-making refinement to simulate the decision-making of judge. (2) we introduce SimuCourt, a judicial benchmark that encompasses 420 Chinese judgment documents, spanning the three most common types of judicial cases. Furthermore, to support this task, we construct a large-scale legal knowledge base, Legal-KB, with multi-resource legal knowledge. (3) Extensive experiments show that our framework outperforms the existing advanced methods in various aspects, especially in generating legal articles, where our model achieves significant improvements of 8.6% and 9.1% F1 score in the first and second instance settings, respectively.
1 Introduction
Existing legal AI research has advanced individual tasks, but complex judicial decisions require connected analysis across trial stages and specialized reasoning. AgentsCourt addresses this gap with court-process simulation, legal-resource retrieval, and decision refinement, supported by SimuCourt and Legal-KB.
1 Introduction
- Existing legal research often treats judicial stages separately, despite complex cases requiring collaboration across court debates, case analysis, and judgment prediction.This fragmentation leaves complex judicial decisions dependent on deep legal-expert involvement.Source: §1 · p. 1
- Judicial decision-making requires agents to handle ethical relationships, specialized legal knowledge, and intertwined logical, factual, and legal reasoning.These challenges arise because cases involve multiple stakeholders, nuanced legal provisions, and ethical and social justice considerations.Source: §1 · p. 2
- SimuCourt contains 420 Chinese judgment documents spanning criminal, civil, and administrative cases across first- and second-instance courts.The benchmark is designed to assess judicial analysis and decision-making across common case types.Source: §1 · p. 2
- Legal-KB combines effective laws and regulations, highly cited judicial papers, and recent precedents to support judicial decision-making.The framework uses retrieved legal resources to address gaps in expert knowledge.Source: §1 · pp. 2–3
- AgentsCourt integrates court debate simulation, precedent retrieval, case analysis, legal-article generation, and judgment delivery in a sequential multi-agent framework.Its modules follow opening remarks, debate, precedent retrieval, and judgment stages.Source: §1 · pp. 2–3
- 8.6% and 9.1% F1 improvements were achieved for legal-article generation in first- and second-instance settings, respectively.The results compare the framework with existing advanced methods across multiple aspects.Source: §1 · p. 3
2 Related Work
Prior legal AI research advances individual tasks but overlooks their interconnection in complex judicial decisions. The paper formulates judicial decision-making as generating analysis, legal articles, and a final judgment from case details across trial stages.
2 Related Work
- Existing legal AI research covers judgment prediction, question answering, language understanding, case retrieval, and document summarization, but mainly treats these as separate tasks.Source: §2 · p. 3
- Neglecting connections among trial stages leaves complex judicial decisions dependent on substantial legal-expert involvement.Source: §2 · p. 3
2.1 Task Formulation
- Judicial Decision-Making requires case analysis, rigorous legal-article generation, and a definitive judgment from case details.Source: §2.1 · p. 3
- SimuCourt evaluates both first-instance fact analysis and second-instance appellate review involving new evidence and legal-error detection.Source: §2.1 · p. 4
3 The AgentsCourt Framework
AgentsCourt models judicial decision-making as a court-trial process combining simulated debate, legal-resource retrieval, and judgment refinement. Its agents reconstruct party arguments, retrieve precedents and legal provisions, and iteratively produce a final judgment.
3 The AgentsCourt Framework
- AgentsCourt follows the real-world court process to study multi-agent collaboration in judicial decision-making.Source: §3 · p. 4
3.1 Court Debate Simulation
- Three role-playing agents simulate the judge, plaintiff, and defendant, using recorded statements to reconstruct debate despite incomplete court transcripts.Source: §3.1 · p. 4
3.2 Legal Resources Retrieval
- The judge assistant retrieves public information, uses BM25 to select 100 candidates, BGE-Large to rerank them, and extracts legal provisions from the top five precedents.Source: §3.2 · p. 5, Figure 4 · p. 5
3.3 Judgement Refinement
- The judge first analyzes case facts and simulated transcripts, then refines the preliminary judgment using precedents, legal regulations, and public opinions.Source: §3.3 · p. 5
4 The SimuCourt Benchmarck
SimuCourt addresses limitations of existing legal datasets by evaluating complete judicial analysis and decision-making across trial stages and case types. The paper also builds Legal-KB from laws, scholarship, and precedents to support retrieval.
4 The SimuCourt Benchmarck
- SimuCourt targets agent-as-judge evaluation beyond factual inputs or final judgments by requiring case analysis, legal-article generation, and judgments.Source: §4 · p. 5
4.1 Data Collection
- The benchmark contains 420 real-world Chinese cases spanning first- and second-instance proceedings and criminal, civil, and administrative matters.Source: §4.1 · p. 5
- Most cases were released after April 2023, reducing potential leakage from the pretraining data of the cited GPT models.Source: §4.1 · p. 5
4.2 Legal Knowledge Base Construction
- Legal-KB combines laws, regulations, judicial interpretations, journal articles, and precedents to provide broad legal information for judicial decisions.Source: §4.2 · p. 6
- Precedents are collected from 2017–2022 judgments and capped at 20k documents per cause of action, retaining the longest texts when distributions are long-tailed.Source: §4.2 · p. 6
5 Experiments
Experiments evaluate AgentsCourt with automatic metrics, human judgments, multiple baselines, and analyses of legal knowledge, court debate, and case-type difficulty. The framework outperforms other models overall, but legal-article generation and several judgment-analysis capabilities remain challenging.
5.1 Automatic Evaluation
- Legal-article evaluation uses strict matching with micro-averaged precision, recall, and F1, while civil and administrative judgments use GPT-4 to assess matching key points.Criminal judgments are evaluated separately for charge, prison term, and fine accuracy.Source: §5.1 · pp. 6–7
5.2 Human Evaluation
- Human evaluation samples 100 entries per setting and rates case analysis for correctness, logicality, and concision using three law graduate students.Correctness requires consideration of all parties, logicality excludes illogical or untrue reasoning, and concision requires necessary information without extras.Source: §5.2 · p. 7
5.3 Baselines
- The baselines include few-shot GPT-3.5 and GPT-4, ReAct, AutoGPT, and LaWGPT, with GPT-3.5 used as the foundation model for agent systems.LaWGPT is pretrained on Chinese legal corpora and fine-tuned on legal instructions.Source: §5.3 · p. 7
5.4 Main Results
- AgentsCourt outperforms other models in all evaluated aspects, with legal-article F1 improvements of 8.6% and 9.1% in first- and second-instance settings.GPT-4 reaches only 13.6% and 22.8% in the corresponding settings.Source: §5.4 · p. 7, Table 5 · p. 7
5.5 Discussion and Analysis
- GPT-4 reaches 35.4% accuracy for case-reason prediction and LaWGPT 43.7%, while all models perform poorly on legal-article generation.All three models perform well on simple case-type prediction, but LaWGPT sometimes produces garbled article-generation output.Source: §5.5 · pp. 7–8
- Court-debate simulation significantly reduces absolute differences in criminal prison terms and fines, indicating improved judicial-decision accuracy in ablation analysis.The comparison is reported in the framework’s first-instance ablation study and Figure 6 discussion.Source: §5.5 · p. 8, Table 6 · p. 8
- AgentsCourt generates more reliable legal articles for criminal cases than for civil and administrative cases, where multiple vested interests create more complex legal and social issues.The comparison is reported for the first-instance setting.Source: §5.5 · p. 8, Table 7 · p. 8
- External legal knowledge improves judicial reasoning by up to 6.2%, while document reranking raises cause-of-action consistency from 62% to 85%.The result is attributed to the automatic retrieval module.Source: §5.5 · p. 8
6 Conclusion
The conclusion presents AgentsCourt as a multi-agent judicial framework that sequentially simulates courtroom debate, retrieves precedents, analyzes cases, generates legal articles, and delivers judgments. It also introduces SimuCourt and reports that the paradigm enhances judicial efficiency.
6 Conclusion
- AgentsCourt sequentially simulates court debate, retrieves precedents, analyzes cases, provides legal articles, and delivers clear judgments.The framework is presented as a multi-agent approach to judicial decision-making.Source: §6 · p. 8
- SimuCourt evaluates agents’ judicial analysis and decision-making power, while the reported multi-agent paradigm significantly enhances judicial efficiency.Source: §6 · p. 8
7 Limitation
The paper’s evaluation is limited to Chinese judgments from China Judgments Online and three major case types. The authors also report that overall agent-system performance remains unsatisfactory despite extensive legal resources.
7 Limitation
- The dataset contains only Chinese documents from China Judgments Online, so testing across different legal systems remains important.Source: §7 · p. 9
- The judgment documents cover criminal, civil, and administrative cases, and broader case coverage is needed for more comprehensive evaluation.Source: §7 · p. 9
- Overall agent-system performance remains unsatisfactory despite the database containing many precedents and legal resources.Source: §7 · p. 9
- The authors identify further exploration of the legal knowledge base as future work.Source: §7 · p. 9
A Retrieval Module
The retrieval module raises cause-of-action consistency between retrieved cases and current cases to 85%, while accompanying materials illustrate simulated court transcripts.
A Retrieval Module
- 85% cause-of-action consistency is achieved between retrieved cases and current cases after rough retrieval and document re-ranking.The comparison is reported in Table 8.Source: A Retrieval Module · p. 11
B Example of Court Transcript
- A court transcript is presented as an example of multi-agent debate.The example is identified as Table 12.Source: B Example of Court Transcript · p. 11
C Data Demonstration
The paper demonstrates trial-stage information and describes a case-selection strategy emphasizing diverse causes of action, clear facts, and reliable judgments.
C Data Demonstration
- The paper provides an information list covering different trial stages and examples for first-instance and second-instance proceedings.The examples are shown in Figures 7 and 8, respectively.Source: C Data Demonstration · p. 11
D.1 Data Description
- Case selection prioritizes diversity because the 15 most common civil causes of action represented 66% of 2022 cases.The dataset seeks to reflect a broader spectrum of legal practice despite this long-tail distribution.Source: §D.1 · p. 11
- Selected judgments provide clear case analysis and facts to support accurate annotation and agents’ understanding of judicial reasoning and legal articles.The selection also prioritizes cases not overturned on appellate review to improve evaluation consistency.Source: §D.1 · p. 11
D.2 Data Quality Inspection
SimuCourt undergoes privacy processing and manual quality inspection, while legal-article evaluation parses generated text and matches it against specific legal provisions.
D.2 Data Quality Inspection
- Judgment documents are anonymized by replacing personal, geographic, institutional, and other privacy-revealing details with generic terms.This processing is performed before or alongside annotation and quality review.Source: §D.2 · p. 11
- Table 8 reports cause-of-action matching after retrieval and re-ranking, while Table 9 reports data-quality analysis.Figure 6 presents the absolute difference change.Source: Table 8 · p. 12, Table 9 · p. 12, Figure 6 · p. 12
- All 420 annotated cases are manually reviewed by three graduate students for case standards, annotation accuracy, and privacy security.The inspection covers clear facts, non-overturned judgments, extracted legal information, and sensitive-information protection.Source: §D.2 · pp. 11–12
E.1 Legal Articles Evaluation
- Legal-article evaluation parses free text and hard-matches it against specific legal provisions using true positives, false positives, and false negatives.The example yields Precision = 2/3 and Recall = 2/4.Source: §E.1 · p. 12
E.2 Judgement Evaluation for Civil and Administrative Cases
The supplied materials document example outputs and evaluation prompts for criminal, first-instance, and second-instance cases, but do not report substantive civil or administrative judgment-evaluation results.
E.2 Judgement Evaluation for Civil and Administrative Cases
- Figures 7 and 8 provide translated examples of first-instance and second-instance stages.These examples are referenced as demonstrations of different trial stages.Source: Figure 7 · p. 13, Figure 8 · p. 14
- The supplementary tables include information by trial stage and an example output for a criminal case.They also include a simulated multi-agent court transcript.Source: Table 10 · p. 15, Table 11 · p. 16, Table 12 · p. 17
- A figure summarizes causes of action for civil cases in 2022.The supplied caption identifies the statistic but not further values.Source: Figure 9 · p. 15
- The materials include a translated comparison of reference and model-generated legal articles and a translated GPT-4 evaluator prompt demonstration.These artifacts document evaluation setup rather than provide civil or administrative judgment scores.Source: Table 13 · p. 17, Table 14 · p. 18