PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents

Guhong Chen, Liyang Fan, Zihan Gong, Nan Xie, Zixuan Li, Ziqiang Liu, Chengming Li, Qiang Qu, Hamid Alinejad-Rokny, Shiwen Ni, Min Yang

arXiv:2408.08089v2cs.CLcs.AI

TL;DR

Existing legal language models and simulation systems do not comprehensively handle dynamic courtroom interactions. AgentCourt uses AdvEvol to evolve lawyer agents through adversarial courtroom simulation and structured knowledge construction, and its evolved agents achieve state-of-the-art performance in dynamic courtroom dialogue. The framework remains bounded by API latency, jurisdictional generalization, and the need for professional human oversight.

  • Problem

    Existing legal models are strong on static tasks but struggle with dynamic courtroom interactions and multi-party legal reasoning.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    AgentCourt uses AdvEvol to simulate courtroom interactions and autonomously evolve three legal knowledge bases from complaints and defense statements.

    Source: §1 · pp. 1–2

  • Results

    GPT-4o-mini-1000 consistently outperformed general-purpose and specialized legal models on dynamic courtroom dialogue tasks.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    CourtBench and AgentCourt support evaluating and developing legal AI through realistic courtroom interactions rather than static legal knowledge retrieval.

    Source: §5 · p. 9

    The framework relies on external APIs, may require adaptation across legal systems and jurisdictions, and requires qualified legal-professional review before practical use.

    Source: Limitations · p. 9, Ethics Statement · p. 9

Abstract

from arXiv · show

Current research in LLM-based simulation systems lacks comprehensive solutions for modeling real-world court proceedings, while existing legal language models struggle with dynamic courtroom interactions. We present AgentCourt, a comprehensive legal simulation framework that addresses these challenges through adversarial evolution of LLM-based agents. Our AgentCourt introduces a new adversarial evolutionary approach for agents called AdvEvol, which performs dynamic knowledge learning and evolution through structured adversarial interactions in a simulated courtroom program, breaking the limitations of the traditional reliance on static knowledge bases or manual annotations. By simulating 1,000 civil cases, we construct an evolving knowledge base that enhances the agents' legal reasoning abilities. The evolved lawyer agents demonstrated outstanding performance on our newly introduced CourtBench benchmark, achieving a 12.1% improvement in performance compared to the original lawyer agents. Evaluations by professional lawyers confirm the effectiveness of our approach across three critical dimensions: cognitive agility, professional knowledge, and logical rigor. Beyond outperforming specialized legal models in interactive reasoning tasks, our findings emphasize the importance of adversarial learning in legal AI and suggest promising directions for extending simulation-based legal reasoning to broader judicial and regulatory contexts. The project's code is available at: https://github.com/relic-yuexi/AgentCourt

1 Introduction

AgentCourt addresses the gap between static legal language-model tasks and dynamic courtroom interaction by simulating multi-party civil proceedings with adversarially evolving agents. It combines automated knowledge evolution with CourtBench evaluation to assess interactive legal reasoning.

1 Introduction

  • Existing legal models often perform well on static legal tasks but struggle to apply knowledge coherently in adversarial courtroom dialogue.The paper cites ChatLaw-33B as an example of overfitting to standardized tasks and failing to generate valid interactive responses.Source: §1 · p. 1
  • AgentCourt introduces a complete courtroom environment where judges, attorneys, and other agents engage in structured legal discourse through AdvEvol.The framework targets dynamic legal interactions rather than isolated retrieval or question-answering tasks.Source: §1 · pp. 1–2
  • Starting from complaints and defense statements, agents autonomously refine statutory, strategic, and precedent knowledge through adversarial court debates.The three knowledge bases are a legal provisions memory, an experience base, and a case library, without extensive manual annotation.Source: §1 · pp. 1–2
  • Through 1,000 simulated civil cases, AgentCourt improves agents’ legal capabilities and reaches performance comparable to GPT-4o while outperforming specialized legal models on dynamic courtroom tasks.The reported contrast is between strong standardized-task performance by existing legal models and weaker dynamic-dialogue behavior.Source: §1 · p. 2
  • CourtBench is introduced as a dedicated benchmark for systematic evaluation of interactive legal reasoning.It is designed to assess capabilities required in dynamic courtroom dialogue.Source: §1 · p. 2

2 Related Work

Related work shows strong progress in specialized legal language models and multi-agent simulation, but existing legal systems remain limited in dynamic, multi-party courtroom reasoning. AgentCourt extends multi-agent simulation to civil court proceedings while adding automated knowledge-base construction through simulation.

2.1 LLMs in the Legal Domain

  • Existing legal language models support tasks such as legal research, consultation, statutory interpretation, and case prediction, but remain largely task-specific.Examples include Lawyer-LLaMA-13B, HanFei-7B, ChatLaw-33B, DISC-LawLLM, PLJP, and DeliLaw.Source: §2.1 · p. 3
  • Despite sophisticated architectures and extensive legal corpora, current legal AI systems struggle with complex queries, multi-party interactions, and realistic court proceedings.The limitation concerns dynamic legal reasoning rather than only static knowledge access.Source: §2.1 · p. 3

2.2 LLMs for Real-World Simulation

  • Multi-agent LLM systems have improved collaborative problem solving across language, robotics, planning, education, and finance.These systems combine knowledge sharing, cognitive synergy, and decision-making across agents.Source: §2.2 · p. 3
  • Agent Hospital demonstrates that autonomous multi-agent simulation can support complex professional-domain modeling and performance evaluation.Its simulated doctors, nurses, and patients perform disease-treatment simulations without manual annotation.Source: §2.2 · p. 3
  • AgentCourt extends multi-agent simulation to civil courts by combining dynamic courtroom interaction with automated knowledge-base construction.This positioning addresses shortcomings of task-specific legal models while broadening professional-domain simulation.Source: §2.2 · p. 3

3 Court Simulation

AgentCourt models courtroom proceedings with role-specific agents and a three-phase simulation workflow, then evolves legal knowledge through adversarial interactions and structured case representations.

3.1 Agent Design

  • The framework uses lawyer, judge, clerk, plaintiff, and defendant agents to support case analysis, evidence organization, procedural control, and courtroom debates.Source: §3.1 · pp. 3–4
  • Lawyer decisions depend on the current state and accumulated knowledge, whereas other agents rely on the current state alone.Source: §3.1 · p. 4

3.2 Simulation Workflow

  • The workflow processes 1,000 real civil cases, runs standard court proceedings, and refines lawyer reasoning after sessions through AdvEvol.Source: §3.2 · p. 4

3.3 AdvEvol Method

  • AdvEvol acquires knowledge through structured adversarial courtroom interactions instead of static knowledge bases or extensive manual annotation.Source: §3.3 · p. 4
  • Its three-tier knowledge structure comprises regulations memory for legal provisions, an experience base for debate strategies, and a case library for precedent analysis.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 5

3.3.3 Case Library Construction

  • The case library extracts case background, category, keywords, quick response points, and response directions into structured representations that support retrieval and application.Source: §3.3.3 · p. 5

4 Experiments

AgentCourt evaluates adversarially evolved lawyer agents in simulated courtroom debates using CourtBench and three dimensions of legal expertise. Across benchmark comparisons, data-scale analysis, ablations, and a case study, the results show gains over baselines and distinct roles for the evolving knowledge bases.

4.1 Experimental Setup

  • The evaluation combines simulated debates over authentic civil-case documents with human and automated win-tie-loss assessments across cognitive agility, professional knowledge, and logical rigor.Legal professionals assessed 40 cases spanning contract, tort, marriage and family, and property-rights disputes.Source: §4.1 · pp. 5–6

4.2 Experimental Results

  • GPT-4o-mini-1000 achieves 70.0% human-evaluation win rates against GPT-4o-mini+RAG in both cognitive agility and professional knowledge.It also reaches a 75.0% professional-knowledge win rate against GPT-4o under human evaluation.Source: §4.2 · p. 6, Figure 2 · p. 7
  • 64.52% CourtBench accuracy nearly matches GPT-4o’s 66.13% and exceeds GPT-4o-mini’s 52.42% and GPT-4o-mini+RAG’s 58.06%.The knowledge-enhanced model achieves this without additional model parameters or extensive pre-training.Source: §4.2 · p. 6, Table 1 · p. 7
  • Specialized legal models perform poorly on dynamic courtroom dialogue, with ChatLaw-33B at 0.81%, Lawyer-LLaMA-13B at 33.87%, and HanFei-7B at 26.61% accuracy.These results contrast with their performance on traditional static legal tasks.Source: §4.2 · pp. 6–7
  • Professional-knowledge gains from 1,000 cases are strongest over 200 cases, with 80.0% human and 72.5% LLM win rates, but narrow to 52.5% and 57.5% over 500 cases.The reported pattern suggests early benefits from increased data followed by a possible plateau.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 Ablation Studies

  • Removing the legal-provisions database lowers human-evaluation win rates to 57.5%, 65.0%, and 65.0% across cognitive agility, professional knowledge, and logical rigor.The ablation evaluates GPTM-1000 after excluding individual knowledge bases.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Removing case knowledge reduces win rates to 52.5%, 60.0%, and 52.5%, while removing the experience base lowers cognitive-agility wins to 62.5%.LLM evaluations show similar patterns, with legal provisions especially important for logical rigor and experience especially influential for cognitive agility.Source: §4.3 · p. 7

4.4 Case Analysis

  • In a contract-dispute case, the evolved model cited relevant provisions, used five-element argumentation and evidence combinations, and handled shipping-agency issues more systematically than the baseline.It secured USD 27,509.40 as plaintiff and defended against all claims as defendant.Source: §4.4 · pp. 7–8

5 Conclusion

AgentCourt combines simulated courtroom debates, adversarial evolution, and a three-tier knowledge base to improve legal agents’ reasoning in dynamic courtroom dialogue. The paper reports state-of-the-art performance against general-purpose and specialized legal models and introduces CourtBench for interaction-focused evaluation.

5 Conclusion

  • AgentCourt uses adversarial evolution and legal-provisions, case-precedent, and strategic-experience knowledge bases to refine agents’ legal reasoning.The framework is intended to address limitations of static legal AI systems through continuous adaptation to new case contexts.Source: §5 · p. 8
  • The work introduced CourtBench, a benchmark designed to evaluate legal AI models based on real courtroom interactions rather than static legal knowledge retrieval, further validating the work’s approach.Source: §5 · p. 9

Limitations

The framework relies on external API calls and generates approximately 3,361 tokens per debate round, creating latency and response-time challenges for large-scale or real-time applications. Its structured legal setting may also require adaptation across jurisdictions.

Limitations

  • API dependence and approximately 3,361 generated tokens per debate round may constrain response speed, long-term knowledge evolution, and real-time legal advisory use.The framework may also require adaptation to generalize across diverse legal systems and jurisdictions.Source: Limitations · p. 9

Ethics Statement

The section describes AgentCourt’s simulated-court workflow and emphasizes that its outputs are intended for training and research, not autonomous legal decisions. The workflow combines role-specific agents, structured proceedings, knowledge retrieval, and post-case reflection.

Ethics Statement

  • AgentCourt is intended as a supplementary tool for legal training and research, not to replace legal professionals or make actual legal decisions.Its generated outputs require qualified legal review before practical application and should not be treated as formal legal advice or used directly in court without human oversight.Source: Ethics Statement · p. 9

A System Overview

  • The simulation uses judge, lawyer, clerk, plaintiff, and defendant roles within a structured court environment.The workflow includes role-specific prompts and procedural responsibilities for courtroom participants.Source: A System Overview · p. 10, §B.1 · p. 10, §B.2 · p. 10

C Simulation Workflow Details

  • Lawyer agents retrieve relevant information from three knowledge bases during debate and reflect after each case to refine their legal reasoning.The reflection process focuses on legal provisions, self-assessment, opponent learning, and case analysis, enabling continuing strategy evolution.Source: Figure 5 · p. 11, C Simulation Workflow Details · p. 11
  • The system runs through pre-trial preparation, court proceedings, and knowledge construction.Proceedings include case presentation, judge-identified disputes, multi-round lawyer debates, a ruling, and trial-record archiving.Source: C Simulation Workflow Details · p. 11
  • The workflow begins with 1,000 real civil cases, using complaints and defense statements as simulation backgrounds and assigning roles randomly.Random assignment exposes lawyer agents to multiple litigation perspectives.Source: C Simulation Workflow Details · p. 11

D Data Settings and Processing

This section explains that access to pleadings is constrained because they are often confidential and proprietary. It frames this restricted access as a motivation for constructing alternative legal data resources.

D Data Settings and Processing

D.1 Accessing Confidential Pleadings

  • Confidential and proprietary pleadings limit access to foundational documents used in legal proceedings.Traditional open-source datasets are insufficient because primary case files are commonly restricted to court filing systems and private records.Source: §D.1 · p. 12

D.2 Dataset Construction and Preprocessing

The dataset pipeline filters public civil judgments into representative cases and generates anonymized legal texts for simulated moot-court training and evaluation.

D.2 Dataset Construction and Preprocessing

  • 1,000 representative cases were selected from 10,000 civil judgments after preprocessing and semantic clustering.The selection retained cases containing both plaintiff claims and defendant defenses, while BERT-based vectorization and K-Means clustering reduced redundancy.Source: §D.2 · p. 12

D.3 Data Generation and Anonymization

  • The resulting generated dataset contains 1,000 training samples and 50 test samples for legal argumentation and judgment prediction.ERNIE-Speed-128K was used to generate and anonymize simulated legal texts tailored to civil judgments.Source: §D.3 · p. 12

E Case Study Analysis

The case-study evaluation compares GPT-4o-mini-1000 with a baseline across plaintiff and defendant roles, while CourtBench assessment uses structured prompts and professional scoring dimensions.

E Case Study Analysis

  • GPT-4o-mini-1000 is compared with a baseline in both plaintiff and defendant roles.The comparisons are presented in Figure 8a and Figure 8b.Source: E Case Study Analysis · p. 12

F.2 CourtBench Assessment Prompts

  • Table 2 provides structured prompt templates for lawyer-agent planning and execution.Source: Table 3 · p. 13

F.1 LLM Evaluation Prompts

  • The evaluation assesses court-debate performance across cognitive agility, professional knowledge, and logical rigor.Each dimension has specific criteria and scoring guidelines intended to support consistent evaluation.Source: §F.1 · p. 13

F.2 CourtBench Assessment Prompts

  • CourtBench uses base prompts for standard evaluation and enhanced prompts that add professional knowledge references for models with external knowledge bases.Source: §F.2 · p. 13

G LawBench Result

Legal-specific models perform decently on standardized LawBench tasks but decline on CourtBench, which evaluates applying legal knowledge in courtroom debates. A representative contract-dispute debate shows the evolved lawyer producing more structured legal arguments and a judgment supporting key claims.

G LawBench Result

  • Legal-specific models show decent LawBench performance but decline on CourtBench, where legal knowledge must be applied in courtroom debate.LawBench covers memorization, understanding, and consultation tasks, whereas CourtBench assesses courtroom debate ability.Source: G LawBench Result · p. 13, Table 8 · p. 15
  • In the debate, the plaintiff’s lawyer grounds the cargo-loss request in contracts, records, and cited maritime law, while the defendant challenges unsupported travel expenses.The opposing arguments illustrate the need to connect legal provisions with case evidence rather than merely recite rules.Source: G LawBench Result · p. 16
  • The evolved lawyer supports the cargo-loss claim and litigation-cost allocation while rejecting the plaintiff’s inadequately documented travel-expense claim.The judgment awards $27,509.40 plus interest for cargo losses, denies 5,000 yuan in travel expenses, and assigns specified litigation fees to the defendants.Source: G LawBench Result · p. 16
  • GPT-4o-mini-1000 uses precise legal citations, systematic arguments, and professional expertise more effectively than GPT-4o-mini in a contract dispute.The comparison is presented through representative excerpts from the most relevant debate round.Source: Figure 8 · p. 16

Related papers

Loading 2408.08089v2…