Source-linked AI summary
LawThinker: A Deep Research Legal Agent in Dynamic Environments
Xinyu Yang, Chenlong Deng, Tongyu Wen, Binyu Xie, Zhicheng Dou
TL;DR
Legal reasoning systems must ensure that intermediate reasoning is accurate and procedurally compliant, because unverified errors can propagate through long-horizon tasks. LawThinker enforces verification after each exploration step through an Explore-Verify-Memorize agent with a three-dimensional DeepVerifier and memory module. It improves performance over direct and workflow-based methods on dynamic evaluation while also improving accuracy on static benchmarks.
Problem
Legal reasoning requires accurate outcomes, fact-grounded reasoning, and procedural compliance, but errors such as inapplicable statute citations can propagate through long-horizon interactions.
Method
LawThinker enforces an Explore-Verify-Memorize strategy in which DeepVerifier checks each exploration result for knowledge accuracy, fact-law relevance, and procedural compliance, while memory stores validated knowledge.
Results
LawThinker achieves a 24% improvement over direct reasoning and an 11% gain over workflow-based methods on J1-EVAL, plus approximately 6% average accuracy improvement over direct reasoning across three static benchmarks.
Takeaways & Limitations
The results support Explore-Verify-Memorize as a strategy for accurate outcomes and legally compliant reasoning processes across dynamic and static legal tasks.
Takeaways & Limitations
The evaluated task setting assumes that relevant information is gathered incrementally across dialogue rounds, with reasoning depending on accumulated prior information.
Abstract
from arXiv · showhide
Legal reasoning requires not only correct outcomes but also procedurally compliant reasoning processes. However, existing methods lack mechanisms to verify intermediate reasoning steps, allowing errors such as inapplicable statute citations to propagate undetected through the reasoning chain. To address this, we propose LawThinker, an autonomous legal research agent that adopts an Explore-Verify-Memorize strategy for dynamic judicial environments. The core idea is to enforce verification as an atomic operation after every knowledge exploration step. A DeepVerifier module examines each retrieval result along three dimensions of knowledge accuracy, fact-law relevance, and procedural compliance, with a memory module for cross-round knowledge reuse in long-horizon tasks. Experiments on the dynamic benchmark J1-EVAL show that LawThinker achieves a 24% improvement over direct reasoning and an 11% gain over workflow-based methods, with particularly strong improvements on process-oriented metrics. Evaluations on three static benchmarks further confirm its generalization capability. The code is available at https://github.com/yxy-919/LawThinker-agent .
1 Introduction
LawThinker addresses legal reasoning settings where correct outcomes must also rest on accurate, fact-relevant, and procedurally compliant reasoning. It combines step-level verification with memory and reports gains on dynamic and static legal benchmarks.
- Legal reasoning requires both correct conclusions and accurate, fact-grounded, procedurally compliant reasoning processes.
- Inapplicable statute citations can yield apparently coherent explanations and even correct final answers while resting on incorrect legal grounds.The motivating example selects Article 985 instead of the applicable Article 122.
- LawThinker enforces an Explore-Verify-Memorize strategy, using DeepVerifier to check knowledge accuracy, fact-law relevance, and procedural compliance after exploration.
- 24% overall improvement over direct reasoning and an 11% gain over workflow-based methods were achieved on J1-EVAL, with particularly strong process-oriented improvements.The dynamic benchmark covers six judicial scenario types.
- Approximately 6% average accuracy improvement over direct reasoning was observed across LawBench, LexEval, and UniLaw-R1-Eval.These static-benchmark evaluations support generalization beyond dynamic settings.
- 15 legal tools support exploration, verification, and memorization for navigating legal knowledge, validating intermediate reasoning, and reusing verified information across long-horizon tasks.
2 Related Work
The related work spans LLM-based legal reasoning, specialized judicial frameworks, process verification, and interactive courtroom-oriented systems. The supplied passages position procedural compliance as an unresolved challenge despite these advances.
- LLM Reasoning and Verification: Recent legal reasoning research extends general chain-of-thought with legal syllogism reasoning and specialized frameworks for judicial complexity.
- LawThinker: LawThinker is presented as a legal reasoning agent with Explore-Verify-Memorize capability.
- LLM Reasoning and Verification: LegalReasoner uses step-wise reasoning and a process verifier, but its verification focuses on contribution to a correct conclusion rather than legally prescribed procedures.
- Dynamic Judicial Scenarios: Interactive legal systems model courtroom procedures, judicial-bench discussions, and multi-agent interactions in dynamic scenarios.
3 Methodology
LawThinker models multiround legal tasks as stepwise reasoning with autonomous exploration, enforced verification, and persistent memory. Its verifier assesses retrieved knowledge, legal relevance, and procedural compliance before the agent proceeds or revises.
- Task Formulation: Dynamic legal tasks require agents to generate reasoning and responses across multiround dialogues using available tools and accumulated history.
- Task Formulation: Errors introduced in early dialogue rounds can propagate through later reasoning, creating a need for accurate and verified long-horizon knowledge.
- Step-Level Verification: Each reasoning step may invoke exploration, after which verification assesses knowledge accuracy, fact-law relevance, and procedural compliance.The assessment guides acceptance, reasoning revision, or query reformulation and re-exploration.
- Framework Overview: LawThinker treats exploration and verification as an enforced atomic operation, with DeepVerifier activated immediately after every exploration.
- Framework Overview: The framework verifies individual reasoning steps and stores validated legal knowledge and case context for retrieval in subsequent rounds.The memory module is intended to avoid redundant exploration.
- Exploration Tools: Seven exploration tools cover statute and charge discovery, precedent retrieval, and task-guided support for drafting and courtroom simulation.
- Exploration Tools: Tool invocation is autonomous: the agent selects tools and constructs queries from perceived knowledge gaps without predefined triggers or fixed sequences.
- DeepVerifier: DeepVerifier uses a separate prompt context to evaluate exploration results from an independent perspective rather than continuing the main agent’s generative flow.
4 Experiments
Experiments evaluate LawThinker across dynamic and static legal settings, showing that explicit verification and memory improve both outcome accuracy and procedural compliance. LawThinker consistently surpasses baseline reasoning paradigms, especially on complex and process-oriented tasks.
- Experimental Setup: J1-EVAL evaluates legal agents across 508 judicial environments spanning six scenario types and three hierarchical complexity levels.Metrics combine rule-based and LLM-based evaluations, including document, judgment, crime, law, format-following, and procedural-following scores.
- Baseline Results: Direct reasoning remains below 50% overall accuracy because models rely on internal knowledge, insufficient exploration, and weak intermediate-step verification.Performance deteriorates further in complex scenarios requiring procedural awareness and long-horizon contextual reasoning.
- Baseline Results: Workflow-based methods improve overall accuracy by over 13% compared with direct reasoning, but unverified external information can introduce noise and reduce procedural compliance.In Complaint Drafting, workflow-based methods can underperform direct reasoning, showing that external exploration alone is insufficient.
- Main Results: 24% improvement over direct reasoning and 11% gain over workflow-based methods are achieved by LawThinker overall on J1-EVAL.Its strongest gains occur on process-oriented metrics such as FOR and PFS, while Level-III results also show stronger court-stage completion and judgment accuracy.
- Ablation Study: Removing DeepVerifier consistently reduces performance across scenarios, especially in knowledge-intensive tasks and Criminal Court, where procedural compliance is strict.The ablation results indicate that explicit verification reduces hallucinations and reasoning errors.
- Ablation Study: Removing memory causes more severe degradation in document drafting and court simulation than in Knowledge Questioning.Long-horizon tasks require retaining information across multiple turns, whereas Knowledge Questioning relies mainly on current-turn information.
- Ablation Study: The Explore-Verify-Memorize strategy yields the largest gains, with removing any component degrading performance across all scenarios.The study attributes robust legal reasoning to the synergy of exploration, verification, and memorization.
- Quantitative Analysis: LawThinker leads on both outcome-oriented and process-oriented metrics and achieves the highest completion rates across all civil and criminal court stages.Other methods often score below 25% in Preparation and Investigation, while workflow-based methods such as ReAct can underperform direct reasoning.
5 Conclusion
The paper proposes LawThinker as an autonomous legal research agent for dynamic judicial environments. Its Explore-Verify-Memorize strategy and specialized tools improve both outcome accuracy and procedural compliance across dynamic and static benchmarks.
- Conclusion: LawThinker integrates iterative knowledge exploration with explicit verification and uses 15 specialized tools spanning exploration, verification, and memorization.Experiments on one dynamic and three static legal benchmarks report stronger outcomes and legally compliant reasoning processes than existing methods.
A Detailed Description of Legal Tools
LawThinker uses specialized tools to support legal knowledge acquisition, intermediate-step checking, and context retention during sustained judicial interactions.
- Tool Functions: The tool suite retrieves and expands relevant legal knowledge, examines intermediate reasoning validity and consistency, and preserves context across multi-turn interactions.The taxonomy covers tools for exploration, verification, and memorization.
B Additional overall performance
Additional J1-EVAL results define metrics for binary and open-ended questions and assess legal-document quality through key component scores. These results show increasingly strong advantages for LawThinker as court simulations become more complex.
- Metrics: J1-EVAL evaluates Level-I responses with Binary Accuracy and Non-Binary Accuracy, and assesses Level-II documents through component-level scores.The reported document components include plaintiff, defendant, and claims information.
- Results: LawThinker’s advantages become increasingly pronounced in more complex court simulation scenarios.The results are interpreted as evidence that Explore-Verify-Memorize verifies intermediate reasoning and improves procedural compliance and judicial decision-making performance.
C Descriptions of Static Benchmarks
The paper evaluates LawThinker on three static legal benchmarks covering broad legal abilities, judicial cognition, and knowledge- versus case-based reasoning. It selects knowledge-intensive tasks and dominant legal domains from these resources.
- LexEval: LexEval organizes 23 legal tasks across six abilities, including memorization, understanding, logical inference, discrimination, generation, and ethics.Its 14,150 questions combine existing benchmarks, legal examination datasets, and expert-annotated samples.
- LawBench: LawBench evaluates legal knowledge memorization, comprehension, and application across 20 tasks and five output categories.The categories include single-label classification, multi-label classification, regression, information extraction, and text generation.
- UniLaw-R1-Eval: UniLaw-R1-Eval contains 800 samples from JEC-QA and Chinese National Judicial Examination data, divided into knowledge-based and case-based subsets.Its source examination data spans 2015 to 2021.
- Evaluation scope: The evaluation selects knowledge-intensive tasks from LexEval and LawBench and dominant domains from UniLaw-R1-Eval.Selected domains include criminal procedure, labor, commercial, and constitutional law, plus jurisprudence.
D Prompts in LawThinker
The paper provides detailed prompts and documents the exploration, verification, and memorization tools used by LawThinker, alongside fine-grained evaluations across J1-EVAL levels.
- Prompts: Detailed LawThinker prompts are provided in Tables 7, 8, 9, and 10.
- Tools: Table 3 overviews the legal agent’s exploration, verification, and memorization tools.
- Evaluation: Table 4 reports fine-grained evaluation at Levels I and II in J1-EVAL.
- Evaluation: Table 5 reports fine-grained evaluation at Level III in J1-EVAL.
- Examples: Figures 7 through 10 present LawThinker instructions, exploration examples, DeepVerifier prompts, and checking-phase examples.