Source-linked AI summary
RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Game
Shijia Xu, Yu Wang, Xiaolong Jia, Zhou Wu, Kai Liu, April Xiaowen Dong
TL;DR
Automated contract revision with LLMs lacks rigorous behavioral constraints and remains vulnerable to hallucinated safety. RCBSF addresses this by using a hierarchical GPA-CRA-LVA Stackelberg game with risk budgets and iterative optimization. On a unified legal benchmark, it achieves state-of-the-art results, including an average RRR of 84.21% and improved token efficiency.
Problem
LLM-based contract revision remains vulnerable to inefficient review, legal hallucinations, and insufficiently rigorous behavioral constraints.
Method
RCBSF uses a hierarchical Stackelberg architecture in which the GPA imposes risk constraints on CRA and LVA for iterative contract optimization.
Results
84.21% average Risk Resolution Rate is achieved with Qwen2.5-7B-Chat, compared with 79.56% for the Iteration baseline and 70.81% for the Standard method.
Takeaways & Limitations
RCBSF balances strict risk compliance with linguistic fidelity in automated contract revision.
Takeaways & Limitations
The benchmark insufficiently represents complex cross-clause conflicts, interdependent risks, niche disputes, and other intricate contract scenarios.
Abstract
from arXiv · showhide
Despite the widespread adoption of Large Language Models (LLMs) in Legal AI, their utility for automated contract revision remains impeded by hallucinated safety and a lack of rigorous behavioral constraints. To address these limitations, we propose the Risk-Constrained Bilevel Stackelberg Framework (RCBSF), which formulates revision as a non-cooperative Stackelberg game. RCBSF establishes a hierarchical Leader Follower structure where a Global Prescriptive Agent (GPA) imposes risk budgets upon a follower system constituted by a Constrained Revision Agent (CRA) and a Local Verification Agent (LVA) to iteratively optimize output. We provide theoretical guarantees that this bilevel formulation converges to an equilibrium yielding strictly superior utility over unguided configurations. Empirical validation on a unified benchmark demonstrates that RCBSF achieves state-of-the-art performance, surpassing iterative baselines with an average Risk Resolution Rate (RRR) of 84.21\% while enhancing token efficiency. Our code is available at https://github.com/xjiacs/RCBSF .
1 Introduction
Contract review is professionally demanding and vulnerable to ambiguity, implicit clause conflicts, hallucinations, and inconsistent quality. RCBSF addresses these challenges by organizing risk-constrained revision through a hierarchical multi-agent Stackelberg framework.
- Contract drafting, revision, and review are time-consuming because textual ambiguity and implicit clause conflicts complicate the process.
- Improper contract handling can cause losses reaching 9.2% of an enterprise’s annual revenue.
- Generic legal models remain vulnerable to hallucinations, false citations, privacy weaknesses, and insufficient logical consistency.
- RCBSF models contract revision as a bilevel Stackelberg game with a Global Prescriptive Agent leading a Constrained Revision Agent and Local Verification Agent.
- The framework imposes strict 5-dimensional risk constraints and theoretically targets a risk-minimized equilibrium with superior utility over standard prompting.
- On a unified benchmark, RCBSF achieves state-of-the-art performance in Risk Resolution Rate and Token Efficiency Score.
2 Related Work
Legal LLMs and multi-agent systems have improved specialized legal processing and collaborative reasoning, but existing approaches leave strategic risk-constrained contract revision insufficiently addressed. RCBSF builds a non-cooperative bilevel game on these foundations.
- Legal LLMs are developed through legal-domain pretraining or fine-tuning on instruction datasets aligned with legal reasoning patterns.
- Retrieval-augmented methods are used to mitigate hallucination and outdated knowledge by retrieving information from external knowledge bases.
- Multi-agent systems use role division and interactive collaboration to support complex tasks and quality enhancement from multiple perspectives.
- Recent Stackelberg-game and no-regret-learning research provides foundations for handling strategic uncertainty.
- RCBSF operationalizes generation-audit interaction as a non-cooperative bilevel game in which the follower is constrained to converge toward a risk-constrained Stackelberg equilibrium.
3 Methodology: A Stackelberg Game Theoretic Framework for Contract Revision
RCBSF formulates contract revision as a hierarchical game: the GPA constrains the follower’s revision space through structured risk guidance, while CRA and LVA iteratively optimize the contract. The framework includes budget penalties, divergence regularization, and convergence guarantees.
- RCBSF models risk prescriptive logic and generative revision as a non-cooperative game on a semantic Riemannian manifold.
- The Leader is the GPA, while the Follower is a revision system comprising the Drafter and Local Verifier.
- The GPA projects contract states onto a structured risk manifold and constructs a five-feature instruction tensor covering Category, Location, Evidence, Issue, and Suggestion.
- The bilevel optimization lets the Leader maximize weighted risk mitigation under token constraints while the Follower generates revisions under the Leader’s induced strategy.
- The Follower maximizes language-model likelihood while regularizing the draft so its audited risk distribution aligns with the Leader’s target distribution.
- Iterative interaction uses a fusion gradient, with Ffusion aggregating outer expectations and inner audits and η representing the learning rate controlled by prompt intensity.
- Theoretical results state strict utility superiority over unguided Nash-equilibrium generation under a non-convex risk manifold and convergence to a stationary point with ∥∇Lrisk(x∗)∥<ϵ.
4 Experiments
The experiments evaluate RCBSF across risk resolution, contract quality, and token efficiency using a unified benchmark assembled from four legal datasets. The setup standardizes data, compares representative baselines, and uses a resource-conscious backbone configuration.
- Experiments assess contract risk resolution, contract quality, and token efficiency.
- The unified benchmark combines PrivacyQA, ContractNLI, MAUD, and CUAD to cover privacy policies, non-disclosure agreements, and merger agreements.
- Raw documents undergo anonymization and template standardization to address personally identifiable information and inconsistent formatting.
- RCBSF is compared with four representative legal text generation and revision baselines, including Standard Zero-Shot and CoT-Refinement.
- Risk Resolution Rate measures the percentage of ground-truth risks mitigated, while Token Efficiency Score measures risks resolved per 1,000 tokens.
- Qwen2.5-7B-Chat serves as the backbone for all RCBSF agents, with CUAD training data used to tune risk weights and token budgets.
5 Results
RCBSF consistently outperforms baseline methods across datasets, backbones, risk resolution, contract quality, and token efficiency. Its strongest reported Qwen2.5-7B-Chat results include 84.21% average RRR and 87.29% TES.
- RCBSF achieves the best overall performance across four datasets and various LLM backbones.
- Contract Quality: RCBSF dominates Contract Quality across settings, measuring clarity, rigor, balance, and professionalism on a 0–100 scale.
- Contract Quality: Across Clarity, Rigor, Balance, and Professionalism, RCBSF outperforms RAG and Iteration in the Figure 3 heatmap.The heatmap uses the Qwen2.5-7B-Chat model for each individual dataset.
- Token Efficiency: 87.29% TES with Qwen2.5-7B-Chat exceeds Iteration at 83.40% and Standard at 74.31%.The result supports the reported utility-cost optimization of the Stackelberg framework.
- Table 1 compares LLM backbones across four datasets using CQ, RRR, and TES, with dark blue and light blue marking first- and second-best results.
6 Analysis and Ablation Studies
The analysis evaluates RCBSF through ablations, sensitivity tests, and qualitative comparisons, showing that detailed risk guidance, budget constraints, iterative refinement, and Q-Score weighting shape effectiveness and efficiency.
- Ablation Studies: Removing 5-dimensional guidance lowers RRR by 11.06% and Win Rate to 15.2%, confirming the importance of Evidence and Suggestion fields.The unguided configuration also permits hallucinated fixes in irrelevant contract clauses.
- Ablation Studies: Removing the budget penalty increases RRR by 0.64% but reduces TES by 12.14%, while BLEU-4 decreases to 83.45.Budget-aware weighting limits minor-issue flags and longer revisions, prioritizing Critical and Major risks.
- Ablation Studies: Reducing the game to K = 1 decreases RRR by 7.76% but increases TES by 3.16%, as single-pass generation misses complex, interdependent risks.Iterative LVA verification catches residual risks or errors introduced during initial revision.
- Ablation Studies: Removing Q-Score weighting decreases RRR by 4.86% and TES by 6.14%, indicating that semantic-severity prioritization guides revisions toward impactful risks.
- Sensitivity Analysis: RRR rises from 74.24% at K = 1 to above 84.21% by K = 3, after which marginal gains diminish while token cost grows linearly.The analysis identifies K = 3 as the Pareto-optimal stopping point; τ = 1.0 provides the best exploration–exploitation balance.
- Qualitative Comparison: In the Force Majeure example, RCBSF converts the GPA’s structured 48-hour suggestion into contract text, whereas the baseline claims a fix without changing the text.
7 Conclusion
RCBSF frames automated contract revision as a hierarchical game that separates strategic auditing from execution. Its GPA-guided follower system iteratively balances risk compliance with linguistic fidelity on legal benchmarks.
- RCBSF uses a GPA to constrain a follower system composed of the CRA and LVA, enabling iterative optimization within precise bounds.
Limitations
The benchmark mainly contains publicly available, structurally consistent clauses and insufficiently tests complex cross-clause interactions and interdependent risks. Validation is also concentrated in United States and Common Law settings and English.
- The benchmark rarely includes highly complex cross-clause conflicts or niche disputes, leaving implicit unfairness and risks spanning multiple sections insufficiently investigated.
- Validation uses predominantly United States and Common Law datasets in English, so performance in Civil Law systems and non-English languages remains unverified.The paper notes that current prompting may encode US-centric legal norms.
Ethical Considerations
The paper treats RCBSF as an assistive system for high-stakes legal work rather than a substitute for qualified counsel. It emphasizes verification, automation-bias risks, and privacy safeguards.
- Because legal LLM deployment carries hallucination and bias risks, users must verify automated revisions and avoid treating RCBSF as a substitute for legal counsel.Experiments used publicly available datasets with permissible licenses and de-identification pipelines to prevent PII exposure.
Appendices
The appendices document the unified benchmark, data-processing pipeline, theoretical guarantees, and sensitivity and quality analyses supporting RCBSF.
- Dataset Construction: The unified benchmark aggregates 711 legal contracts across 41 categories from four legal NLP datasets, including CUAD, MAUD, ContractNLI, and PrivacyQA.The category distribution is long-tailed, with Merger at 18.4% and Disclosure at 8.6%.
- Dataset Construction: The preprocessing pipeline classifies raw headers, standardizes contract structure, anonymizes personally identifiable information, and generates machine-readable text.Header normalization uses noise reduction and hierarchical keyword mapping before template standardization.
- Theoretical Analysis: The framework theoretically guarantees Stackelberg equilibrium existence, convergence of the inner loop, and superior Leader utility over Nash equilibrium under stated conditions.These claims rely on continuity, compactness, upper hemicontinuity, and a contraction condition.
- Sensitivity Analysis: 84.21% RRR is reached at K = 3, while increasing rounds to K = 8 yields only 84.24% and raises normalized token cost from 2.8× to 7.3×.The results characterize a diminishing-returns trade-off between risk resolution and computational efficiency.
- Sensitivity Analysis: 84.21% RRR is achieved at τ = 1.0, whereas RRR falls to approximately 76.54% at τ = 0.1 and 72.32% at τ = 2.0.Low temperatures miss subtle risks, while high temperatures destabilize the instruction vector.
- Quality Analysis: RCBSF improves fine-grained contract quality across backbones, with QW-7B-Chat + RCBSF reaching average CQ 86.87 and LawLLM-7B Rigor increasing from 65.81 to 79.81.LexiLaw-6B average quality also increases from 60.40 to 72.30.
E.2 Ablation Study Breakdown
The ablation and qualitative analyses show that explicit risk dimensions, evidence grounding, actionable suggestions, and iterative auditing are central to RCBSF’s revision behavior.
- Ablation Study: Removing Evidence increases Hallucination Rate from 4.23% to 18.68%, causing revisions to target unrelated clauses with similar keywords.Evidence anchors generation to the text under review.
- Ablation Study: Removing Suggestion increases Fix Failure Rate from 7.35% to 19.86%, producing generic mitigations instead of concrete constraints.The cited example contrasts vague language with enforcing liability caps.
- Ablation Study: Removing Location increases Hallucination Rate from 4.23% to 12.47% by weakening target-scope boundaries.The model may then modify non-target sections or fabricate irrelevant clauses.
- Qualitative Cases: In a liability example, the LVA flags a missing financial cap after the first revision, and the CRA ultimately adds a cap of 12 months’ fees.The case illustrates iterative audit feedback enforcing a Leader constraint.
- Qualitative Cases: In a software-services example, a three-round interaction resolves Payment and Termination risks first, then resolves IP ownership after LVA feedback.The final clause balances fairness, clarity, and asset protection.
F.2 Failure Case Analysis
The failure analysis identifies context-length and jurisdictional limitations, while the framework’s prompts operationalize structured risk extraction, constrained revision, and local auditing.
- Failure Cases: RCBSF’s main failure modes arise from context limitations and jurisdictional ambiguity.The analysis frames these as boundaries despite the framework’s strong reported performance.
- Failure Cases: In documents longer than 15,000 tokens, the Leader may misattribute evidence across sections, causing redundant revisions and lower conciseness.The example involves confusing a definition on page 5 with a closing condition on page 80.
- Failure Cases: US-centric training data can produce jurisdictional overfitting when the system handles Civil Law contracts.The German-law example incorrectly suggests adding a consideration clause that is not required for validity.
- Framework Operation: The Leader extracts a five-dimensional risk vector containing Category, Location, Evidence, Issue, and Suggestion to ground revision instructions.Location supplies pointers, Evidence quotes the risky text, and Suggestion provides an actionable instruction.
- Framework Operation: Risk quantization evaluates Validity/Compliance, Liability Scope, Control Allocation, and Remediability to construct the weighted instruction vector.These dimensions cover invalidation or penalties, financial exposure, power balance, and repair difficulty.
- Framework Operation: The CRA optimizes revisions using Leader constraints, LVA feedback, and budget control, while the LVA audits each iteration for residual risk.The inner-loop audit produces structured feedback that guides subsequent generations.