PaperlayerView exact revision on arXiv

Source-linked AI summary

CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method

Jin Su, Zhuofeng Zhao, Huanhuan Wang, Hao Chen

arXiv:2608.17536v1cs.CLcs.AI

TL;DR

Legal questions vary in complexity, but fixed or indiscriminate retrieval strategies can trade off reasoning quality, interpretability, and efficiency. CoAL-RAG routes retrieval adaptively using query reasoning demand and retrieval consistency, and outperforms baselines across Chinese and English legal benchmarks while balancing accuracy and efficiency.

  • Problem

    Legal queries vary from single-provision questions to multi-step, condition-heavy cases, while existing vector, graph, and hybrid retrieval strategies provide limited reasoning or introduce redundancy, noise, or conflicts.

  • Method

    CoAL-RAG combines multidimensional query-complexity assessment with retrieval-consistency analysis to route retrieval strategies adaptively and truncate noisy context.

  • Results

    0.2815 BLEU on LawBench was CoAL-RAG’s highest reported Chinese-benchmark score, while experiments across Chinese and English benchmarks showed significant baseline outperformance and cross-jurisdictional validation.

  • Takeaways & Limitations

    CoAL-RAG offers a complexity-aware legal question-answering framework that balances deep reasoning, generation quality, and response efficiency across civil- and common-law benchmarks.

  • Takeaways & Limitations

    CoAL-RAG can misclassify edge cases requiring implicit legal hierarchy and struggles with local policies absent from its statutory knowledge graph.

Abstract

from arXiv · show

Legal consultation questions exhibit multi-level complexity. A single retrieval strategy often leads to over-reasoning for simple questions and poor interpretability for complex ones, making it difficult to meet the requirements for both answer quality and efficiency in high-risk scenarios. To address this issue, this paper proposes CoAL-RAG, a complexity-aware legal retrieval-augmented generation method, which constructs a multi-dimensional evaluation mechanism based on ``question essence'' and ``retrieval consistency'' to enable adaptive routing of retrieval strategies. First, the reasoning demand is quantified according to the logical structure of the question. Then, the discrepancy between semantic retrieval and keyword retrieval is utilized to indirectly reflect problem complexity, thereby selecting the most appropriate retrieval strategy and dynamically filtering contextual information. Experimental results demonstrate that the proposed method significantly outperforms baseline models not only on Chinese legal benchmarks (SocialLawQA, LawBench) but also demonstrates strong cross-jurisdictional generalization on English datasets (LexGLUE, CaseHold). Specifically, on Chinese datasets, the BLEU score improves by 42.5\% and ROUGE-L reaches 3.6 times that of knowledge graph-based methods. On English benchmarks, CoAL-RAG maintains highly competitive accuracy, achieving an optimal balance between generation quality, deep logical reasoning, and system efficiency across different legal systems.

1 Introduction

CoAL-RAG addresses the wide variation in legal-query complexity by adaptively routing retrieval strategies using internal query logic and retrieval consistency. It combines complexity awareness, hybrid retrieval, and knowledge-graph coordination to balance efficient responses with deep logical accuracy across jurisdictions.

  • Problem: Legal consultation questions vary substantially in complexity, creating different reasoning requirements for simple statutory queries and more demanding cases.High-stakes legal answering also requires accuracy and traceability, motivating external knowledge integration to reduce hallucination and knowledge obsolescence.
  • Problem: Vector retrieval provides fragmented provisions, graph reasoning introduces redundancy and noise, and direct hybrid retrieval can produce inconsistency and reasoning conflicts.These limitations motivate adaptive selection rather than indiscriminate use of a single or combined retrieval strategy.
  • Method: The proposed multi-dimensional mechanism evaluates query complexity through internal logic and external retrieval consistency.Its competition-function algorithm supports adaptive routing with numerical stability and probabilistic interpretability.
  • Method: CoAL-RAG synergizes complexity awareness, hybrid retrieval, and knowledge-graph coordination to tailor retrieval strategies for efficient and accurate legal question answering.It dynamically filters context and selects strategies according to complexity scores, balancing response speed with deep logical accuracy.
  • Validation: Experiments on Chinese civil-law datasets and English common-law benchmarks show that CoAL-RAG outperforms existing baselines while improving complex-query accuracy and maintaining low-latency responses.The evaluated datasets are SocialLawQA, LawBench, LexGLUE, and CaseHold.

2 Related Works

Prior legal language models improved comprehension, intent recognition, and reasoning but remained inadequate for highly complex, multi-step consultations. Retrieval-augmented and knowledge-graph methods address knowledge and reasoning needs, yet face legal-specific limitations in complexity assessment, interpretability, noise, latency, and statutory structure.

  • Legal language models: General-purpose LLMs suffer from legal hallucinations, while later domain, instruction-tuned, and knowledge-augmented models still struggle with highly complex, multi-step legal consultations.Domain models improved comprehension; instruction-tuned and knowledge-augmented models improved intent recognition and reasoning.
  • Retrieval-augmented generation: RAG methods use hybrid retrieval, re-ranking, and dynamic routing, but token confidence misses rigorous judicial deduction and one-dimensional classifiers overlook multifaceted legal queries.The passage contrasts these limitations with methods such as FLARE and Adaptive-RAG in open-domain settings.
  • Knowledge-graph reasoning: Knowledge graphs support complex RAG reasoning through explicit paths or subgraph extraction, but statutory applications introduce retrieval noise, latency, and loss of chapter-and-clause structure.The passage attributes these challenges to indiscriminate retrieval and general models that fail to capture legal-text hierarchy.

3 Method

CoAL-RAG models legal-query complexity from both question essence and retrieval consistency, then adaptively routes queries across retrieval strategies and filters context. Its pipeline combines a hierarchical legal knowledge graph, multi-dimensional scoring, competitive retrieval gating, and score-cliff truncation.

  • Framework: CoAL-RAG represents each input as a query, unstructured legal corpus, and hierarchical legal knowledge graph, enabling complexity-aware dynamic retrieval routing.The system is implemented with LangGraph and uses the graph alongside corpus and query information.
  • Question essence: The intrinsic complexity score combines query dimensions including reasoning-chain length, knowledge integration, domain span, relational reasoning, and conditional-constraint density.Complexity scores increase with relevant units or logical structures, while dimension-specific saturation marks highly complex queries.
  • Retrieval consistency: BM25 and vector retrieval provide external feedback through QSI and RDI, measuring literal-match reliability and disagreement between keyword and semantic results.Higher QSI indicates lower retrieval complexity, whereas higher RDI indicates greater disagreement between retrieval viewpoints.
  • Retrieval consistency: Competitive gating nonlinearly fuses QSI and RDI into simple and complex evidence energies before calculating retrieval-consistency complexity.The method sets p = 1.5 to penalize weak literal matches and q = 0.3 to smooth responses to retrieval divergence.
  • Complexity fusion: The final complexity score equally weights question structure and retrieval feedback, combining internal reasoning demands with external evidence consistency.The balancing coefficient is γ = 0.5.
  • Routing and context filtering: Adaptive routing uses θlow = 0.25, θmedium = 0.45, and θhigh = 0.7 to organize processing into four tiers, while score-cliff truncation removes tail noise.With σ = 0.2, truncation selects the first index whose adjacent-document score decline exceeds 20%, forming the final context set.

4 Experimental

The experiments evaluate CoAL-RAG on Chinese social-law benchmarks and English cross-jurisdictional datasets, comparing it with retrieval, reasoning, reranking, and knowledge-graph baselines. Evaluation covers generation quality, legal reasoning, and response-time efficiency, with hyperparameters calibrated for robustness.

  • Chinese Benchmarks: Experiments use SocialLawQA and LawBench to assess CoAL-RAG on Chinese social-law consultation scenarios.SocialLawQA contains 1.5k Q&A pairs across 16 statutes, while LawBench contributes a 1k social-law subset.
  • Hyperparameter Calibration: Hyperparameters α, β, p, q, γ and threshold θ were calibrated by grid search on a stratified expert-annotated validation set of N = 120.Sensitivity analysis reports stable performance within ±10% parameter variance.
  • Baselines: Baselines span retrieval-free reasoning, retrieval-augmented generation, unified retrieval-and-reranking models, and knowledge-graph augmentation.Compared systems include Direct inference, CoT, LawGPT_zh, RAG, Hybrid RAG, CLERAG, IRCoT, Search-o1, bge-reranker-v2-m3, Qwen3-Reranker-4B, G-Retriever, and LeanRAG.
  • Evaluation Metrics: Metrics include ROUGE-1/2/L, BLEU-4, and BERTScore for Chinese generation, Accuracy for CaseHold, Micro-F1 and Macro-F1 for LexGLUE, and Average Response Time for efficiency.The metrics assess token overlap, semantic alignment, multi-class logical reasoning, and system latency.
  • English Benchmarks: English evaluation uses LexGLUE and CaseHold to test cross-jurisdictional legal understanding, logical deduction, and long-text holding identification.LexGLUE emphasizes legal NLU and logical deduction; CaseHold is a challenging multiple-choice benchmark.

5 Results · 5.1 Generation Quality and Generalization

CoAL-RAG delivers the strongest reported generation quality on Chinese civil-law benchmarks and robust cross-jurisdictional generalization on English common-law datasets. Its advantages include statistically significant precision gains and performance exceeding Search-R1 despite limitations from a civil-law-centric knowledge graph.

  • 5.1 Generation Quality and Generalization: 0.4954 CaseHold Accuracy illustrates the difficulty pure parametric models face without common-law grounding.The English evaluation covers LexGLUE and CaseHold.
  • 5.1 Generation Quality and Generalization: CoAL-RAG outperforms the RL-tuned Search-R1, which reports 0.6885 Accuracy and 0.7186 Micro-F1.The comparison supports CoAL-RAG’s robust generalization on English benchmarks.
  • 5.1 Generation Quality and Generalization: CoAL-RAG consistently surpasses generic rerankers and Hybrid RAG without requiring costly reinforcement learning.This advantage is reported despite the method’s marginal Macro-F1 lag.
  • 5.1 Generation Quality and Generalization: The marginal Macro-F1 lag is attributed to the Civil-Law-centric knowledge graph lacking precedent indexing.This limitation appears specifically in the English common-law evaluation.

5.2 Efficiency Analysis

CoAL-RAG’s complexity-aware routing reduces response time for simple queries while preserving generation quality. It achieves a favorable balance between efficiency and accuracy compared with standard, language-model, and graph-based retrieval methods.

  • Efficiency Analysis: 4.76 s and 5.09 s average response times are achieved on LawBench and SocialLawQA, respectively, by avoiding redundant computation for simple queries.The efficiency gain is attributed to complexity-aware routing.
  • Efficiency Analysis: ∼2.2× faster than LawGPT and faster than complex graph methods like LeanRAG, CoAL-RAG improves retrieval efficiency.These comparisons underscore the benefit of adaptive routing over slower baselines.
  • Efficiency Analysis: ∼2 s of added latency compared to Standard RAG yields a 61.8% LawBench BLEU improvement, balancing generation quality with real-time performance.The additional latency is associated with a substantial quality gain.

5.3 Ablation Study · The Effectiveness of Intrinsic Assessment

The ablation study evaluates CoAL-RAG’s intrinsic complexity assessment, retrieval consistency assessment, and dynamic document selection on LawBench. Removing intrinsic assessment reduces Article F1 and ROUGE-L, highlighting its role in recognizing logical depth and supporting coherent, relevant answers.

  • 5.3 Ablation Study: Ablation experiments on LawBench test three CoAL-RAG variants that remove intrinsic assessment, retrieval consistency, or dynamic document selection.All scores are averaged across multiple runs to reduce random variance.
  • 5.3 Ablation Study: The w/o Intrinsic variant routes queries using retrieval consistency alone instead of intrinsic complexity assessment.This variant tests whether question-level complexity estimation contributes beyond retrieval consistency.
  • 5.3 Ablation Study: The w/o Consistency variant omits retrieval consistency assessment and routes using query features alone.This variant isolates the contribution of retrieval consistency to adaptive routing.
  • 5.3 Ablation Study: The w/o Dynamic variant replaces adaptive document selection with a fixed Top-10 document set for generation.This variant evaluates the effect of dynamically filtering retrieved context.
  • 5.3 Ablation Study: Article F1 and LawConcept Recall are added to measure statutory article retrieval and legal concept coverage.Table 3 reports these additional ablation metrics.
  • The Effectiveness of Intrinsic Assessment: Article F1 falls from 0.5308 to 0.4977 after removing intrinsic complexity assessment, a relative decrease of 6.24%.The result underscores the importance of anticipating logical depth when identifying complex queries.
  • The Effectiveness of Intrinsic Assessment: ROUGE-L decreases from 0.4162 to 0.4025 without intrinsic assessment, indicating weaker structural coherence and relevance in generated answers.The finding links fine-grained perception of question types to generation quality.

The Effectiveness of Retrieval Consistency · The Effectiveness of Dynamic Top-K

Ablations show that retrieval consistency improves routing by filtering noise, while dynamic top-K selection avoids the semantic degradation caused by indiscriminately retrieving more documents. Together, these modules balance answer quality, evidence precision, and response efficiency.

  • The Effectiveness of Retrieval Consistency: Article F1 drops by 1.48 percentage points when retrieval consistency evaluation is removed.ROUGE-L also declines by 1.55 points, while LawConcept Recall and BERTScore decrease as well.
  • The Effectiveness of Retrieval Consistency: Removing retrieval consistency causes ROUGE-L to fall by 1.55 points.The results confirm that multi-perspective consistency filters retrieval noise and improves routing accuracy.
  • The Effectiveness of Retrieval Consistency: Retrieval consistency evaluation improves routing accuracy by filtering retrieval noise through multi-perspective consistency.Its removal also decreases LawConcept Recall and BERTScore.
  • The Effectiveness of Dynamic Top-K: The w/o Dynamic variant, which retrieves a fixed top-10 documents, achieves the lowest LawConcept Recall.This indicates that fixed-volume retrieval is less effective than adaptive document selection.
  • The Effectiveness of Dynamic Top-K: 0.8034 LawConcept Recall and 0.7653 BERTScore are reported for the w/o Dynamic variant.Indiscriminately increasing document volume introduces noise, whereas CoAL-RAG dynamically filters low-relevance documents.
  • The Effectiveness of Dynamic Top-K: Multi-dimensional evaluation combined with dynamic filtering identifies key evidence while balancing answer quality and response efficiency.The ablation results demonstrate a synergistic effect among the three core modules.

5.4 Sensitivity Analysis

Sensitivity analysis on a stratified N = 120 validation set shows that CoAL-RAG remains robust to moderate changes in routing thresholds, fusion weight, gating exponents, and critical thresholds.

  • Parameter robustness: On a stratified N = 120 validation set, ±10% shifts in routing thresholds θ_low, θ_med, θ_high or fusion weight γ caused < 1.5% fluctuation in ROUGE-L and BERTScore.Gating exponents p and q showed < 1.0% performance variance across p ∈[1.2, 1.8] and q ∈[0.2, 0.4].
  • Efficiency stability: Average Response Time remained consistent within ±0.4s under critical threshold variations.This result supports the efficiency stability of CoAL-RAG under tested sensitivity conditions.

5.5 Case Study and Error Analysis

CoAL-RAG is effective across different complexity levels, but its error analysis identifies knowledge gaps and priority conflicts as key failure modes. These failures can cause reasoning voids or misclassification when implicit legal hierarchy is required.

  • Case Study and Error Analysis: CoAL-RAG demonstrates effectiveness across varying question complexities.Table 4 reports the method’s effectiveness across complexities.
  • Case Study and Error Analysis: Knowledge gaps arise when queries involve local policies outside the statutory knowledge graph, producing reasoning voids.Case 3 exemplifies this failure mode.
  • Case Study and Error Analysis: Priority conflicts occur when overlapping laws require resolving hierarchical logic, such as General versus Special laws.The model occasionally struggles with these conflicts.
  • Case Study and Error Analysis: The routing mechanism may systematically misclassify edge cases requiring implicit legal hierarchy.This limitation follows from the observed difficulty resolving hierarchical logic among overlapping laws.

6 Conclusion

CoAL-RAG dynamically selects retrieval strategies and constructs context by combining multidimensional reasoning-depth assessment with retrieval consistency, achieving generality across Chinese Civil Law and English Common Law benchmarks. Remaining challenges concern dynamic adaptation, specialized-domain extension, cross-document conflict resolution, and scaling to larger LLMs.

  • Conclusion: CoAL-RAG combines multidimensional reasoning-depth assessment with retrieval consistency to select retrieval strategies and adaptively construct context for legal queries.The method is designed for queries with variable complexity.
  • Conclusion: Evaluations across Chinese Civil Law and English Common Law benchmarks confirm the generality of CoAL-RAG’s complexity-aware mechanism.The conclusion characterizes the method as balancing quality and efficiency.
  • Future work: Future work will extend CoAL-RAG to specialized domains such as criminal law and finance.This addresses broader domain coverage beyond the evaluated settings.
  • Future work: Future work will enhance cross-document reasoning to resolve conflicts among overlapping legal provisions.The goal is to improve handling of conflicting information across documents.
  • Future work: Future work will scale CoAL-RAG to larger LLMs, including 7B/14B models, to investigate performance ceilings.The proposed scale-up explicitly includes 7B/14B models.
Loading 2608.17536v1…