PaperlayerView exact revision on arXiv

Source-linked AI summary

LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning

Qingjing Chen, Junkai Zhang, Shaochun Wang, Jiahao Ding, Siyuan Zheng, Yukun Yan, Zhi Zheng, Antonino Rotolo, Yun Liu, Weixing Shen

arXiv:2609.27009v1cs.CLcs.IR

TL;DR

Legal retrieval methods can miss normative relations among provisions, while vanilla Chain-of-Thought does not enforce legal reasoning structure. LEGO combines expert-graph retrieval with structured Provision–Fact–Conclusion reasoning; with Qwen3-8B, it reaches 40.53% exact-match accuracy on LawExamQA_Civil and leads evaluated baselines on two open-ended benchmarks.

  • Problem

    Legal RAG and GraphRAG can overlook normative links among provisions, and vanilla Chain-of-Thought may produce plausible reasoning without following legal order.

    Source: §1 · p. 2

  • Method

    LEGO retrieves instance-specific provision subgraphs from an expert-annotated Civil Code graph and organizes the provisions and case facts into Provision–Fact–Conclusion reasoning.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    40.53% exact-match accuracy on LawExamQA_Civil with Qwen3-8B outperforms evaluated RAG and CoT baselines, while LEGO also leads evaluated baselines on two open-ended benchmarks.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The structured syllogistic output makes the rationale linking provisions, facts, and conclusions easier to inspect and audit.

    Source: §1 · p. 3

    The experiments evaluate answer accuracy but do not establish the faithfulness of generated analyses to internal reasoning or their correctness as post-hoc explanations.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models are increasingly applied to high-risk domains such as law, yet complex legal reasoning remains limited by two structural challenges. First, existing RAG and GraphRAG methods emphasize lexical or semantic similarity while overlooking normative relations among legal provisions. Second, vanilla Chain-of-Thought prompting may generate plausible rationales without enforcing the normative structure of legal reasoning. To deal with the bottleneck of pipelines in the legal reasoning domain, we propose LEGO, a dual-module framework that synergizes Legal Expert GraphRAG and expert Chain-of-thought for complex legal reasoning. ExpertGraphRAG uses an expert-annotated civil code graph encoding these normative relations with a greedy normative-coverage retrieval algorithm to dynamically extract instance-specific provision subgraphs, while ExpertCoT organizes the retrieved provisions and case facts into structured Provision-Fact-Conclusion reasoning. With a Qwen3-8B backbone, LEGO achieves 40.53% exact-match accuracy on LawExamQA_Civil, outperforming the evaluated RAG and CoT baselines and performing comparably to the evaluated larger models, while remaining robust on multi-hop questions. It also achieves the best results among the evaluated baselines on the open-ended benchmarks. Ablation studies confirm the individual and complementary contributions of both modules, demonstrating LEGO's effectiveness in improving LLMs' complex legal reasoning ability. Code and dataset can be found in the link: https://github.com/BLK-WHT/LEGO

1 Introduction

LEGO addresses gaps in legal retrieval and reasoning by combining expert-structured provision retrieval with a structured Provision–Fact–Conclusion analysis. With Qwen3-8B, it outperforms evaluated RAG and CoT baselines, performs comparably to larger evaluated models, and remains robust on multi-hop questions.

1 Introduction

  • Legal reasoning remains difficult because similarity-based retrieval can miss normative links between provisions, while vanilla Chain-of-Thought does not ensure the required normative order.Relevant provisions may be textually distant yet linked through prerequisites, exceptions, general–special relations, or legal effects.Source: §1 · pp. 1–2
  • ExpertGraphRAG uses an expert-annotated Civil Code graph and greedy normative-coverage retrieval to select a compact, instance-specific provision subgraph by marginal legal utility.ExpertCoT then organizes the retrieved provisions and case facts into structured Provision–Fact–Conclusion reasoning.Source: §1 · pp. 2–3
  • 40.53% exact-match accuracy on LawExamQA_Civil lets LEGO outperform evaluated RAG and CoT baselines while performing comparably to larger evaluated models.The structured syllogistic output also makes the rationale linking provisions, facts, and conclusions easier to inspect and audit.Source: §1 · p. 3

2 Related Work

Prior GraphRAG and legal CoT research provides relevant retrieval and reasoning techniques, but the cited work leaves gaps in evaluating and combining them for legal reasoning across related provisions.

2.1 GraphRAG and Legal-Domain Retrieval

  • Existing general-purpose and legal GraphRAG work does not evaluate retrieval on complex or multihop legal reasoning involving normative relations among provisions.GraphRAG structures corpus information as graphs, but retrieving relevant provisions alone does not ensure correct deduction across them.Source: §2.1 · p. 3

2.2 Chain-of-Thought and Legal Reasoning

  • Legal CoT studies explore structured reasoning, but the cited work lacks reliable retrieved-evidence grounding and does not address inter-provision relations or legal syllogisms.Prior RAG–CoT systems and expert-guided medical QA combine retrieval and reasoning, while legal reasoning must retrieve provisions and apply their normative links.Source: §2.2 · p. 3

3 Methodology

LEGO pairs graph-structured provision retrieval with syllogistic generation, organizing retrieved rules, case facts, and option judgments into a Provision–Fact–Conclusion analysis.

3.1 ExpertGraphRAG

  • Normative Coverage Greedy selects Civil Code articles by the marginal gain of relevance-weighted coverage of expert rule units, with a redundancy regularizer to discourage repeated coverage.The selected articles induce a compact, instance-level provision subgraph used as the legal major-premise context.Source: §3.1 · p. 4

3 Methodology

  • LEGO serializes each case, question, and answer options as a retrieval query, retrieves provisions from a global Provision Graph, then generates a structured P–F–C analysis and final answer.The resulting rationale graph is a conceptual description rather than a separately materialized graph object.Source: §3.2 · pp. 4–5, §3 · p. 4

3.2 ExpertCoT

  • ExpertCoT applies retrieved rules as major premises and relevant case facts as minor premises to derive a judgment for each answer option.Its instructions direct the model to identify controlling rules, account for priorities and exceptions, and check each option’s elements and asserted legal effect.Source: §3.2 · p. 5

4 Experiments

Across civil-law benchmarks, LEGO leads the evaluated systems on LawExamQA_Civil and RAG baselines on two open-ended benchmarks. The experiments also examine multi-hop robustness, component contributions, provision retrieval, and remaining error patterns.

4.1 Setup

  • The evaluation covers three civil-law datasets, while the expert Civil Law graph encodes 318 concepts in 409 rule cards linked to 1,211 of 1,260 Civil Code articles.RAG baselines share the Qwen3-8B backbone and Civil Code corpus; pipelines use method-specific retrieval configurations rather than numerically equalized retrieval budgets.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 38.71% accuracy on ≥4-hop questions exceeds the second-best result of 32.26% by 6.45 points, while LEGO’s accuracy remains relatively stable across hop counts.The reported patterns include LEGO’s 1/2/3/4+-hop accuracies of 41.8 → 40.5 → 37.5 → 38.7, compared with a 12.2-point GPT-5 drop from 1-hop to 4+-hop.Source: Figure 2 · p. 6, §4.2 · p. 7
  • 40.53% exact-match accuracy makes LEGO the highest-scoring evaluated system on LawExamQA_Civil, significantly exceeding same-backbone G-Retriever (31.67%; ∆=8.86 pp).Its point estimate is higher than DeepSeek-V3 (39.97%) and GPT-5 (37.48%), but paired tests do not establish significant superiority over those larger models.Source: Table 1 · p. 7, §4.2 · p. 7
  • LEGO achieves the highest scores among RAG baselines on LexRAG_Civil and PLawBench_Civil, including LexRAG overall quality (5.164) and PLawBench reasoning (47.40) and overall (59.68).Against HippoRAG 2, LEGO improves LexRAG overall score by 0.335 points and completeness by 0.407 points.Source: §4.2 · pp. 7–8, Table 2 · p. 9

4.4 Recall of Legal Retrieval

  • 27.99% to 72.53% Recall@8, 34.70% to 74.58% Recall@10, and 50.52% to 81.07% Recall@20 show higher gold-provision coverage for CivilCode-RAG + ExpertGraph over Naive RAG.With Qwen3-8B, QA accuracy rises from 25.45% in the zero-shot setting to 30.29%; LEGO’s accuracy is 40.53% against 41.22% for the gold-article reference, a 0.69 pp gap.Source: §4.4 · p. 8, Table 4 · p. 9

4.3 Ablation Study

  • The full LEGO system outperforms the evaluated baselines and variants in Accuracy and F1; generic RAG pipelines trail by 6.23–8.30 pp in Accuracy and 3.56–6.13 pp in F1.ExpertGraphRAG’s Accuracy gain is 1.25 pp with Plain CoT and 4.71 pp with ExpertCoT; ExpertCoT gains 1.52 pp with Naive RAG and 4.98 pp with ExpertGraphRAG.Source: §4.3 · p. 8, Figure 3 · p. 8, Table 3 · p. 9

4.5 Error Analysis

  • LEGO mainly reduces generic RAG errors involving legally irrelevant retrieval, partial single-provision answers, and hedged conclusions, while remaining errors concentrate on provision selection and prerequisite violations.Priority misapplication, concept conflation, and exception bypass account for most of the remaining errors.Source: §4.5 · p. 9

5 Conclusion

LEGO combines expert-structured legal retrieval with Provision-Fact-Conclusion reasoning. The paper reports stronger performance than evaluated RAG and CoT baselines, including on multi-hop questions and open-ended benchmarks.

5 Conclusion

  • LEGO selects provisions from an expert-annotated Civil Code graph and organizes them with case facts into Provision-Fact-Conclusion reasoning.The paper describes ExpertGraphRAG as selecting by normative coverage rather than similarity alone.Source: §5 · p. 9
  • With an 8B backbone, LEGO outperforms evaluated RAG and CoT baselines, performs comparably to evaluated larger models, and remains robust on multi-hop questions.The paper also reports that LEGO leads the evaluated baselines on two open-ended benchmarks.Source: §5 · p. 9
  • Ablations confirm individual and complementary contributions from both modules, and the paper concludes LEGO improves legal reasoning and the interpretability of analyses.Source: §5 · p. 9

Limitations

The paper evaluates a methodological framework rather than introducing a new legal reasoning benchmark. Its evaluation has limits in hop measurement, explanation validation, and scalability across jurisdictions and legal domains.

Limitations

  • The study focuses on a methodological framework, not a new legal reasoning benchmark.Source: Limitations · p. 10
  • LawExamQA_Civil is used only for evaluation, and its hop count approximates complexity by counting provisions cited in official rationales rather than fine-grained inter-provision transitions.Those transitions include prerequisite, exception, priority, condition–consequence, and general–special relations.Source: Limitations · p. 10
  • Answer-accuracy experiments do not establish the faithfulness of generated analyses to internal reasoning or their correctness as post-hoc explanations.Source: Limitations · p. 10
  • Building and maintaining the expert normative graph requires substantial expert effort, potentially limiting scalability across jurisdictions and legal domains.The Fact Graph is conceptual; implementation passes case text to the model without constructing an explicit fact graph.Source: Limitations · p. 10

Ethics Statement

The work is a controlled methodological evaluation, not a legal-advice or automated-decision system. Its questions come from publicly available judicial-examination materials, and the Civil Law ExpertGraph was built independently of benchmark answers and rationales.

Ethics Statement

  • LEGO is intended to evaluate expert-structured retrieval and syllogistic reasoning, not to provide legal advice or replace professionals or automated legal decision-making.Source: Ethics Statement · p. 10
  • Evaluation questions come from public judicial-examination materials and are used only for controlled experiments, without promoting a particular legal interpretation.Source: Ethics Statement · p. 10
  • The Civil Law ExpertGraph was constructed independently of benchmark questions, answer labels, and official rationales, and expert annotators received fair-market compensation.Source: Ethics Statement · p. 10

Use of AI Assistants

The appendices document LEGO's prompts, worked example, and baseline configurations, alongside the authors' disclosure of AI assistance. They include materials on expert-structured retrieval and syllogistic reasoning as well as detailed case studies and error analysis.

Use of AI Assistants

  • The authors used Claude, Gemini, GPT, Qwen, and GitHub Copilot for text editing and code development, verified the outputs, and retain accountability.Source: Use of AI Assistants · p. 10

Appendix Contents

  • Appendices A–F cover the syllogistic CoT prompt, baseline details, P/F/C case studies, dataset, expert-graph construction, and graph source statistics.Source: Appendix Contents · p. 13
  • Appendices G–L provide error analysis, multi-hop and component-synergy cases, civil-law case studies, and statistical validation of main results.Source: Appendix Contents · p. 13

A Syllogistic CoT Prompt Card

A.1.2 User prompt template

  • The user template fills retrieved law, case, and question blocks, then separates controlling provisions, legally relevant facts, and option judgments while checking provision relations.The prompt also distinguishes legal effects and directs limiting clauses to be interpreted as limitations.Source: §A.1.2 · pp. 13–14

A.1.3 End-to-end example (qid 0)

  • The qid 0 example presents a transport-delay and cargo-damage question with options and an expected gold answer of A.Source: §A.1.3 · pp. 14–15

B.1 RAG Baselines

B.2 CoT Baselines

  • CoT baselines use retrieved articles as law context but vary the reasoning scaffold, including free-form step-by-step reasoning and an Issue–Rule–Application–Conclusion trace.Source: §B.2 · p. 16

B.3 Experiment Settings

The settings fix the generator, decoding, retrieval weights, and article budget for reproducibility. Case studies show how retrieval and structured Provision–Fact–Conclusion reasoning address distinct legal-reasoning errors.

B.3 Experiment Settings

  • The LawExamQA_Civil provision retrieval settings use normalized similarities, fixed seed 42, and eight selected articles passed into the Syllogistic-CoT prompt.Open-source runs use greedy decoding at temperature 0 with max_tokens 1800, except the zero-shot setting uses 512; the Qwen3-8B reader runs with thinking disabled.Source: §B.3 · p. 17

C P/F/C Case Studies with Pipeline-Level Reasoning

  • The six case studies compare pipeline-level Provision–Fact–Conclusion reasoning, marking correct, erroneous, and partially correct steps.Green, red, and yellow cells distinguish those three categories.Source: C P/F/C Case Studies with… · p. 17

C.1 QID 19: Contract Rights and Obligations Transfer

  • QID 19 turns on consent for transferring contractual rights and obligations; silence and later performance do not supply statutory consent by default.The analysis distinguishes implied expression from treating silence as consent and uses a relation check to prevent performance or efficiency from becoming consent.Source: §C.1 · p. 18

C.2 QID 4: Stamped Blank Contract and Apparent Agency

  • QID 4 treats stamped blank contracts as a typical appearance of authority, allowing the apparent-agency special rule to override bare no-authority reasoning.The passage cautions that effective receipt and understanding of the removal notice could weaken reasonable reliance; the example follows the benchmark gold, not a universal conclusion.Source: §C.2 · p. 18

C.3 QID 1: Mandate Termination and Loss Compensation

  • QID 1 distinguishes Article 933’s loss compensation from agreed remuneration, so ExpertCoT checks the option’s asserted consequence against the statutory consequence.Both parties may terminate the mandate, but the two payment consequences are not interchangeable.Source: §C.3 · p. 19

C.4 QID 36: Emergency Rescue and Negotiorum Gestio

  • QID 36 turns on a negative legal effect: Article 979 permits necessary-expense reimbursement and appropriate loss compensation, but creates no remuneration claim.ExpertGraph supplies this boundary; plain RAG and RAG+ExpertCoT miss it when they focus on expense or compensation facts.Source: §C.4 · p. 20

C.5 QID 7: Misdelivered Milk and Good-Faith Recipient

  • QID 7 illustrates synergy: LEGO Full combines an exception with fact filtering to avoid both overapplying unjust enrichment and over-reading negligence.The graph supplies the good-faith recipient’s present-benefit limitation, while the structured reasoning applies it to the facts.Source: §C.5 · p. 20

C.6 QID 16: Bank Overpayment and Business Profit

The bank-overpayment case separates the statutory return of principal and interest from downstream business profit, illustrating the need to constrain legal consequences to the controlling rule. The surrounding material summarizes cross-case lessons, benchmark construction, and the expert graph’s normative structure.

C.6 QID 16: Bank Overpayment and Business Profit

  • QID 16’s gold explanation awards the bank principal plus one month’s interest, not the downstream business profit; LEGO Full checks whether the option’s consequence is authorized by the rule and benchmark explanation.The passage cautions that this illustrates legal-effect boundary control, not a universal profit-return rule; the graph also records a broader returnable-benefit view.Source: §C.6 · p. 21

C.7 Cross-Case Lessons

  • Across six cases, retrieval helps when a missing major premise is the bottleneck, ExpertCoT when rule-to-option mapping is needed, and the full system when both are required.The cross-case account attributes QID 7 and QID 16’s challenges to combining the right rule neighborhood with P/F/C control of legal consequences.Source: §C.7 · p. 21

LawExamQA_Civil

  • LawExamQA_Civil combines objective items with subjective questions converted to option form, and counts distinct provisions required by official explanations as a proxy for reasoning hop count.LexRAG_Civil evaluates multi-turn consultation, while PLawBench_Civil uses practical case analysis scored across fact, provision, reasoning, and conclusion components.Source: LawExamQA_Civil · p. 22

Expert-annotated legal graph

  • The Civil Law ExpertGraph encodes concepts, provisions, elements, exceptions, priorities, and legal effects in typed nodes and normative relations, independently of benchmark questions and answers.Concept, element, distinction, exception, effect, and priority nodes target different reasoning errors.Source: Expert-annotated legal graph · pp. 23–24

F Civil Law ExpertGraph: Source Statistics

The source corpus is a hierarchical Markdown document whose deeper headings contain material mapped to normative edges. Its structure and concept distribution characterize the input corpus, not the deployed graph itself.

F Civil Law ExpertGraph: Source Statistics

  • The source is a single hierarchical Markdown document with 8,211 heading nodes and 217 distinct Civil Code articles spanning articles 1–1260.Its roughly 391K characters are arranged over 17,190 lines, of which 8,717 are non-empty.Source: F Civil Law ExpertGraph: Source… · p. 28
  • H6–H9 contain much of the source tree’s legal-effect statements, exceptions, and prerequisites—the material mapped onto normative edges.The source-corpus heading-depth figures should not be read as deployed-graph statistics because construction changes the unit of analysis.Source: F Civil Law ExpertGraph: Source… · p. 28
  • Contracts, General Provisions, and Property Rights account for roughly 71.8% of the source corpus’s hierarchical heading nodes.Table 16 reports counts across ten top-level sections.Source: F Civil Law ExpertGraph: Source… · p. 28, Table 16 · p. 28
  • The source’s concept vocabulary includes procedural and defensive terms alongside core civil-law concepts, capturing operational reasoning machinery as well as first-order concepts.Examples include limitation period, claim right, lien, and defence, alongside validity, transfer, damages, and rescission.Source: F Civil Law ExpertGraph: Source… · p. 28, Table 17 · p. 28
  • Construction types heading nodes, normalizes vocabulary, resolves article references, and adds normative edges, so source-corpus figures are not deployed-graph statistics.The downstream pipeline changes the unit of analysis through rule-unit typing and article linking.Source: F Civil Law ExpertGraph: Source… · p. 28

G Detailed Error Analysis

The error analysis finds that wrong answers cluster in structural reasoning failures, especially provision selection and prerequisite checking. The accompanying multi-hop case study examines cases LEGO answered correctly when all 13 baselines failed.

G Detailed Error Analysis

  • Incorrect provision selection (36.74%) and prerequisite violation (20.70%) together account for 57.44% of errors; the top seven categories cover 91.4%.The figure and table summarize errors on items mis-answered by full LEGO.Source: Figure 8 · p. 29, G Detailed Error Analysis · p. 29, Table 18 · p. 30
  • Incorrect provision selection is the largest category, with 158 errors (36.74%), often caused by choosing lexically or topically similar provisions that are legally inapplicable.The passage argues the selected provision must occupy the right legal-topology position and be licensed by the facts.Source: G Detailed Error Analysis · p. 29
  • Prerequisite violations account for 89 errors (20.70%), while priority misapplication accounts for 42 (9.77%); both motivate explicit checks before committing to a conclusion.The former applies downstream consequences before upstream legal states are verified; the latter applies general and special rules in the wrong order or treats them as parallel.Source: G Detailed Error Analysis · p. 29
  • Concept conflation (33 errors, 7.67%) and exception bypass (30, 6.98%) reflect errors in logical operators and failure to test whether exceptions defeat general rules.Other substantive categories include legal-effect misinterpretation and fact subsumption, which occur even after provision retrieval succeeds.Source: G Detailed Error Analysis · pp. 29–30

H Multi-Hop Case Studies on LawExamQA-Civil

  • The multi-hop case study selects four examples from 20 items LEGO answered correctly while all 13 baselines failed, spanning four hop strata and distinct reasoning patterns.It examines the opposite population from the error taxonomy, which codes LEGO’s 430 mis-answered items.Source: H Multi-Hop Case Studies on… · p. 30, Table 19 · p. 30

H.1 QID 7 (1-hop): Good-Faith Recipient Exception to Unjust Enrichment and Tort

Across four case studies, LEGO’s structured legal reasoning is contrasted with baselines that overlook exceptions, thresholds, and rule priority. The cross-case analysis links success to explicit rule-hierarchy resolution and constitutive-element checking.

H.1 QID 7 (1-hop): Good-Faith Recipient Exception to Unjust Enrichment and Tort

  • Every baseline assigns liability, but LEGO applies the good-faith-recipient limitation and concludes Zhang owes no restitution and commits neither tort nor unjust enrichment.LEGO’s retrieved set includes Arts. 122 and 985–988; its conclusion tests the general rule against the limitation.Source: §H.1 · p. 31

H.2 QID 716 (2-hop): Threshold-Gated Mental-Distress Claim

  • Eleven of thirteen baselines select all options; LEGO rejects C because the facts fall below Art. 1183’s serious-mental-harm threshold, while accepting A, B, and D.The major-premise expansion makes the remedy threshold explicit for checking against the facts.Source: §H.2 · pp. 31–32

H.3 QID 544 (3-hop): Joint Tort vs. Separate Tort with Divisibility

  • Nine of thirteen baselines select inconsistent options BCD; LEGO distinguishes the tort regimes and applies Art. 1172, yielding proportional liability when shares are provable and equal shares otherwise.LEGO rejects Art. 1168 because the supervision agreement was not executed, so there was no concerted action.Source: §H.3 · p. 32

H.4 QID 22 (4+-hop): Independent Mortgage Transfer (Art. 406 vs. Art. 407)

  • Art. 407 bars transferring a mortgage independently of its secured debt, overriding general transferability and registration-effect rules; registration cannot cure substantive invalidity when only the mortgage, not debt, was transferred.LEGO identifies Art. 406 as the general rule and Art. 407 as the special rule that prevails.Source: §H.4 · p. 33

H.5 Cross-Case Synthesis

  • LEGO’s 4+-hop margin is +6.4 pp over the strongest LLM baseline; the analysis attributes it to retrieving the right provision configuration and resolving hierarchy, not retrieving more provisions.The mortgage-transfer case is a unanimous baseline failure because Art. 407 is doctrinally required but lexically remote from the facts.Source: §H.5 · p. 33

I Component-Synergy Case Studies on LawExamQA-Civil

The section examines how ExpertGraph retrieval and ExpertCoT reasoning contribute separately and together to LEGO’s gains, using case studies to show their role in doctrinal disambiguation, legal-rule hierarchy, multi-provision reasoning, and substantive limits. LexRAG_Civil examples also illustrate dialogue-aware, actionable consultation responses.

I Component-Synergy Case Studies on LawExamQA-Civil

  • 40.53% accuracy for Full LEGO is a +6.22 pp gain over Naive RAG + plain CoT; the estimated synergy is +3.46 pp, with a 95% interval including zero.ExpertGraph alone reaches 35.55% (+1.24 pp), while ExpertCoT alone reaches 35.82% (+1.52 pp); combining them yields larger aggregate and 1-hop gains than the sum of the individual gains, and ExpertCoT improves 2-hop accuracy by +7.5 pp over plain CoT with Naive RAG.Source: I Component-Synergy Case Studies on… · p. 34

I.1 QID 237 (2-hop): Constitutum Possessorium

  • On QID 237, Full LEGO selects BD, while eight of eleven ablations select CD; even ExpertGraph + IRAC-CoT retrieves Arts. 228 and 224 but fails to apply their general–special priority.ExpertCoT’s provision-relation prompt resolves the conflict by applying Art. 228 over Art. 224: ownership transfers when the parties agree that A will keep possessing the jade, and delivery is constitutum possessorium.Source: §I.1 · p. 35

I.2 QID 509 (3-hop): Title-Retention Sale plus Good-Faith Third Party

  • On QID 509, Full LEGO selects ACD, while six of eleven ablations select AD and miss Art. 634’s full-price branch.The case requires Art. 311 for good-faith acquisition, Art. 634 for the full-price and rescission-plus-use-fee branches, and Art. 642 to reject recovery without notice and a reasonable period; the Full system retrieves and composes all three rules.Source: §I.2 · pp. 35–36

I.3 QID 652 (4+-hop): Will Validity vs. Mandatory Reservation for a Fetus

  • On QID 652, Full LEGO selects ABC, while all eleven ablations fail to select B, treating a formally complete will as fully valid despite Art. 1155’s mandatory fetus-share reservation.ExpertGraph alone retrieves Art. 1155 but does not apply it as a substantive limit; ExpertCoT alone offers element checking but lacks the article, so both components are necessary in this case.Source: §I.3 · pp. 36–37

I.4 Cross-Case Synthesis

  • Across the three cases, each component provides a necessary but insufficient capability, and the Full system’s wins arise from their interaction.The cases isolate complementary needs: resolving provision priority, composing three rules, and connecting the unborn fetus to the mandatory-reservation requirement.Source: §I.1 · p. 35, §I.2 · p. 36, §I.4 · p. 37

Method Marriage-prohibition article 157? Overall

  • +21.7 Overall margin, LEGO’s largest PLawBench win, accompanies its Arts. 1042→157→964 chain, avoiding Art. 1048 and linking prohibition, invalidity, restitution, and fault allocation.Four of five baselines retrieve the doctrinally unrelated Art. 1048; LEGO’s normative-edge retrieval avoids that near-neighbour error and its CoT composes the provisions.Source: Method Marriage-prohibition article… · p. 40

J.5 Cross-Case Synthesis for PLawBench-Civil

  • Across four PLawBench cases, LEGO addresses hedged conclusions, off-target anchor statutes, missing controlling statutes under distractors, and near-neighbour confusion through normative edges, element decomposition, and multi-statute composition.The cases include Art. 1048 versus 1042 confusion; the cross-case synthesis identifies these mechanisms as sources of LEGO’s gains.Source: §J.5 · p. 40

K LexRAG_Civil Case Studies

  • LexRAG_Civil assesses multi-turn consultations on five dimensions plus Overall; four cases were selected from 140 samples to span consultation categories and baseline failure modes.The dimensions are Factuality, Satisfaction, Clarity, Coherence, and Completeness.Source: K LexRAG_Civil Case Studies · pp. 40–41

K.1 972_turn2 (Real Property): The Effect-of-Registration Doctrine

  • All five baselines confuse Art. 209’s registration-for-transfer rule with Art. 225’s third-party opposability rule, while LEGO separates a valid contract from ownership transfer requiring registration.LEGO is the only method described as anchoring its answer in the separation between the obligatory act and the dispositive act.Source: §K.1 · p. 41

K.2 173_turn4 (Public Security): Joint Liability among Multiple Tortfeasors

  • 9 Overall for LEGO, versus 6 for HippoRAG 2, reflects combining Arts. 1170 and 178 to distinguish tortfeasor-identification cases and explain recourse on the user’s narrowed liability question.LEGO uses dialogue history to avoid repeating comparative-fault analysis already covered in earlier turns.Source: §K.2 · pp. 41–42

K.3 301_turn2 (Inheritance): Separating the Inheritance Act from Property Transfer

  • LEGO separates inheritance acceptance from later property-title transfer, explaining that inheritance itself needs no transfer procedure but registration completes the real-property ownership change.Three baselines give a flat “no,” which omits the practical need to register inherited real estate.Source: §K.3 · p. 42

K.4 639_turn1 (Land Dispute): Indefinite-Term Lease of Private Land

  • LEGO composes lease provisions into actionable advice—sign a written contract, specify a term of ≤20 years, and prioritize the existing tenant on renewal—where baselines fail to assemble the required chain.RAPTOR’s Art. 721 retrieval is a specific wrong-article error; the reference chain also includes termination rights for indefinite-term leases.Source: §K.4 · pp. 42–43

K.5 Cross-Case Synthesis for LexRAG_Civil

  • Across LexRAG_Civil and PLawBench-Civil, normative edges disambiguate lexically close but doctrinally distinct provisions, while Syllogistic CoT composes provisions into structured advisory chains.The synthesis attributes both rubric-based and dialogue-quality gains to these two design choices.Source: §K.5 · p. 43

L Statistical Validation of the Main Results

Paired statistical analyses support LEGO’s advantage over same-backbone RAG systems and the contributions of both expert modules. Comparisons with larger models and evidence for statistical super-additivity remain inconclusive.

Statistical procedure

  • Comparisons use paired item-level exact McNemar tests and 10,000 item-level bootstrap resamples, with Holm correction applied separately to system and component families.The families address different questions: performance against external systems and the contribution of LEGO’s components.Source: L Statistical Validation of the Main… · p. 44

System-level results

  • Five same-backbone RAG systems are significantly outperformed by LEGO, while comparisons with Qwen3-30B-A3B, GPT-5, and DeepSeek-V3 are inconclusive.Under a shared Qwen3-8B reader and Civil Code corpus, the supported claim is that LEGO improves on every evaluated RAG pipeline.Source: Table 24 · p. 44, L Statistical Validation of the Main… · p. 44

Component-level results

  • Ten of eleven non-full configurations are significantly worse than full LEGO; the exception, ExpertGraphRAG + IRAC-CoT, is 3.18 pp lower (pHolm = 0.067).Its interval just includes zero, so the added benefit of replacing this strong generic legal prompt with ExpertCoT is not separable from noise at this sample size.Source: Table 25 · p. 44, L Statistical Validation of the Main… · p. 44

Interaction between the two modules

  • The crossed ablation estimates a +3.46 pp interaction, but its 95% interval [−0.97, +7.75] includes zero, so statistical super-additivity is not established.Conditional gains increase across both module substitutions, supporting complementary gains without establishing a super-additive effect.Source: L Statistical Validation of the Main… · p. 45

Component-level results

  • 4.70 pp: with ExpertCoT fixed, replacing ExpertGraphRAG with Naive RAG lowers accuracy from 40.53% to 35.82% (95% CI [+1.52, +7.88], pHolm = 0.019).ExpertGraphRAG raises Recall@8 over the Civil Code corpus from 27.99% to 72.53%, while ExpertCoT structures application of the retrieved provisions.Source: L Statistical Validation of the Main… · p. 45

Discordant-pair counts

  • Against G-Retriever, LEGO is correct where it is wrong on 116 items and wrong where it is correct on 52; against Naive RAG, the split is 129 to 45.These discordant-pair counts underlie the paired tests and are distinct from the stringent 20-case intersection used for qualitative mechanism analysis.Source: L Statistical Validation of the Main… · p. 45
Loading 2609.27009v1…