Source-linked AI summary
GANDR: Claim Auditing for Verifiable Legal Answer Generation
Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos
TL;DR
Legal answer evaluation can miss fabricated or weakly matched citations when it scores answers in aggregate. GANDR builds claim-level auditing into generation and pairs it with strict citation correctness, leading baselines on shared-backbone tests while remaining bounded to supplied authority.
Problem
Aggregate answer scores can treat a correct conclusion as successful despite fabricated or loosely matched citations, even though legal readers verify claims against cited sources.
Method
GANDR uses a structured Drafter and a separate Critic that audits each claim against its cited source, while strict correctness requires every citation to resolve to a retrieved passage.
Results
GANDR leads the strongest baseline by 11.3 strict points on the shared backbone and remains positive by +3.2 to +6.5 points across three further backbones.
Takeaways & Limitations
The lead reflects the Drafter configuration and protocol-anchored commit rather than rewriting, while the per-claim audit provides a validated citation-discipline signal.
Takeaways & Limitations
The rewrite threshold is fitted to this corpus rather than a held-out split, although headline strict metrics do not depend on it.
Abstract
from arXiv · showhide
In high-stakes domains such as legal practice, a language-model answer is only useful to the extent that a reader can verify each claim against the source the system cites. Current grounded-generation pipelines score the answer as a whole, so a correct conclusion can rest on fabricated or loosely matched citations and still score well. Closing this gap requires both a system built for per-claim verification and an evaluation that measures it. We introduce GANDR (Grounded ANswer DRafter), a two-agent system in which a Drafter writes an answer in a structured legal-reasoning format and a separate Critic, with the same view as a human verifier, audits each claim against its cited source and emits a per-claim audit trace on every round. We pair it with a strict correctness criterion requiring every citation to resolve to a passage the retriever returned. On a 185-item legal benchmark where all six systems share one backbone, one retrieval surface, and one citation instruction, GANDR ranks first on every primary metric, reaching 70.8% strict accuracy and leading the strongest baseline by 11.3 points (p<0.01). Reverting the protocol-anchored commit rule lowers strict accuracy by 22.7 points, and the strict lead stays positive on three further backbones, at +3.2 to +6.5 points. This lead traces to the Drafter configuration and the protocol-anchored commit, not to rewriting. Against two law-trained annotators the audit flags under-supported claims at F1 0.84 as a binary detector, while its four-way verdict labels agree only weakly and are advisory. Code is available upon request.
1 Introduction
GANDR addresses the gap between correct legal conclusions and verifiable citations by embedding per-claim auditing into generation and evaluating grounding strictly.
- 58% to 88% of open-domain legal queries produce hallucinations from general-purpose LLMs, while retrieval-grounded legal-AI products still hallucinate on 17% to 43%.
- Aggregate evaluation can reward correct conclusions supported by fabricated or loosely matched citations, so verifiability must be built into the system.
- GANDR uses a Drafter and separate Critic to generate structured legal answers with per-claim audit traces.
- Strict correctness requires the right disposition and every emitted citation to resolve to a passage returned by the retriever.
- On a 185-item benchmark with shared backbone, retrieval, and citation instructions, GANDR leads every baseline and remains positive across four backbones.
2 Related Work
Prior work improves retrieval, prompting, attribution, or self-correction, but GANDR distinguishes itself by making per-claim auditing a first-class generation artifact with a structural commit gate.
- Per-claim attribution is established for post-hoc evaluation, while GANDR carries the audit into generation and gates commitment structurally.
- Better retrieval is necessary but insufficient because a correctly retrieved passage may still fail to support the claim cited to it.
- Same-context self-correction lets the generator see the reasoning it must critique, whereas GANDR separates the Critic’s context to reduce anchoring and rubric-stamping.
3 The GANDR System
GANDR combines CREAC-structured drafting, a separate-context Critic, and a protocol-anchored orchestrator that fails closed when structural checks do not pass.
- GANDR rests on CREAC generation, separate-context claim auditing, and an orchestrator anchored to structural checks rather than Critic verdicts.
- CREAC as a verification contract: CREAC orders five legal-reasoning blocks for verification, requiring both conclusions, retrieved rule references, and citations for explanation and application claims.
- Drafter and Critic in disjoint contexts: The Critic sees the question, retrieved passages, and final draft but not the Drafter’s chain of thought, then emits per-claim support verdicts and rewrite directives.
- Protocol-anchored orchestrator: The orchestrator commits when protocol_check passes, retries failures for up to T=3 rounds, and returns passed=False with the full trace after exhaustion.
- Protocol-anchored orchestrator: 98.6% of runs commit on Round 1, while the rewrite threshold is corpus-fitted and no headline metric depends on the rewrite path.
4 Experimental Setup
The evaluation controls retrieval, backbone, citation instructions, and judging across six systems on a 185-item legal benchmark, while strict metrics test citation resolvability.
- Benchmark and retrieval: The 185-item benchmark spans LegalBench and LegalBench-RAG across eight domain buckets, with one BM25 index delivering the same top-8 passages to every system.
- Systems, backbone, and judge: Six systems share one backbone, retrieval surface, grounding requirement, and separate-lineage judge, enabling an architecture-focused comparison.
- Metrics and evaluation parity: Strict scoring requires a leniently correct holding, at least one inline citation marker, and resolution of every cited rule_id to a retrieved top-8 header.
5 Results
GANDR leads the benchmark on strict grounding, remains ahead across additional backbones, and pairs its answers with an audit trace. The protocol-anchored commit rule is load-bearing, while auditability is supported by claim-level detection results.
- Benchmark results: GANDR leads on every primary metric, with its benchmark comparison against B2 significant at p<0.01.Table 1 reports the headline comparison on the 185-item benchmark.
- Benchmark results: 11.3 pp is GANDR’s strict-accuracy lead over B2 on the shared backbone.The lead reflects the Drafter configuration and protocol-anchored commit; 98.6% of items commit at Round 1.
- Backbone transfer: +3.2 to +6.5 pp is GANDR’s positive strict lead over B2 across three further backbones.These are single-run point estimates, so the paper interprets them as directional.
- Commit-rule ablation: 22.7 pp is the strict-accuracy loss when the protocol-anchored commit rule is reverted to the original 0.95 gate.The reverted variant scores 48.1% strict accuracy and returns passed=False on 92.7% of runs.
- Auditability: The audit flags under-supported citations against two law-trained annotators at precision 0.80 and recall 0.88.Its four-way verdict labels remain advisory.
- Statistical comparison: Among disagreement items, GANDR is correct on 38 versus 17 for B2, with paired McNemar p=0.006.Per-domain buckets are too small to separate systems individually, so the claim rests on the aggregate.
6 Conclusion
GANDR builds claim-level verification into legal answer generation and evaluates grounding strictly. It leads the strongest baseline on the shared benchmark, remains positive across further backbones, and provides an audit trace whose binary detection reaches F1 0.84.
- Conclusion: GANDR builds claim-by-claim verification into generation and uses strict citation grounding rather than aggregate scoring.Its strict criterion requires every citation to resolve to a retrieved passage.
- Conclusion: 11.3 strict points is GANDR’s lead over the strongest baseline, with a positive +3.2 to +6.5-point lead on three further backbones.The shared-backbone comparison has p<0.01.
- Conclusion: The per-claim audit trace flags under-supported claims at F1 0.84 as a binary detector.The four-way verdict labels are advisory.
B Why only one ablation is reported
The Critic cannot be cleanly ablated because the orchestrator commits Round-1 drafts without reading its verdict, making observed accuracy differences non-causal. The isolated commit-rule ablation is therefore the only interpretable one.
- Critic ablation: Disabling the Critic leaves committed answers unchanged on 98.6% of full-GANDR runs because the Round-1 return path ignores its synthesis decision.The protocol_check, atomic cite_verify, and sef_rubric_score tools still run, so the audit trace remains available.
- Critic ablation: The −Critic variant’s accuracy difference reflects temperature-0.7 sampling draws rather than an effect of removing the Critic.A causal Critic ablation would require the component to change the committed text.
- Why the commit rule is isolable: The original 0.95 atomic-verification gate is confounded with repeated rewrites, so reverting it does not isolate the Critic or decomposition mode.The reverted gate commits at Round 1 on only 3.6% of runs and returns passed=False on 92.7% of them.
- Commit behavior: 98.6% of GANDR runs commit the Round-1 draft through the protocol anchor, bypassing the Critic’s PASS/REWRITE verdict.The Critic still logs its decision and directives on every round, but those outputs do not control the committed text.
- Comparison designs: B2 is a single-LLM baseline, whereas B5 reviews and rewrites within the same context; neither design supplies a clean GANDR Critic ablation.GANDR’s separate-context Critic is the intended contrast with same-context self-correction.
I Benchmark and retrieval construction
The benchmark combines LegalBench and LegalBench-RAG items under a shared, fixed retrieval surface and strict grounding checks. This design makes system comparisons primarily reflect generation and orchestration behavior rather than retrieval variation.
- Benchmark: 185 items span 15 LegalBench task tags and LegalBench-RAG contract-NLI and CUAD subsets across eight domain buckets.The benchmark includes statutory, case-law, mixed, and open-ended contract-grounding questions.
- Retrieval: 13,090 passages are indexed once with BM25, preserving rule_id strings from both source collections.Every system receives the same top-8 retrieved passages.
- Evaluation: Strict correctness requires judge correctness ≥4 plus valid grounding for every citation marker.A single fabricated or out-of-context marker fails the grounding check even when other markers resolve correctly.
- Audit protocol: Atomic cite_verify decomposes GANDR explanations and applications into claim–citation–span tuples labeled SUPPORTED, PARTIAL, UNSUPPORTED, or MISCITED.These labels populate the audit trace but are not consumed by the headline strict metric.
- Rewrite protocol: The Round-1-anchor ablation uses the original 0.95 gate, while the rewrite path uses a corpus-calibrated 0.50 atomic-verification threshold.The reported ablation identifies the protocol-anchored commit as the load-bearing choice.
L Extended judge calibration
The extended calibration evaluates judge agreement across systems and correctness bands. Agreement is strong overall but reveals a specific disagreement at the judge’s 4-to-5 boundary.
- Calibration design: 120 calibration items contain 20 items per system, and both annotators clear the preregistered agreement thresholds against the judge.Pooled agreement reaches r=0.846 and κq=0.845; annotators agree with each other at κq=0.979.
- Per-system agreement: Every system clears the per-system κq≥0.60 threshold, so no single system drives pooled agreement.Table 4 reports mean ratings and annotator–judge r and κq values for each system.
- Correctness bands: Agreement collapses on the judge’s 4-band because both annotators round every such item up to 5.Exact and within-one agreement remains high at rubric ends 1 and 5.
- Correctness bands: Collapsing scores {4,5} raises exact agreement to 86% without changing any pass/fail decision.The discrepancy is localized to the rubric anchor between 4 and 5 rather than a systematic scale-wide bias.
M Cost accounting
Cost accounting separates per-round Critic calls from observed end-to-end spending. Because most runs commit on Round 1, the dollar overhead is much smaller than the raw call-count overhead.
- Per-round cost: Each Critic round costs 4 + NLLM calls, including drafting, decomposition, per-claim verification, rubric scoring, and Critic synthesis.N is the atomic-assertion count.
- Observed cost: The Round-1 anchor makes GANDR’s dollar factor over B2 far smaller than its call-count factor.Most items commit in a single round rather than consuming the full rewrite budget.
- Accounting basis: Closed-source cost figures use OpenAI list prices dated 2026-05-22 and token usage from k=5 artifacts.Self-hosted backbones incur no per-token charge.
- Closed-source comparison: GANDR on gpt-5.4-mini costs approximately $2.45 and reaches 77.8% strict accuracy.This is compared with B2 on gpt-5.4 at approximately $3.10 and 76.8% strict accuracy.
N Baseline failure taxonomy
Under the shared grounding requirement, baselines fail cite-validity through distinct marker, identifier, and multi-agent coordination errors. These failures can leave dispositions correct while citations remain invalid or fabricated.
- Cross-baseline pattern: The four non-CREAC baselines fail cite-validity in qualitatively different ways under the same grounding requirement block.The comparison isolates how each system follows the citation instruction rather than differences in the requirement itself.
- B1 zero-shot: B1 zero-shot loses 18.9 pp because some outputs omit the canonical cite: marker or use unparseable bracket forms despite often reaching the right disposition.Its main failure is citation syntax rather than necessarily the legal conclusion.
- B3 advanced RAG: B3 advanced RAG loses 17.3 pp through missing markers and citations to rule_ids outside the retrieved top-8.Re-ranking improves retrieved evidence but does not change the model’s citation behavior.
- B4 CrewAI: B4 CrewAI loses 36.8 pp when its Researcher supplies nonexistent doctrine-label identifiers that the Writer cites through well-formed markers.The extra agent hop introduces the fabrication, so additional calls without auditing amplify the failure.
- B5 LangGraph: B5 LangGraph loses 29.8 pp by corrupting retrieved identifiers with subsection suffixes and nested brackets, then ratifying that corruption in its same-context Review node.This provides the concrete mechanism behind the paper’s separate-context design.
O Per-domain significance and marker normalization
Per-domain strict results support the aggregate GANDR comparison rather than broad domain-by-domain separation. Marker normalization shows that GANDR’s advantage remains after forgiving citation-form variation, indicating the strict lead is not merely punctuation-sensitive.
- Per-domain significance: At n≈20–30, per-domain confidence intervals are wide and overlap almost everywhere, so the aggregate carries the comparison.Only employment separates the systems on its own.
- Per-domain significance: The aggregate GANDR-versus-B2 comparison has McNemar b=38, c=17, p=0.006, remaining p=0.03 under Bonferroni correction across five baselines.B2 is the pre-specified primary comparison; the four non-B2 comparisons clear correction at p<10^-8.
- Marker normalization: Marker normalization forgives bracket variation, optional cite: prefixes, and identifier suffixes while leaving generation unchanged.The control tests whether strict accuracy penalizes invented provenance rather than unconventional punctuation alone.
- Marker normalization: After normalization, GANDR still leads the strongest baseline by +11.9 pp, slightly above the +11.3 pp headline gap.The surviving difference reflects how often answers are right once citation form is forgiven.