When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

1 Introduction

• LLMs are reshaping legal research, drafting, and interpretation by enabling fast retrieval, summarization, and synthesis of large volumes of legal material.These capabilities promise substantial efficiency gains, but their deployment in high-stakes legal settings introduces serious risks.Source: §1 · p. 1

M. Abdullah Canbaz University at Albany, SUNY

• Existing legal benchmarks often miss granular, citation-driven verification against a specific source, limiting what their performance says about practice-critical legal work.RAG outputs may appear grounded while still including unsupported inferences.Source: M. Abdullah Canbaz University at… · p. 1
• PARCEL uses 2024–2025 New York Court of Appeals decisions to create 3,000+ claims labeled Supported, Refuted, or Not Found.It adapts claim-level verification to legal authorities and the closed-domain task practitioners face.Source: M. Abdullah Canbaz University at… · p. 1
• The benchmark uses explanatory parentheticals from real appellate briefs to construct adversarial claims that remain stylistically plausible and partly grounded.This targets subtle factual or logical deviations from the underlying decision.Source: M. Abdullah Canbaz University at… · p. 2
• The paper contributes an adversarial benchmark and a reproducible pipeline for generating synthetic legal claims that test groundedness and robustness.Its research questions examine detection reliability, the most damaging adversarial categories, systematic failures, and what error analysis reveals about closed-book legal reasoning.Source: M. Abdullah Canbaz University at… · p. 2

2.1 Hallucinations in Legal Context

• Legal hallucinations can distort source materials or overgeneralize them, and even subtle deviations in holdings or authority attribution can alter legal analysis.Dahl [4] distinguishes failures internal to model reasoning from failures relative to external legal reality and reports hallucinations across jurisdictions and model families.Source: §2.1 · p. 2

2.2 Benchmarking LLMs in Law

• LexGLUE [2] and LegalBench [7] mainly assess document-level understanding or abstract legal knowledge, not whether individual claims are supported by cited authorities.RAG outputs may appear grounded while containing unsupported inferences [10].Source: §2.2 · p. 2

2.3 Systematic Claim Generation for Fact-Checking Tasks

• Existing claim-generation frameworks are largely domain-agnostic and do not account for legal writing’s stylistic, logical, and citation-specific conventions.PARCEL grounds adversarial claims in real appellate decisions and aligns evaluation with legal evidentiary standards.Source: §2.3 · p. 2

3 Methodology

• The methodology combines authentic explanatory parentheticals with controlled synthetic generation of supported, refuted, and not-found claims aligned to long-context opinions.Figure 1 presents the seed extraction, synthetic claim generation, and construction pipeline.Source: §3 · pp. 2–3, Figure 1 · p. 3

3.1 Unit of Analysis: Explanatory Parentheticals

• Explanatory parentheticals compactly convey a cited decision’s relevance and expose the author’s professional interpretation of its legal reasoning.Their regular structure supports automated harvesting, while their varied functions include summarizing holdings, rationales, procedural outcomes, and applications of rules.Source: §3.1 · p. 3

3.2 Task Definition

• Parenthetical Verification maps a decision and claim to entailment, contradiction, or neutral labels, with neutral corresponding to Not Found.A claim is entailed when the decision provides sufficient evidence for it; neutral means the decision contains insufficient information to support or refute it.Source: §3.2 · p. 3

3.3.1 Harvesting Parentheticals from Real Briefs.

• The team harvested parentheticals from 85 public appellate briefs, filtered out quotations and basic factual matches, and retained 180 reasoning-dense claims as exemplars.Candidate examples were extracted with a targeted prompt, then manually inspected and filtered using an LLM-as-a-judge approach.Source: §3.3.1 · pp. 3–4

3.3.2 Creating Synthetic Supported Claims from Judicial Decisions.

• From 85 recent Court of Appeals decisions, the team generated 849 supported claims, using few-shot prompts and a rubric to diversify parenthetical usage types.Decisions shorter than 2,000 characters and those triggering safety filters were excluded; the process instructed the model to include each usage type at least twice.Source: §3.3.2 · p. 4

3.3.3 Creating Refuted and Not Found Claims.

• Refuted claims alter entities, outcomes, qualifiers, or majority–dissent attribution, while Not Found claims introduce plausible material absent from the opinion.Not-found strategies include phantom citations, unaddressed arguments, and missing procedural details; the latter are plausible but omitted facts.Source: §3.3.3 · p. 4
• The completed dataset contains 3,396 claims with a controlled 1:2:1 balance across Supported, Refuted, and Not Found.The construction process generates adversarial claims conditioned on the full judicial decision, keeping them topically consistent and aligned with the judge’s writing style.Source: §3.3.3 · pp. 4–5

3.4 Verification Setup

• Verification is a zero-shot NLI task: each model receives the full judicial decision as premise and one parenthetical claim as hypothesis.The minimalist prompt defines the task and label space, requiring models to rely on their internal representations, long-context processing, and implicit legal reasoning.Source: §3.4 · p. 5

4.1 RQ1: Reliability of Hallucination Detection

• 97% accuracy and 0.97 macro F1: gemini-3-flash-preview leads; all but llama-3.3-70b exceed 90% accuracy with full long-context opinions.The evaluation reports overall accuracy, macro F1, and class-wise precision, recall, and F1.Source: §4.1 · p. 5, Table 2 · p. 6
• 15.4% of Not Found claims versus 4.3% of Refuted claims were marked Supported; glm-4.7's FER−1 = 1.18% versus FER0 = 9.55%.The aggregate confusion matrix reports percentages within each ground-truth class; FER measures false entailment.Source: §4.1 · pp. 5–6, Table 4 · p. 6, Table 3 · p. 6

4.2 RQ2 (Failure Modes by Adversarial Strategy)

• 0.86–0.97 average accuracy across refuted strategies; Authority Swap was hardest (0.77 for gemini-2.5-flash and gpt-oss-120b), while glm-4.7 scored 1.00 on Scope Contradiction.Authority Swap requires discourse-level attribution, including distinguishing majority opinions from dissents or concurrences.Source: Table 5 · p. 6, §4.2 · p. 6
• 0.72 Phantom Citation versus 0.81 Missing Detail average accuracy; Phantom Citation was hardest (llama-3.3-70b: 0.24), while gemini-3-flash-preview scored 0.93–0.96 across not-found strategies.The results characterize fabricated but legally plausible content as more challenging than explicitly false statements.Source: §4.2 · pp. 6–7, Table 6 · p. 7

4.3 RQ3 (Systematic Failures and Error Diagnosis)

• 290 claims (8.54% of the dataset) were misclassified by at least three models, but only 8 (0.24%) defeated all seven.The sharp decline across failure thresholds indicates that most errors are model-specific rather than universal.Source: Table 7 · p. 7, §4.3 · p. 7
• Because most hard claims are classified correctly by at least some models, majority voting or verifier cascades could substantially reduce hallucination risk.Only a very small fraction of claims remain universally difficult.Source: §4.3 · p. 7
• On 27 instances where at least six models failed, models sometimes accepted legally accurate abstract claims absent from the decision, letting parametric legal knowledge override textual constraints.The example claim restates a statutory code in plain English, although its descriptive wording does not appear in the opinion.Source: §4.3 · p. 7

• Models frequently over-validate unsupported claims, so aggregate accuracy substantially overstates their trustworthiness in legal settings.Source: §5 · p. 8
• Models are better at detecting explicit contradictions than missing support, a key weakness for RAG systems facing fabricated details or uncited authorities.Source: §5 · p. 8
• Full-opinion access does not resolve the problem: models may validate claims that are abstractly accurate but unsupported by the specific decision.Source: §5 · p. 8
• Model diversity may help mitigate hard claims, with ensemble verification or secondary checking offered as possible strategies.Source: §5 · p. 8

• PARCEL reflects practical legal review by checking whether a proposition is supported by cited authority, while exposing vulnerabilities to fabricated citations, missing details, and unsupported abstractions.Source: §6 · p. 8
• The authors call for evaluation frameworks that prioritize groundedness and safety over aggregate accuracy.Source: §6 · p. 8
• Future work includes uncertainty-aware evaluation, stronger citation verification, and verifier- or ensemble-based mitigation strategies.Source: §6 · p. 8

• The study’s prompts and dataset are available at the PARCEL GitHub repository.Source: A Prompts and Dataset · p. 9
• The repository includes templates for generating Supported, Refuted, and Not Found claims.Source: A Prompts and Dataset · p. 9
• The repository also provides templates for model evaluation.Source: A Prompts and Dataset · p. 9