PaperlayerView exact revision on arXiv

Source-linked AI summary

Is this Citation on Point?

Apurv Verma

arXiv:2608.12571v1cs.DLcs.CL

TL;DR

Current evaluations largely overlook whether real legal citations actually support the propositions they accompany. This paper tests proposition-level verification through controlled citation corruptions and finds that models detect wrong cases far better than wrong pinpoint pages, with reasoning helping but not closing the gap.

  • Problem

    Existing evaluations largely overlook whether real legal citations support the propositions for which they are offered, a task distinct from detecting fabricated citations.

  • Method

    The paper evaluates fourteen model configurations on real legal citations corrupted by replacing cited cases or changing pinpoint pages within the same case.

  • Results

    37–61% recall on CLERC and 52–83% on BriefMe for Hard examples shows that models struggle substantially more with proposition-level verification than easier citation errors.

  • Takeaways & Limitations

    Current models recognize legal topics more reliably than they verify page-level support, while prompting for page-grounded checks improves recall but increases false positives.

  • Takeaways & Limitations

    The study focuses on substantive citations in U.S. federal court opinions and briefs and treats citation support as a binary decision.

Abstract

from arXiv · show

In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.

1. Introduction

The paper studies whether LLMs can verify that real legal citations support the propositions for which they are offered, a harder task than detecting nonexistent authorities. Across controlled corruptions, models readily detect wrong-case citations but frequently miss errors that change only the pinpoint page, relying on topical overlap rather than page-level support.

  • Problem: Real citations can still misrepresent authority when the cited case, reporter, and page exist but do not support the quoted proposition.This support-verification failure is harder to detect than a nonexistent citation.
  • Scope and method: The study focuses on substantive citations that provide direct doctrinal support because errors in them change the meaning of an argument.The authors use controlled probes rather than claiming the corruptions reproduce how LLM drafting errors arise.
  • Results: 93–100% recall on Easy corruptions shows that models nearly saturate when a citation is replaced with a different case across both datasets.Easy corruptions replace the cited authority, making them distinct from same-case pinpoint errors.
  • Results: 37–61% recall on court opinions and 52–83% on briefs shows that models struggle when only the pinpoint page changes within the same case.These Hard corruptions preserve the cited authority while altering its pinpoint reference.
  • Results: GPT-5.4 with extended reasoning gains 11 pp on court opinions and 18 pp on briefs but still misses 40% of Hard corruptions on court opinions.Page-grounded prompting improves recall but also raises the false positive rate, so neither intervention closes the gap.

2. Task Definition and Citation Taxonomy

The task tests whether a cited authority supports the proposition for which it is offered, using the marked source paragraph and cited-document content. The evaluation focuses on substantive citations, which assert legal rules, while filtering out procedural and secondary citations.

  • Task definition: The model receives a source paragraph with the target citation wrapped in <CITE> tags and the content of the cited document.It must determine whether the cited authority is on point for the surrounding proposition.
  • Citation taxonomy: Only substantive citations make support claims by asserting a legal rule and offering a case as its basis.Procedural-history citations record prior events, while secondary citations provide background, comparison, or contrary authority.
  • Automated classification: 0.83 F1 was achieved by a prompt-based LLM classifier filtering non-substantive citations on a held-out expert-annotated validation set.The same legal experts developed the taxonomy and annotated the validation set.
  • Dataset scope: 69–70% of citations in both datasets were substantive, so the evaluation restricts analysis to substantive case-law citations.Statutes and regulations are excluded from the case-law evaluation.

3. Evaluation Design

The evaluation uses real citations from court opinions and legal briefs, applying controlled perturbations that preserve legal context while selectively breaking citation support. Human validation indicates that the resulting corrupted examples generally reflect genuine not-on-point citations.

  • Datasets: The study draws on CLERC (∼2,000 citations from court opinions) and BriefMe (∼750 citations from legal briefs).CLERC includes preceding paragraphs as context, whereas BriefMe does not.
  • Corruption Design: Controlled perturbations create three difficulty levels: random-document, same-document, and wrong-page-number corruption.Same-document corruption is unavailable for BriefMe because it lacks preceding paragraphs.
  • Corruption Design: The study uses controlled perturbations rather than LLM-generated corruptions to preserve original legal prose and citation contexts.Pilot LLM-based corruptions sometimes altered text associated with other citations or introduced unrelated defects.
  • Corruption Design: Wrong-page corruption changes only the pinpoint page within the same case, preserving case-level overlap while breaking page-level support.This is designed as the hardest setting because the cited case remains unchanged while the page discusses a different legal issue.
  • Human Validation: Annotator agreement with heuristic labels averaged 88% for CLERC and 84% for BriefMe, supporting the validity of the corrupted examples.Agreement remained high across corruption types, reducing concerns that harder corruptions were unusually ambiguous.

4. Experiments

The experiments evaluate fourteen frontier-model configurations spanning three model families and thirteen base models, including two GPT-5.4 reasoning settings. Performance is assessed by recall on corrupted citations and false positive rate on valid citations, where effective guardrails require high recall and low FPR.

  • Model configurations: 14 configurations span three model families and 13 base models, including GPT-5.4 with standard inference and high reasoning effort.The evaluated models include systems from OpenAI, Anthropic, and Google DeepMind.
  • Evaluation metrics: Recall measures the fraction of erroneous citations correctly flagged, while false positive rate measures the fraction of valid citations incorrectly flagged.These metrics jointly capture detection effectiveness and over-flagging.
  • Evaluation metrics: A good citation guardrail requires high recall and low false positive rate.The evaluation therefore considers both successful detection of corrupted citations and preservation of valid citations.

5. Results

Models nearly saturate easy and wrong-case examples, but same-case pinpoint mismatches remain difficult, especially in court opinions. Higher scale and reasoning improve recall, yet no model combines high hard-example recall with low false-positive rates.

  • Difficulty: 94.8–99.8% CLERC and 93.0–100.0% BriefMe recall on Easy examples shows near-saturation.On CLERC, Hard-example recall falls to 36.5–60.6%; on BriefMe, it ranges from 51.5–82.7%.
  • Scale and capability: 55.5% vs. 38.4% and 77.0% vs. 51.5% Hard-example recall show GPT-5 improving over GPT-4.1 on CLERC and BriefMe.Performance does not increase monotonically with nominal model capability: GPT-5.4 without reasoning trails GPT-5, and Claude Opus 4.6 trails both Sonnet variants.
  • Recall–FPR tradeoff: 60.6% baseline recall with 34.9% FPR for GPT-4o contrasts with Claude Opus 4.6’s 4.1% CLERC FPR and 36.5% Hard-example recall.The same recall–false-positive tradeoff appears on BriefMe, where the highest Hard-example recall is 82.7%.
  • Document type: 12–26 pp higher Hard-example recall on BriefMe than CLERC occurs for every matched model.For GPT-5.4, high reasoning adds 18.0 pp on BriefMe and 11.3 pp on CLERC.
  • Reasoning: 82.0% vs. 64.0% and 59.8% vs. 48.5% Hard-example recall show high reasoning improving GPT-5.4 on BriefMe and CLERC.High reasoning also lowers FPR from 15.6% to 13.1% on CLERC and from 14.6% to 14.4% on BriefMe, but CLERC still misses about 40% of pinpoint mismatches.

6. Diagnosing the Failure Mode

Models miss wrong-page citations primarily by relying on topical overlap, inventing support rationales, and failing to verify quoted text. Page-grounded prompting improves recall on Hard examples but also increases false positives because models become more skeptical overall.

  • Recurring error patterns: Three recurring errors are invented support rationales, topical matching, and failure to check quoted text.These patterns are illustrated through GPT-5 false negatives on corrupted citations.
  • Recurring error patterns: Roughly two-thirds of missed corruptions include rationales claiming the cited page “expressly states” or “explicitly says” absent language.The model invents confirmation instead of treating the missing language as grounds to reject the citation.
  • Recurring error patterns: 92% of GPT-5 false negatives with verbatim quotes contain quoted text absent from the cited page.This finding motivates explicitly asking models to verify whether quoted language appears in the target content.
  • Page-grounded prompting: Page-grounded prompting raises recall by 9.7–27.8 pp on CLERC and 6.5–35.5 pp on BriefMe across models.On GPT-5, recall improves by +14.1 pp for cases with verbatim quotes versus +10.5 pp for cases without.
  • Page-grounded prompting: False positive rates rise by 1.0–24.7 pp across the two datasets, indicating that prompting makes models more skeptical overall rather than more selective.The increase occurs across every model and accompanies improved recall on Hard examples.

7. Discussion

The discussion finds that models detect easy citation errors but struggle with proposition-level support, especially when wrong pinpoint pages remain topically coherent. Larger models and reasoning help only modestly, while page-grounded prompting improves recall at the cost of false positives, underscoring the need for page-specific verification and supervised use.

  • RQ1: Detection by Difficulty: Hard-example recall is 37–61% on CLERC and 52–83% on BriefMe, a 16–63 pp drop from Easy examples that signals difficulty with proposition-level verification.The discussion frames this performance gap as the central detection limitation across both corpora.
  • RQ2: Model Scale and Reasoning: GPT-5 outperforms GPT-4.1 on Hard examples, reaching 55.5% versus 38.4% on CLERC and 77.0% versus 51.5% on BriefMe, but gains are modest and non-monotonic across models.GPT-5.4 without reasoning trails GPT-5 on both datasets, while Claude Opus 4.6 trails both Sonnet variants.
  • RQ3: Document Type: Hard-example recall is 12–26 pp higher on briefs than on court opinions across all models, with extended reasoning producing the largest brief improvement, +18 pp for GPT-5.4.The proposed explanation is that briefs often state narrower propositions, but proposition specificity was not directly measured.
  • The Failure Mode and Its Limits: Models favor topical coherence over proposition-level support because a wrong pinpoint page preserves doctrinal and topical relatedness without actually supporting the cited proposition.In roughly two-thirds of missed pinpoint mismatches, models claim the cited page expressly states supporting text; among missed Hard corruptions with verbatim quotes, that language is absent 92% of the time.
  • The Failure Mode and Its Limits: Page-grounded prompting improves recall on Hard examples but increases false positives, making the intervention diagnostic rather than corrective.The result suggests baseline prompting did not elicit proposition-level verification and that reliable tools must verify support on the cited page.
  • Implications for Legal Practice: Citation verification tools remain useful under lawyer supervision, but checking authority existence or accepting model rationales at face value can miss real citations that misrepresent their supporting propositions.The discussion identifies this risk in contrast to fabricated-case errors that database lookups can catch.

8. Limitations

The study focuses on binary support judgments for substantive citations in U.S. federal opinions and briefs, leaving broader legal evaluation dimensions and realistic review workflows unmeasured. Its corruption strategies and measurements also have important caveats, including coarse coverage, possible recall understatement, and limited human-validation agreement.

  • Scope: The evaluation covers substantive citations in U.S. federal court opinions and briefs, treating support as binary rather than accounting for authority, factual analogy, or later treatment.The study targets the clearest question: whether the cited page supports the text’s proposition.
  • Measurement caveats: 84–88% annotator agreement indicates that human validation was imperfect, and chance-valid corruptions may make reported recall on corrupted examples appear understated.A model accepting a corruption that remains on point is counted as missing an error.
  • Coverage: The corruption strategies are coarse and omit relational substitutions among connected cases, which preliminary experiments suggest may be harder to detect than wrong-pinpoint errors.Systematic evaluation of this failure mode remains for future work.
  • Workflow validity: The study neither compares model-assisted legal work with human-review workflows nor measures end-to-end auditing costs.False positives increase reviewer burden, while false negatives create false reassurance, so per-citation results do not capture realistic supervised utility.

9. Conclusion

Verifying that a cited case exists does not establish proposition-level support, the harder citation-verification problem studied here. Current models often identify the correct legal topic without confirming page-level support, and neither stronger models nor extended reasoning closes the gap.

  • Conclusion: Database lookup largely solves fake citations, but real cases that fail to support the cited proposition remain the harder verification problem.The paper frames this as support verification at the proposition level.
  • Conclusion: 16–63 pp: recall falls on Hard examples despite near-saturated recall on Easy examples across fourteen model configurations.The evaluation spans three model families and two document types.
  • Conclusion: 40%: GPT-5.4 with high reasoning effort still misses pinpoint mismatches on court opinions, while page-grounded prompting improves recall but raises false positive rates.More capable models and extended reasoning do not close the gap, and skeptical prompting has a precision cost.
  • Conclusion: Legal review must test proposition-level support, because current models often recognize the right topic without verifying the cited proposition.Closing this gap may require more than prompt tuning, larger models, or extended reasoning.

Reproducibility Statement

The evaluation is designed to be reproducible through detailed methodological documentation, public prompt templates, and openly available source corpora.

  • Reproducibility Statement: The methodology, prompt templates, and both source corpora are publicly documented or available in specified sections and repositories.Methodological details appear in Sections 3 and F; prompt templates in Section B; CLERC and BriefMe are publicly available.

Ethics and Impact Statement … B. Prompts

The paper reports an ethics tradeoff: exposing citation-verification failures may improve legal AI safety while enabling harder-to-detect errors. Its appendices document datasets, errors, evaluation formats, legal-citation background, related work, and baseline versus page-grounded prompts.

  • Ethics and Impact Statement: The study uses public court opinions and briefs, compensates experienced legal analysts at market rates, and weighs improved safety against adversarial misuse risks.The authors note that exposing limitations could help adversaries craft harder-to-detect errors, but judge the benefits greater.
  • Appendix Contents: The appendices include related work, prompts, dataset statistics, human annotation details, error analysis, evaluation-data formats, diagnostic results, and a primer on legal citations.The contents list identifies baseline and page-grounded prompts, processing and schema examples, sensitivity results, and distinguished-authority diagnostics.
  • F. Evaluation Data Format: The appendices separately organize processing beyond source datasets, schema definitions, corruption examples, prompt-sensitivity results, legal document types, citation anatomy, and common patterns.These materials support interpretation of the evaluation data and the U.S. legal-citation conventions used throughout the paper.
  • A. Related Work: Prior legal NLP work addresses authority retrieval and citation formatting, whereas this study tests whether a citation supports a passage, with Hard-example recall of 37–61%.The cited comparison reports 59% recall for the best LePaRD retrieval models and describes support verification as a distinct task.
  • A. Related Work: The paper targets misattributed rather than fabricated hallucinations and distinguishes topical authority-finding from proposition-level support verification.Related work characterizes misattributed support as harder to detect and emphasizes that fabricated authorities and misattributed support create different verification problems.
  • A. Related Work: Citation-evaluation research motivates testing support beyond citation presence, while noting that binary supports/does-not-support judgments miss support nuance.The paper situates its methodology alongside ALCE and critiques of binary NLI-based citation evaluation.
  • B.1. Citation Verification Prompt: The baseline prompt asks models to evaluate citation support for a source claim using IRAC, identify relevant rules, and answer whether the citation is on point.Inputs include the source paragraph and cited-document content or pinpoint pages; the output requires a definitive YES/NO and brief rationale.
  • B.2. Page-Grounded Verification Prompt: The page-grounded intervention adds verbatim-quote, page-level, and specific-claim checks, rejecting citations supported only by topical relevance.It requires verifying the cited pages rather than the case generally and locating asserted language or close paraphrases in the target content.

C. Dataset Statistics · D. Human Annotation Details

The study retains case-law citations from CLERC and BriefMe, characterizes their pinpoint coverage and citation roles, and uses proposition-level human annotation to assess whether citations support their accompanying propositions. Annotator agreement is generally strong on Medium and Hard examples, though conventional metrics can understate agreement under class imbalance.

  • C. Dataset Statistics: The evaluation filters out statutes, regulations, and other non-case authorities from the public CLERC and BriefMe datasets.The retained citations are analyzed by pinpoint-page inclusion and functional role.
  • C. Dataset Statistics: Non-pinpoint citations are more common in BriefMe, whereas court opinions more often cite specific pages that enable Hard corruptions.Hard corruptions keep the cited case fixed and alter only the cited page.
  • C. Dataset Statistics: 70% of CLERC citations and 69% of BriefMe citations are substantive, while procedural citations comprise 11% and 4%, respectively.Secondary citations account for 20% in CLERC and 27% in BriefMe.
  • C. Dataset Statistics: The citation-role classifier uses GPT-4.1-mini and compares predictions with legal-expert labels on a held-out CLERC set.The classifier assigns retained citations to substantive, procedural, or secondary categories; BriefMe is omitted from the Medium-split breakdown.
  • D. Human Annotation Details: Three legal analysts with litigation experience and U.S. federal case-law expertise labeled citations as “on point” or “not on point” using the source passage and full cited document.They were instructed to judge proposition-level support rather than mere topical relevance.
  • D. Human Annotation Details: For BriefMe, Fleiss’ κ is -0.07 while Gwet’s AC1 is 0.85, reflecting class imbalance and indicating genuine agreement.The paper treats Gwet’s AC1 as more robust than traditional agreement metrics for skewed annotation distributions.
  • D. Human Annotation Details: 86% three-way agreement was observed for both CLERC Hard examples and BriefMe Medium examples, with AC1 = 0.81 reported for CLERC.Each subset had 7 of 10 overlapping cases; Easy and Positive categories lacked sufficient overlap for reliable estimation.
  • D. Human Annotation Details: Human labels were also compared with anticipated labels designating original citations as on point and corrupted citations as not on point.This analysis measures agreement with the labels implied by the dataset construction rather than agreement among annotators.

E. Error Analysis … I.4. Other Terms Used in This Paper

The paper’s error analysis shows that models often replace proposition- and page-level verification with topical matching, sometimes fabricating verification claims. The appendices define the evaluation data, diagnostic treatment-relation tests, citation anatomy, and terminology underlying these analyses.

  • E. Error Analysis: Roughly two-thirds of false negatives contain fabricated claims that the cited page expressly or directly supports the proposition.Models construct plausible legal rationales asserting verification of content absent from the cited page.
  • E. Error Analysis: 92% of GPT-5 false negatives involving verbatim-quoted source passages cite pages where the quoted language is absent.These cases provide an objective page-content stress test for citation support verification.
  • F. Evaluation Data Format: Hard evaluation examples change only the pinpoint page within the same case, while Easy and Medium examples replace citations with unrelated or different-section sources.“Not on point” is the positive class because detection targets erroneous citations.
  • G. Full Prompt Sensitivity Results: Page-grounded prompting produces modest Easy and Medium recall improvements and the largest gains on Hard recall in the full CLERC results.The corresponding BriefMe results omit Medium because that dataset lacks the difficulty level.
  • H. Diagnostic Results on Distinguished and Criticized Authorities: Treatment-relation replacements are challenging because models both over-reject valid citations and generally gain not-on-point recall from page-grounded prompting.The diagnostic sets cover authorities that distinguish or criticize other authorities, whose legal relation does not itself establish citation support.
  • I. Primer on U.S. Legal Citation and Document Types: U.S. legal writing uses Bluebook conventions, and the paper distinguishes document types, authorities, citation signals, and their relationships to legal claims.The Bluebook standardizes source identification and usage but is not itself law.
  • I.2. Anatomy of a Case Citation: A standard case citation identifies the case, reporter, first page, pinpoint page, court, and year, with the pinpoint page central to proposition-level verification.A case may be generally relevant while its cited page fails to support the specific surrounding proposition.
  • I.4. Other Terms Used in This Paper: The paper defines an on-point citation as authority supporting the offered proposition and distinguishes holdings, citechecking, substantive citations, and procedural-history citations.These terms frame the paper’s analysis of whether legal citations directly support the claims for which they are offered.
Loading 2608.12571v1…