PaperlayerView exact revision on arXiv

Source-linked AI summary

Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection

Mingzhe Li, Zhiqiang Lin, Shiqing Ma

arXiv:2605.08583v2cs.CL

TL;DR

Citation hallucination detection is limited by plausible fabricated references and binary, weakly auditable verification. The paper addresses this with a 12-code taxonomy and cascading multi-agent detector, achieving 97.1% synthetic accuracy and 97.1% real-world recall. Its evaluation is concentrated on computer-science and ML papers, with connector coverage and rate limits constraining broader robustness.

  • Problem

    Citation hallucination detection needs finer-grained, field-level adjudication because existing systems provide binary decisions and incomplete bibliographic audits.

  • Method

    CiteTracer combines a 12-code taxonomy with structured extraction, cascading evidence retrieval, deterministic field matching, and specialist judgers for ambiguous cases.

  • Results

    97.1% accuracy on the synthetic set and 97.1% recall on the real-world set were reported.

  • Takeaways & Limitations

    The taxonomy and benchmark provide field-level diagnostic distinctions for different citation errors across controlled mutations and real-world fabrications.

  • Takeaways & Limitations

    Evaluation focuses on Computer Science and ML papers, while limited connector coverage and high-concurrency rate limits may reduce robustness elsewhere.

Abstract

from arXiv · show

Large language models are increasingly used in scientific writing, yet they can fabricate citation-shaped references that appear plausible but fail bibliographic verification. Existing detectors often reduce verification to binary found/not-found decisions and rely on brittle parsing or incomplete retrieval, offering little field-level signal to auditors. We reframe citation hallucination detection as taxonomy-aligned field-level adjudication and introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations. Based on this taxonomy, we build CiteTracer, a cascading multi-agent detector that extracts structured citations from PDF and BibTeX, retrieves evidence through cache lookup, URL fetch, scholar connectors, and web search, applies deterministic field matching, and routes ambiguous cases to class-specialist judgers. We release a benchmark of 2,450 synthetic citations built from real seeds with controlled LLM mutations, paired with 957 real-world fabricated citations drawn from ICLR 2026 and an anonymous conference desk-rejected submissions. CiteTracer reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and detects 97.1% of fabrications on the real-world set without abstaining. Code: https://github.com/aaFrostnova/CiteTracer.

1 Introduction

Citation hallucinations arise when fluent language models generate plausible but bibliographically incorrect references, while existing detectors provide limited field-level and ambiguity-aware auditing. The paper introduces CiteTracer, a taxonomy-aligned multi-agent detector and benchmark for this problem.

  • Motivation: Bibliographic errors can involve mismatched titles, authors, venues, or other citation fields despite plausible surface form.Such errors undermine credit attribution and the traceability of evidence.
  • Limitations of Existing Detectors: Existing citation auditors often collapse ambiguous cases into binary Real-or-Fake decisions and leave several bibliographic fields unaudited.Reportedly unaudited fields include venue, year, DOI, pages, and publisher.
  • Contributions: The paper introduces a 12-code taxonomy spanning REAL, POTENTIAL, and HALLUCINATED citation classes, with field-level failure modes.The supplied contribution passage also specifies a 2,450-citation synthetic benchmark spanning five rendering styles.
  • Contributions: CiteTracer combines structured citation extraction, cascading evidence collection, deterministic field matching, and class-specialist adjudication.The system evaluates PDF and BibTeX inputs and emits taxonomy-aligned verdicts with offending fields and reasons.
  • Results: 97.1% accuracy was achieved on the synthetic benchmark, alongside 97.1% recall on the real-world set.The contribution passage reports that CiteTracer surpassed every baseline on every class.

2 Related Work

Related work largely verifies whether citations can be found, but provides limited visibility into which fields are wrong and how ambiguous cases should be adjudicated. The paper positions field-level taxonomy and auditing as the missing capability.

  • Hallucination in Academic Writing: Citation hallucinations are especially consequential in academic writing because references are structured factual claims that should resolve to real publications.Language models can produce references that look plausible while failing bibliographic verification.
  • Citation Hallucination Detection: Commercial and academic auditors commonly emit binary Real-or-Fake verdicts that hide field-specific errors and collapse POTENTIAL cases.The POTENTIAL middle ground includes nickname variants, non-academic sources, and peripheral metadata gaps.
  • Citation Hallucination Detection: Hallucinator and GPTZero audit only subsets of bibliographic fields, while GPTZero additionally accepts only PDF input and places throughput behind a paid subscription.Hallucinator focuses on title and author; GPTZero audits title, author, date, URL, and publisher.
  • Citation Hallucination Detection: Existing systems do not expose a per-field taxonomy that supports auditing which citation field is wrong and why.The paper identifies this as the gap addressed by its taxonomy and field-level multi-agent approach.

3 Benchmark

The benchmark combines controlled mutations of real bibliographic seeds with real-world fabricated citations, while organizing errors by field and adjudication class. This design supports both consistent evaluation and diagnostic analysis.

  • Construction: 2,450 synthetic citations are grounded in real bibliographies and paired with a 957-citation real-world test set from two conference sources.The real-world set includes 807 ICLR 2026 desk-rejection citations and 150 citations from another anonymous conference.
  • Taxonomy: The taxonomy defines 12 codes grouped into REAL, HALLUCINATED, and POTENTIAL classes according to field correctness and external verifiability.HALLUCINATED codes localize errors in title, authors, venue, year, identifier, or peripheral metadata.
  • Taxonomy: REAL codes cover exact matches and normalizable formatting variants, while POTENTIAL codes cover nickname variants, non-academic sources, and unverifiable peripheral fields.The taxonomy separates ambiguous cases from confirmed field-level hallucinations.
  • Construction: Synthetic entries begin with seed BibTeX records and receive code-specific mutations while preserving untouched fields.An LLM proposes candidate values, and a deterministic post-processor enforces each mutation operator’s field schema.
  • Real-world Test Set: Each real-world entry carries a venue chair’s fabrication verdict and the cited bibliographic record for cross-checking.The two slices were collected from desk-rejected submissions flagged for fabricated references.

4 Methodology

CiteTracer decomposes citation hallucination detection into structured extraction, cascading evidence retrieval, deterministic field matching, and specialist adjudication. The pipeline produces field-level, taxonomy-aligned citation verdicts while escalating unresolved cases to flexible agents.

  • Reference Extractor: CiteTracer parses each paper reference into a structured citation record containing bibliographic fields for downstream verification.The Reference Extractor uses OCR to propose citation blocks, then a parsing agent rechecks cropped images against OCR text and repairs boundaries before finalizing records.
  • Cascading Evidence Collector: Evidence collection proceeds from Memory and URL Fetch to parallel Scholar Connectors and Web Search, balancing retrieval cost against source coverage.The cascade retrieves cached or directly linked records first, then expands to bibliographic connectors and unstructured web sources when needed.
  • Field Matcher: The Field Matcher compares citation fields with retrieved evidence using deterministic, field-specific normalizers before invoking a Matcher Agent for unresolved cases.Supported normalizations include case folding, punctuation removal, DOI canonicalization, page-range normalization, author-order normalization, and known venue abbreviations.
  • Field Matcher: The matcher emits a residual field-status profile that distinguishes normalizable variation, missing metadata, and true field contradictions for later taxonomy adjudication.Examples include reordered authors, venue abbreviation matches, and missing publisher or page information.
  • Class-Specialist Judgers: Class-specialist judgers route unresolved citations according to their residual field patterns instead of applying one general-purpose judge across all taxonomy codes.This design addresses differing evidence thresholds for REAL, POTENTIAL, and HALLUCINATED cases.
  • Class-Specialist Judgers: Table 2 reports label-level performance for CITETRACER and baselines on BibTeX and PDF inputs.The supplied passage identifies the table’s input modes but does not provide its complete contents here.

5 Evaluation

CiteTracer is evaluated on synthetic and real-world citation sets using label-level, subtype-level, extraction, ablation, and real-world metrics. It outperforms baselines across input modes and identifies both broad classes and fine-grained bibliographic failure codes.

  • Label-level Performance: CiteTracer surpasses every baseline on all three classes under both PDF and BibTeX input.On BibTeX, it achieves F1 scores of 97.0 for REAL, 95.8 for POTENTIAL, and 98.5 for HALLUCINATED; PDF scores are 95.1, 95.5, and 96.9.
  • Per-subtype Performance: CiteTracer attains the highest in-bucket TPR and lowest out-of-bucket FPR on every reported BibTeX subtype.It reports R (94.3/0.1), P1 (100.0/0.6), P3 (99.4/0.4), and H-average (99.0/0.3) as TPR/FPR pairs.
  • PDF Extraction: LLM reparsing produces the larger PDF-extraction gain, while cropped page images further improve title and identifier accuracy.A-to-B reparsing raises AUTHORS from 85.6 to 98.2 and LOCATION from 81.6 to 100.0; B-to-C imagery raises TITLE from 96.5 to 98.5 and IDENTIFIER from 93.1 to 96.5.
  • Impact of Web and Scholar Connectors: Removing the Web Agent lowers F1 across all three classes, and removing Scholar Connectors reduces performance further.The reported Web-Agent ablation drops REAL F1 from 97.0 to 79.6, POTENTIAL from 95.8 to 79.0, and HALLUCINATED from 98.5 to 85.8.
  • Real-World Evaluation: On real-world sets, CiteTracer flags 796 of 807 ICLR citations as hallucinated and labels all 150 confirmed fabrications across both venues.The ICLR set yields 98.6% recall, while the second set contains 133 FAKE-REFERENCE and 17 POTENTIAL labels; correctly detected citations average 2.24 error codes.

6 Conclusion

The paper reframes citation hallucination detection as taxonomy-aligned, field-level adjudication and implements it through a cascading multi-agent detector. CiteTracer achieves strong synthetic accuracy and real-world recall while attributing improvements to specific design choices.

  • 6 Conclusion: CiteTracer replaces binary citation verification with a 12-code taxonomy spanning REAL, POTENTIAL, and HALLUCINATED classes.The taxonomy supports per-field verdicts and manual routing for ambiguous cases.
  • 6 Conclusion: The four-module detector combines deterministic matching, an eight-connector evidence cascade, and three specialist agents.The rule matcher handles VALID and HALLUCINATED cases near-zero cost, while specialist agents adjudicate disjoint taxonomy slices using calibrated evidence thresholds.
  • 6 Conclusion: 97.1% accuracy is achieved on the synthetic set, alongside 97.1% recall on the 957-citation real-world set.The evaluation is designed to attribute improvements to specific system design choices.

A Benchmark Details

The benchmark defines auditor-facing citation classes and constructs controlled field-level mutations from real bibliographic seeds. Its taxonomy distinguishes exact or normalizable citations, verifiable errors, and cases requiring manual inspection.

  • Taxonomy: The 12 taxonomy codes are grouped into REAL, POTENTIAL, and HALLUCINATED auditor-facing classes.The codes map citation verdicts to concrete auditor actions.
  • REAL: REAL codes cover exact matches and normalizable variants such as venue abbreviations, punctuation, capitalization, initials, and et al. truncation.R1 is character-for-character matching; R2 permits surface normalization; R3 preserves named-author correctness under et al. truncation.
  • HALLUCINATED: HALLUCINATED codes localize externally verifiable bibliographic errors to title, authors, venue, year, identifiers, or peripheral metadata.H1 through H6 each target a specific field-level corruption.
  • POTENTIAL: POTENTIAL citations lack sufficient stable evidence for automatic resolution and are routed for manual inspection.P1 includes recognized author-name variants such as nicknames, spelling variants, and transliterations.

A.2 Source Selection

The benchmark uses richly populated BibTeX records from public bibliographic repositories and applies controlled mutation operators to create taxonomy-labeled instances. Quality-control audits verify field boundaries, source resolvability, and author-variant validity before retention.

  • Source Selection: Official BibTeX entries are collected from Crossref, DBLP, and arXiv, prioritizing seeds with many populated bibliographic fields.The selected fields include title, authors, venue, year, identifiers, and peripheral metadata.
  • Seed-Pool Composition: 2,270 synthetic entries derive from real publications, while 180 P3 entries are pure fabrications without real seeds.Crossref and DBLP supply 41.6% and 35.2% of the real-derived seed pool, respectively.
  • Mutation Construction: Each non-R1 mutation changes only documented fields corresponding to its named failure mode, with deterministic post-processing enforcing field boundaries.The generator proposes candidate values, while operators leave all other seed fields unchanged.
  • Quality Control: 2,450 taxonomy-labeled instances remain after round-trip, verifiability, and author-curated boundary checks.The checks verify operator diffs, source resolution for R1 and P3 cases, and recognized P1 author variants.

B Efficiency Analysis

CiteTracer uses a cascade that resolves most citations through fast cache or deterministic matching, reserving costly specialist adjudication for the remainder.

  • 61.7% of citations are closed by deterministic Field Matcher rules without an LLM call.Cache hits and non-academic short-circuits close another 3.6% within seconds.
  • 34.7% of citations reach Class-Specialist Judgers, whose sequential LLM passes and external-API checks account for most per-citation latency.
  • 0.50 citations per second is CiteTracer's approximate end-to-end throughput, with the long tail driven primarily by external-API round trips.

C Implementation Details

CiteTracer combines layout-aware extraction, structured parsing, deterministic field matching, and specialist adjudication with explicit field-equivalence rules and taxonomy-based escalation.

  • Pipeline: CiteTracer uses layout-aware OCR, a parsing agent, a field matcher, and class-specialist judgers to process citation records and unresolved discrepancies.The Parser Agent rechecks OCR text against the cropped reference image and extracts structured bibliographic fields.
  • Reference extraction: The Parser Agent outputs structured fields including title, authors, venue, year, volume, pages, publisher, location, DOI, arXiv ID, and URL.Unknown values are left empty, as [] or null, depending on field type.
  • Field matching: The Field Matcher classifies authors as exact, r2_initial, p1_variant, or h2_error, distinguishing formatting, name-form, and genuine identity discrepancies.Venue and publisher receive exact, alias, or different labels.
  • Potential adjudication: P1 covers plausible same-person name variants, P2 covers unverifiable non-academic sources, and P3 covers unsupported peripheral fields when core identity fields match.P3 applies only to volume, pages, publisher, or location and not to explicit contradictions or core-field mismatches.
  • Escalation rules: Author count or set mismatches always escalate to HALLUCINATED rather than being explained as P1.

D Per-Subtype TPR and FPR Heatmaps

Figure 4 visualizes subtype recall and false-positive rates across chatbot baselines and CiteTracer for PDF and BibTeX inputs, using shared heatmap encodings.

  • Figure layout: Figure 4 places methods on the vertical axis and nine scoring buckets on the horizontal axis, grouped into REAL, POTENTIAL, and HALLUCINATED classes.
  • Encodings: The left panel encodes in-bucket TPR, while the right panel encodes out-of-bucket FPR using a shared amber-to-green scale with inverted FPR coloring.Green indicates favorable values and amber unfavorable values; the FPR axis is capped at 20%.
  • Observed comparison: CiteTracer rows are uniformly deep green across TPR buckets, except REAL on PDF input at 90.8 due to Stage 1 extraction noise.
  • Observed comparison: Chatbot baselines show weak POTENTIAL recall and large REAL false-positive costs, while GPTZero is omitted because three output buckets are not applicable.Gemini reaches 41.5% and 46.3% REAL false-positive rates on PDF and BibTeX inputs, respectively.

E Limitations

The evaluation is concentrated on computer-science and machine-learning citations, while CiteTracer is positioned for author self-checking and editorial triage.

  • Scope boundary: The evaluation focuses on Computer Science papers, especially machine learning, limiting demonstrated coverage of other citation formats and fields.Less-standard formats, complex structures, or limited connector coverage may cause missed candidates and incorrect HALLUCINATED verdicts.
  • Operational limitation: High-concurrency verification can trigger API rate limits and drop candidate evidence from the eight Scholar Connectors.The paper proposes routing citations to the most appropriate connector as a future robustness improvement.
  • Broader impacts: CiteTracer is intended for authors' pre-submission self-checks and for conference chairs and journal editors conducting desk-review triage.
Loading 2605.08583v2…