Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences

1. Introduction

• Because technical claims can require expert judgment, the paper treats references as a more auditable surface for measuring citation failures at scale.LLM-assisted scientific writing can produce polished bibliographies whose entries do not identify the works they purport to cite.Source: §1 · p. 1
• The audit counts non-existent works and substantial author-list mismatches, while excluding ordinary venue, year, publication-status, and minor-name variation.It does not establish whether surrounding text was LLM-generated or whether cited results are true.Source: §1 · pp. 1–2
• RefChecker checks references against multiple bibliographic sources and sends unresolved or suspicious cases to LLM-driven deep web search.The audit covers accepted camera-ready papers from top AI and security venues.Source: §1 · p. 2
• In 2025, roughly one in twenty accepted NeurIPS and USENIX Security papers contained at least two likely hallucinated academic-paper-like references, with some papers reaching 5+ failures.Likely hallucinated citations also appeared in award-winning papers.Source: §1 · p. 2
• After ChatGPT's public release, likely hallucinated-citation rates rose in several venue families, though timing alone does not establish that LLM-assisted writing caused the increase.The series is not perfectly monotonic.Source: §1 · p. 2
• A scan of 3,703 accepted papers and 221,281 references cost approximately $157, or roughly four cents per paper.RefChecker is open-sourced for reproducible analysis and extension.Source: §1 · p. 2
• The authors report findings in aggregate rather than identifying papers, authors, or institutions, citing the risk of disproportionate reputational harm when a failure may reflect workflow breakdown rather than misconduct.They frame the work as exposing a failure mode that current publication workflows are not designed to catch.Source: §1 · p. 2

2. Related Work

• Figure 1 summarizes visible paper-level prevalence under the conservative definition and a higher-count tail in the post-ChatGPT period.The caption describes accepted papers with multiple likely hallucinated references in one bibliography.Source: Figure 1 · p. 3
• ICLR 2026 and ICML 2026 policies identify hallucinated references as potential desk-rejection grounds, with ICLR emphasizing concrete evidence over detector scores,.These policies frame citation hallucination as a publication-integrity concern.Source: §2 · p. 3
• Controlled-prompting studies measure fabricated references from models, whereas this paper measures what survives peer review into the archival record.Reported rates vary by model and domain in Naser [17]; retrieval pairing improves BibTeX generation in Rao and Callison-Burch [8].Source: §2 · p. 3
• Citation-verification infrastructure includes CiteAudit [19] and GhostCite's CiteVerifier [13], which also resolve references against authoritative databases before escalating suspicious cases.CiteAudit [19] pairs a benchmark with a multi-agent extraction, retrieval, and judging pipeline.Source: §2 · p. 3
• Broader work studies hallucination benchmarks, attribution, grounded citations, and automated fact verification,,, –.The paper positions its audit alongside Zhao et al. and GhostCite [13], emphasizing camera-ready proceedings and paper-level structure.Source: §2 · p. 3
• The closest audits include Zhao et al. and GhostCite [13], but their work differs in corpus, candidate selection, paper-level analysis, and reported cost.Zhao et al. focus mainly on preprints; GhostCite [13] covers AI/ML and security venues from 2020–2025 and uses less conservative candidate selection.Source: §2 · pp. 2–3

3. Background and Definitions

• A citation serves both to credit prior work and to give readers enough structured information to recover the cited artifact from the public record.The paper focuses on the pointer role because it makes a bibliography auditable.Source: §3 · p. 4
• A hallucinated citation fails to identify the claimed work through either a nonexistent work or a substantial mismatch in the author list.Minor name variants, spelling mistakes, and venue, year, or publication-status changes are not counted unless author evidence substantially diverges.Source: §3 · p. 4
• Checking whether a work exists and its authors match the citation is more tractable than judging whether a technical claim is hallucinated.The study asks whether the bibliography points to a real work by the named authors, not whether the cited claim is true.Source: §3 · p. 4

• Each bibliography entry is treated as a claim that should resolve to a compatible source record, and Figure 2 maps the pipeline from parsing through six final statuses.Source: §4 · p. 4, Figure 2 · p. 5
• The corpus includes accepted ICLR, ICML, NeurIPS, and USENIX Security papers with retrievable camera-ready PDFs from OpenReview or official proceedings, excluding unavailable PDFs and author-hosted substitutes.Each analyzed manuscript is tied to a venue-controlled or platform-controlled publication record.Source: §4 · p. 4
• RefChecker extracts bibliography fields from camera-ready PDFs, preserves original reference strings for auditing, and normalizes metadata to absorb harmless formatting variation.LLM-assisted parsing or GROBID fallback can be used for noisy bibliographies.Source: §4 · p. 4
• Multiple bibliographic sources and identifiers are compared; unresolved or suspicious references are escalated after deterministic checks, then candidate-source metadata is re-verified before a likely-hallucination label.Minor year or venue variation alone does not trigger escalation.Source: §4 · pp. 4–5
• The audit separates final statuses, evidence issue values, and deep-check hallucination verdicts; uncertain or merely unverified references are not automatically counted as hallucinations.Source: §4 · p. 5
• LLMs assist with noisy-bibliography parsing and targeted searches, but external evidence and metadata re-verification constrain final citation-validity decisions.The pipeline uses LLM search as an escalation and evidence-gathering step, not as a direct labeler.Source: §4 · p. 5
• PDF extraction noise, incomplete or stale records, and configured LLM behavior can yield false positives and negatives, so measured rates are estimates rather than exact counts.Source: §4 · p. 5

5. Evaluation

• The audit covers 48,095 accepted papers and 2,614,992 references, counting only missing works or substantial author-list mismatches, not ordinary bibliographic drift.The strict definition makes the measured rates a lower bound on broader citation noise.Source: §5 · p. 5

5.1. Metrics

• Metrics use all-reference and academic-paper-like denominators, paper thresholds of one and two failures, high-count tails, and identity-level failure modes.OpenReview-covered papers are also joined to reviewer ratings, acceptance tiers, and research areas.Source: §5.1 · p. 6

5.2. Findings

• Reference-level rates of likely hallucinated references stayed below 1% in essentially every venue-year, yet roughly one in four accepted NeurIPS 2025 papers and one in three accepted USENIX Security 2025 papers contained at least one unreconciled reference.Source: §5.2 · pp. 6–7
• In 2025, 5.1% of NeurIPS and 4.8% of USENIX Security papers had at least two likely hallucinated academic-paper-like references.The all-reference shares were 6.9% and 12.1%, respectively.Source: Figure 4 · p. 7, §5.2 · p. 7
• Rates rise in the post-ChatGPT window in three of four venue families; the ICLR 2026 point is the highest ICLR rate observed, at 3.1% for all references and 2.4% for academic-paper-like references.Source: §5.2 · p. 7
• Most affected 2025 papers had exactly one flagged reference, yet rare papers with five or more account for a disproportionate share of failures; the NeurIPS maximum was 20.Across the corpus, single-paper maxima were 13 at both ICML and USENIX Security and 8 at ICLR.Source: §5.2 · pp. 7–8, Figure 6 · p. 9, Figure 7 · p. 9
• Across 14 research areas, affected-paper prevalence ranged from 15.5% to 25.7%, with no area exempt from the failure mode.LLM and foundation-model papers were slightly less affected, possibly due to more careful checking or narrower tooling.Source: §5.2 · p. 8
• Affected papers received nearly the same mean reviewer ratings as clean papers, and prevalence was similar across poster, spotlight, and oral tiers.At ICLR 2023, rejected papers were also affected at 16.9%, close to 16.0% for accepted papers despite lower mean ratings.Source: §5.2 · pp. 8–9, Figure 9 · p. 10, Figure 10 · p. 10

• Hallucinated citations have entered the archival record, and peer review does not reliably remove them.Source: §6 · p. 10
• Most manually examined flags were recoverable errors; retained hallucinations instead identified no work or bound a real title to unrelated authors.Examples include nonexistent works and real titles with unrelated author lists.Source: §6 · pp. 10–11
• Submission-stage verification can surface a small set of suspicious references with evidence for human confirmation, leaving final judgment to people.Source: §6 · pp. 11–12
• Authors contacted about papers with at least five likely hallucinated references described workflow errors, including unverified LLM-generated entries, rather than deliberate fabrication.Source: §6 · p. 12
• Auditing 3,703 accepted papers and 221,281 references cost roughly four cents per paper, while human adjudication remains the dominant cost.Source: §6 · p. 12
• A continuously updated canonical citation service could improve verification precision and recall while helping authors retrieve correct entries at the source.Source: §6 · p. 12

• Most examined flags were recoverable references, so an individual flag is not conclusive without review.The authors treat counts of at least two, and especially five or more, as more reliable signals than a single flag.Source: §7 · p. 12
• Results and costs depend on the configured LLM, whose capabilities and behavior can change over time.Source: §7 · p. 12
• PDF extraction errors can make real references appear malformed; source bibliographies such as .bib files can avoid this noisy pipeline stage.Source: §7 · p. 12
• Unpublished, newly accepted, or obscure works may be flagged because bibliographic records do not yet contain them, requiring human review and author justification.Source: §7 · p. 13
• RefChecker verifies a cited work’s existence and authorship, not whether it supports the citing sentence or whether the paper’s scientific claims are correct.Source: §7 · p. 13
• Flags should inform an evidence-backed human process that lets authors justify or correct references before punitive action such as desk rejection.Source: §7 · p. 13

• Roughly one quarter of accepted NeurIPS 2025 papers had an unverifiable citation, despite reference-level rates being well under one percent in nearly every venue-year.Source: §8 · p. 13
• Distinguishing fabrication from copy-and-paste errors, misspellings, or missing index records remains difficult, so automated flags need human adjudication.Source: §8 · p. 13
• RefChecker resolves references against bibliographic records, escalates difficult cases, and gives reviewers evidence; the tool costs cents per paper and is open source.Source: §8 · p. 13