PaperlayerView exact revision on arXiv

Source-linked AI summary

GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models

Zuyao Xu, Yuqi Qiu, Lu Sun, Fasheng Miao, Fubin Wu, Xiang Li, Xinyi Wang, Haozhe Lu, Zhengze Zhang, Yuxin Hu, Jialu Li, Luo Jin, Feng Zhang, Rui Luo, Xinran Liu, Yingxian Li, Jiaji Liu

arXiv:2602.06718v2cs.CRcs.AI

TL;DR

Evidence remains limited on how often LLMs fabricate citations, how prevalent invalid citations are in published work, and how they pass through peer review. GhostCite develops an open-source verification framework and studies these questions, finding every tested LLM hallucinated citations, with rates from 14.23% to 94.93%.

  • Problem

    The scale of citation fabrication, scalable ways to detect invalid citations, and how they pass through drafting and peer review remain insufficiently understood.

    Source: §I · p. 1

  • Method

    The paper develops CITEVERIFIER and combines a benchmark of citation hallucinations across 13 LLMs, an audit of published papers, and a researcher survey.

    Source: C. Research Questions · p. 3

  • Results

    2.6% in CV to 52.5% in OH for DeepSeek illustrates wide domain variation in citation-hallucination rates, with no model uniformly reliable.

    Source: B. Domain Sensitivity · p. 7

  • Takeaways & Limitations

    Ghost citations threaten academic trust because fabricated references can enter and persist in the literature amid sparse, largely trust-based verification practices.

    Source: §VIII · p. 12

    The pipeline verifies only title similarity, so it may undercount hallucinations resembling real papers and flag legitimate but poorly indexed works.

    Source: C. Limitations · p. 13

Abstract

from arXiv · show

Citations provide the basis for trusting scientific claims; when they are invalid or fabricated, this trust collapses. With the advent of Large Language Models (LLMs), this risk has intensified: LLMs are increasingly used for academic writing, but their tendency to fabricate citations (``ghost citations'') poses a systemic threat to citation validity. To quantify this threat, we develop \citeb, an open-source framework for large-scale citation verification, and conduct a comprehensive study of citation validity in the LLM era through three complementary experiments. First, we benchmark 13 LLMs on citation generation task in various research domains, finding that all models hallucinate citations at rate from 14.23\% to 94.93\%. Second, we analyze 2.2 million citations from 56,381 papers at AI/ML and Security venues (2020--2025), finding that 1.07\% of papers contain invalid citations, with an 80.9\% increase in 2025. Third, we survey 97 researchers, finding that 87.2\% use AI-powered tools in their workflows, 76.7\% of reviewers do not thoroughly check references, and 74.5\% view peer review as ineffective at catching citation errors. Based on these findings, we argue that ghost citations represent a systemic threat to academic integrity, and call for coordinated efforts from community to address this challenge.

I. INTRODUCTION · II. BACKGROUND AND RELATED WORK · A. Ghost Citations in the Wild

The paper frames LLM-generated “ghost citations” as a threat to scholarly trust and addresses gaps in scalable verification and empirical evidence through a verification framework and three complementary studies. It examines citation hallucinations in LLMs, invalid citations in published papers, and researcher verification practices.

II. BACKGROUND AND RELATED WORK

  • Invalid citations collapse the trust that citations provide,; LLMs can combine correlated names, titles, and venues into plausible fabricated references, termed “ghost citations”,.Such fabrications can misattribute foundational ideas, create phantom prior work, misdirect research investment, and undermine evidentiary chains.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • Awareness has prompted ICLR 2026 guidance on AI-generated content and IEEE disclosure policies, while reports document hallucinated citations in submissions, accepted papers, and domain-specific analyses –[11].The introduction argues that systematic measurement and coordinated responses remain necessary.Source: §I · pp. 1–2
  • CITEVERIFIER is an open-source framework for large-scale citation verification, developed to address unknown prevalence, lack of scalable detection, and weak understanding of how errors pass through review.The authors call for coordinated interventions by researchers, venues, and tool developers.Source: §I · pp. 1–2
  • Across 40 domains, 13 LLMs hallucinated citations at rates of 14.23%–94.93%, with 375,440 generated citations evaluated.The domains align with arXiv CS subject classes.Source: §I · pp. 1–2
  • 1.07% of 56,381 papers contained invalid citations; the rate rose 80.9% in 2025, from a 2020–2024 average of 0.89% to 1.61%.The analysis covered 2.2 million citations from eight AI/ML and Security venues; manual verification identified 739 invalid citations across 604 papers.Source: §I · p. 2
  • 76.7% of reviewers did not thoroughly inspect references, 74.5% viewed peer review as ineffective at catching citation errors, and 70.2% strongly supported automated checks.The survey received 94 valid responses from 97 researchers; 41.5% of authors admitted copy-pasting BibTeX entries without checking.Source: §I · p. 2

A. Ghost Citations in the Wild

  • Earlier citation errors included incorrect metadata and nonexistent sources, while copied citations could propagate errors; LLMs add systematic fabrication through autoregressive token prediction.Their ability to mimic citation syntax can make fabricated references difficult to distinguish from authentic ones.Source: A. Ghost Citations in the Wild · p. 2
  • Prior work includes hallucination benchmarks, [28], reported citation errors –[11], and watermarking or probability-curvature detection,, but these approaches do not establish citation accuracy at scale.The paper identifies gaps in prevalence measurement across models and published literature, and in understanding why verification behaviors fail.Source: A. Ghost Citations in the Wild · p. 2

B. The Risk to Scientific Research · C. Research Questions · III. AUTOMATED VERIFICATION FRAMEWORK

Ghost citations threaten research integrity by corrupting scholarship, peer review, and citation-based assessment. The paper poses three research questions and introduces CITEVERIFIER to support large-scale detection and empirical study.

B. The Risk to Scientific Research

  • Ghost citations misdirect researchers toward nonexistent literature, waste resources, and create “Epistemic Pollution” by giving fabricated claims an illusion of evidence.Source: B. The Risk to Scientific Research · p. 3
  • Plausible ghost citations exploit peer review’s trust gap, while checking 30–50 references per paper and manually reviewing 2,530 flagged citations took 16 researchers about a month.The authors argue that publishers and conference organizers lack automated tools for detecting fabricated citations, leaving reference verification to time-consuming manual review.Source: B. The Risk to Scientific Research · p. 3
  • Ghost citations distort citation graphs and impact metrics, may be reproduced by future LLMs trained on affected papers, and risk shifting scholarship from trust to costly skepticism.Source: B. The Risk to Scientific Research · p. 3

C. Research Questions

  • The paper frames three unresolved questions: scalable detection amid heterogeneous formats and incomplete indices, prevalence across published literature, and why verification fails before publication.Source: C. Research Questions · p. 3
  • CITEVERIFIER combines robust parsing, cascaded multi-source retrieval, and calibrated similarity matching, supporting a benchmark of 13 LLMs, an audit of 56,000+ papers, and a behavioral user study.Source: C. Research Questions · p. 3

III. AUTOMATED VERIFICATION FRAMEWORK

  • CITEVERIFIER was developed to identify invalid citations reliably at scale and enable subsequent analyses across large academic corpora.Source: §III · p. 3
  • Section III presents CITEVERIFIER’s design and implementation as an automated framework for robustly identifying invalid citations across large academic corpora.Source: §III · pp. 3–4

A. Framework Design … IV. EXPERIMENT DESIGN

CITEVERIFIER is a modular, three-stage framework designed to verify citations at scale despite noisy parsing and incomplete bibliographic coverage. It uses cascading verification and title similarity to classify citations, and the paper outlines three experiments to study hallucination, prevalence, and researcher practices.

A. Framework Design

  • The design addresses citation-format and PDF-extraction noise, uneven database coverage, and the throughput and cost demands of verifying millions of citations.Its modular cascade queries sources in order of cost and precision and uses similarity matching to tolerate minor variations while limiting false positives.Source: A. Framework Design · p. 4

B. Framework Implementation

  • CITEVERIFIER uses parsing, verification, and classification stages, with GROBID as its primary parser and a Qwen3-Flash [45] LLM reparser when initial parsing fails.The reparser extracts metadata via a JSON schema and reruns the verification chain.Source: B. Framework Implementation · p. 4
  • Verification checks a local cache, cascades through DBLP and Google Scholar data, falls back to general web search, then reparses and retries if no match is found.The cache reduces repeated searches and API costs; the fallbacks cover recent, non-indexed, and non-traditional academic content as well as parsing errors.Source: B. Framework Implementation · p. 4

C. Implementation Details

  • At θ = 0.9, only 0.4% of real-paper similarities and 50.3% of LLM-generated similarities are at or below threshold, supporting its use to filter low-similarity hallucinations.Real-paper similarities remain near 1.0; the small tail below 1.0 is largely attributable to OCR errors, hyphenation, and minor typos.Source: Figure 2 · p. 5, C. Implementation Details · p. 5

B. Framework Implementation

  • Citations are classified Valid or Invalid using the highest title-similarity score among retrieved candidates and an empirically calibrated threshold, targeting untraceable or nonexistent references.The method focuses on ghost citations rather than citation errors,.Source: B. Framework Implementation · p. 5

C. Implementation Details

  • The Python implementation uses asyncio and semaphore-based rate limiting, defaulting to 10 concurrent requests, and supports directory batch processing, incremental export, and CSV results.CSV output includes status, diagnosis, similarity scores, and matched metadata.Source: C. Implementation Details · p. 5

IV. EXPERIMENT DESIGN

  • The experiment design pairs an LLM benchmark on hallucination rates and accuracy factors with archival analysis of published-paper invalid citations and a user study of researchers’ practices and attitudes.The benchmark evaluates 13 models across major vendors (Table I).Source: §IV · p. 5, Table I · p. 5

A. Experiment I: LLM Benchmark … A. Overall Hallucination Rates

The paper evaluates citation generation with a controlled LLM benchmark, examines citations in published papers, and surveys researchers; in the benchmark, half of extracted citations were classified as invalid.

A. Experiment I: LLM Benchmark

  • The controlled benchmark tested 13 LLMs across 40 computer science domains, generating 375,440 citations in 22,800 interactions; its rates are a controlled baseline, not real-world prevalence.Prompts requested fixed-size blocks of domain-specific references, with batch sizes of 10, 20, or 30 and conditions with or without online search and chain-of-thought prompting.Source: A. Experiment I: LLM Benchmark · pp. 5–6
  • In an auxiliary validator experiment, each model classified 100 entries with known ground truth—50 valid and 50 invalid—one entry per prompt.The entries came from the archival analysis; the supplied passage describes the test design but not its results.Source: A. Experiment I: LLM Benchmark · p. 6

B. Experiment II: Archival Analysis

  • The archival analysis covered 56,381 accepted papers from eight AI/ML and Security venues from 2020–2025, flagging citations with similarity scores below θ = 0.9 for manual review.Sixteen trained research assistants reviewed flagged citations, each independently checked at least twice and classified as non-academic source, valid, or invalid.Source: B. Experiment II: Archival Analysis · p. 6
  • A manual review of 400 citations sampled from the valid pool found no additional invalid citations; researchers further divided invalid citations into error citations and ghost citations.Ghost citations were defined as untraceable after extensive manual search; two researchers reviewed all manually classified invalid citations.Source: B. Experiment II: Archival Analysis · p. 6

C. Experiment III: User Study

  • The researcher survey covered AI adoption, citation practices, and author- and reviewer-level verification; 3 of 97 responses were excluded as inconsistent, leaving 94 valid responses.The voluntary, anonymized survey received IRB approval and included paired reverse-worded items as a consistency check.Source: C. Experiment III: User Study · p. 6

V. MEASURING THE UNRELIABILITY: LLM BENCHMARK

  • 50.29% of 331,809 extracted citations were invalid, while 49.71% were valid; JSON format compliance was 90.58% across 22,800 interactions.Manual checks found 100% accuracy for the valid sample and 98% (392/400) for the invalid sample.Source: §V · p. 6

A. Overall Hallucination Rates

  • Hallucination rates ranged from 14.23% for DeepSeek to 94.93% for Hunyuan, a roughly 6.7× difference across models.Table II reports JSON format compliance and citation hallucination rates for 13 LLMs, sorted by hallucination rate.Source: A. Overall Hallucination Rates · p. 6, Table II · p. 7
  • Online search, chain-of-thought prompting, and batch size showed no consistent effect on hallucination rates across models.The authors interpret this as suggesting that hallucination reflects a limitation in bibliographic knowledge rather than a specific prompting strategy or output quantity.Source: A. Overall Hallucination Rates · p. 6

B. Domain Sensitivity · C. Patterns of Generated Citations · D. LLMs as Citation Judges

Citation hallucination varies across research domains and publication years, and generated citations differ in repeatability. LLMs also perform poorly when asked to validate citations.

B. Domain Sensitivity

  • Domain-average hallucination rates range from 28.80% in Computation and Language to 80.19% in Digital Libraries, a 51.39 percentage-point gap.Grok 4 reaches 100% hallucination in AI; manual checks found outputs mostly stitched from prompted example papers, suggesting prompt overfitting.Source: B. Domain Sensitivity · p. 7

D. LLMs as Citation Judges

  • LLM validators averaged 38% accuracy versus 50% random guessing; ERNIE reached 56% accuracy, with 100% precision but mislabeling 88% of valid entries as fabricated.ERNIE correctly identified all 50 invalid citations but misidentified 88% of valid citations as fabricated.Source: Table III · p. 7, D. LLMs as Citation Judges · pp. 7–8

C. Patterns of Generated Citations

D. LLMs as Citation Judges

  • The citation-validation test prompted 13 models to classify 100 entries—50 valid and 50 invalid—with online search and chain-of-thought enabled.Entries came from manually verified ground truth in the archival analysis.Source: D. LLMs as Citation Judges · p. 7

B. Domain Sensitivity

  • Across models, hallucination rates span 14.23–94.93% (a 6.7× gap), and no model is uniformly reliable: DeepSeek ranges from 2.6% in CV to 52.5% in OH.In Hardware Architecture, Grok 4 hallucinates at 99.2%, while DeepSeek’s rate is 8.6%; GPT-5 and Claude 4 reach 93.0% and 97.2% in Digital Libraries.Source: B. Domain Sensitivity · p. 7, Figure 3 · p. 8, D. LLMs as Citation Judges · p. 8

C. Patterns of Generated Citations

  • Hallucination rates rise from 27.61% for citations dated 2000 to 98.75% for 2025; an exponential fit models hallucination count with R2 = 0.94.Valid citations are relatively uniform from 2000 to 2014, rise through 2020, then decline, reflecting the models’ training-data cutoff.Source: C. Patterns of Generated Citations · p. 7, Figure 4 · p. 8

VI. HUNTING FOR GHOSTS: ARCHIVAL ANALYSIS · A. Detection of Invalid Citations

The archival analysis quantified invalid citations in published AI/ML and Security literature from 2020–2025. After manually reviewing citations flagged by CITEVERIFIER, the authors found that 1.07% of papers contained at least one invalid citation.

VI. HUNTING FOR GHOSTS: ARCHIVAL ANALYSIS

  • The corpus covered 56,381 papers from eight top-tier AI/ML and Security venues (2020–2025), yielding 2,199,409 citations and 2,530 potentially problematic matches.Potential problems were flagged when title similarity was below θ = 0.9, after filtering OCR extraction errors.Source: §VI · p. 8

A. Detection of Invalid Citations

  • Researchers manually reviewed all 2,530 flagged citations by searching bibliographic databases, checking metadata when found, and treating unlocated citations as untraceable.The review took approximately one month, and two experts subsequently reviewed all citations classified as invalid.Source: A. Detection of Invalid Citations · p. 8
  • The review classified flagged citations as non-academic sources, valid after manual matching, or invalid through metadata errors or untraceability.Source: A. Detection of Invalid Citations · p. 8
  • Of the flagged citations, 490 (19.4%) were non-academic sources and 1,301 (51.4%) were verified as valid through manual search.Source: A. Detection of Invalid Citations · pp. 8–9
  • 1.07% of the 56,381 papers contained invalid citations; 0.24% contained error citations and 0.86% contained ghost citations.Fifteen papers had both types.Source: A. Detection of Invalid Citations · p. 9
  • Manual search confirmed 739 citations (29.2%) as invalid, comprising 136 error citations and 603 ghost citations; 604 of 56,381 papers (1.07%) contained at least one invalid citation.Source: A. Detection of Invalid Citations · p. 9

B. Case Studies of Invalid Citations

Five case studies show invalid citations ranging from misleading metadata mismatches to untraceable fabricated or record-only ghosts, plus repeated errors that can propagate through citation tools and copying.

Case Study 1: Metadata Error Citation

  • The [Kenny and Keane, 2019] citation in an IJCAI 2021 paper had a significantly mismatched title, though manual searching found a work with matching authors and venue.The example shows the system can identify misleading title mismatches as well as entirely untraceable citations.Source: B. Case Studies of Invalid Citations · p. 9

Case Study 2: Untraceable Ghost Citation

  • The citation [3] for “Heimdallr” in a USENIX Security 2022 paper had no author information, and researchers found no such paper in the complete OSDI 2021 proceedings.Google Scholar and DBLP searches also yielded no results.Source: B. Case Studies of Invalid Citations · p. 9

Case Study 3: Ghost Citation with Clear Fabrication

  • The citation [4] for “Adaptive watermarking for source code protection” had no database results and generic names and a plausible-sounding journal title characteristic of LLM-style fabrication.It appeared in a NeurIPS 2025 paper; searches covered Google Scholar, DBLP, and major bibliographic databases.Source: B. Case Studies of Invalid Citations · p. 9

Case Study 4: Ghost Citation Only in Records

  • The [9] citation appeared only in Google Scholar records, with no paper found in major databases or journal archives and volume and page numbers inconsistent with publication records.The researchers hypothesize it may refer to an internal or restricted-access paper, but classify it as a ghost citation.Source: B. Case Studies of Invalid Citations · p. 9

Case Study 5: Repeated Invalid Citations

  • Repeated incorrect metadata for [Hendrycks et al., 2020] appeared in 16 AAAI, IJCAI, and NeurIPS papers and 220 Google Scholar citations.The actual title differs from the cited title; OpenReview’s cite button may have seeded the error, which later papers copied, including along a traced path from a top-tier paper.Source: B. Case Studies of Invalid Citations · pp. 9–10

C. Overall Distribution of Invalid Citations · D. Temporal Trends and Propagation

Invalid citations appear across venues and papers, with a sharp rise in 2025 and evidence that errors can recur across publications. The findings suggest possible AI-assisted generation and citation copying, but do not establish AI use as the cause.

C. Overall Distribution of Invalid Citations

D. Temporal Trends and Propagation

  • Invalid-citation rates stayed between 0.76% and 0.98% in 2020–2024, then reached 1.61% in 2025, an 80.9% increase over the 0.89% average.The rise coincides with wider adoption of autonomous AI-agent workflows, but the data do not establish causality.Source: Figure 5 · p. 10, D. Temporal Trends and Propagation · p. 10
  • The authors hypothesize three periods: pre-LLM errors from oversight or restricted-access literature, limited reference-generation impact in 2023–2024, and scale-enabling agentic workflows in 2025.This is a proposed explanation for the trend, not evidence that AI caused the observed increase.Source: D. Temporal Trends and Propagation · p. 10
  • AI/ML venues show a more pronounced increase in invalid-citation counts than Security venues, which the authors attribute tentatively to earlier and broader LLM-tool adoption.The proposed adoption explanation is qualified as likely, rather than established causation.Source: D. Temporal Trends and Propagation · p. 10
  • The same erroneous “AugMix” title appears in 16 papers across AAAI, IJCAI, and NeurIPS, illustrating repeated invalid citations across independent papers.Such repetition suggests researchers may copy erroneous citations from earlier papers, compounding mistakes.Source: D. Temporal Trends and Propagation · pp. 10–11, Table VI · p. 11

VII. ANALYZING HUMAN FACTORS: SURVEY FINDINGS · A. AI Adoption and Workflow Integration · B. Verification Practices

A survey of researchers found widespread AI use alongside weak citation verification: many authors and reviewers reported practices that leave citation errors and hallucinations unchecked.

VII. ANALYZING HUMAN FACTORS: SURVEY FINDINGS

  • The study received 97 survey responses and included paired reverse-worded BibTeX verification items to flag inconsistent responses.Figure 6’s demographic breakdown covers 94 respondents.Source: §VII · p. 11

A. AI Adoption and Workflow Integration

  • 87.2% of respondents who answered the AI-use question (75/86) use AI-powered tools for research, across career stages and research areas.Among AI users, 46.7% use AI for specific difficult paragraphs, 29.3% for almost every sentence, 22.7% only for final proofreading, and 1.3% not for polishing.Source: A. AI Adoption and Workflow Integration · p. 11
  • 72.6% of respondents rely on Google Scholar’s “Cite” button for references, while 17.9% export directly from publishers and 4.8% copy BibTeX from other papers.The paper notes that reliance on reference sources can expose researchers to propagation of upstream metadata errors.Source: A. AI Adoption and Workflow Integration · p. 11
  • Among AI users, 46.7% use AI for specific difficult paragraphs, 29.3% for almost every sentence, and 22.7% only for final proofreading.Only 1.3% (1/75) report not using AI for polishing.Source: A. AI Adoption and Workflow Integration · p. 11

B. Verification Practices

  • 76.7% of reviewers do not thoroughly check references, and 74.5% rate peer review as not very effective or ineffective at detecting citation errors.Additionally, 80.0% of reviewers had never suspected fake or hallucinated references in a submission.Source: B. Verification Practices · p. 11
  • 41.5% of authors copy-paste BibTeX without checking, while 17.3% cite AI-suggested papers without reading them.Among AI users, 41.3% encounter hallucinated citations often or very often.Source: B. Verification Practices · p. 11
  • 44.4% choose no-action options when encountering suspicious references, either verifying privately or ignoring them.Source: B. Verification Practices · p. 11

C. Perception of Severity and Responsibility · VIII. DISCUSSION AND MITIGATION · A. The Lifecycle of Ghost Citations

The authors frame ghost citations as a systemic threat sustained by sparse, trust-based verification, trace their propagation through a four-stage pipeline, and call for coordinated intervention. Survey respondents recognize the problem’s severity, yet checking is limited and responsibility is concentrated on authors.

C. Perception of Severity and Responsibility

  • 76.6% of respondents consider hallucinated citations a major problem or critical crisis, including 44.7% who call it a critical crisis.The remaining 31.9% rate it as a major problem.Source: C. Perception of Severity and… · p. 12
  • 87.2% of researchers use AI-powered tools, while 41.5% copy-paste BibTeX without checking.Researchers often trust citations by default rather than independently verifying them.Source: C. Perception of Severity and… · p. 12
  • 76.7% of reviewers do not thoroughly inspect references, and 74.5% view peer review as ineffective at catching citation errors.Source: C. Perception of Severity and… · p. 12

VIII. DISCUSSION AND MITIGATION

  • Ghost citations threaten academic trust because sparse, trust-based verification lets fabricated references enter and persist in the literature.The authors propose targeted interventions across the citation lifecycle.Source: §VIII · p. 12

A. The Lifecycle of Ghost Citations

  • LLMs can produce structurally realistic nonexistent references, which time-pressed or unfamiliar researchers may copy after superficial checks, mistaking coherent formatting for accuracy.Participants using AI assistance inserted invalid citations at significantly higher rates than those who did not.Source: A. The Lifecycle of Ghost Citations · p. 12
  • Reviewers lack time and tools to verify every reference, while good-faith assumptions and submission systems without automatic flags let ghost citations pass undetected.Source: A. The Lifecycle of Ghost Citations · p. 12
  • After publication, ghost references enter databases and recur in up to 16 papers; repeated appearance makes errors seem credible and increasingly difficult to detect or correct.The authors call for coordinated action at multiple pipeline stages before contamination becomes intractable.Source: A. The Lifecycle of Ghost Citations · pp. 12–13

B. Mitigation and Recommendations … ETHICAL CONSIDERATIONS

The paper recommends coordinated, multi-stakeholder action to reduce ghost citations, while qualifying its findings with methodological limitations. It concludes by urging the community to prioritize citation integrity and describes safeguards used to conduct the study responsibly.

B. Mitigation and Recommendations

  • Researchers should treat AI outputs as unverified, prefer retrieval-grounded tools, verify citation metadata, and read at least an abstract or source-based summary before citing.The authors flag missing DOIs and inconsistent metadata, and caution against unverified BibTeX entries and AI-only summaries.Source: B. Mitigation and Recommendations · p. 13
  • Conference and journal organizers should automate citation checks, require structured reference metadata and author verification attestations, and give reviewers compact citation-risk summaries.The survey found 70.2% strong support for automated verification measures.Source: B. Mitigation and Recommendations · p. 13
  • AI tool developers should ground citations in verified retrieval, distinguish retrieved from generated content, provide evidence fields, and signal when a source cannot be found.The authors also recommend prompting users to verify outputs before finalizing them.Source: B. Mitigation and Recommendations · p. 13
  • The research community should coordinate shared verification infrastructure, detection research, regular measurement, open APIs, benchmark datasets, and standardized reporting to address ghost citations.The authors warn that fragmented efforts may leave gaps exploitable by hallucination and complacency.Source: B. Mitigation and Recommendations · p. 13

C. Limitations

  • The study’s limitations make its findings controlled or conservative estimates, rather than definitive measures of real-world citation hallucination and behavior.Title-similarity verification may miss fabricated citations resembling real papers and may flag poorly indexed legitimate work; manual double-checking mitigates but does not eliminate misclassification. The benchmark’s fixed-number prompts differ from drafting use, while self-reported survey rates (86.7% verification and 41.5% risky practices) may reflect social-desirability bias; the authors interpret these as perceived norms.Source: C. Limitations · p. 13

IX. CONCLUSION

  • The authors present CITEVERIFIER as an open-source measurement baseline and call for immediate efforts to safeguard the scholarly record from ghost citations.They frame citation validity as a systemic threat that flows from generation through adoption, review failure, and propagation.Source: §IX · p. 13

ETHICAL CONSIDERATIONS

  • The study reports safeguards for external-service use, paper access, and survey privacy, including rate limits, authorized access, IRB approval, and de-identified response storage.External services were used under their terms and without harmful prompts, following the Menlo report. Papers were used only for research and not redistributed; participants received study details and an opt-out option, and responses were not shared outside the research.Source: ETHICAL CONSIDERATIONS · p. 14
  • The authors withheld identities of papers with invalid citations to focus on systemic patterns rather than blame, noting that invalid references do not establish intent.They cite meta-science precedent and acknowledge that causality and attribution are complex, while inviting reviewer feedback on balancing transparency and proportionality.Source: ETHICAL CONSIDERATIONS · p. 14

APPENDIX … E. Domain-Level Hallucination Rates

The appendix provides supplementary materials on the study’s domain coverage, prompting, and analyses of factors affecting citation hallucination rates. It reports domain-level results across 40 computer-science subfields and examines API-level prompting settings and batch size.

APPENDIX

  • The appendix provides additional study details omitted from the main text due to space constraints.Source: APPENDIX · p. 15

A. Research Domain Codes

B. LLM Citation Generation Prompt

  • The citation-generation prompt requests exactly the specified number of realistic, credible references related to a representative paper, returned as JSON.It requests reference metadata including authors, title, venue, year, URL, DOI, and reference type.Source: B. LLM Citation Generation Prompt · pp. 15–16

C. LLM Citation Validity Judgment Prompt

  • The auxiliary validity-judgment prompt asks whether a bibliographic entry’s title, authors, year, and venue accurately match a real publication, with a true-or-false JSON judgment and reason.Source: C. LLM Citation Validity Judgment Prompt · p. 16

D. Factors Affecting Hallucination Rates

  • Batch sizes of 10, 20, and 30 citations per prompt show no consistent correlation with hallucination rates.The authors suggest hallucination reflects limits in models’ bibliographic knowledge rather than output-quantity constraints. Online-versus-offline and thinking comparisons use API-level toggles, not each vendor’s full retrieval or reasoning stack.Source: D. Factors Affecting Hallucination Rates · p. 16, Table VIII · p. 16

E. Domain-Level Hallucination Rates

  • Citation hallucination rates are reported for 40 computer-science subfields, aggregated across all 13 models and ranked from highest to lowest.The passage describes Table X but does not provide its domain-level values.Source: E. Domain-Level Hallucination Rates · p. 16

F. Citation Stability and Cross-Model Overlap · G. Survey Response Details (Full)

Valid citations were more reproducible across repeated prompts than hallucinated citations, while the survey appendix supplies the complete response breakdown and question map. The appendix also documents the consistency check used to exclude three respondents.

F. Citation Stability and Cross-Model Overlap

  • Citation stability measures whether a model generates identical citations across repeated runs for each model-and-configuration pair.A configuration combines research topic, requested reference count, chain-of-thought setting, and online-search setting; scores range from 0 for maximal variability to 1 for perfect consistency.Source: F. Citation Stability and Cross-Model… · p. 17
  • Valid citations reached mean stability scores of 0.58 for DeepSeek and 0.57 for Qwen-3, compared with up to 0.23 for hallucinated citations.The paper suggests that frequently encoded real papers are recalled repeatedly, whereas fabricated references are generated more arbitrarily.Source: F. Citation Stability and Cross-Model… · p. 17
  • Across 40 research domains, the most frequently generated valid citations include NeRF in Graphics, RAG in Computation & Language, and U-Net in Computer Vision.These are described as references LLMs consistently recall, suggesting strong representation in parametric memory.Source: F. Citation Stability and Cross-Model… · p. 17, Table XI · p. 19

G. Survey Response Details (Full)

Related papers

Loading 2602.06718v2…