PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM hallucinations in the wild: Large-scale evidence from non-existent citations

Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, Yian Yin

arXiv:2605.07723v1cs.DLcs.AIcs.CYphysics.soc-ph

TL;DR

Evidence on real-world hallucinations has been limited, although scientific citations provide a traceable object for large-scale study. The paper audits citation records across four corpora and finds rising, diffusely distributed hallucinated citations; existing checks allow many to enter and persist in the scholarly record.

  • Problem

    The real-world prevalence of hallucinations remains poorly understood, and measuring them at scale is difficult because many forms lack a clear, scalable ground truth.

    Source: Discussion · p. 9

  • Method

    The authors use a corpus-level reference-verification pipeline across arXiv, bioRxiv, SSRN, and PubMed Central to estimate prevalence and examine who produces and benefits from hallucinated citations.

    Source: Discussion · p. 9

  • Results

    Hallucinated citations rose across the studied corpora, appeared sparsely across many papers, and most non-existent citations passed arXiv moderation.

    Source: The rise and distribution of… · pp. 4–5, Systemic responses and failure modes · p. 8

  • Takeaways & Limitations

    Hallucinated citations can enter the scholarly record, persist through publication, and compound through citation networks and model-training loops.

    Source: Discussion · pp. 9–10

    The analysis focuses on detectable hallucinated titles; real citations used to support claims they do not make remain harder to detect, and the pipeline may miss niche venues or heavily mathematical documents.

    Source: Discussion · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are known to generate plausible but false information across a wide range of contexts, yet the real-world magnitude and consequences of this hallucination problem remain poorly understood. Here we leverage a uniquely verifiable object - scientific citations - to audit 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central. We find a sharp rise in non-existent references following widespread LLM adoption, with a conservative estimate of 146,932 hallucinated citations in 2025 alone. These errors are diffusely embedded across many papers but especially pronounced in fields with rapid AI uptake, in manuscripts with linguistic signatures of AI-assisted writing, and among small and early-career author teams. At the same time, hallucinated references disproportionately assign credit to already prominent and male scholars, suggesting that LLM-generated errors may reinforce existing inequities in scientific recognition. Preprint moderation and journal publication processes capture only a fraction of these errors, suggesting that the spread of hallucinated content has outpaced existing safeguards. Together, these findings demonstrate that LLM hallucinations are infiltrating knowledge production at scale, threatening both the reliability and equity of future scientific discovery as human and AI systems draw on the existing literature.

Estimating hallucinated references at scale

The study verifies whether reference titles exist by matching parsed citations against scholarly indexes and checking remaining unmatched entries. Across four large bibliometric datasets, unmatched rates were stable through late 2022 and rose sharply in 2023; the excess over the pre-LLM baseline is treated as the population-level signature of hallucination.

Verification pipeline

  • The verification pipeline matches parsed reference titles against an Elasticsearch index, then cleans and refines unmatched entries before a final Google Scholar lookup.The initial pipeline matches 95.1% of references; cleaning and title refinement reduce the unmatched rate to 1.54% before the additional lookup.Source: Estimating hallucinated references at… · pp. 3–4
  • The final unmatched pool comprises references still unverifiable after checking Google Scholar for a matching record or evidence that the same-titled work was cited elsewhere.The approach focuses on non-existent titles, a specific type of hallucinated reference.Source: Estimating hallucinated references at… · pp. 3–4

Datasets

  • The four datasets cover arXiv, bioRxiv, SSRN, and a 10% random sample of recent PubMed Central papers, totaling more than 111 million citations.They differ in scientific fields, publication formats, and citation-parsing techniques.Source: Estimating hallucinated references at… · p. 4

Baseline and trend

  • Unmatched-reference rates stayed relatively stable through late 2022, then increased sharply in 2023; the study treats post-baseline excess as estimated hallucinations, not individually classified cases.Manual inspection attributes baseline unmatched cases to edge cases such as formatting artifacts or works absent from mainstream citation databases.Source: Estimating hallucinated references at… · p. 4

The rise and distribution of hallucinated citations

Estimated hallucinated citations rose across four datasets, with growth accelerating from mid-2024 and a projected 146,932 citations in 2025. The increase is distributed across many papers, with higher rates in social science and computer science and an association with estimated LLM use in arXiv writing.

The rise and distribution of hallucinated citations

  • By August 2025, estimated hallucination rates reached 0.39% for arXiv, 0.21% for bioRxiv, 1.91% for SSRN, and 0.27% for PMC.The steepest rise began in mid-2024, roughly 18 months after ChatGPT's late-2022 release.Source: The rise and distribution of… · p. 4
  • 146,932 hallucinated citations are estimated across the four corpora in 2025, assuming the observed trends persisted through year end.The August 2025 monthly estimates were 3,353 for arXiv, 478 for bioRxiv, 767 for SSRN, and 8,140 for PMC; the authors describe the estimate as a likely lower bound.Source: The rise and distribution of… · p. 5
  • Papers with at least one hallucinated reference increased by 2–23 percentage points across datasets, while the share with at least 50% unmatched references rose much more slowly.The authors interpret this distribution as sparse contamination across many seemingly legitimate papers rather than a few heavily contaminated manuscripts.Source: The rise and distribution of… · p. 5
  • Hallucination rates were markedly higher in social science and computer science, fields where LLM-assisted writing is reported to be particularly prevalent.The authors describe adoption of some unvetted LLM citation suggestions as a plausible explanation, not a demonstrated causal mechanism.Source: The rise and distribution of… · p. 5
  • In arXiv, estimated LLM use in writing correlated with hallucination rates at the subfield level (Pearson r = 0.441, P < 0.001), and the relationship also held at the paper level.The authors state that these results are consistent with LLM use being a major driver of the rise.Source: The rise and distribution of… · pp. 5–6

Hallucination citers and their beneficiaries

The analysis compares papers containing unmatched citations with matched controls and models unmatched citations as a mixture of LLM hallucinations and conventional matching failures. Hallucination citers are disproportionately less experienced, while the citations tend to credit prominent scholars and deviate from conventional authorship patterns.

Hallucination citers and their beneficiaries

  • Hallucination citers had 62.1%–73.2% fewer prior publications in arXiv, bioRxiv, and SSRN, and 27.4% fewer in PMC than matched controls.The share with zero prior publications was 32.1% in arXiv, 9.3% in bioRxiv, 51.0% in SSRN, and 3.7% in PMC.Source: Hallucination citers and their… · p. 6
  • By 2025, the productivity gap had closed; relative productivity increases among hallucination citers versus matched authors ranged from 1.33× in PMC to 3.13× in SSRN.Hallucination rates also attenuated sharply with team size.Source: Hallucination citers and their… · p. 6
  • Authors named in hallucinated citations were 22.2% less likely to match an existing publication profile, indicating that such titles often co-occur with invented author names.Author profiles were linked using text-based, paper-level embeddings and restricted to cases close to the cited author's existing body of work.Source: Hallucination citers and their… · p. 7
  • When hallucinated titles matched real author profiles, they credited scholars with 68.8% more publications and 58.3% more citations than matched real citations; male names were overrepresented by 6.4 percentage points.The authors characterize the gender result as suggestive evidence of reinforced disparity.Source: Hallucination citers and their… · p. 7
  • Hallucinated references tended to credit smaller teams, and the first–last author productivity or impact hierarchy was 12 percentage points weaker than conventional patterns.The authors interpret these deviations as differing from the typical social organization of scientific authorship.Source: Hallucination citers and their… · p. 7
  • Even valid references in hallucination-containing papers skewed toward more productive and impactful authors than references in closely matched control papers.Controls were matched on year, month, field, subfield, and reference-list length; the authors suggest visible hallucinations may be only part of a broader reallocation of credit.Source: Hallucination citers and their… · p. 7

Systemic responses and failure modes

Existing safeguards detect some hallucination-containing manuscripts but fail to prevent most unmatched citations from entering or remaining in the scholarly record. Such citations appear across the journal-impact spectrum and are also accumulating in bibliographic databases.

Systemic responses and failure modes

  • 78.8% of non-existent citations were estimated to pass arXiv moderation and appear on the platform, despite rejected manuscripts reaching a 2.2% hallucination rate by August 2025.The 2.2% rate was 4.5 times that of accepted manuscripts; moderation rejected a disproportionate share but was not designed to systematically audit references.Source: Systemic responses and failure modes · p. 8
  • 85.3% of hallucinations present in bioRxiv preprints persisted into their published versions in PubMed Central.The analysis traced 2,241 bioRxiv preprints containing unmatched references to published versions.Source: Systemic responses and failure modes · p. 8
  • Hallucinations appeared across the journal-impact spectrum: the lowest and highest impact deciles had higher and lower rates, respectively, but there was no consistent gradient between them.Many relatively high-impact journals remained susceptible.Source: Systemic responses and failure modes · p. 8
  • A growing number of unmatched references already appear as entries in Google Scholar and in other papers, a pattern replicated across all four corpora.The authors warn that accumulation in bibliometric databases may erode mechanisms for detecting hallucinations.Source: Systemic responses and failure modes · p. 8

Discussion

A cross-disciplinary audit finds citation hallucinations rising sharply from mid-2024, with diffuse contamination and unequal credit allocation. Existing safeguards leave substantial leakage, while the paper identifies both limits to citation verification and risks beyond science.

Discussion

  • More than 146,932 hallucinated references were conservatively estimated in 2025, with citation hallucinations rising from mid-2024 through late 2025 without signs of plateauing.The audit covers arXiv, bioRxiv, SSRN, and PubMed Central.Source: Discussion · p. 9
  • Hallucinated references are diffusely distributed across papers and disproportionately channel credit toward already prominent, predominantly male scholars.Small-team papers have higher hallucination rates than larger-team papers.Source: Discussion · p. 9, Figure 1 · p. 11, Figure 2 · p. 12
  • Once published, hallucinated citations may compound as they enter bibliographic databases and are learned by future language models.The paper identifies citation propagation and model training on shared open-access corpora as two possible channels.Source: Discussion · p. 9
  • Citation validators may reduce structured-reference errors if adopted, but pipeline coverage is limited and detecting real sources misused to support claims remains harder.The pipeline may miss niche venues and heavily mathematical documents; the harder detection problem remains under development.Source: Discussion · pp. 9–10
  • Because science has unusually rich verification infrastructure yet still admits persistent hallucinations, the paper argues that less structured knowledge domains may face a broader, harder-to-detect problem.The authors point to government, legal, clinical, corporate, and journalistic knowledge work as domains without comparable verification infrastructure.Source: Discussion · p. 10
  • Moderation and journal impact do not prevent substantial leakage: most hallucinated references pass arXiv moderation, and rates vary without a clear gradient across most journal-impact levels.Unmatched references appearing as Google Scholar entries have also increased steeply since 2024.Source: Figure 3 · p. 13

Related papers

Loading 2605.07723v1…