Source-linked AI summary
HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences
Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
TL;DR
Hallucinated references threaten scientific reliability, yet their prevalence and impact have been insufficiently characterized. The paper systematically analyzes ACL, NAACL, and EMNLP papers using OCR, database matching, and manual verification, finding nearly 300 affected papers, with a large concentration at EMNLP 2025. It argues for preventive author checks rather than immediate penalties while noting limitations in peer-review transparency and domain scope.
Problem
Hallucinated references threaten scientific reliability and may reduce the quality and credibility of published conferences and journals.
Method
The study analyzes ACL, NAACL, and EMNLP papers using OCR-based citation extraction, database matching, and manual verification of candidates.
Results
Nearly 300 papers contain at least one HalluCitation, with more than half appearing at EMNLP 2025 and over 100 in its Main and Findings tracks.
Takeaways & Limitations
The paper recommends author toolkits and pre-submission checks rather than immediate penalties for papers containing HalluCitations.
Takeaways & Limitations
The analysis is limited by restricted peer-review transparency and focuses on ACL conferences rather than broader research domains.
Abstract
from arXiv · showhide
Recently, we have often observed hallucinated citations or references that do not correspond to any existing work in papers under review, preprints, or published papers. Such hallucinated citations pose a serious concern to scientific reliability. When they appear in accepted papers, they may also negatively affect the credibility of conferences. In this study, we refer to hallucinated citations as "HalluCitation" and systematically investigate their prevalence and impact. We analyze all papers published at ACL, NAACL, and EMNLP in 2024 and 2025, including main conference, Findings, and workshop papers. Our analysis reveals that nearly 300 papers contain at least one HalluCitation, most of which were published in 2025. Notably, half of these papers were identified at EMNLP 2025, the most recent conference, indicating that this issue is rapidly increasing. Moreover, more than 100 such papers were accepted as main conference and Findings papers at EMNLP 2025, affecting the credibility.
1 Introduction
The paper examines hallucinated references, termed HalluCitations, as a growing threat to citation accuracy and scholarly credibility. An analysis of over 17,000 ACL, NAACL, and EMNLP papers identifies nearly 300 affected papers, while proposing detection support and cautioning against immediate author penalties.
- Motivation: AI-supported research and rapid submission growth have made rigorous checking of manuscript factual correctness more difficult.Reviewers increasingly evaluate unfamiliar topics under strict time constraints and heavy workloads.
- Motivation: HalluCitations are non-existent references that may threaten scientific reliability and undermine the credibility of published venues.The paper uses “HalluCitation” for hallucinated references and “HalluCited” for papers containing at least one.
- Scope: Over 17,000 papers from ACL, NAACL, and EMNLP in 2024 and 2025 were analyzed, covering main conference, Findings, and workshop papers.The dataset spans papers published across the three conferences and two years.
- Findings: Almost 300 papers contain at least one HalluCitation, mostly published in 2025; half were identified at EMNLP 2025, including over 100 main and Findings papers.The findings indicate a rapidly increasing issue affecting conference credibility.
- Detection: The proposed detection method combines OCR and database matching, with papers having four or more detected candidates showing a high incidence of HalluCitations.This threshold is presented as a practical guideline for automated detection toolkits.
- Implications: HalluCitations cannot be immediately attributed to AI-generated content, so the paper favors prior preventive measures such as author toolkits and pre-submission checks.Secondary sources may also introduce citation errors, making their causes more complex.
2 Analysis Methods
The analysis screens more than 17,000 ACL-community papers through citation extraction, database matching, and manual verification. A conservative workflow produces HalluCitation candidates while acknowledging that reported counts are lower bounds and that candidate rates increased over time.
- 2.1 Data Collection: The study collects over 17,000 papers from ACL, NAACL, and EMNLP in 2024 and 2025, including main, Findings, and workshop papers.The collected PDFs are archival papers registered in the ACL Anthology, excluding tutorials and proceedings.
- 2.2 Citation Extraction: Citation extraction combines MinerU OCR for reference blocks with GROBID parsing and normalization to obtain structured bibliographic information.The workflow explicitly supplies reference blocks to GROBID to mitigate under-coverage across multi-page reference lists.
- 2.3 Candidate Creation: The pipeline narrows citations using ACL or arXiv keywords, searches ACL Anthology and arXiv metadata, and additionally uses DBLP and OpenAlex.This heuristic screening focuses on likely target citations while retaining possible citations outside ACL venues or arXiv.
- 2.3 Candidate Creation: A citation becomes a candidate when character-level fuzzy title matching finds no database title with similarity of 0.9 or higher.Similarity uses normalized Levenshtein distance scaled to [0, 1].
- 2.3 Candidate Creation: Candidate citations increased over time in average and maximum counts, while candidate percentages also showed a consistent upward trend.The paper argues that the increase exceeds what can be explained by coarse screening and simple database-matching noise alone.
- 2.4 Manual Verification: Candidate citations are manually checked against original PDFs and web searches, and a paper is marked HalluCited after one HalluCitation is identified.Verification uses links, DOIs, arXiv IDs, pages, venues, or title searches, applying a conservative precision-over-recall criterion.
3 Results
HalluCited papers increased sharply in 2025, with EMNLP 2025 accounting for 154 cases and main-track papers increasingly affected. Papers with multiple candidates are easier to flag, but most HalluCited papers contain only one or two candidates, making manual detection difficult.
- 20 HalluCited papers in 2024 increased to 275 in 2025, with EMNLP 2025 accounting for 154 cases.The proportion rose from around 0.28% to 2.59% overall, reaching 3.7% at EMNLP 2025.
- Area-level percentages can be sensitive to individual papers in venues and tracks with small numbers of papers.They remain useful for tracking trends.
- Nearly 70% of EMNLP 2025 HalluCited papers appeared in Main and Findings rather than workshops.This indicates that the issue increasingly affects main tracks rather than remaining confined to side venues.
- Around four HalluCitation candidates identify papers containing actual HalluCitations at a rate of nearly three out of four.Three or four candidates are recommended as a threshold for further verification, while fewer candidates often reflect OCR, parsing, or fuzzy-matching noise.
- Nearly 200 papers with one or two candidates account for around two-thirds of all HalluCited papers.Because these hallucinations are embedded among otherwise proper citations, reviewers outside the cited area may find them nearly infeasible to detect manually within limited review time.
4 Further Analysis and Discussions
Further analysis examines area and title patterns, review-process evidence, database errors, and possible mitigation. The findings point to emerging-topic review challenges, incomplete disclosure, and the need for careful use of primary sources and automated support.
- 4.1 Trend Analysis: Low-Resource NLP, LLM Efficiency, and AI/LLM Agents show relatively high counts and proportions of HalluCited papers.Several high-proportion areas were newly introduced at EMNLP 2025, where securing qualified reviewers may be difficult.
- 4.1 Trend Analysis: HalluCited titles more often contain reinforcement-learning-related terms, abbreviations such as “LLM,” and efficiency-related terminology.General papers more often use terms such as “Human,” “Reasoning,” and “Preference” and spell out “Large Language Model.”
- 4.2 How about the Peer-Review Process?: ARR preprints show a high proportion of papers containing HalluCitation candidates, although such papers are filtered out to some extent during peer review.Because the disclosure rate is around 20%, it remains unclear whether opt-in papers are uniformly representative; the analysis assumes uniform distribution.
- 4.2 How about the Peer-Review Process?: The analysis does not assess the quality of HalluCited papers.The authors nonetheless note increasing difficulty filtering such cases among accepted EMNLP 2025 papers and suggest automated flagging to assist reviewers.
- 4.3 Citation Sources: Secondary databases contain truncated titles, non-existent papers, incorrect arXiv identifiers, and missing authors.An incorrect arXiv identifier propagated to hundreds of subsequent citations, while entirely hallucinated papers were also referenced by other papers.
- 4.3 Citation Sources: Authors are encouraged to obtain citation information from primary sources or use citation-normalization tools such as Ribiber.The recommendation follows cases where secondary-source records were inaccurate or entirely fabricated.
5 Suggestions and Recommendations
The paper recommends automatic HalluCitation verification, clearer correction traceability, and peer-review structures that prioritize rigorous, constructive engagement over speed.
- Automatic verification: Automatic verification systems should support both author pre-submission checks and organizer-side review scans.These systems can provide early warnings, reduce reviewer workload, and improve citation reliability.
- Definitions and traceability: HalluCitations should be defined using cited-title existence and consistency of key identifiers such as IDs and page numbers.The paper distinguishes severe HalluCitations from minor citation errors that may warrant correction rather than punishment.
- Definitions and traceability: Revision and peer-review processes need traceability so corrected citation information can be verified in camera-ready papers.The paper emphasizes that unclear correction tracking is a serious process problem.
- Peer-review reform: Review systems should provide sufficient time and manageable workloads for constructive discussion that improves papers.The paper contrasts this goal with incentives for rapid completion under strict time constraints.
- Peer-review reform: Conference and journal-equivalent models may need reconsideration as submission growth and review burdens threaten conference credibility.One proposed direction is asynchronous, rigorous peer review with conferences focused on presenting accepted work.
6 Conclusion
The study finds nearly 300 HalluCited papers across recent ACL conferences, while arguing for corrective and preventive responses rather than immediate post hoc penalties.
- Conclusion: Nearly 300 HalluCited papers were identified among ACL, NAACL, and EMNLP publications from 2024 and 2025.More than half appeared at EMNLP 2025.
- Conclusion: HalluCitations should not automatically trigger penalties because the errors can arise unintentionally rather than through deliberate misconduct.The paper emphasizes author toolkits, pre-submission checks, and voluntary corrections instead.
- Conclusion: Acceptance should be treated as part of an ongoing process of improving and communicating research, not as a binary reward or punishment.The conclusion connects this view to correction and erratum mechanisms.
- Conclusion: Peer review should retain a collaborative nature that tolerates minor errors while promoting responsible correction.The paper frames conferences as venues for sharing work rather than rewarding perfection under strict constraints.
Limitations
The study’s limitations concern its NLP-focused venue coverage, restricted access to review materials, and precision-oriented detection methodology that yields a lower-bound estimate.
- Scope: The analysis covers six recent top-tier NLP conferences and excludes venues unavailable publicly at the time of writing.Broader investigations across other conferences and journals are left for future work.
- Analysis: Restricted access to review materials and the general unavailability of rejected papers limit analysis of review quality and outcomes.The paper argues that greater transparency would facilitate more thorough analyses.
- Methods: The methodology prioritizes precision, so the reported HalluCitation results should be interpreted as a lower bound.Additional HalluCitations may exist beyond those detected.
- Methods: The study uses the ACL Anthology and OpenReview because broader domains lack comparable unified bibliographic infrastructure.OCR and engineering choices are described as implementation details rather than the primary contribution.
Ethical Considerations
The paper describes its data handling, licensing, privacy safeguards, and tool usage for a study based on publicly available records and published papers.
- Information: All information reflects publicly available data as of January 4, 2026, so later updates, corrections, or removals are not included.This date bounds the links, metadata, and dataset contents used in the study.
- Licenses: The ACL Anthology, OpenReview, and additional datasets are described as publicly available and permissively licensed.The authors state that human verification was conducted solely by the paper’s authors.
- Privacy and potential harm: The study preserves reproducibility by using public HalluCitation records while omitting author names and paper titles from Table 7.Each HalluCited paper is represented through reference-entry links and one example HalluCitation.
- Tool usage: DeepL, ChatGPT, and Grammarly were used for translation and grammatical improvement, while OCR processing used an NVIDIA A6000 GPU.HalluCitation records were managed locally and were not transmitted to external APIs.
A More Suggestions
The paper proposes making citation verification easier through machine-friendly reference formats and more transparent area information. These changes are intended to facilitate automatic verification and support more effective review.
- Toward easy verification formats and proper citation practices: Machine-friendly reference sections could facilitate automatic citation verification and reduce manual effort.Suggested changes include clearer citation delimiters, explicit key fields, printed URLs, clickable links, and DOIs.
- Eliminating area selection or improving area transparency: The paper suggests either eliminating fixed area selection or publicly releasing area information for accepted papers.The authors argue that fixed area selection can create reviewer mismatches as cross-area research becomes more common.
- Eliminating area selection or improving area transparency: Reviewer mismatches and reduced review rigor may hinder effective review, including HalluCitation detection.The paper connects improved reviewer matching and area transparency with more effective reviews.
B List of HalluCited Papers
Table 7 lists identified HalluCited papers together with the corresponding HalluCitations, presenting evidence in reference format rather than directly displaying paper details. The listed records include examples from EMNLP 2024 and NAACL 2025.
- B List of HalluCited Papers: Table 7 lists identified HalluCited papers and their corresponding HalluCitations.The table is described as containing the hallucinated papers and the evidence associated with them.
- B List of HalluCited Papers: The evidence is presented in reference format rather than by directly displaying HalluCited paper details.Readers are directed to the provided link or the reference section to identify the corresponding paper.