HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences

Introduction

• HalluCitations are non-existent citations that threaten scientific reliability and can undermine the quality and credibility of published venues.Time-constrained peer review makes it harder to inspect manuscript rigor and factual correctness carefully.Source: §1 · pp. 1–2
• The analysis covers more than 17,000 ACL, NAACL, and EMNLP papers published in 2024 and 2025, including main conference, Findings, and workshop papers.The study calls papers containing at least one HalluCitation “HalluCited” papers.Source: §1 · p. 2
• Nearly 300 papers contain at least one HalluCitation, mostly from 2025; half were identified at EMNLP 2025, including over 100 accepted main and Findings papers.The findings indicate that HalluCitations are rapidly increasing and may affect conference credibility.Source: §1 · p. 2
• The proposed detection method combines OCR, fuzzy citation-title matching against a reference database, and manual verification of flagged candidates.Papers with four or more detected candidates show a high incidence of HalluCitations, suggesting a practical threshold for automated toolkits.Source: §1 · p. 2, Figure 2
• HalluCitations cannot be immediately attributed to AI-generated content because they may originate from secondary sources, so authors should not face immediate penalties.The study instead recommends prior consensus on preventive measures such as author toolkits for pre-submission checks.Source: §1 · p. 2

2.1 Data Collection

• The dataset covers over 17,000 papers from ACL, NAACL, and EMNLP conferences held in 2024 and 2025, including main, Findings, and workshop papers.Tutorials and proceedings were excluded from the archival PDFs collected from the ACL Anthology.Source: §2.1 · pp. 2–3

2.2 Citation Extraction

• More than 740,000 citations were extracted with MinerU for reference-block OCR extraction and GROBID for parsing and normalization; the median was 39 citations per paper.The citation quartiles remained stable across venues, and reference blocks were explicitly supplied to GROBID to improve coverage of multi-page reference lists.Source: §2.2 · p. 3, §2.3 · p. 3

2.3 Canditate Creation

• Citations were screened using venue or arXiv keywords, database searches, and title fuzzy matching, with scores below 0.9 sent for manual verification.Matching used normalized Levenshtein similarity across ACL Anthology, arXiv, DBLP, and OpenAlex metadata.Source: §2.3 · p. 3
• Candidate citations increased over time in both average and maximum counts per paper, while candidate-paper and candidate-citation proportions also rose, suggesting the increase exceeds methodological noise.The authors attribute the remaining increase most notably to HalluCitation, despite possible noise from coarse screening and simple database matching.Source: §2.3 · p. 4

2.4 Manual Verification

• All candidates from the 2,950 target papers were manually verified, and a paper was marked “HalluCited” once one HalluCitation was identified.A citation was classified as hallucinated when no corresponding work could be found or at least two key attributes failed to match a reliable source; the criterion prioritizes precision over recall.Source: §2.4 · p. 4

Finding 1

• 275 papers contained HalluCitation in 2025 versus 20 in 2024, with EMNLP 2025 accounting for 154 papers and 2.59% of papers overall versus 0.28% in 2024.EMNLP 2025 reached 3.7%, and nearly 70% of its HalluCited papers appeared in Main and Findings; all EMNLP 2025 tracks exceeded 3%.Source: §3 · p. 4, Table 4 · p. 4
• The increase was not confined to workshops or side tracks, as HalluCited papers increasingly affected main tracks, especially at EMNLP 2025.At NAACL and ACL, workshops accounted for roughly half of HalluCited papers, whereas EMNLP’s Main and Findings accounted for nearly 70%; small-track percentages should be interpreted cautiously.Source: §3 · p. 4

Finding 2

• Nearly three out of four papers with around four HalluCitation candidates contained an actual HalluCitation, making them suitable for further verification.Simple title matching against reference databases can detect such cases, while isolated candidates are more vulnerable to OCR, parsing, and fuzzy-matching noise.Source: §3 · pp. 4–5, Table 5 · p. 5

Finding 3

• Nearly 200 papers with one or two candidates accounted for around two-thirds of all HalluCited papers, so most cases contain only a small number of hallucinated citations.These citations are often disguised among proper references and may be nearly impossible to detect manually outside a reviewer’s expertise during limited review time.Source: §3 · p. 5
• Because hallucinated citations may originate from secondary sources and remain embedded among proper citations, the results motivate automated validation systems.Potential sources include Google Scholar, reference-management tools, and LLM-based recommendation systems, whose entries may already contain hallucinations.Source: §3 · p. 5

4.1 Trend Analysis

• Newer areas such as Safety and Alignment in LLMs, AI/LLM Agents, and LLM Efficiency show relatively high HalluCitation proportions, possibly reflecting reviewer shortages.Low-Resource NLP, LLM Efficiency, and AI/LLM Agents also have relatively high counts, while NLP Applications, Interpretability, and Resources and Evaluation have high counts but lower proportions.Source: §4.1 · p. 6

4.2 How about the Peer-Review Process?

• For ARR iterations linked to conferences since 2024, HalluCitation-candidate proportions are high but are filtered out to some extent during peer review.Such papers are frequently assigned to meta-reviewers, sometimes at a rate of at least one paper per reviewer, while the roughly 20% preprint disclosure rate limits interpretation.Source: §4.2 · p. 7
• The analysis does not assess paper quality, but increasing HalluCitations among accepted EMNLP 2025 papers indicate that reviewer effort alone may be insufficient, motivating automated flagging.Automated systems could assist reviewer workloads and help maintain rigor and quality.Source: §4.2 · p. 7

4.3 Unintentional HalluCitation: Databases Are Often Contaminated

• Secondary databases can introduce HalluCitations through truncated titles, non-existent papers, incorrect arXiv identifiers, missing authors, and inaccurate citation information.Figure 5 illustrates these failure cases in Google Scholar and Semantic Scholar.Source: §4.3 · p. 7, Figure 5 · p. 7
• Database contamination means HalluCitations do not necessarily indicate AI-generated references, and authors should verify sources directly or use citation-normalization tools such as Ribiber.Incorrect entries can propagate through subsequent citations, while retrieval APIs used by AI-Scientist agents may amplify the problem.Source: §4.3 · pp. 7–8

Automatic verification

• Automatic HalluCitation detection in author toolkits and review checks could warn authors early, scan submissions at scale, reduce reviewer workload, and improve citation reliability.HalluCitations may arise from secondary-source errors, so their presence cannot be attributed solely to LLMs or AI-generated papers.Source: §5 · p. 8

Definition and traceability

• HalluCitations should be defined using cited-title existence and key-identifier consistency, while minor errors receive correction rather than disproportionate penalties.Traceable revision and peer-review records are needed to verify that corrections appear in camera-ready versions and to analyze emerging issues rapidly.Source: §5 · p. 8

Purpose of conferences and peer review

• Peer review should prioritize sufficient time and manageable workloads for constructive feedback, because rapid reviews and growing burdens risk undermining conference credibility and sustainability.The paper contrasts conferences with preprints’ immediacy and journals’ rigor, arguing that conferences should reconsider their purpose if they provide neither.Source: §5 · p. 8

ARR and Findings reform

• ARR and Findings could be redefined as journal-equivalent venues, with conference presentations certifying or presenting accepted work in a more flexible model similar to TMLR.This change would not resolve submission-volume growth but could help preserve credibility as the research community evolves.Source: §5 · p. 9

• Nearly 300 HalluCited papers were identified across ACL, NAACL, and EMNLP publications from 2024–2025, with more than half appearing at EMNLP 2025.The analysis covered all main conference, Findings, and workshop papers published by these conferences during both years.Source: §6 · p. 9
• HalluCitations should prompt author toolkits, pre-submission checks, and encouraged erratum revisions rather than penalties, because errors may be unintentional and papers were accepted for substantive contributions.The conclusion states that HalluCited authors should not be penalized post hoc and that voluntary corrections should be better encouraged through ACL Anthology errata.Source: §6 · p. 9
• Peer review should remain collaborative and tolerant of minor errors while promoting responsible correction and self-improvement through mechanisms such as Findings papers and ARR.The conclusion frames conferences as venues for sharing work rather than rewarding perfection under batch-review constraints, and calls on NLP to shape peer review in the LLM era.Source: §6 · p. 9

Scope

• The analysis covers six recent top-tier NLP conferences because HalluCitation cases were negligible earlier and the sharp increase occurred in 2025.AACL 2025 and the ARR October 2025 cycle were unavailable when the study was written, while other conferences and journals remain future directions.Source: Limitations · p. 9

Analysis

• The study primarily analyzes accepted papers because rejected papers and detailed review materials are generally inaccessible, limiting assessment of review quality and correction traceability.The authors argue that publicly releasing review materials, including rejected papers, would enable more comprehensive analyses of review outcomes (Church et al., 2025; Yang, 2025; Kim et al., 2025d).Source: Limitations · pp. 9–10

Methods

• The ACL-focused data sources and MinerU OCR implementation limit generality, while precision-prioritized detection means the reported HalluCitation results should be interpreted as a lower bound.The study identifies nearly 300 HalluCited papers despite its restricted scope and positions broader domains, alternative OCR tools, and improved detection as future work.Source: Limitations · p. 10

• The study’s information reflects publicly available data as of January 4, 2026, so later updates, corrections, or removals are not included.Source: Ethical Considerations · p. 10
• ACL Anthology, OpenReview, and additional datasets are publicly and permissively licensed; human verification was performed solely by the authors.Source: Ethical Considerations · p. 10
• The analysis does not attribute HalluCitations to misconduct or advocate penalties, instead focusing on review-system limitations and constructive solutions.The study avoids targeting individuals and presents findings as neutral academic research.Source: Ethical Considerations · p. 10
• Author protection is balanced with reproducibility by listing public-record HalluCitations while omitting author names and paper titles from Table 7.Each paper is represented through links to reference entries and one example HalluCitation, and the study reports compliance with the ACL Ethics Policy21.Source: Ethical Considerations · p. 10
• DeepL, ChatGPT, and Grammarly supported language improvement, while HalluCitation records were managed locally without transmission to external APIs or services.Experiments used an NVIDIA A6000 GPU for OCR; other analyses used a MacBook Pro, spreadsheets, and Google Chrome.Source: Ethical Considerations · p. 10

Toward easy verification formats and proper citation practices

• Reference sections should use clearer citation delimiters, explicit bibliographic fields, printed URLs, and mandatory clickable links or DOIs to enable automatic verification.These changes would reduce the manual effort required to detect HalluCitations, which are often disguised in reference sections and currently require OCR-based parsing and manual verification.Source: A More Suggestions · p. 32

Eliminating area selection or improving area transparency

• Research-area selection should be eliminated or made transparent by publicly releasing accepted-paper area information to reduce reviewer mismatches and support HalluCitation detection.Fixed area assignments may be problematic for cross-area research, while public area records could improve reviewer matching and review effectiveness.Source: A More Suggestions · p. 32
• Modern review systems could replace explicit area selection with automated reviewer matching, while area archives such as the ACL Anthology would preserve transparency.The passage notes that area information is currently only partially disclosed and that archiving it could support more effective reviews.Source: A More Suggestions · p. 32

B List of HalluCited Papers

• For privacy, the entries present reference-format information rather than directly displaying HalluCited paper details.Readers are instructed to follow the provided link or consult the reference section to identify the corresponding paper.Source: B List of HalluCited Papers · p. 32
• Table 7 lists the identified hallucinated papers and their corresponding HalluCitations.The table is explicitly presented as a list of HalluCited Papers and their supporting evidence.Source: B List of HalluCited Papers · p. 32, Table 7 · p. 33
• The cited records include an EMNLP 2024 industry paper, a NAACL 2025 AmericasNLP workshop paper, and an ACL 2025 demo paper.Source: B List of HalluCited Papers · pp. 34, 37

ACL 2025 records

• Records 80–91 cover ACL 2025 papers citing purported work on African languages, retrieval-augmented medical QA, pretrained models, biomedical classification, factual consistency, and instruction following.Examples include Wura, rationale-guided RAG, SwahBERT, AlignScore, and Alpaca.Source: B List of HalluCited Papers · pp. 39–40

ACL 2025 Findings

• Records 92–108 include ACL 2025 Findings entries referencing information retrieval, direct preference optimization, abstractive summarization, DeepSeek, OpenOrca, and GPT-4o through the cited works.Source: B List of HalluCited Papers · pp. 40–41

ACL 2025 workshops and tracks

• The cited entries cover retrieval-augmented ICD coding, where a two-stage system beats vanilla LLMs by 94, evaluation-quality criteria, low-resource speech-to-text translation across modalities, cryptocurrency fraud detection, language-model safety and red teaming, and NLP-involved causal inference.Source: B List of HalluCited Papers · p. 42

ACL 2025 SemEval and workshops

• The cited records include papers on nominal compound compositionality, text-based emotion detection, generative pre-training, lexicon-based emotion classification, and Mistral 7B.Source: B List of HalluCited Papers · pp. 43–44

EMNLP 2025 records

• Records 142–146 begin the EMNLP 2025 portion, covering Arabic question answering, morphological generalization, language-model pretraining, BERT, and collaborative multiagent debate.These entries identify purported references associated with ArabicNLP, BabyLM, CODI, and related EMNLP 2025 papers.Source: B List of HalluCited Papers · pp. 44–45

EMNLP 2025 demo and industry entries

• Entries 147–152 are EMNLP 2025 demo or industry records citing Jang et al. (2025), Liu et al. (2025f), Chao et al. (2025), Khabiri et al. (2025), Zhang (2025), and Ethiraj et al. (2025).Source: B List of HalluCited Papers · p. 45

EMNLP 2025 industry entries

• Entries 153–158 contain bibliography records citing Yu et al. (2025b), Beyranvand et al. (2025), Chawla et al. (2025), Azeez et al. (2025), Swamy et al. (2025), and Bougie and Watanabe (2025).Source: B List of HalluCited Papers · pp. 45–46

EMNLP 2025 main-paper entries

• Entries 180–217 continue the EMNLP 2025 main-paper list with citations spanning semantic modeling, summarization, table generation, multilingual data, instruction following, alignment, reasoning, dialogue state tracking, vision-language models, and multi-agent systems.The listed entries include works attributed to Liu et al. (2025e), Yuan et al. (2025b), Sadia et al. (2025), Gao et al. (2025), Wang et al. (2025d), Kang et al. (2025), Trifan et al. (2025), Wang et al. (2025c), and Chen et al. (2025a).Source: B List of HalluCited Papers · pp. 47–50

B List of HalluCited Papers

• The cited passages show EMNLP 2025 records numbered 218–220 and identify cited works including Scale efficiently, Ciphergan, Vicuna, and Counselingeval, but do not establish a complete 218–246 span.Source: B List of HalluCited Papers · p. 51
• Entries 267–278 cover additional EMNLP 2025 workshop and main-conference papers, including math, multilingual retrieval, legal NLP, and NLP perspectives venues.The cited works include GPT-4, InternVL 1.5, DeepSeek-Prover-V2, CrossRAG, LegalSeg, and Human label variation.Source: B List of HalluCited Papers · pp. 55–56
• The supplied passages identify EMNLP 2025 StarSem records 279–282 and TSAR record 283, naming Kadam and Ferraro, Gasan and Pais, Rai et al., Bailleux et al., and Hayakawa et al.Source: B List of HalluCited Papers · p. 56
• Records 285–287 identify EMNLP 2025 UncertainLP and WinLP papers associated with Dossou and Aïdasso (2025), Vinjamuri and Sun (2025), and Kolli et al. (2025).The cited passages also include Zhang et al. (2023), Li et al. (2019), and Tan et al. (2023b).Source: B List of HalluCited Papers · pp. 56–57
• Records 290–292 cover EMNLP 2025 WMT papers attributed to Zhou et al. (2025c), Zhu et al. (2025c), and Bell et al. (2025).The corresponding references include Kunchukuttan and others (2023), Vechev et al. (2024), and Gliozzo and Strapparava (2006).Source: B List of HalluCited Papers · p. 57
• Records 293–295 list additional EMNLP 2025 WMT papers by Pang et al. (2025), Kim (2025), and Acharya et al. (2025).Their cited works are Chiang et al. (2023), Jiang et al. (2022), and Ramesh et al. (2023), respectively.Source: B List of HalluCited Papers · p. 57