Source-linked AI summary
CheckIfExist: Detecting Citation Hallucinations in the Era of AI-Generated Content
Diletta Abbonato
TL;DR
AI-generated citation hallucinations create a bibliographic-integrity gap because plausible references may not correspond to published work, while existing reference managers do not provide real-time authenticity validation. CheckIfExist addresses this gap with cascading searches across CrossRef, Semantic Scholar, and OpenAlex, similarity-based confidence scoring, and unified single-reference and batch workflows. It returns database-derived APA citations and exportable BibTeX records for validated references.
Problem
LLMs can generate plausible but nonexistent citations, while reference-management systems organize bibliographies without providing real-time authenticity validation.
Method
CheckIfExist uses cascading validation across CrossRef, Semantic Scholar, and OpenAlex with normalized string similarity and cross-source author verification.
Results
The tool provides immediate verification feedback and generates database-derived APA citations and exportable BibTeX records.
Takeaways & Limitations
Researchers can validate suspected citations, audit bibliographies, and replace problematic entries with corrected records within contemporary research workflows.
Takeaways & Limitations
Threshold and penalty values were empirically calibrated, with author-related discrepancies weighted more heavily because their diagnostic signal is stronger for AI-generated fabrications.
Abstract
from arXiv · showhide
The proliferation of large language models (LLMs) in academic workflows has introduced unprecedented challenges to bibliographic integrity, particularly through reference hallucination -- the generation of plausible but non-existent citations. Recent investigations have documented the presence of AI-hallucinated citations even in papers accepted at premier machine learning conferences such as NeurIPS and ICLR, underscoring the urgency of automated verification mechanisms. This paper presents "CheckIfExist", an open-source web-based tool designed to provide immediate verification of bibliographic references through multi-source validation against CrossRef, Semantic Scholar, and OpenAlex scholarly databases. While existing reference management tools offer bibliographic organization capabilities, they do not provide real-time validation of citation authenticity. Commercial hallucination detection services, though increasingly available, often impose restrictive usage limits on free tiers or require substantial subscription fees. The proposed tool fills this gap by employing a cascading validation architecture with string similarity algorithms to compute multi-dimensional match confidence scores, delivering instant feedback on reference authenticity. The system supports both single-reference verification and batch processing of BibTeX entries through a unified interface, returning validated APA citations and exportable BibTeX records within seconds.
1 Introduction
CheckIfExist addresses citation hallucinations by verifying bibliographic references against multiple scholarly databases in real time. It targets a validation gap left by reference-management tools that organize citations but do not authenticate them.
- CheckIfExist is an open-source tool for real-time bibliographic reference validation against multiple scholarly databases.
- LLMs can generate plausible citations with realistic authors, journals, and publication years that correspond to no published work.
- More than 100 AI-hallucinated citations were found across at least 53 NeurIPS 2025 papers in an analysis of over 4,000 papers.
- Reference managers organize and format citations but are fundamentally designed for organization rather than authenticity validation.
- The tool queries CrossRef, Semantic Scholar, and OpenAlex and returns validation results within seconds.
2 Background and Motivation
LLM-generated citation hallucinations differ from ordinary transcription errors because they systematically fabricate references that imitate legitimate scholarly conventions. This shifts verification costs onto readers, reviewers, and editors while making detection difficult.
- Traditional citation errors primarily arise from transcription mistakes, whereas LLM hallucinations are systematic fabrications produced by language-generation regularities.
- Hallucinated references can combine plausible authors, journals, titles, and publication details, making manual detection cognitively demanding.
- Reported hallucination rates range from 6% to over 30% depending on model, prompting strategy, and domain.
- Generating plausible citations costs little, while verification costs remain substantial for readers, reviewers, and editors.
3 System Architecture
The system uses a cascading, multi-source verification architecture that preprocesses references, searches scholarly databases, compares metadata with string-similarity methods, and returns confidence-scored results. Cross-source author validation, penalties, and confirmation bonuses help determine whether a reference exists.
- The application separates preprocessing, BibTeX parsing, multi-source search, and result-presentation functions into four modules.
- CrossRef is queried first, with Semantic Scholar and OpenAlex serving as fallback sources in the validation cascade.
- The verification algorithm filters LaTeX commands, searches for candidates, cross-validates authors, and initiates fallback validation when scores or issues warrant it.
- The procedure returns an existence decision, confidence, detected issues, formatted APA output, BibTeX data, and source information.
- Normalized Levenshtein similarity compares reference strings after lowercasing and removing non-alphanumeric characters.
- The composite confidence score combines field similarities with penalties for mismatches, fabricated authors, and a multi-source confirmation bonus.
4 Features and Usage
CheckIfExist provides unified quick-check and batch-check workflows for individual references and bibliographies. It produces database-derived APA citations and BibTeX records while positioning verification as complementary to reference-management systems.
- Quick check accepts free-form citations or informal reference descriptions, while batch check processes BibTeX entries or newline-separated citation lists.
- Batch processing displays progressive validation outcomes as references are processed sequentially with API-compliant rate limiting.
- Thresholds and penalties were empirically calibrated, with author discrepancies weighted more heavily because they provide a stronger diagnostic signal for AI-generated fabrications.
- Each verified reference yields APA 7th-edition citations and valid BibTeX records derived from retrieved scholarly metadata.
- The system complements rather than replaces reference managers by allowing exported libraries to be validated and corrected BibTeX records reimported.
5 Conclusions
CheckIfExist supports contemporary research workflows by helping users verify references before submission, during LLM-assisted literature review, and in manuscript or bibliometric audits. Its open-source, multi-source architecture provides rapid authenticity checks and workflow-compatible exports.
- CheckIfExist supports reference-list validation before manuscript submission and enables rapid replacement of uncorrected entries with validated BibTeX records.
- Researchers can verify LLM-suggested citations before incorporating them into their work, addressing hallucinated references in AI-generated content.
- Reviewers can audit manuscript citations for potential fabrications, while publishers can integrate multi-source validation into submission pipelines.
- Large-scale bibliometric analyses can filter potentially spurious references to protect the validity of citation network analyses.
- The open-source tool combines CrossRef, Semantic Scholar, and OpenAlex verification with a unified interface, automatic LaTeX filtering, and BibTeX export.