Source-linked AI summary
Testing Our Foundations: Citation Trends, Errors, and Emerging Hallucinations in the Computing Education Literature
Paul Denny, Gweneth Barbre, Musa Blake, Yan Cathy Hua, Juho Leinonen, Andrew Luxton-Reilly, James Prather, Brent N. Reeves
TL;DR
The paper asks how reference integrity has changed in computing education and how often published references contain errors or fabricated information. It analyzes ACM Digital Library metadata and recent computing-education references using trend analysis, automated matching, classification, and manual verification. The findings show that hallucinated references are a recent, conservative lower-bound concern, often combining real and fabricated bibliographic details, while automated checking is complicated by metadata discrepancies.
Problem
The study addresses the lack of large-scale evidence on reference integrity across computing education literature spanning pre- and post-LLM eras.
Method
The paper analyzes ACM Digital Library publications and references, classifies bibliographic errors, and manually verifies candidate hallucinated references after automated field parsing and matching.
Results
The identified hallucinations include entirely fabricated references and hybrid references combining real titles with fabricated or incorrect authorship, venues, or years.
Takeaways & Limitations
Hallucinated references are no longer hypothetical in computing education research, but the reported findings are a conservative lower-bound estimate.
Takeaways & Limitations
Automated reference checking is limited by apparent mismatches between ACM XML metadata and references printed in camera-ready PDFs.
Abstract
from arXiv · showhide
Accurate references are foundational to scholarly work, enabling verification, attribution, and systematic review. However, the rapid adoption of large language models has introduced a serious integrity concern: plausible-looking but fabricated citations. Although hallucinated references are widely discussed, their visibility within specific research communities remains unclear. We address this gap by examining reference integrity at key computing education venues using ACM Digital Library data. We analyze referencing trends across 24,751 computing education papers and compare them with the broader ACM corpus of more than 723,000 papers and 15 million references. We then examine reference lists from these venues, classify common bibliographic errors, and manually identify LLM-generated hallucinations containing verifiably false information, including impossible page ranges, invented titles, and misattributed authors. In 2025, hallucinated references appeared across five SIGCSE-sponsored or in-cooperation venues. At the Technical Symposium alone, verified hallucinated references increased from 3 in 2025 to 17 in 2026, appearing in 2.3\% of 2026 proceedings papers. Although still relatively rare for now, this growth poses an integrity risk our community should not ignore.
1 Introduction
References support verification, attribution, and cumulative knowledge, yet computing education literature contains increasingly visible plausible-looking but unreliable citations. This paper addresses the gap by analyzing reference trends, bibliographic errors, and verifiably fabricated information across computing education venues.
- References enable readers to verify claims, trace ideas, credit prior work, and build cumulative knowledge.
- Citation errors range from incorrect years and author names to wrongly attributed or nonexistent publications.The introduction gives examples involving impossible page information, an invented title, incorrect publication year, and misattributed authors.
- LLMs can generate plausible-looking references by combining realistic authors, titles, venues, and metadata without grounding them in real sources.The paper frames this as a growing concern in academic writing support and notes increasing visibility in published literature.
- The study analyzes ACM Digital Library publications spanning 1951–2026 to examine computing-education trends, classify bibliographic errors, and identify hallucinations containing verifiably false information.Its research questions address publication and reference-list trends, error types, and fabricated bibliographic information.
2 Related Work
Prior computing education bibliometric work largely describes the community, while newer studies examine publication integrity, hallucinated references, and verification methods across disciplines. Existing evidence motivates systematic study of hallucination trends and manual validation in computing education.
- Earlier computing education studies use bibliometric, keyword, and collaboration analyses to describe a growing community and broad research topics.
- Prior work also raises concerns about citation integrity because authors may not accurately represent the conclusions of cited papers.The emergence of hallucinated references has renewed concerns about scholarly publication rigor.
- Large-scale studies estimate rising hallucinated-reference rates after widespread LLM adoption, including substantial increases in biomedical and AI/ML literature.These studies compare pre- and post-LLM unmatched-reference rates or audit structured references against bibliographic databases.
- Proposed detection systems combine similarity matching, deterministic field matching, and LLM-based adjudication for ambiguous citations.
- The closest methodological comparison combines large-scale citation extraction, fuzzy title matching against databases, and manual verification of candidate hallucinations.
3 Methods
The study builds a complete ACM Digital Library metadata corpus covering publications and reference lists from 1951–2026. A Python workflow extracts structured publication data and plain-text references for subsequent analysis.
- The dataset contains 1,304,236 ACM metadata XML files covering publications from 1951 to 2026.ACM supplied the complete metadata collection as 28,269 zip archives on April 1, 2026.
- ACM XML files provide publication metadata such as authors, titles, years, venues, and identifiers, alongside plain-text reference-list entries.
- A Python script extracts publication and reference data into Pubs and Refs datasets, links references to parent publications, and removes non-publications and duplicates.
Publication and Reference Data Processing.
The processing pipeline identifies computing education venues, constructs a recent reference dataset, and partitions it for automated matching and manual checking. Structured fields are parsed from raw references, while manual review returns to the original plain text.
- Publication and Reference Data Processing.: The study tags ACM publications associated with 13 computing education venues using manually identified venue-name and acronym variations.The venue set includes ACE, CSERC, CompEd, ICER, ITiCSE, Koli Calling, SIGCSE Bulletin, SIGCSE TS, SIGCSE Virtual, TOCE, UKICER, WCCCE, and WiPSCE.
- Publication and Reference Data Processing.: The CS-Ed dataset contains 24,751 computing education publications, including 5,225 publications since 2021 with 113,588 reference entries.The 2021–2026 range balances pre-widespread-AI coverage with a feasible manual checking workload.
CS-Ed-Refs dataset.
The CS-Ed-Refs dataset is parsed into structured bibliographic fields and matched against ACM publication records to separate matched from unmatched references.
- A locally hosted Qwen3.5-4B model parses raw references into authors, titles, venues, years, and DOIs for automated matching.Manual checking uses the original plain-text references rather than the parsed fields.
- The matching procedure prioritizes normalized DOI matches, then title variants, and finally boundary-aware title-substring searches.
- Figure 2 defines SET A as references matched by DOI or title and SET B as references without such a match.
- 39,620 references matched ACM metadata records, while 73,968 remained unmatched.
3.1 Publication and Referencing Trends (RQ1)
The study compares publication and referencing trends in computing education with those in other ACM domains over time.
- 3.1 Publication and Referencing Trends (RQ1): The trend analysis plots publication counts and median reference-list lengths for CS Ed and other ACM venues.
3.2 Common Bibliographic Errors (RQ2)
The common-error analysis combines automated author-discrepancy detection with manual review of candidate references and publication trends.
- 3.2 Common Bibliographic Errors (RQ2): The author-overlap pipeline used exact surname matching, fuzzy fallback, given-name or initial confirmation, and handling for et al. truncation.
- 3.2 Common Bibliographic Errors (RQ2): Figure 3 displays yearly publication counts and median reference-list lengths for CS Ed and other venues, excluding incomplete 2026 data.
- 3.2 Common Bibliographic Errors (RQ2): 1,278 references with too little author overlap were flagged for manual checking after automated comparison with ACM metadata.False positives included similar-title matches that actually cited URL-only or non-ACM sources.
- 3.2 Common Bibliographic Errors (RQ2): Two coders evenly classified 828 records within each year using adapted hallucination-taxonomy codes, including H for common hallucination errors and R for real references.They consulted raw references, matched metadata, DOI pages, source PDFs, and external bibliographic records.
3.3 Hallucinated References (RQ3)
The hallucination analysis uses a conservative definition requiring verifiably fabricated bibliographic information and manually investigates candidates from matched and unmatched references.
- 3.3 Hallucinated References (RQ3): Hallucinated references are counted only when bibliographic information is verifiably fabricated, not when correct but incomplete information merely omits fields.
- 3.3 Hallucinated References (RQ3): The candidate set contained 225 matched references with inconsistent core metadata and 904 unmatched references appearing to cite CS Ed venues.
- 3.3 Hallucinated References (RQ3): Semantic Scholar matching found 239 unmatched candidates without a title match, but these remained candidates because indexing gaps and parsing errors could explain missing matches.
- 3.3 Hallucinated References (RQ3): The final 464 candidates were manually searched online, and entries lacking credible publication records received second-review checks using bibliographic databases, publisher pages, DOI searches, and general web search.
4 Results
Computing education publication and reference volumes have grown, while manual review found both metadata mismatches and author-field errors. Conservatively verified hallucinated references appeared across multiple venues and increased sharply in SIGCSE TS from 2025 to 2026.
- 4.1 Publication and Referencing Trends (RQ1): Publication volume and reference-list length both increased over time, raising the number of citations requiring verification.Computing education reference lists grew less sharply than those in the broader ACM corpus.
- 4.2 Common Bibliographic Errors: The most common reviewed issue was an ACM XML–PDF metadata mismatch, with 229 cases, rather than an error in the published paper.Author-field errors classified as hallucinated were the next largest category, with 225 cases.
- 4.2 Common Bibliographic Errors: Valid bibliographic variants accounted for 188 cases, while working-group and curriculum reports accounted for 150 cases despite varied citation formats.Minor typos and acceptable formatting variations were also recorded.
- 4.3 Hallucinated References: 30 hallucinated references were identified among 464 manually reviewed candidates, spanning 14 papers.Hallucinations contained verifiably fabricated bibliographic information and were identified using conservative manual verification.
- 4.3 Hallucinated References: Hallucinated references appeared in 2025 papers from five computing education venues, with seven found in one UKICER paper.The venues were SIGCSE TS, UKICER, CompEd, TOCE, and ITiCSE.
- 4.3 Hallucinated References: 17 verified hallucinated references appeared in seven SIGCSE TS 2026 publications, up from 3 references across the 2025 proceedings.The 2026 snapshot includes only publications available by April 1, 2026.
5 Discussion
The study finds that hallucinated references are a real and recently increasing concern in computing education, but identifying them remains difficult because plausible errors overlap with metadata problems and ordinary citation mistakes. It therefore argues for shared responsibility and human-in-the-loop checking rather than relying on reviewers or automation alone.
- Thirteen of the 30 references were entirely fabricated, while 17 combined real titles with fabricated or incorrect authorship, venue, or year information.Seven hybrid cases included at least one correct author, making many hallucinations superficially plausible.
- Distinguishing hallucinations from valid variants, ACM metadata mismatches, and ordinary human errors often requires substantial manual checking beyond typical reviewer capacity.Increasing publication volumes, longer reference lists, and stretched reviewer capacity further complicate verification.
- Automated checking is constrained by imperfect metadata ground truth and may miss subtle hybrid hallucinations.The authors report that ACM XML often disagreed with clean references printed in camera-ready PDFs, sometimes because of special-character handling.
- The authors recommend human-in-the-loop tools that flag suspicious references, explain the reasons, and provide feedback before peer review.Their identified error categories are proposed as a starting point for such systems.
- The analysis covers published papers that passed peer review and publication checks, so the submission-stage problem may be larger than observed.The paper presents the published literature as only the visible part of a potentially broader problem.
- The paper proposes shared responsibility: authors should verify citations, venues should target high-risk cases, and publishers should improve metadata and validation support.Reviewers and program committees are not expected to manually audit every reference.
6 Conclusions
The paper combines large-scale ACM metadata analysis with conservative manual verification to assess reference integrity in computing education. It finds that hallucinated references remain uncommon but are real, recent, and growing, requiring shared responsibility across the publication process.
- Thirty hallucinated references appeared across 14 papers in 2025 and 2026, and 2.3% of SIGCSE TS 2026 proceedings papers contained at least one.Most identified hallucinated references appeared in the SIGCSE TS 2026 proceedings.
- Most suspicious references were metadata mismatches, valid variants, or ordinary citation irregularities rather than hallucinations.
- The paper concludes that the problem is small but real, recent, and growing, and calls for shared responsibility among authors, reviewers, program chairs, editors, and publishers.