Source-linked AI summary
Do Language Models Know When They're Hallucinating References?
Ayush Agrawal, Mirac Suzgun, Lester Mackey, Adam Tauman Kalai
TL;DR
Language models can generate credible but unfounded information, and the paper asks whether they can recognize hallucinated references without external resources. It treats references as a tractable test case, applies direct and indirect consistency checks, and finds that models often produce inconsistent author lists for hallucinated references while recalling authors of real references more consistently.
Problem
Language models frequently generate credible but unfounded information, creating a need to understand and detect hallucinated references without relying on external resources.
Method
The paper treats hallucinated references as a model organism and tests direct binary queries alongside indirect queries that compare independently generated author responses for consistency.
Results
Models often produce inconsistent author lists for hallucinated references while often accurately recalling the authors of real references.
Takeaways & Limitations
The findings suggest hallucination can be addressed partly through the generation pipeline and studied using the model’s internal representations without external resources.
Takeaways & Limitations
The evaluation cannot determine definitively what is grounded because the models’ training data are inaccessible, and results may depend on prompt wording and computer-science references.
Abstract
from arXiv · showhide
State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.
1 Introduction
The paper studies hallucinated references as a tractable model organism for broader language-model hallucination research. It investigates whether LMs can detect their own fabricated references through questioning strategies that do not rely on external tools.
- Hallucinated outputs can undermine reliability and cause serious harm when they enter domains such as healthcare, finance, law, or academic publishing.
- The paper focuses on hallucinated computer-science book and article references because they are easier to classify than many other hallucination types.
- Reference titles are suitable for study because they are static, broadly searchable, and likely represented in language-model training corpora.
- The experiments compare direct and indirect questioning strategies for detecting hallucinated references across GPT- and Llama-based models without external tools.
- The authors propose hallucinated references as a model instance, demonstrate automatic classification, compare hallucination rates, and introduce indirect queries.
2 Preliminaries and Background
The paper distinguishes groundedness from factual correctness and uses Web search as an approximate groundedness test for references. It builds on black-box confidence estimation and introduces indirect consistency queries alongside direct queries.
- The paper defines hallucination as fabricated text with little or no grounding in the model’s training data.
- Groundedness differs from correctness because information present in training data can still be factually false.
- Exact-match Web search provides an approximate groundedness evaluation because published article titles are usually discoverable and sufficiently specific.
- Direct queries ask whether a reference exists, whereas indirect queries ask open-ended questions such as who authored it and compare independent answers for consistency.
3 Related Work
The work addresses open-domain groundedness, where hallucinations are difficult to measure, and relates its black-box approach to prior confidence and attribution research. It also distinguishes the studied ungrounded errors from grounded factual errors.
- Open-domain hallucinations are difficult to measure, motivating reference-based evaluation with search heuristics and black-box detection methods.
- The paper studies ungrounded errors, which are distinct from factually incorrect information that is nevertheless grounded in flawed training data.
- Prior work includes attribution studies, medical-domain evaluations of hallucinated references, and black-box methods for measuring confidence in LM generations.
- Unlike approaches that inspect model weights or consult databases, this work evaluates hallucinations through generation-only black-box queries.
4 Methodology: Consistency Checks
The methodology uses direct and indirect consistency checks to estimate whether candidate references are grounded. Direct queries elicit repeated binary judgments, while indirect queries compare independently generated author responses for overlap.
- Direct queries: Direct queries ask whether a candidate reference exists and use three prompt templates with varying contextual information.
- Direct queries: For each direct query, repeated completions estimate the model’s probability that the reference exists by dividing “yes” responses by all completions.
- Indirect queries: Indirect queries first ask about a reference’s authors in independent sessions, then estimate overlap between the resulting answers using the LM.
- Indirect queries: Consistent indirect answers suggest familiarity with a reference, whereas varied answers may signal that the model lacks intrinsic knowledge of its authors or content.
- Ensembling: The study also averages direct and indirect scores in an ensemble check and avoids external resources such as Google Scholar or Semantic Search.
5 Experimental Details
The study builds a computer-science reference corpus by prompting language models across randomly sampled ACM topics, then labels titles through exact-match Bing searches. It evaluates automated labeling reliability and several consistency-check and filtering metrics.
- Corpus construction: The corpus samples 200 of 543 ACM Computing Classification System topics and prompts each language model to generate five related reference titles per topic.This produces 1,000 titles per language model.
- Automatic labeling: Bing labels each generated title as grounded when an exact quoted-title search returns results and hallucinated when it returns none.The reference title is searched in quotation marks through the Bing Search API.
- Verification: Four expert annotators agreed with Bing on 99% or 100% of sampled GPT-4 labels, supporting the automatic pipeline’s reliability.The manual check covered 10% of GPT-4-generated references.
- Language models and scoring: The experiments evaluate GPT-3, ChatGPT, GPT-4, and Llama 2 Chat models using indirect and direct consistency-query scores.Indirect experiments average three query results, while direct experiments use ten judgments at temperature 1.0.
- Evaluation metrics: ROC curves measure the grounded-versus-hallucinated classification trade-off, while FDR curves measure preserved references against hallucination tolerance.The ROC analysis summarizes performance with AUC, and the FDR framing treats the classifier as a literature-review filter.
6 Results and Discussion
Newer models show higher grounding rates, and consistency checks classify references effectively, with performance generally improving with model size. Ensembling improves classification across models, while qualitative analysis exposes plausible title constructions and method-specific errors.
- Quantitative results: Newer models have higher grounding rates, or equivalently lower hallucination rates, than older models in the six-model comparison.The study reports this pattern in Table 1, whose labels are based on Bing-search ground truth.
- ROC analysis: For GPT-4, DQ1 achieves AUC 0.887 and IQ achieves AUC 0.878, while performance generally improves as model size increases.For L2-70B and ChatGPT, IQ performs best overall by AUC.
- FDR analysis: IQ provides lower FDR and a better FDR-preservation trade-off for L2-70B and ChatGPT, while GPT-4’s IQ and DQ methods offer comparable low-FDR trade-offs.The FDR curves compare the fraction of references preserved with the false discovery rate.
- Ensembling: The IQ+DQ ensemble obtains the best AUC and lower FDR curves for every language model, improving on individual consistency checks.IQ is generally stronger than DQ1-3 for ChatGPT and L2-70B, whereas both are similarly effective for GPT-4.
- Qualitative findings: Qualitative analysis finds hallucinated title mashups, plausible-sounding nonexistent titles, and direct-query false positives involving surveys or book chapters.The Bing quoted-search heuristic is more lenient than exact matching for capitalization and punctuation.
- Qualitative findings: Indirect checks can also falsely label hallucinations as grounded when a model consistently generates likely authors from a related real book or classic paper.Examples include hallucinated titles borrowing authors associated with Introduction to Operations Research and Exploratory Data Analysis.
- Error analysis: The indirect method may falsely classify grounded references as hallucinated when the model cannot generate authors that are not encoded in its weights.The paper notes that falsely marking hallucinations as grounded is more problematic for applications focused on detecting potential hallucinations.
7 Conclusions
The paper concludes that reference hallucinations provide a measurable setting for studying open-domain hallucination, while suggesting generation-time interventions as a path toward reducing it.
- Conclusions: Open-domain hallucination is difficult to measure, but reference search enables quantitative comparison across models and black-box detection methods.The authors note that direct publication-index lookup could achieve higher detection accuracy, but their goal is to study self-detection.
- Conclusions: The study suggests hallucination is not entirely a training problem and may be addressed through different generation procedures using the same internal representation.
- Conclusions: Direct and indirect query methods are only partially reliable and impractically expensive, limiting their immediate use for hallucination detection.
- Future work: Improved non-black-box decoding techniques are identified as a crucial direction because reducing hallucination may be a generation-time problem.
- Future work: Future work includes indirect questions about years or venues and testing whether consistency-based methods generalize beyond references.The paper notes that year and venue questions introduce ambiguity because identical titles and authors can appear in multiple venues and years.
8 Limitations
The paper’s limitations concern inaccessible training data, ambiguous hallucination boundaries, prompt sensitivity, domain-specific references, and potential demographic bias.
- Limitations: Because training data are inaccessible, the study cannot determine what model outputs are truly grounded versus hallucinated.The web is used as a contending proxy for training data.
- Limitations: Binary labels do not cleanly capture references that are substrings or superstrings of existing titles.
- Limitations: Findings comparing direct and indirect queries may depend on specific prompt wording because language models are prompt-sensitive.
- Limitations: Using ACM Computing Classification System topics biases results toward computer science references, although the procedure can be rerun for other topic lists.
- Limitations: The procedure may perform worse for certain racial groups and may fail to recognize some names as likely authors, motivating broader bias and accuracy checks before deployment.
A Bing Search Reliability
The supplementary reliability analysis reports that automated Bing exact-match labels closely agree with expert human judgments.
- Human annotation: Expert annotators received labeling instructions and corresponding Google search links for each reference title.
- Human annotation: Human labelers agreed on labels for four exemplar references before assigning manual labels.
- Reliability analysis: Cohen’s κ comparisons show that automated Bing exact-match labeling reliably matches human expert judgments.
B Supplementary Experimental Details
Supplementary experiments report weak detection performance for several models, document the evaluation setup and costs, and provide annotation materials and implementation details.
- Experimental results: The procedures were not effective for detecting hallucinations in L2-13B, L2-7B, and GPT-3, with the worst performance for L2-7B.IQ helped most for GPT-3, DQ2 helped most for both L2 models, and the IQ+DQ ensemble performed best overall among the tested approaches.
- Data handling: The paper states that search-query results were not stored; only whether each query returned results was retained.
- Computation and cost: The reported computation used approximately 2.2M GPT-4 tokens, 2.3M ChatGPT tokens, and 2.1M GPT-3 tokens, plus 3,000 Bing Search API queries, for a total estimated cost of $412.
- Annotation materials: The appendix includes the labeling instructions, sample annotated titles, exemplar labels, and human-annotation materials.
E Examples of Hallucinations and References
Manual inspection of sampled titles found that Bing-based H and G classifications usually corresponded to hallucinated and real references, respectively. However, title similarity could produce misleadingly high IQ scores for hallucinated titles.
- Manual searches found that the vast majority of Bing-classified H titles were hallucinations, while the vast majority of G titles were real references.The inspection covered 30 randomly sampled candidate titles classified as H or G.
- Figure 9 presents ROC curves for the IQ and DQ approaches together with ensemble approaches.
- Figure 10 relates false discovery rate to the fraction of references preserved for each groundedness filter and language model.Preservation rate is the fraction retained after filtering, while FDR is the fraction of retained references that are hallucinations.