PaperlayerView exact revision on arXiv

Source-linked AI summary

HALoGEN: Fantastic LLM Hallucinations and Where to Find Them

Abhilasha Ravichander, Shrusti Ghela, David Wadden, Yejin Choi

arXiv:2501.08292v1cs.CLcs.AI

TL;DR

Hallucinations are difficult to measure at scale because human verification of open-ended generations is costly, while their sources remain unclear. HALOGEN benchmarks diverse use cases with automatic atomic-unit verification and uses the results to study error sources. The evaluation finds task-dependent hallucination even in the best-performing models, with GPT-4 scores ranging from 4% to 86%.

  • Problem

    Hallucinations can cause downstream harms, but their causes are unclear and human verification makes measuring them costly and time-consuming.

    Source: §1 · p. 1

  • Method

    HALOGEN benchmarks diverse use cases with automatic verifiers that decompose generations into atomic units and check factuality.

    Source: §1 · p. 1

  • Results

    4% to 86%: GPT-4 hallucination scores vary by task, and no single domain strongly predicts hallucination in others.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The framework provides a systematic foundation for studying hallucination sources and truthful LLMs.

    Source: §7 · p. 10

    The metrics measure factual precision and appropriate abstention but do not account for whether responses contain all information they should.

    Source: HALOGEN · p. 10

Abstract

from arXiv · show

Despite their impressive ability to generate high-quality and fluent text, generative large language models (LLMs) also produce hallucinations: statements that are misaligned with established world knowledge or provided input context. However, measuring hallucination can be challenging, as having humans verify model generations on-the-fly is both expensive and time-consuming. In this work, we release HALoGEN, a comprehensive hallucination benchmark consisting of: (1) 10,923 prompts for generative models spanning nine domains including programming, scientific attribution, and summarization, and (2) automatic high-precision verifiers for each use case that decompose LLM generations into atomic units, and verify each unit against a high-quality knowledge source. We use this framework to evaluate ~150,000 generations from 14 language models, finding that even the best-performing models are riddled with hallucinations (sometimes up to 86% of generated atomic facts depending on the domain). We further define a novel error classification for LLM hallucinations based on whether they likely stem from incorrect recollection of training data (Type A errors), or incorrect knowledge in training data (Type B errors), or are fabrication (Type C errors). We hope our framework provides a foundation to enable the principled study of why generative models hallucinate, and advances the development of trustworthy large language models.

1 Introduction

HALOGEN addresses the difficulty and cost of measuring hallucinations by benchmarking diverse LLM tasks and tracing potential sources of errors. Its results show substantial task-dependent hallucination and suggest that error sources vary across domains.

1 Introduction

  • HALOGEN measures hallucinations across diverse use cases with automatic verifiers that decompose generations into atomic units and check their factuality.The benchmark includes response-based and refusal-based tasks.Source: §1 · p. 1
  • 4% to 86%: GPT-4 hallucination scores vary by task, and no single domain strongly predicts hallucinations in others.The evaluation spans 14 models and 150,000 generations.Source: §1 · p. 1
  • 29%: even the best-performing model responds in scenarios where it should abstain.The paper says this highlights a need to improve calibration.Source: §1 · pp. 1–2
  • The paper classifies hallucinations as Type A when correct facts were in pretraining, Type B when training data contained incorrect or out-of-context facts, and Type C when neither fact was present and the model over-generalized.Source: §1 · p. 2
  • Hallucination sources vary across domains: code-package errors often involve incorrect training-data facts, while senator-affiliation errors often occur despite correct information being available.The paper classifies these examples as Type B and Type A errors, respectively.Source: §1 · p. 2
  • The authors offer HALOGEN as a systematic foundation for studying hallucination sources and truthful LLMs.Source: §1 · pp. 1–2

2 Related Work

Prior work detects hallucinations in grounded and open-ended generation, and benchmarks factuality using several evaluation strategies. HALOGEN extends this landscape with broader task coverage, refusal settings, and varied verifiers.

2 Related Work

  • Prior grounded-task detection identifies factual units and checks them against source text with entailment-based or QA-based systems.Source: §2 · p. 2
  • Open-ended detection uses reference sources such as Wikipedia or web search, or reference-free methods that compare responses or inspect logits.Source: §2 · p. 2
  • Existing benchmarks evaluate responses with stronger LLMs, likelihoods, or human annotators, while other benchmarks test knowledge-base completion and multiple-choice factuality.Source: §2 · pp. 2–3
  • HALOGEN spans grounded, open-ended, and bespoke tasks, including both response and refusal settings, with verifiers ranging from entailment approaches to package and reference searches.Source: §2 · p. 3
  • Prior attribution often uses search engines or fixed sources such as Wikipedia, whereas HALOGEN uses WIMBD to cross-reference hallucinations against pretraining corpora.Source: §2 · p. 3

3 Building a Benchmark for Hallucinated Content

This section defines hallucinations and describes HALOGEN’s nine-scenario benchmark, which decomposes generations into atomic units for verification. It also introduces metrics for measuring hallucination, response, and utility.

3 Building a Benchmark for Hallucinated Content

  • Hallucination means a generated fact misaligned with established knowledge or user context; in content-grounded tasks, even true facts count if the text does not entail them.The benchmark focuses open-domain generation on knowledge-oriented rather than creative or subjective tasks.Source: §3 · p. 3
  • Because established knowledge can be pluralistic or conflicting, the benchmark specifies one source of truth per scenario; other sources or interpretations can change factuality judgments.Source: §3 · p. 3

HALOGEN

  • HALOGEN covers 10,923 prompts across nine scenarios, with prompt sets, decomposition engines, and factuality verifiers for response- and refusal-based tasks.The benchmark’s tasks either call for a response or require the model to refuse.Source: HALOGEN · p. 3, Table 1 · p. 4
  • Verifiers are task-specific: scientific reference titles use Semantic Scholar, packages use PyPI, summaries use GPT-3.5 entailment, and historical-event responses are judged for refusal.Source: HALOGEN · pp. 3–5
  • 91% for summarization, 92% for simplification, and 83% for historical events were the verifier agreement rates in manual checks.The study sampled and manually annotated 100 atoms for each task.Source: HALOGEN · p. 5

3.2 Evaluation Metrics

  • The three metrics are response ratio, hallucination score, and utility score; utility combines response and hallucination scores, with scoring conditioned on response- versus refusal-based tasks.The metrics address flexible outputs and the possibility that a model abstains.Source: §3.2 · pp. 5–6

4 Results

Across 14 language models, hallucination rates varied substantially by task, and model rankings were not consistently predictive across domains. GPT-3.5 and GPT-4 were comparably factual on response-based tasks, while GPT-4 showed better appropriate refusal behavior.

Quantifying Hallucination Rate

  • 4%-86% of facts generated by even the best-performing LLMs were hallucinations, depending on the domain.Tables 2 and 3 report hallucination rates, response ratios, and utility scores for response-based and refusal-based categories.Source: §4 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • GPT-3.5 and GPT-4 were comparably factual on response-based tasks, while GPT-4 showed better appropriate refusal behavior.Source: §4 · p. 6

Hallucination patterns by domain

  • Model rankings varied by domain: summarization and simplification were highly correlated, but biographies were not perfectly predictive of other domains.Mistral 7b hallucinated the fewest code packages; Alpaca 7b hallucinated none but often produced unhelpful programs; GPT-4 and Alpaca 7b more rarely hallucinated scientific references, while GPT-3.5 and GPT-4 were most factual on summarization, simplification, and biographies.Source: §4 · p. 6, Figure 2 · p. 6

Refusal Behavior

  • Llama models and GPT-3.5/4 had high refusal rates on queries that should be refused, while Mistral models and OLMo often accepted them and hallucinated.Source: §4 · p. 6

Open-Source vs Closed Models

  • GPT-3.5 and GPT-4 were clear winners over open-source and open-pipeline models on response-based and refusal-based tasks; Llama-3-70B led among open models.Source: §4 · pp. 6–7

Do larger models hallucinate less?

  • Larger models generally hallucinated less on response-based tasks in four of six tasks, but no similar trend appeared on refusal-based tasks.Within model families, Llama-2 70B ≤ 13B ≤ 7B and Llama-3 70B ≤ 8B; Mixtral 8x7B hallucinated less than Mistral 7B on average in both settings.Source: §4 · p. 7

5 Why Do Models Hallucinate?

The section traces hallucinations to training-corpus evidence and finds that their likely origins vary across domains and tasks. Its analyses distinguish errors despite correct training evidence, errors associated with incorrect or out-of-context evidence, and fabrication or unsupported inference.

5 Why Do Models Hallucinate?

  • Hallucinations are classified as Type A when correct facts occur in training data, Type B when incorrect or out-of-context facts occur there, and Type C when neither occurs and the model over-generalizes.Content-grounded tasks add unsupported inferences from the provided context as another source.Source: §5 · pp. 7–8

5.1 Open-Ended Tasks

  • For historical-meeting hallucinations, entity names rarely co-occur in the same pretraining document across the three analyzed models.The sampled cases were drawn from models with at least 100 hallucinations in this category.Source: §5.1 · p. 8
  • Correct senator educational affiliations are commonly present in C4 for Llama models even when models predict incorrect affiliations.The analysis covers 500 instances for Llama-2 7B/13B/70B, Llama-3 8B, and OLMo.Source: §5.1 · p. 8
  • Up to ∼72% of hallucinated software packages in one case appear in pretraining corpora, with possible explanations including local imports, renamed or deprecated packages, and non-package symbols.The analysis examines matched documents for five packages hallucinated by each model.Source: §5.1 · p. 8, Table 4 · p. 9

5.2 Content-Grounded Tasks

  • 83% of hallucinations in high-utility summaries stem from incorrectly processing context, while 17% stem from introducing external facts.The authors manually annotated 100 hallucinated statements, excluding ambiguous entailments and atomization errors.Source: §5.2 · p. 9
  • In high-utility summaries, incorrect entities or predicates are common hallucination sources, while grammatical errors are rare.The fine-grained categories also include relation, circumstance, coreference, and discourse-link errors.Source: §5.2 · p. 9
  • 49% of simplified-text samples feature insertion errors, 49% substitution errors, and 7% deletion errors.An atomic fact can contain multiple error types.Source: §5.2 · p. 9
  • Among simplification errors, 91.8% of substitution errors are severe, substantially altering the complex text’s main idea; among 49 samples with verifiable hallucinated terms, 65.3% occur in pretraining data.Source: §5.2 · p. 10

6 Discussion and Future Work

The discussion presents hallucinations as arising from multiple sources, so it speculates that mitigation may need complementary approaches. It also notes that semantic errors persist in summarization and calls for more robust frameworks for meaning.

6 Discussion and Future Work

  • The benchmark aims to measure hallucinations rigorously to support progress toward more trustworthy models, given potential harms in user-facing applications.Source: §6 · p. 10

What will it take to have truthful AI systems?

  • Hallucinations may arise from incorrect training information or fabrication, and the authors speculate that effective mitigation would require complementary approaches.Retrieval may help with long-tailed information but not when the datastore lacks relevant information; verbalized uncertainty may help in that case.Source: What will it take to have truthful AI… · p. 10
  • Current models also make semantic errors in summarization when context is completely provided, indicating a need for more robust frameworks for semantic meaning.Source: What will it take to have truthful AI… · p. 10

7 Conclusion

The conclusion presents HALOGEN as a resource for measuring hallucinations across scenarios and uses its generations to trace hallucinations to training data. The resulting analysis highlights varied potential causes and proposes a foundation for scientific study.

7 Conclusion

  • HALOGEN measures and identifies hallucinations across a broad range of scenarios.Source: §7 · p. 10
  • The authors use 150,000 generations from 14 language models to trace hallucinations to training data and classify three types of errors.They discuss potential mitigation strategies based on the error types.Source: §7 · p. 10

8 Limitations

The limitations concern verifier reliability, attribution to undisclosed training data, incomplete coverage measurement, and the choice of a single factual source of truth. The accompanying prompt-construction details describe datasets and task setups used in the benchmark.

HALOGEN

  • Benchmark-score reliability is limited by the accuracy of automated verifiers, especially for subjective open-ended tasks.Code-generation verification is more accurate because it exactly searches available Python packages.Source: HALOGEN · p. 10
  • Undisclosed training sources for many evaluated models limit the accuracy of training-data attributions.Source: HALOGEN · p. 10
  • The metrics assess factual precision and appropriate abstention but do not measure whether responses include all information they should.Source: HALOGEN · p. 10
  • A singular source of truth operationalizes established knowledge, although alternative sources or interpretations can yield different factuality judgments.Source: A Prompt Construction Details · p. 15

A Prompt Construction Details

  • Prompt construction draws on tasks and datasets including senator-search questions, factual summarization, and text simplification.Senator-search prompts use state–college combinations whose correct answer is always “No”; summarization uses 1,278 filtered CNN/DailyMail instances.Source: A Prompt Construction Details · pp. 15–16

B Detecting Model Abstention

The section describes task-specific rules for identifying abstentions, including phrase matching and response checks. It also presents factual-density statistics for response-based and refusal-based tasks.

B Detecting Model Abstention

  • For scientific-attribution and historical-event prompts, abstention is detected from refusal phrases or empty responses, with a verifier also checking denials of historical meetings.For scientific attribution, responses with no references count as abstentions; for historical events, Llama-2-70b-chat is asked whether the response confirms a meeting.Source: B Detecting Model Abstention · p. 17
  • False-presupposition responses count as abstentions when they use refusal phrases, are empty, or say “no response.”The prompt explicitly asks for “no response” when the model cannot answer.Source: B Detecting Model Abstention · p. 17
  • Tables 5 and 6 report factual-density statistics for response-based and refusal-based tasks, respectively.Both captions describe totals and average, minimum, and maximum atomic-unit counts, with corresponding hallucinated-unit statistics in parentheses.Source: Table 5 · p. 18, Table 6 · p. 18

Code

The methods decompose model generations into task-specific atomic units and verify those units against prompt constraints, source passages, or external knowledge sources. Extraction and verification procedures vary by task.

Code

  • Code-package names are extracted from import statements and checked against PyPI, then Python’s official documentation if needed.A package not found in either source is marked hallucinated.Source: Code · p. 18
  • Summaries are decomposed into independent facts and checked for entailment against the source passage; simplification uses the same process, while biographies use Factscore.Unsupported summary facts are classified as hallucinated atomic units.Source: Code · pp. 18–19
  • For binary rationalization prompts, the pipeline extracts yes/no answers and task-specific factors, senator names, or flight paths, then applies category-specific hallucination checks.Factors are parsed from a prompted list, senator names are extracted with named-entity recognition, and directed flight paths are extracted as tuples.Source: Code · p. 19
  • Rationalization verification flags incorrect binary answers, generated factors or senator names, and flight paths absent from the prompt context.The listed checks treat “no” as hallucinated for the primality category and “yes” for the senator-search category.Source: Code · p. 19
  • For numerical tasks, the method extracts a number and entity list, then flags items that differ from the prompt’s predefined valid entities.If one of the number or list units is missing, only the available unit is retained.Source: Code · p. 19
  • For scientific references, extracted titles are queried against Semantic Scholar; titles with no matching entry are marked hallucinated.Titles are the atomic units for this task.Source: Code · p. 20
  • For historical-event prompts, a model checks whether the response confirms the proposed meeting, and confirmation without abstention is flagged as hallucinated.The verifier is prompted for a yes-or-no answer.Source: Code · p. 20
  • For false-presupposition prompts, entity lists are extracted from responses and checked against the prompt’s valid-entity list.The list is formed by splitting the parsed response at commas.Source: Code · p. 20

D Factual Density

The section reports factual density but does not treat it as a measure of how much information a generation contains, nor prescribe an optimal density for each task.

D Factual Density

  • The hallucination, response-ratio, and utility scores do not measure how much information a model generation contains.The authors say that interpreting response detail is not always clear and depends on task context.Source: D Factual Density · p. 20
  • The appropriate amount of detail differs by task: attribution may call for all relevant references, while summaries should not exceed the source’s information.The section gives these contrasting tasks as examples of why response detail is difficult to interpret uniformly.Source: D Factual Density · p. 20
  • The authors report factual density and average generated-fact counts but do not prescribe an optimal factual density for any task.The statistics are reported separately for response-based and refusal-based tasks.Source: D Factual Density · p. 20

E Examples of Model Generations

The section collects qualitative model-generation examples across several prompt types, including code, scientific attribution, summarization, and reasoning tasks.

E Examples of Model Generations

  • A Python example narrows a list by repeatedly asking the user to choose between two randomly selected options until one remains.The section includes both code fragments and a prose description of the function.Source: E Examples of Model Generations · p. 20
  • For a figure-detection programming prompt, an OLMo response proposes OpenCV, NumPy, and scikit-image imports.The excerpt shows imports including cv2, numpy, and several skimage modules.Source: E Examples of Model Generations · pp. 20–21
  • A scientific-attribution example pairs a request for references about Venice canal pollution with a model response listing APA-style citations.The excerpt includes several purported 2020 references.Source: E Examples of Model Generations · p. 21
  • Summarization and simplification examples show model responses to a Bob Barker article and two text-rewriting prompts.The examples include a summary response and simplified versions of text about stripe colors and literacy rates.Source: E Examples of Model Generations · p. 21
  • Other examples include biography responses about Rohit Sharma and Rafael Nadal, and a response describing a supposed meeting between Joan Rivers and Irene Stefani.The section also includes a separate prompt about Einstein and Paulo Schroeber.Source: E Examples of Model Generations · p. 21
  • Reasoning examples cover false-presupposition lists and binary or numerical questions, with model outputs shown alongside the prompts.The excerpts include planet and continent lists, a primality answer, a senator question, and a planet-count response.Source: E Examples of Model Generations · p. 22

F Examples of Model Hallucinations

The section presents qualitative examples of model hallucinations across HALoGEN prompt types.

F Examples of Model Hallucinations

  • Asked whether 5023 is prime, Llama 2 7B answers yes but gives 5 × 10047 as its factorization.The stated product is not 5023, so the response contradicts its own requested factorization.Source: F Examples of Model Hallucinations · p. 23
  • The planet-count response says “3” but includes “Melee,” explicitly identified as not a planet.The example pairs a numerical answer with a listed item that does not meet the prompt’s condition.Source: F Examples of Model Hallucinations · p. 23

Related papers

Loading 2501.08292v1…