PaperlayerView exact revision on arXiv

Source-linked AI summary

HalluLens: LLM Hallucination Benchmark

Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, Pascale Fung

arXiv:2504.17550v1cs.CLcs.AI

TL;DR

LLMs can produce outputs inconsistent with user input or training data, creating trust and evaluation challenges. HalluLens separates hallucination from factuality, introduces a taxonomy and dynamic extrinsic benchmark tasks alongside intrinsic tasks, and analyzes existing evaluations; results vary across subtasks and models, while intrinsic evaluation remains constrained by judge-related hallucination concerns.

  • Problem

    Inconsistent hallucination definitions and benchmark categories prevent unified evaluation, while hallucination is often conflated with factuality.

  • Method

    HalluLens defines extrinsic and intrinsic hallucination, combines new extrinsic tasks with existing intrinsic tasks, dynamically generates extrinsic test sets, and analyzes existing benchmarks.

  • Results

    Results vary across subtasks, with Kendall’s τ correlations of 0.5897 between subtasks and 0.7436 and 0.8462 between each task and their average.

  • Takeaways & Limitations

    Distinguishing hallucination from factuality supports dedicated evaluation, while dynamic test generation targets leakage and benchmark obsolescence.

  • Takeaways & Limitations

    Intrinsic hallucination evaluation does not use dynamically generated test sets because an LLM judge can introduce intrinsic hallucination.

Abstract

from arXiv · show

Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination." These hallucinations undermine user trust and hinder the adoption of generative AI systems. Addressing hallucinations is essential for the advancement of LLMs. This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks, built upon clear taxonomy of hallucination. A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations. We disentangle LLM hallucination from "factuality," proposing a clear taxonomy that distinguishes between extrinsic and intrinsic hallucinations, to promote consistency and facilitate research. Extrinsic hallucinations, where the generated content is not consistent with the training data, are increasingly important as LLMs evolve. Our benchmark includes dynamic test set generation to mitigate data leakage and ensure robustness against such leakage. We also analyze existing benchmarks, highlighting their limitations and saturation. The work aims to: (1) establish a clear taxonomy of hallucinations, (2) introduce new extrinsic hallucination tasks, with data that can be dynamically regenerated to prevent saturation by leakage, (3) provide a comprehensive analysis of existing benchmarks, distinguishing them from factuality evaluations.

1 Introduction

HalluLens addresses inconsistent hallucination benchmarking by separating hallucination from factuality and combining new extrinsic with existing intrinsic tasks. It uses dynamically generated extrinsic test sets and analyzes existing benchmarks to reduce leakage and clarify evaluation coverage.

  • Motivation: Hallucinations can conflict with user input, previous outputs, or existing knowledge, undermining trust and downstream decision making.The paper frames comprehensive, reliable, and ungameable evaluation as a first step toward mitigation.
  • Benchmarking gap: Inconsistent definitions and categories prevent a unified framework for comprehensive hallucination evaluation.Existing benchmarks often leave hallucination types unspecified or use inconsistent categories, producing uneven coverage.
  • Taxonomy: HalluLens disentangles hallucination from factuality and provides a taxonomy intended to promote consistent research.Hallucination is evaluated against the model’s training corpus or input context, whereas factuality requires an external ground-truth oracle.
  • Benchmark design: The benchmark combines newly introduced extrinsic hallucination tasks with existing intrinsic hallucination tasks.Extrinsic test sets are dynamically generated, while intrinsic evaluation uses established tasks.
  • Benchmark design: Dynamic test-set generation reduces leakage risk and obsolescence as training datasets continuously expand.The approach is designed to preserve reliable evaluation over time.
  • Existing benchmarks: The paper analyzes major hallucination and factuality benchmarks to identify their evaluated challenges and limitations.The analysis includes TruthfulQA, SimpleQA, and HaluEval2.0, with TruthfulQA reported as saturated and affected by incorrect gold answers and excessive metric penalties.

2 Overview of LLM Hallucination

The paper defines hallucination by consistency with a model-accessible source, distinguishing extrinsic and intrinsic forms from factuality. It uses this distinction to organize benchmark categories, explain hallucination sources, and set design criteria for realistic evaluation.

  • Existing categorizations: Existing surveys are described as conflating hallucination with factuality and overlooking extrinsic hallucination.Figure 2 separates new HalluLens benchmarks, adaptable factuality-related tasks, and benchmarks suitable for intrinsic hallucination.
  • Hallucination versus factuality: Hallucination and factuality assess reliability against different reference sources and therefore represent distinct evaluation concepts.Factuality uses established verification sources, whereas hallucination uses training data or inference-time input context.
  • Hallucination versus factuality: A response can be factually incorrect without being hallucinated when it remains consistent with the model-accessible source.The paper uses time-sensitive knowledge, such as the latest Summer Olympics, to illustrate this distinction.
  • Taxonomy: The proposed taxonomy contains extrinsic hallucination, inconsistent with training data, and intrinsic hallucination, inconsistent with input context.Extrinsic cases may arise in free-form generation or knowledge-gap filling, while intrinsic cases contradict or lack support from the input.
  • Sources: Potential hallucination sources include unseen or limited knowledge, contradictory or noisy data, and modeling error.Limited knowledge can prompt fabrication, while conflicting information can produce both extrinsic and intrinsic hallucinations.
  • Evaluation criteria: Benchmark design should emphasize real-world applicability, broad domain and task coverage, and resistance to narrow optimization.Diverse topics, prompts, and response formats help measure underlying capabilities rather than benchmark gaming under Goodhart’s law.

3 HalluLens (a): Extrinsic Hallucination Evaluation

HalluLens evaluates extrinsic hallucination through three tasks covering precise answers, long-form consistency, and refusal on nonexistent information, using dynamically generated test questions. Results show trade-offs among refusal, hallucination, correctness, and factual grounding across models and tasks.

  • Task design: The benchmark introduces three extrinsic-hallucination tasks spanning precise short answers, long-form consistency, and refusal on nonexistent information.PreciseWikiQA and LongWiki target modeling errors, while NonExistentRefusal targets knowledge gaps from unseen or limited information.
  • Task design: Dynamic test-question generation reduces leakage and memorization risks while seeking stable evaluation across test-set versions.The benchmark generates new questions during evaluation rather than relying on a fixed test set, balancing unpredictability with reproducibility.
  • PreciseWikiQA: In PreciseWikiQA, GPT-4o achieves the highest correct answer score at 52.59%, while Llama-3.1-405B-Instruct has the lowest hallucination rate when answering at 26.84%.Llama-3.1-405B-Instruct also falsely refuses 56.77% of the time, illustrating the trade-off between avoiding hallucinations and answering questions.
  • PreciseWikiQA: Across difficulty levels, Llama-3.1-405B-Instruct refuses more on long-tail knowledge but maintains relatively consistent hallucination rates, whereas GPT-4o and Claude-3-Sonnet hallucinate more there.GPT-4o nevertheless provides correct answers more frequently because its refusal rate is lower.
  • NonExistentRefusal: NonExistentRefusal results show consistent but task-dependent model behavior: false acceptance rates correlate across subtasks, while larger models generally better identify nonsensical or nonexistent entities.The Llama-3.1-405B-Instruct model records 11.48% false acceptance on MixedEntities and 2.28% on GeneratedEntities.

4 HalluLens (b): Intrinsic Hallucination Evaluation

Intrinsic hallucination measures whether generated content remains consistent with an input context, a concern closely tied to user trust. HalluLens reviews established evaluations spanning summarization, reference-grounded question answering, and noisy or contradictory contexts.

  • Definition: Intrinsic hallucination occurs when generated content contradicts or deviates from the provided input context.The source may be an original document in summarization or another user-provided context.
  • Motivation: Intrinsic hallucination is comparatively well studied because source-input tasks have clear verification references, although it remains important for maintaining user trust.Existing intrinsic benchmarks are described as not saturated and still relevant.
  • HHEM leaderboard: HHEM evaluates factual inconsistency in text summaries relative to their source documents using a dedicated evaluation model.Its evaluation documents come from the CNN/Daily Mail Corpus, and inference uses temperature 0 for consistency.
  • HHEM leaderboard: As of December 2024, GPT-4o has a 1.5% hallucination rate on HHEM, while smaller models remain an improvement target.The passage also reports 3.9% for Llama-3.1-405B-Instruct and 4.6% for Claude-3-5-sonnet.
  • ANAH 2.0: ANAH 2.0 with references tests consistency between answers and factually accurate contexts, but does not claim robustness against training-data leakage.Answers are evaluated sentence by sentence for contradictory or unverifiable hallucinations.
  • ANAH 2.0: Qwen1.5-14B achieves a 5.33% hallucination rate on ANAH-v2, whereas LLaMA-7B reaches 58.16%.The reported difference is associated primarily with language-dependent discrepancies for LLaMA-7B.
  • FaithEval: FaithEval targets noisy, unanswerable, inconsistent, and counterfactual contexts, where models often revert to commonsense or answer confidently instead of acknowledging uncertainty.Its 4.9K problems are evaluated using strict- and non-strict-matching accuracy.

5 Revisiting Existing Benchmarks

The section revisits factuality and hallucination benchmarks, arguing that their evaluation targets and suitability for hallucination assessment must be distinguished. It identifies limitations in TruthfulQA, explains when factuality benchmarks can be adapted, and highlights missing refusal measures.

  • TruthfulQA: TruthfulQA is primarily a factuality benchmark and is not easily adaptable to hallucination evaluation.Its errors often reflect noisy training data rather than inconsistency with training data or input context.
  • TruthfulQA: More than 200 MC1 samples, approximately 25% of the test set, were judged incorrect despite potentially being factually correct.The analysis attributes this partly to log-probability preferences for answer tone rather than factual accuracy.
  • TruthfulQA: TruthfulQA contains problematic ground-truth answers, including claims disputed by studies and responses that penalize nuanced answers.Examples include kindergarten’s developmental importance, Freud’s attribution, and advice about waking sleepwalkers.
  • TruthfulQA: TruthfulQA’s prompts and answers can be subjective, insufficiently nuanced, or effectively require refusal, undermining its factuality measurement.The authors conclude that the benchmark may need revision or updating.
  • Adaptability of factuality benchmarks: SimpleQA and PopQA can be adapted into extrinsic hallucination benchmarks when their data remain within training knowledge and refusal behavior is measured.PopQA could incorporate a refusal-rate measure, while benchmarks using current or post-cutoff knowledge are unsuitable for hallucination evaluation.
  • Adaptability of factuality benchmarks: HaluEval 2.0 reports Micro Hallucination Rate and Macro Hallucination Rate but lacks a measure of refusal.The benchmark covers biomedicine, finance, science, education, and open-domain questions.

6 Related Work

Related work spans hallucination benchmarks, automatic hallucination detection, and factuality metrics. These lines of research address evaluation or detection from different perspectives, including knowledge boundaries, unanswerable questions, and factual accuracy.

  • Taxonomy: Recent hallucination surveys broadened definitions and proposed categorizations, but they conflate hallucination with factuality.Their expanded frameworks nevertheless provide valuable coverage of hallucination research.
  • Extrinsic and Intrinsic Hallucination Benchmark/Dataset: Prior datasets assess whether models recognize knowledge boundaries, including unanswerable mathematics and inherently unanswerable questions.UMWP evaluates arbitrary or unreasonable answers to unanswerable math questions, while SelfAware targets inherently unanswerable questions.
  • Hallucination and/or Factuality Detection: Hallucination and factuality detection develop automatic judges or metrics, a task distinct from hallucination evaluation itself.These systems may support benchmark judging or mitigation processes.
  • Hallucination and/or Factuality Detection: FactScore emphasizes precision, while Veriscore and SAFE develop metrics for evaluating factual accuracy in long-form generated answers.The cited detection work focuses on factuality detection rather than the benchmark construction problem addressed here.

7 Conclusion

The paper concludes by separating hallucination from factuality and organizing hallucinations into intrinsic and extrinsic types. HalluLens combines new extrinsic tasks with existing intrinsic tasks and uses dynamic test generation to address leakage.

  • Conclusion: HalluLens presents a taxonomy distinguishing hallucination from factuality and categorizing hallucinations as extrinsic or intrinsic.The taxonomy is intended to clarify evaluation and support consistent research.
  • Conclusion: The benchmark includes three newly proposed extrinsic hallucination tasks alongside three existing intrinsic hallucination tasks.The extrinsic tasks assess generations against the model’s training data across diverse scenarios.
  • Conclusion: Dynamic test-set generation makes the proposed extrinsic tasks robust against data leakage while maintaining stable model evaluation.The conclusion emphasizes both leakage robustness and evaluation stability.
  • Conclusion: The paper revisits existing benchmarks and emphasizes the need for a distinct extrinsic hallucination benchmark.Appendix materials describe PreciseWikiQA, LongWiki, and NonExistentRefusal in detail.

A Overview of HalluLens

HalluLens combines dynamically created extrinsic hallucination tasks with existing intrinsic tasks and evaluates benchmark properties including leakage robustness, applicability, stability, sensitivity, and reproducibility.

  • Table 5: Table 5 compares HalluLens with factuality benchmarks using leakage robustness, real-world applicability, stability, sensitivity, and reproducibility.HalluLens is highlighted in blue, while factuality benchmarks are shown in gray and treated as outside the hallucination study’s scope.
  • HalluLens: HalluLens’s dynamically created tasks are designed to resist intentional data leakage while preserving low variance across trials.The benchmark also reports high sensitivity for distinguishing intra-model rankings.
  • HalluLens: The proposed tasks span diverse domains and scenarios to improve real-world applicability.The overview contrasts this breadth with narrow optimization for a benchmark.
  • HalluLens: HalluLens includes existing intrinsic hallucination tasks covering multiple scenarios.Dynamic test-set construction is more challenging for intrinsic tasks because they require human-annotated gold answers.

B Discussion and Implementation Details

Models are evaluated with a shared deterministic decoding setup, and reported scores average three trials. Table 6 lists the models used for prompt generation and evaluation.

  • All models use temperature 0 and top-p 1 under the established benchmark decoding setup.
  • Reported scores are averages across three trials, with standard deviations provided for individual tasks.The three trials showed low variance across runs.
  • Table 6 lists the models used for prompt generation and LLM-as-a-judge evaluation.

B.1 PreciseWikiQA

PreciseWikiQA examines model behavior across controlled question difficulties, showing that refusal and hallucination patterns vary by model and difficulty. The evaluation also checks generated gold answers and supports version flexibility through GoodWiki.

  • Evaluation and difficulty breakdown: The PreciseWikiQA results report standard deviations from three evaluation runs, while Figure 6 breaks performance down by question difficulty.
  • Model performance on different difficulty levels: 5,000 GoodWiki documents are sampled across 10 harmonic-centrality difficulty levels for question generation.Each level contributes 500 pages.
  • Model performance on different difficulty levels: Models generally refuse more often on difficult questions, while hallucination and correct-answer rates are generally better on easier questions.Correct-answer rates are higher on easier questions across all tested models.
  • Model performance on different difficulty levels: GPT-4o and Mistral Nemo Instruct tend to refuse less frequently regardless of question difficulty.
  • Model performance on different difficulty levels: Llama-3.1-405B-Instruct maintains a relatively consistent hallucination rate across difficulty levels when it answers rather than abstains.
  • Evaluation validation and data version: Human annotations are used to verify the accuracy of automatically generated question-answer gold answers, and GoodWiki’s Wikipedia version can be adapted to a model’s training set.

B.2 LongWiki

LongWiki evaluates dynamically generated test-set stability and an automated verification pipeline. Results show low cross-run variation, but Wikipedia-only retrieval and verification remain imperfect.

  • Stability of dynamically generated test set: 1.85%, 0.95%, 1.20%, and 0.84% are the average standard deviations across models for recall, precision, F1, and false refusal rate, respectively.These values come from three runs for each model.
  • False acceptance evaluation: Figure 7 compares false acceptance rates for GeneratedEntities and MixedEntities, with bars representing variance across three trials.Model rankings and trends remain consistent across trials, with low inter-trial variance.
  • Evaluation pipeline: The pipeline restricts source references to Wikipedia pages and manually annotates 500 claims to assess validity.This aligns verification with an approximation of training data rather than internet searches.
  • Evaluation pipeline: 76.8% of final verification decisions align with human annotators, while 5% of claims are not verifiable within Wikipedia pages.Wikipedia does not encompass the entirety of models’ training data.
  • Evaluation pipeline: 15.4% of retrieval attempts fail to provide relevant evidence, although verification remains correct in 6.8% of those cases.The evaluator’s internal knowledge may compensate for some retrieval failures.
  • Evaluation assumption: Unverifiable claims are counted as hallucinations because they cannot be verified with the available sources.

B.3.1 MixedEntities

MixedEntities results show substantial variation in false acceptance across models and domains. The strongest reported model avoids hallucinations most consistently, while refusal behavior complicates domain comparisons.

  • Medicine-domain refusal: Gemma family models have 0.00 false acceptance rates for non-existent medicine because they refuse to provide medical advice.
  • Model and domain variation: False acceptance rates vary across models and domains, with low agreement involving the Medicine domain potentially associated with Gemma’s absolute refusals there.
  • Model and domain variation: 11.48% is Llama-3.1-405B-Instruct’s lowest average false acceptance rate across domains.The result indicates strong avoidance of hallucinations for information beyond its training data.
  • Model and domain variation: 94.74% and 90.87% are the highest average false acceptance rates reported for Mistral-7B-Instruct-v0.3 and Mistral-Nemo-Instruct-2407, respectively.
  • Overall findings: Larger models within a family generally perform better on non-existent-entity queries, although this pattern is not consistent across all families.

B.3.2 GeneratedEntities

GeneratedEntities evaluates false acceptance of nonexistent entities, with generator choice and place frequency affecting results. A round-robin design reduces generator-specific bias, while human judgments validate the automatic evaluator.

  • Entity generation: A single entity generator produces unstable false acceptance rates across tested models, motivating a three-generator round-robin approach.Mistral-Nemo-Instruct-2407 generated entities yielded a 74.62% lowest false acceptance to response, while Llama-3.1-405B-generated entities challenged both GPT-4o and Llama-3.1-405B-Instruct.
  • Entity generation: The rankings of tested models remain consistent when GPT-4o is replaced by Claude-3-Sonnet in the round-robin generator group.
  • Place frequency: False acceptance is highest for places with middle-level N-gram frequency, compared with low- and high-frequency places.
  • Place frequency: Middle-frequency places are near the model’s knowledge boundary, where uncertainty leads models to refuse less and hallucinate more.
  • Evaluation: Human assessments agree with automatic abstention judgments for 94.77% of 440 sampled responses from 11 models.

C LLM Factuality Benchmark

The paper distinguishes hallucination from factuality while reviewing factuality benchmarks and their limitations. Existing benchmarks test absolute correctness, changing knowledge, misconceptions, and long-form factuality, but vary in difficulty, saturation, and evaluation design.

  • Conceptual distinction: Hallucination concerns consistency with user input or training data, whereas factuality concerns absolute correctness against established verification sources.
  • TruthfulQA: TruthfulQA evaluates responses to commonly misunderstood questions but has incorrect or outdated gold answers and strict log-probability metrics.
  • TruthfulQA: TruthfulQA’s original results found larger models tended to be less truthful, while GPT-4 reached 60% on MC1.
  • SimpleQA: SimpleQA contains 4,326 short, single-answer fact-seeking questions and classifies responses as correct, incorrect, or not attempted.
  • SimpleQA: Both GPT-4o and Claude-3.5-Sonnet achieved less than 40% accuracy on SimpleQA, while models often answered incorrectly instead of abstaining.
  • FreshQA: FreshQA accuracy ranges from 0.8% to 32.0% in Strict mode and 0.8% to 46.4% in Relaxed mode, with knowledge cutoffs contributing to difficulty.
  • HaluEval2.0: HaluEval2.0 covers 8,770 questions across five domains and emphasizes factuality hallucinations using an automatic evaluator trained on human annotations.
  • LongFact: Long-form factuality evaluation uses SAFE and LongFact’s F1@K, with GPT-4 Turbo attaining F1@64 of 95.0 and F1@178 of 66.4.

D Prompts

The appendix specifies prompts for generating, answering, judging, and evaluating factuality and hallucination tasks. It also defines templates for nonexistent entities and abstention decisions across short- and long-form settings.

  • D.1 PreciseWikiQA: PreciseWikiQA generates concise, objective questions answerable solely from a supplied reference document.
  • D.1 PreciseWikiQA: PreciseWikiQA answerability judging returns the answer when the reference supports it and “unanswerable” otherwise.
  • D.1 PreciseWikiQA: PreciseWikiQA hallucination judgment labels responses as CORRECT, INCORRECT, or UNVERIFIABLE against a provided correct answer.
  • D.1 PreciseWikiQA: Abstention prompts classify whether a chatbot withholds an answer, excluding responses that correct the question or provide an answer.
  • D.2 LongWiki: LongWiki generates concise factual essay questions whose answers should exceed three sentences, using one reference document per question.
  • D.3.1 Prompt template for inference: Inference prompts instantiate entity type, place, and name fields across varied phrasings; MixedEntities uses no place variable.
  • D.3.2 GeneratedEntities: GeneratedEntities prompts request familiar-sounding nonexistent names for businesses, events, or products, separated by commas.
  • D.3.3 Prompt template for evaluation MixedEntities: MixedEntities evaluation asks whether an AI believes a species exists and returns a Boolean JSON field.
Loading 2504.17550v1…