PaperlayerView exact revision on arXiv

Source-linked AI summary

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token

Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun, Jiawei Zhou

arXiv:2603.05465v1cs.CV

TL;DR

VLM hallucination detection is usually performed after generation, creating a need for earlier risk estimation. HALP probes visual and multimodal internal states in one forward pass, finding that pre-generation prediction is effective while the most informative representation varies across architectures.

  • Problem

    Existing hallucination detection methods typically require completed generation, making early assessment costly and untimely.

  • Method

    HALP uses lightweight probes on visual-only, vision-token, and query-token representations extracted during a single pre-generation forward pass.

  • Results

    Across eight VLMs and diverse benchmarks, late query-token states are most predictive for seven of eight models, while visual or vision-token features lead in some architectures.

  • Takeaways & Limitations

    Hallucination risk is detectable before generation, supporting rapid risk assessment and potential early abstention, routing, or adaptive decoding.

  • Takeaways & Limitations

    Reliance on VQA benchmarks may limit generalization to other vision-language tasks and real-world scenarios.

Abstract

from arXiv · show

Hallucinations remain a persistent challenge for vision-language models (VLMs), which often describe nonexistent objects or fabricate facts. Existing detection methods typically operate after text generation, making intervention both costly and untimely. We investigate whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. Across a diverse set of vision-language tasks and eight modern VLMs, including Llama-3.2-Vision, Gemma-3, Phi-4-VL, and Qwen2.5-VL, we examine three families of internal representations: (i) visual-only features without multimodal fusion, (ii) vision-token representations within the text decoder, and (iii) query-token representations that integrate visual and textual information before generation. Probes trained on these representations achieve strong hallucination-detection performance without decoding, reaching up to 0.93 AUROC on Gemma-3-12B, Phi-4-VL 5.6B, and Molmo 7B. Late query-token states are the most predictive for most models, while visual or mid-layer features dominate in a few architectures (e.g., ~0.79 AUROC for Qwen2.5-VL-7B using visual-only features). These results demonstrate that (1) hallucination risk is detectable pre-generation, (2) the most informative layer and modality vary across architectures, and (3) lightweight probes have the potential to enable early abstention, selective routing, and adaptive decoding to improve both safety and efficiency.

1 Introduction

VLMs remain vulnerable to hallucinations, while existing detection and mitigation methods generally require completed or ongoing generation. HALP instead probes pre-generation internal states to estimate hallucination risk before output tokens are produced.

  • VLM hallucinations include nonexistent objects, invented attributes, and unsupported claims that undermine reliability in high-stakes applications.
  • Existing metrics require complete caption generation, while decoding-time interventions still cannot estimate hallucination risk before generation begins.
  • HALP uses a single forward pass over an image-query pair to extract pre-generative internal representations for hallucination prediction.
  • Simple probes estimate hallucination likelihood from visual features, vision-token states, and query-token states without decoding or hallucination labels at inference time.
  • HALP is evaluated on a 10k-example benchmark spanning object presence, reasoning, compositional knowledge, illusion traps, and targeted hallucination cases across eight open-source VLMs.

2 Background and Related Work

VLM hallucinations are outputs inconsistent with visual inputs, reflecting a persistent disconnect between fluent language and visual grounding. Prior work motivates lightweight probing and contrasts pre-generation prediction with costly post-generation assessment.

  • VLM hallucinations include object, attribute, and relational errors that reduce reliability in safety-critical settings.
  • Representation probing trains lightweight classifiers or regressors on internal activations to assess whether specific properties are encoded.
  • Prior studies found truthfulness and hallucination-related information in language-model internal states before generation.
  • Post-generation methods require complete sequence generation, making them computationally expensive and unsuitable for real-time applications requiring early intervention.

3 HALP: Hallucination Prediction via Pre-Generation Probing

HALP probes internal VLM representations before decoding to predict hallucination risk. It labels generated responses, extracts visual, vision-token, and query-token features, and trains lightweight probes on each representation.

  • 3.1 Vision-Language Model Preliminaries: VLMs process images through a vision encoder, project visual features into language-model space, concatenate them with query tokens, and pass the sequence through a decoder.
  • 3.1 Vision-Language Model Preliminaries: HALP asks whether internal states encode hallucination propensity before text generation and intercepts representations at three stages of the multimodal pipeline.
  • 3.2 Hallucination Evaluation: Hallucination labels are binary indicators assigned after standard response generation by comparing outputs with reference answers through automated LLM judging.
  • 3.3 Pre-Generation Feature Extraction: Visual features are pooled encoder outputs, vision-token states come from final visual positions, and query-token states are final multimodally contextualized decoder positions.
  • 3.4 Probing: A separate lightweight three-layer MLP probe is trained for each representation and layer, producing a score from 0 to 1 where higher values indicate greater hallucination likelihood.

4 Data and Experiments

The experiments evaluate HALP across a broad 10,000-sample VQA collection and eight contemporary VLMs. AUROC provides the threshold-independent primary measure, with standardized judging and stratified training-validation splits.

  • Data: The dataset contains 10,000 image-question pairs from six VQA benchmarks covering diverse vision-language challenges.
  • Data: Attribute Recognition and Visual Understanding comprise 60% of samples, while Spatial Reasoning contributes 17.7%.
  • Models: The evaluated models include Gemma3-12B, LLaVA-1.5-8B, Llama-3.2-11B, Phi4-VL-5.6B, Molmo-7B, Qwen2.5-VL-7B, SmolVLM2-2.2B, and FastVLM-7B.
  • Evaluation: Post-generation hallucinations are assessed with an LLM-as-a-judge framework to maintain consistent labeling across the 10,000-sample dataset.
  • Evaluation: AUROC is the primary metric, with 0.5 representing random chance and 1 representing perfect prediction.

5 Results and Analysis

HALP detects hallucination risk before generation by probing internal VLM representations, with query-token states generally providing the strongest signals. Results also show that optimal representations and layers vary across architectures and error types.

  • Main Results: Seven of eight models achieve peak AUROC between 0.90 and 0.94 using deep query-token representations.Query-token representations dominate across the evaluated models.
  • Main Results: Visual-feature performance varies substantially by architecture, reaching 0.7873 AUROC for Qwen2.5-VL and 0.7703 for Llama-3.2-11B but only 0.6108 for LLaVA-Next and 0.6166 for Phi4-VL.FastVLM-7B instead peaks at 0.7093 AUROC with vision-token features.
  • Main Results: 0.8898 average AUROC makes query-token vectors more reliable than vision tokens at 0.7090 and visual features at 0.6955.The comparison identifies post-fusion query states as the strongest general intervention point before text generation.
  • Practical Implications: HALP supports pre-generation risk assessment by extracting representations at potentially optimal layers rather than defaulting to final outputs.The reported layer-wise patterns are framed as a way to optimize deployment efficiency while retaining robust prediction from mid-to-deep query states.
  • Layer-wise Performance: Query-token performance generally improves from shallow to deep layers, whereas vision-token performance remains comparatively stable and limited.Most vision-token results plateau around 0.65-0.70 AUROC across layers.
  • Layer-wise Performance: 0.9349 AUROC is achieved by Gemma3-12B at the final layer, while Molmo-V1 peaks at 0.9365 AUROC at L/2.Optimal query-token extraction depth therefore differs across models.
  • Hallucination Types and Application Domains: Attribute-Related and Other errors are the most frequent and hardest to detect, while Object-Related errors yield 0.878 AUROC with query-token probes.Relationship errors occur at 6.06% and have 0.708 AUROC; query-token probes show their largest gains over visual features for Attribute and Relationship errors.
  • Hallucination Types and Application Domains: Temporal & Video has a 46.9% error rate and 0.456 QT AUROC, whereas Attribute Recognition has a 7.1% error rate and 0.863 QT AUROC.These domain results identify substantially different hallucination risk and prediction performance across application areas.

6 Discussion

HALP reveals that hallucination-predictive representations and optimal extraction layers vary substantially across VLM architectures. Its pre-generation risk scores can support lightweight monitoring and control policies, with overhead below 1% of full generation in the reported testbed.

  • Architectural heterogeneity: The most predictive representation type and layer are not universal across VLM architectures.Some models rely more on visual-only features, while others rely on post-fusion query-token states.
  • Practical use: HALP scores can serve as control signals for early refusal or deferral and selective routing without modifying the underlying VLM.Inputs can be ranked by predicted hallucination risk so high-risk cases receive refusal, deferral, or alternative processing.
  • Efficiency: 10–15 ms is the reported probe inference time, with relative overhead below 1% compared with full generation on an NVIDIA RTX 4090.The comparison uses a 3-layer MLP and typical generation of 100 tokens.

7 Conclusion

HALP provides pre-generation hallucination prediction across eight modern VLMs and diverse multimodal benchmarks using internal representations extracted in a single forward pass. The results show that hallucination risk is detectable before decoding, while the most informative signals vary across architectures, enabling rapid risk assessment without costly generation.

  • Conclusion: HALP predicts hallucination risk before decoding by probing global visual, decoder-level vision, and query-token states in a single forward pass.The framework examines these representations across eight modern VLMs and diverse multimodal benchmarks.
  • Conclusion: Hallucination risk is detectable before generation, but the most informative internal signals vary across architectures.The approach supports rapid, real-time risk assessment without costly decoding.

Ethical Considerations

HALP is presented as a safety-oriented tool for assessing VLM reliability and reducing risks from generated false content. The paper also emphasizes misuse risks, evaluation and infrastructure limitations, and the need for human judgment in high-stakes applications.

  • Beneficial applications: Early hallucination detection can support AI safety and reliability by helping prevent misinformation and reduce harm from false generated content.The probing approach is described as transparent and usable without access to proprietary architectures.
  • Misuse potential: Adversaries could attempt to circumvent hallucination detectors or exploit detection mechanisms to craft more sophisticated deceptive content.The paper identifies this as a potential misuse of the technology.
  • Evaluation concerns: Reliance on existing VQA benchmarks may limit representativeness across diverse populations and scenarios.The paper recommends more diverse evaluation frameworks and human annotation validation.
  • Accessibility: Internal-representation extraction and probing require substantial computational infrastructure, which may hinder adoption in resource-constrained settings.This creates an accessibility barrier despite the paper's commitment to reproducible research.
  • Human oversight: Hallucination detection should complement, not replace, human judgment in high-stakes applications.The paper links this boundary to responsible deployment of VLMs in real-world settings.

Limitations

The paper identifies limitations involving benchmark bias, judge-model bias, deployment cost, coverage of nuanced hallucinations, and the scale of evaluated VLMs. These constraints limit how broadly the reported findings should be generalized.

  • Scope and evaluation: VQA benchmark reliance may introduce dataset bias and limit generalization to other vision-language tasks and real-world scenarios.This is the first limitation identified in the paper's limitations discussion.
  • Scope and evaluation: GPT-4-based judging may introduce bias and occasional misclassifications that affect hallucination-label quality.Lenient criteria do not eliminate possible judge-model bias.
  • Deployment constraints: High computational costs for extracting and probing internal representations may hinder deployment in resource-constrained settings.The limitation concerns both representation extraction and probing.
  • Coverage and scale: The taxonomy may miss nuanced context-dependent or culturally specific hallucinations, and larger VLMs remain underexplored.The experiments use small-to-medium VLM sizes and do not cover every possible hallucination case.

A Model Analysis

Across models, query-token states are usually the strongest hallucination predictors, but representation usefulness and optimal depth vary substantially by architecture. Visual-only or vision-token features dominate in specific outliers, including FastVLM-7B and parts of SmolVLM-2.2B.

  • Layer-wise behavior: Gemma3-12B shows query-token performance rising from 0.7165 at Layer 1 to 0.9349 at Layer L, while vision-token performance declines with depth.Its visual features score 0.6736, and vision tokens fall from 0.6698 to 0.5956 in the cited progression.
  • Architectural outlier: 0.7093 AUROC is FastVLM-7B’s best score, achieved by vision-token representations at Layer L/2 rather than query-token or visual features.Vision tokens outperform both alternatives in this architecture.
  • Architectural outlier: 0.6136–0.6715 AUROC summarizes FastVLM-7B’s query-token performance, which degrades in deeper layers and remains below its vision-token best score.This contrasts with the query-token dominance observed in most other evaluated architectures.
  • Visual representations: SmolVLM-2.2B’s visual features reach 0.7238 AUROC, outperforming vision tokens across layers despite the model’s compact size.Vision-token performance peaks at 0.6894 AUROC and remains relatively flat across depth.
  • Visual representations: 0.7873 AUROC is Qwen2.5-VL-7B’s visual-feature score, while its query-token peak reaches 0.9215 AUROC at Layer 3L/4.This model therefore combines a strong visual baseline with stronger mid-to-deep query-token detection.

B Dataset Composition and Characteristics

The evaluation set contains 10,000 image-question pairs sampled from six VQA benchmarks, covering varied domains, answer formats, and hallucination categories. Labels are produced by a lenient GPT-4 judge, while model results reveal contrasting representation strategies across architectures.

  • Dataset construction: 10,000 image-question pairs form the evaluation set, sampled from six established VQA benchmarks.The benchmark is designed to cover multimodal reasoning challenges and hallucination patterns.
  • Domain composition: Attribute Recognition and Visual Understanding comprise nearly 60% of the dataset, with specialized domains including Temporal & Video and Math & Calculation.The reported specialized-domain counts are 170 Temporal & Video samples and 628 Math & Calculation samples.
  • Hallucination composition: Object-Related errors dominate the hallucination taxonomy with 3,493 instances, including 2,810 fabricated-object cases and 1,720 Relationship hallucinations.The category distribution is described as broad coverage of output error patterns.
  • Answer composition: Yes/No questions dominate the answer formats with 6,568 samples, or 65.7%, followed by 2,008 Open-Ended responses and 731 Unanswerable questions.Numeric responses account for 657 samples, while Selection tasks account for 36.
  • Labeling protocol: GPT-4 labels a response as hallucinated only when it invents nonexistent entities, contradicts ground truth, or provides completely incorrect information.Accurate paraphrases and correct added details are not flagged.
  • Model contrasts: Gemma3-12B concentrates hallucination signals in late query-token states, whereas FastVLM-7B preserves stronger signals in vision tokens.Gemma query tokens rise from 0.7165 to 0.9349, while FastVLM vision tokens peak at 0.7093 and query tokens decline from 0.6715 to 0.6136.

G Threshold Analysis for Early Refusal

Threshold analysis evaluates early refusal using probe scores from different representation stages. Query-token probes provide the broadest reported refusal coverage, while visual features offer the earliest intervention point and vision tokens provide an intermediate option.

  • Query-token refusal: Seven of eight models achieve AUROC greater than 0.89 with query-token representations, enabling early refusal of 57% to 80% of hallucinations.At optimal thresholds from 0.1 to 0.4, reported precision ranges from 48% to 57%.
  • Earliest intervention: Pure visual features enable the earliest intervention before LLM decoder processing, although their precision is lower than query-token probes.The reported strongest visual-feature scores are 0.787 AUROC for Qwen2.5-VL-7B and 0.770 for Llama-3.2-11B.
  • Vision-token refusal: Vision tokens detect 23% to 66% of hallucinations with precision between 15% and 36%, with Phi4-VL achieving the strongest vision-token performance.The analysis uses vision-token representations at layer L.
  • Deployment guidance: Query-token probes with thresholds τ between 0.1 and 0.2 maximize hallucination coverage for safety-critical applications, reaching up to 80% before token generation.The deployment guideline specifically cites Molmo-7B for this upper coverage figure.
  • Deployment guidance: Thresholds τ between 0.2 and 0.4 provide a balanced deployment setting with 48% to 57% precision and 57% to 71% recall.These values are reported for query-token probes.
  • Practical implication: HALP scores can serve as control signals for early refusal policies, trading response coverage for higher reliability without text generation.The threshold analysis operationalizes this trade-off across models and representation types.
Loading 2603.05465v1…