PaperlayerView exact revision on arXiv

Source-linked AI summary

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani

arXiv:2608.30480v1cs.CVcs.LG

TL;DR

Object hallucination detectors can be source-confounded: internal support for an object may reflect scene plausibility, related visual cues, or generated-prefix continuation rather than object-specific visual evidence. VisER addresses this with a training-free two-sided metric that combines Visual Evidence and Visual Reliance, and it consistently improves AUROC and AUPR over baselines across multiple LVLMs and datasets. The method is scoped primarily to object-existence hallucinations and requires access to internal LVLM representations.

  • Problem

    Existing internal signals may strongly support hallucinated objects without distinguishing object-specific visual evidence from scene, cue, or generated-prefix support.

  • Method

    VisER combines Visual Evidence of object-specific image-token support with Visual Reliance comparing image-derived and generated-prefix support, without additional object-level verification generations.

  • Results

    VisER consistently improves AUROC and AUPR over strong baselines across multiple LVLMs and datasets, with complementary contributions from both signals.

  • Takeaways & Limitations

    VisER provides an effective post-hoc detector for visually unsupported object mentions in open-ended generations.

  • Takeaways & Limitations

    VisER primarily evaluates COCO/VOC-style object-existence hallucinations and requires internal LVLM representations, limiting direct assessment of other factual errors and fully black-box APIs.

Abstract

from arXiv · show

Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.

1 Introduction

VisER frames object hallucination detection as a source-validation problem: high internal support may reflect visual evidence, scene plausibility, misleading cues, or generated-prefix continuation. Its two-sided score combines object-specific Visual Evidence with image-versus-prefix Visual Reliance and improves over baselines across LVLMs and benchmarks.

  • Object hallucinations are difficult to detect because absent objects can remain fluent, contextually plausible, and naturally embedded in otherwise correct descriptions.
  • Existing internal signals can indicate visual grounding or merely plausibility, association, scene priors, related visual cues, or generated-prefix continuation.
  • VisER measures Visual Evidence by testing whether object-context compatibility is backed by object-specific image-token evidence.
  • VisER measures Visual Reliance by comparing image-derived support with support from the generated text prefix.
  • VisER combines both signals into a training-free, source-aware grounding score without model training or additional object-level verification generations.
  • VisER consistently improves AUROC and AUPR over existing baselines, with complementary benefits from both components.

2 Related Work

Object hallucination detection methods range from externally supervised or verified evaluation to training-free internal-signal detectors. Related work shows that likelihood, representations, image-text similarity, co-occurrence, instruction frequency, and scene priors can support hallucinated objects without direct object evidence.

  • External evaluation methods use annotations, references, or auxiliary judge models, but are less suitable when object-level references are unavailable.
  • Training-free detectors use inference-time signals including object-token likelihood, image-hidden-state confidence, internal representations, and global or local image-text similarity.
  • Hallucinated objects can appear plausible through object co-occurrence, instruction frequency, scene priors, statistical bias, or unimodal priors.

3 Method

VisER assigns each generated object mention a grounding score by combining object-specific visual evidence with the relative contribution of image- versus prefix-derived support. This two-sided design suppresses scene- or cue-driven compatibility and favors objects supported directly by image tokens and more strongly by the image than the generated prefix.

  • VisER overview: VisER evaluates each generated object mention using Visual Evidence (VE) and Visual Reliance (VR), whose combination forms the final grounding score.VE captures object-specific image-token evidence, while VR compares image-derived support with generated-prefix support.
  • Design rationale: The method is training-free and avoids additional object-level verification generations.Its score is built from internal image-token and generated-prefix support proxies.
  • Visual Evidence: Visual Evidence gates object-context compatibility by object-specific evidence aggregated across image tokens.Weak object-specific image-token evidence reduces the contribution of compatibility to VE.
  • Visual Reliance: Visual Reliance compares weighted image-derived support with support from selected generated-prefix tokens.Image-token weights reflect object-specific evidence, while prefix support is estimated from tokens most similar to the object embedding.
  • Visual Reliance: A larger VR indicates that object support is more image-derived than prefix-derived, reducing the influence of text-reinforced mentions.VR is a bounded transform of a smoothed image-versus-prefix support ratio.
  • Why VE and VR are complementary: The two components address complementary false-support cases: low VR flags scene- or text-supported “toothbrushes,” while low VE flags cue-supported “ski.”The examples show why high compatibility or high image-versus-text support alone can remain misleading.

4 Experiments

VisER is evaluated across multiple LVLM architectures, benchmarks, baselines, ablations, and efficiency settings. It achieves strong detection performance, benefits from both visual components, and avoids additional object-level verification generations.

  • Main results: VisER achieves the best average performance across MSCOCO and Pascal VOC, reaching 84.89 AUROC and 95.82 AUPR on MSCOCO and 83.89 AUROC and 93.88 AUPR on Pascal VOC.On MSCOCO, it improves over GLSim by 3.63 AUROC points.
  • Extended models: Across extended models, VisER reaches 81.66 average AUROC on MSCOCO, improving over the strongest average baseline, PAS, by 4.24 points.The largest gains are 6.15 points on Qwen2.5-VL and 5.36 points on InternVL3.
  • Comparison with POPE: VisER improves over POPE from 67.77 to 78.67 ACC and from 54.75 to 79.42 hallucinated-object F1, while reducing verification time from 3.19s to 2.36s.Unlike POPE, VisER does not require additional yes/no generations.
  • Sensitivity analysis: The combined VisER score peaks at 85.66 AUROC around α = 0.4, outperforming Visual Reliance alone at 78.22 and Visual Evidence alone at 83.88.The results indicate complementary information from the two components.
  • Sensitivity analysis: VisER remains above PAS and GLSim across the tested K range, improving from 84.85 AUROC at K = 2 to 85.66 at K = 10 before saturating.Later image layers provide stronger evidence, with the best performance at layer 32.
  • Ablation study: Evidence gating produces consistent AUROC gains by filtering object-context compatibility through object-specific image-token evidence.Removing prefix support leaves VE unchanged but reduces VR to chance-level discrimination, while blanking or shuffling images reduces VE and full VisER.

5 Conclusion

VisER is a training-free object-level hallucination detector that addresses source confounding by combining Visual Evidence with Visual Reliance. Across multiple LVLMs and datasets, it improves AUROC and AUPR over strong baselines, with complementary ablation signals.

  • VisER combines Visual Evidence from object-specific image-token evidence with Visual Reliance comparing image-derived and generated-prefix support.Together, these signals address source confounding in object-level hallucination detection.
  • Across multiple LVLMs and datasets, VisER consistently improves AUROC and AUPR over strong baselines.
  • Ablations show that Visual Evidence and Visual Reliance provide complementary signals for detecting visually unsupported object mentions.

Limitations

VisER’s evaluation and representation choices impose clear scope boundaries. It targets object-existence hallucinations in open-ended descriptions, relies on internal LVLM representations, and does not directly assess several other factual-error types.

  • VisER evaluates object-existence hallucinations in open-ended image descriptions rather than open-vocabulary hallucinations or other fine-grained factual errors.The evaluation primarily follows COCO/VOC-style object categories and excludes attributes, relations, counts, actions, and similar errors.
  • The method requires internal LVLM representations, making it more suitable for white-box or gray-box settings than fully black-box APIs.
  • Object-token representations may be sensitive to multi-token expressions, subword tokenization, and repeated object mentions.
  • Baseline Scoring Functions: Baseline scores are oriented so larger values indicate stronger visual grounding and smaller values indicate higher hallucination likelihood.
  • Baseline Scoring Functions: Contextual Lens and GLSim measure compatibility between generated object representations and visual representations, using strongest or selected local visual-token evidence.GLSim combines global and local image-text compatibility, while Contextual Lens uses the strongest compatible visual-token representation.
  • Baseline Scoring Functions: PAS measures text-prefix attention, whereas Entropy, Internal Confidence, and SVAR use uncertainty, visual-token evidence, or image-token attention as grounding-oriented signals.PAS treats stronger prefix attention as evidence that object generation may be text-driven; Entropy uses negative entropy, and SVAR measures image-token attention.

B Implementation Details

VisER is applied post hoc to generated captions without modifying decoding. The implementation uses bounded generation, model-specific hyperparameters, and first-token scoring for multi-token or repeated object mentions.

  • VisER is a post-hoc object-level detector applied to generated captions and does not modify the decoding process.Each model generates at most 512 new tokens, with greedy decoding as the default main-experiment setting.
  • Greedy decoding is used by default for determinism and comparability, while additional decoding strategies are evaluated in ablations.
  • Hyperparameters ℓI, ℓT, K, and α are fixed per model, respectively controlling image-token evidence, text/object representation, prefix-token selection, and signal balancing.
  • For multi-subword object words, VisER uses the first matched token, and for repeated object words it scores only the first generated occurrence.The first occurrence is used because it often captures the object’s core semantic meaning.
  • Table 7 reports the grounded and hallucinated object mentions evaluated using CHAIR object-level labels.

C Additional Ablation Studies

The multi-token aggregation ablation compares first-token, last-token, and all-token averaging strategies. First-token aggregation performs best on both models, while all-token averaging remains close.

  • The ablation compares first-token, last-token, and all-token averaging strategies for multi-token object representations.
  • The first-token strategy achieves the highest AUROC on both models, while averaging over all object tokens remains close.
  • These results indicate that VisER is not highly sensitive to aggregation strategy and support first-token representation as a simple default.

C.2 Evidence Gate Distribution

The evidence gate is designed as a soft validator that modulates compatibility using object-specific visual evidence rather than acting as a standalone detector. Empirically, grounded objects receive higher gate values than hallucinated objects, and VisER remains strong across decoding strategies.

  • Evidence gate design: The evidence gate softly modulates object-context compatibility according to object-specific visual evidence.It does not completely suppress compatibility and is not intended to serve as a standalone detector.
  • Gate distribution: Grounded objects receive consistently higher mean gate values than hallucinated objects on both model scales.The empirical gate distribution is non-degenerate, providing meaningful variation in object-specific visual evidence.
  • Decoding robustness: VisER remains consistently strong under beam search, top-k sampling, and nucleus sampling in addition to greedy decoding.The evaluated settings use Nbeams = 2, k = 10, and p = 0.9, respectively.
  • Score distributions: VisER produces a clearer separation between grounded and hallucinated objects than NLL in object-level score distributions.NLL scores remain more concentrated and overlapping, limiting token likelihood as a visual-faithfulness signal.

E.2 Examples of Source-Confounded Support

Qualitative examples show that VisER detects hallucinated objects that baseline support signals miss when contextual plausibility or generated-prefix continuation creates source-confounded support. Its two component scores suppress both scene-prior-driven and weakly evidenced hallucinations.

  • Baseline failures: VisER correctly detects hallucinated objects that PAS, GLSim, or both incorrectly classify as grounded.The examples include cases where both baselines fail, as well as cases where only PAS or GLSim fails.
  • Source confounding: The examples illustrate that contextual plausibility, object co-occurrence, and generated-prefix continuation can mislead baseline support signals.VisER is designed to distinguish these sources from direct visual grounding.
  • Scene-prior-driven support: Scene plausibility can make hallucinated objects such as “sandwich” and “clock” appear supported despite weak object-specific visual evidence.VisER lowers their final scores because contextual compatibility alone does not provide reliable visual evidence.
  • Insufficient evidence: Related visual cues can support hallucinated mentions such as “chairs” and “train” without making their evidence sufficiently reliable.“Chairs” receives moderate support from a dining scene, while “train” is associated with an elevated rail structure; both remain below the final VisER threshold.

F Source-Confounding Analysis

The analysis formalizes two complementary false-support regimes: evidence-limited hallucinations, where compatibility lacks object-specific image evidence, and prefix-dominated hallucinations, where the generated prefix explains the object better than the image. Under nonnegative margins in both components, VisER ranks grounded objects above hallucinated ones.

  • Complementary false-support regimes: The source-validation analysis distinguishes evidence-limited and prefix-dominated hallucinations as complementary false-support regimes.The former concerns weak object-specific image-token evidence; the latter concerns stronger support from the autoregressive prefix than from the image.
  • Evidence-limited regime: High contextual compatibility alone cannot produce a high Visual Evidence score when object-specific image-token evidence is weak.The compatibility term is restricted by the object-specific evidence gate.
  • Prefix-dominated regime: Visual Reliance is small when autoregressive prefix support dominates image-derived support.The analysis assumes I_og ≥ κT_og, T_oh ≥ κI_oh, and κ > 1 to obtain contrasting reliance bounds for grounded and hallucinated objects.
  • Combined score: If ΔE ≥ 0, ΔR ≥ 0, and at least one inequality is strict, VisER gives a positive grounded-over-hallucinated margin for any α ∈ (0, 1).This formalizes the complementarity of Visual Evidence and Visual Reliance.
Loading 2608.30480v1…