PaperlayerView exact revision on arXiv

Source-linked AI summary

From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models

Hyeonggeun Han, Taehyun Cho, Sangwoo Hong

arXiv:2610.17706v1cs.CVcs.AI

TL;DR

Object hallucination challenges LVLM reliability because generated claims can be unsupported by the image yet linguistically plausible. EGVCA compares claim likelihood under observed and visual-null inputs to guide semantic claim admission. Across three backbones, it reduces CHAIR hallucination, improves POPE performance, and preserves or improves caption quality and MME performance.

  • Problem

    Object hallucination produces object-centric claims unsupported by the image, challenging the reliability of large vision-language models.

    Source: §1 · p. 1

  • Method

    EGVCA compares a claim’s likelihood under the observed image and a visual-null input, using the contrast to guide claim admission rather than alter candidate generation.

    Source: §1 · p. 2

  • Results

    Across three LVLM backbones, EGVCA reduces CHAIR hallucination, improves POPE performance, and preserves or improves caption quality and MME performance.

    Source: §1 · p. 2, §5.2 · pp. 6–7, §6 · p. 8

  • Takeaways & Limitations

    The results support claim-level admission as an approach for controlling visual hallucination across open-ended generation and binary visual question answering.

    Source: §5.2 · p. 7, §6 · p. 8

    EGVCA requires additional original and visual-null scoring, and its experiments cover three LVLM backbones and widely used English-language benchmarks; it does not guarantee factuality in high-stakes settings.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large vision-language models often exhibit object hallucination, generating claims that are not supported by the input image. To mitigate this issue, we propose Evidence-Guided Visual Claim Admission (EGVCA), a training-free framework that admits a semantic claim only when it receives sufficient image-dependent support. EGVCA measures this support by comparing the claim likelihood under the observed image and a visual-null input. For captioning, it controls semantic expansion without altering token generation. For binary visual question answering, it combines atomic visual support with the image-induced change in the model's native Yes--No margin. Experiments on CHAIR, POPE, and MME across three LVLM backbones show that EGVCA substantially reduces object hallucination while preserving caption quality. It achieves the highest POPE F1 in eight of nine backbone--sampling settings and improves the native MME total score on all three backbones. These results demonstrate the effectiveness of claim-level admission as a simple alternative to token-level distribution correction.

1 Introduction

Object hallucinations are visually unsupported object-centric claims that may remain linguistically plausible, challenging LVLM reliability. EGVCA evaluates image-dependent support to regulate semantic claim admission without changing candidate generation, and experiments report reduced hallucination with competitive generation quality.

1 Introduction

  • Object hallucinations include absent objects, incorrect attributes, and unsupported relations, and can remain linguistically plausible despite lacking visual grounding.Their mitigation is presented as important for faithful scene interpretation and trustworthy real-world deployment.Source: §1 · p. 1
  • Existing training-free approaches contrast visual contexts or use internal decoding signals, but generally mitigate hallucination by modifying token distributions or intervening during decoding.The paper notes that such correction may also alter grounded information and favor safer but less specific responses.Source: §1 · pp. 1–2
  • EGVCA compares claim likelihood under the observed image and an all-zero visual-null input, using the contrast to estimate image-dependent evidence.The comparison aims to separate image-supported plausibility from plausibility induced by linguistic context alone.Source: §1 · p. 2
  • Across three LVLM backbones and CHAIR, POPE, and MME, EGVCA reduces object hallucination while maintaining competitive generation quality.Unlike token-level correction, it uses evidence to regulate commitment to claims while leaving candidate generation unchanged.Source: §1 · p. 2

2 Related Work

Related work describes object hallucination as a visual-grounding problem and surveys training-free inference-time strategies. EGVCA shares their goal of improving faithfulness but uses visual evidence to govern semantic claims rather than directly correcting next-token predictions.

Object Hallucination in LVLMs

  • Object hallucination (OH) is object-centric content not faithfully grounded in the given image, appearing as incorrect object identification or erroneous attributes, quantities, or relations (Rohrbach et al., 2018; Li et al., 2023b; Wang et al., 2023).Prior studies link OH to biased object co-occurrences, strong language priors, and insufficient vision-language alignment.Source: §2 · p. 2

Training-Free Decoding for Hallucination Mitigation

  • Training-free inference-time methods aim to improve visual faithfulness without modifying the underlying LVLM.The review distinguishes such methods from approaches requiring retraining.Source: §2 · p. 3
  • VCD contrasts original and distorted visual inputs, OPERA penalizes over-trusted attention, DeCo corrects predictions using intermediate-layer visual information, and MoLE combines layer-wise experts (Leng et al., 2024; Huang et al., 2024; Wang et al., 2025).The passage also notes analyses finding greater reliance on linguistic context and weaker visual dependency later in generation (Min et al., 2025; Anand et al., 2026).Source: §2 · p. 3
  • EGVCA shares the goal of deriving signals from a frozen model but uses original-versus-null likelihood contrast as semantic-claim evidence instead of modifying next-token predictions.It uses that evidence to determine whether a claim should be admitted.Source: §2 · p. 3

3 Claim-Level View of Object Hallucination

This section treats hallucination as an insufficiently grounded semantic commitment rather than solely a token-level error. It compares claim likelihood under the observed image and a visual-null input, then uses the contrast as admission evidence in open-ended generation and binary VQA.

3 Claim-Level View of Object Hallucination

  • A response is represented as a sequence of semantic claims, each describing an object’s existence, a visible object’s attribute, or a relation among visual entities.Each claim is conditioned on the preceding claims in the response.Source: §3 · p. 3
  • A claim’s likelihood alone cannot establish visual grounding because linguistic plausibility or learned priors can make it likely even with little image evidence.The method therefore distinguishes overall plausibility from the increase attributable to the observed image.Source: §3 · p. 3
  • The log-likelihood contrast compares the same claim under the observed image and a visual-null input with task-relevant visual information removed.A positive contrast means the observed image increases the model-assigned likelihood relative to the visual-null input.Source: §3 · p. 3
  • The contrast serves as evidence for admitting semantic commitments rather than as a signal to modify next-token probabilities.The principle applies to open-ended generation and binary visual question answering, though their admission mechanisms differ.Source: §3 · pp. 3–4

4 Evidence-Guided Visual Claim Admission

EGVCA estimates image-dependent claim evidence from likelihood contrast against a visual-null input, then regulates semantic commitments through prefix admission for open-ended generation and decision adjustment for binary VQA.

4.1 Image-Attributable Visual Evidence

  • Image-attributable evidence is the per-token likelihood contrast between the observed image and a same-shaped zero-valued visual-null input, interpreted relative to that reference rather than as absolute correctness.A positive score means the observed image increases average claim likelihood; a negative score means it decreases it.Source: §4.1 · p. 4

4.2 Claim Admission for Open-Ended Generation

  • Open-ended generation retains the first claim, then tests subsequent sentence-like claims in order against a minimum evidence threshold; the first failure discards that claim and all later claims.Stopping at the first rejection preserves the preceding context for retained claims; the threshold controls how much of the response is admitted beyond the first claim.Source: §4.2 · p. 4

4.3 Evidence-Guided Claim Admission for Binary VQA

  • Question components are objects, attributes, or relations extracted by a rule-based parser and scored under a neutral visual-description context; equivalent forms use the highest evidence score, and aggregation averages component scores.The averaging reduces dependence on the number of components.Source: §4.3 · p. 5
  • Because component evidence alone does not establish the complete proposition, proposition-level evidence measures how the observed image changes the model’s relative preference for Yes versus No.This signal is the difference between the native margin under the observed image and the corresponding margin under the visual-null input.Source: §4.3 · p. 5
  • Binary VQA combines evidence for question components and the complete proposition to adjust the native Yes–No margin; the final decision depends on both the native margin and evidence adjustment.The joint evidence score is equally weighted, and evidence above or below its reference level shifts the margin toward Yes or No, respectively.Source: §4.3 · p. 5

5 Experiments

Across three LVLM backbones, EGVCA reduces open-ended object hallucination and improves binary object-presence prediction, while preserving or improving caption quality and MME performance. Ablations show that admission thresholds trade off hallucination reduction and caption quality, and that visual evidence works best when combined with the model’s native decision margin.

5.1 Experimental Setup

  • Experiments use three frozen LVLM backbones, CHAIR-500 captioning, POPE’s three sampling settings, and the full MME benchmark, comparing EGVCA with standard decoding and inference-time baselines.CHAIR uses 500 MS COCO validation images per run with three seeds; evaluation includes caption-quality metrics, POPE accuracy and F1, and MME perception, cognition, and total scores.Source: §5.1 · pp. 5–6

5.2 Experimental Results

  • CHAIR-S is lowest with EGVCA on all three backbones; on LLaVA-1.5, its score is 10.87, 5.33 percentage points below DeCo.On MiniGPT-4 and mPLUG-Owl2, CHAIRS falls from 20.33 to 12.73 and from 21.47 to 12.60, respectively.Source: Table 1 · p. 6, §5.2 · p. 6
  • EGVCA achieves the highest POPE F1 in eight of nine backbone–setting combinations and improves F1 over native predictions.On MiniGPT-4, Popular F1 rises from 69.94 to 77.92; on mPLUG-Owl2, EGVCA leads in accuracy and F1 across all settings.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • EGVCA improves MME Total over the native model on all three backbones, with the largest gain, 249.21 points, on MiniGPT-4.It also achieves the best matched BLEU-4, METEOR, ROUGE-L, and SPICE scores on all three backbones.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 Ablation and Analysis.

  • DoLa and HALC yield unstable binary outputs with MiniGPT-4 on POPE and MME under the matched protocol, indicating sensitivity to backbone–decoding interaction.Source: §5.3 · p. 7
  • On LLaVA-1.5, increasing τ consistently reduces CHAIRS and CHAIRI; τ = 0.25 is selected because it substantially reduces hallucination with negligible SPICE difference from τ = 0.125.At τ = 0.25, CHAIRS falls from 21.93 to 10.87 and CHAIRI from 7.07 to 5.77 versus greedy generation, while SPICE rises from 21.44 to 22.19; more aggressive thresholds lower SPICE.Source: §5.3 · p. 8, Table 3 · p. 8
  • Visual support alone improves MME but degrades POPE, while native-margin gain alone improves POPE and leaves MME nearly unchanged; the full admission head performs best jointly.Using joint evidence as an independent classifier performs poorly, particularly on MME; combining it with the native margin raises POPE F1 from 84.11 to 86.48 and MME Total from 1681.92 to 1740.13.Source: §5.3 · p. 8, Table 4 · p. 8
  • On 200 CHAIR images, EGVCA processes 21.90 final-output tokens per second using 15.27 GB peak memory, exceeding DoLa, OPERA, and HALC in throughput while matching greedy decoding’s memory use.The reported end-to-end timing includes candidate generation, claim segmentation, original and visual-null claim scoring, and claim admission.Source: Table 5 · p. 8, §5.3 · p. 8

6 Conclusion

EGVCA admits visual claims according to their evidence under the observed image relative to a visual-null input. Across three LVLM backbones, it reduces CHAIR hallucination and improves POPE, caption-quality, and MME performance.

6 Conclusion

  • EGVCA is a training-free framework for admitting visual claims according to their evidence under the observed image relative to a visual-null input.Source: §6 · p. 8
  • Across three LVLM backbones, EGVCA reduces CHAIR hallucination, improves POPE performance, and preserves or improves caption quality and MME performance.Source: §6 · p. 8
  • The reported results demonstrate the effectiveness of claim-level admission for controlling visual hallucination.Source: §6 · p. 8

Limitations

EGVCA requires additional scoring and is evaluated on a limited set of backbones and English-language benchmarks; its outputs may retain errors and are not a high-stakes factuality guarantee.

Limitations

  • EGVCA requires additional original and visual-null scoring, resulting in higher inference cost than standard greedy decoding.Source: Limitations · p. 8
  • Experiments cover three LVLM backbones and widely used English-language benchmarks.Source: Limitations · p. 8
  • Because EGVCA mitigates rather than verifies outputs, residual errors remain and the method should not be treated as a factuality guarantee in high-stakes settings.Source: Limitations · p. 8

A.1 Hallucination across Claim Positions

In the analyzed CHAIR-500 captions, object hallucination rates rise across later claims. The analysis measures both hallucinated object mentions and object-bearing claims containing hallucinated objects.

A.1 Hallucination across Claim Positions

  • The analysis uses 1,500 frozen LLaVA-1.5 Greedy captions from three CHAIR-500 runs and includes candidate claims after the first operational rejection.Source: §A.1 · p. 11
  • Object-mention rate counts recognized COCO object mentions labeled hallucinated by CHAIR; claim-level rate counts object-bearing claims with at least one hallucinated object.Source: §A.1 · p. 11
  • Object-mention hallucination rose from 4.12% in the first claim to 17.92% at position four or later; claim-level hallucination rose from 6.67% to 19.90%.Source: §A.1 · p. 11

A.2 Visual-Null Construction

On LLaVA-1.5 CHAIR-500, the zero-tensor visual-null reference retained the highest semantic quality among the tested references without stochastic corruption.

A.2 Visual-Null Construction

  • The zero-tensor reference retained the highest semantic quality and avoided a stochastic corruption process; the shuffled-image reference had the highest hallucination rates at comparable SPICE.Source: §A.2 · p. 11, Table 6 · p. 11
  • The ablation compared the zero tensor with a normalized-black image, diffusion noise, and a randomly shuffled COCO image while generating captions from the original image.Source: §A.2 · p. 11
  • The normalized-black reference behaved similarly to the zero tensor, with slightly lower CHAIR scores but slightly lower SPICE; diffusion noise was worse on both CHAIR metrics and SPICE.Source: §A.2 · p. 11, Table 6 · p. 11

A.3 Binary Parameter Sensitivity

This appendix reports local calibration robustness, claim-admission evidence, inference costs, and complete general-capability results.

A.3 Binary Parameter Sensitivity

  • All displayed POPE settings exceed native F1 84.11, while eight of nine MME settings exceed native Total 1681.92; the selected setting is best locally on both.The sensitivity grids vary τ and λ.Source: §A.3 · p. 11, Table 7 · p. 12, Table 8 · p. 12

A.4 Caption Admission Statistics

  • Across 1,500 captions, the first claim is admitted deterministically; among later claims, admitted claims have substantially stronger counterfactual evidence than rejected claims.The statistics pool runs over seeds 42, 777, and 2024.Source: §A.4 · p. 11, Table 9 · p. 12

A.6 Complete General-Capability Results

  • Tables 12 and 13 provide complete results behind Figure 2; MiniGPT-4 DoLa and HALC results are omitted because their binary outputs were unstable under the matched protocol.The tables cover full-MME perception, cognition, and total scores, and reference-based caption quality across three backbones.Source: §A.6 · p. 12, Table 12 · p. 13, Table 13 · p. 14

A.5 Inference Efficiency

  • 1.49× the end-to-end latency of greedy decoding, EGVCA has higher throughput than DoLa, OPERA, and HALC while matching greedy peak memory.The lower final-output token count reflects shorter captions produced by claim admission.Source: §A.5 · p. 12, Table 10 · p. 13, Table 11 · p. 13

Related papers

Loading 2610.17706v1…