PaperlayerView exact revision on arXiv

Source-linked AI summary

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

Lingfeng Ren, Weihao Yu, Runpeng Yu, Xinchao Wang

arXiv:2602.22144v1cs.CVcs.AIcs.CL

TL;DR

Object hallucinations in LVLMs raise the question of whether visual perception or language generation contributes more to unsupported objects. The paper analyzes both components, finds language-decoder priors predominant, and proposes NoLan, which dynamically suppresses those priors using multimodal and text-only output differences. Across benchmarks and architectures, NoLan reduces object hallucinations, including gains of up to 8.38 accuracy points and 8.77 F1 points on POPE.

  • Problem

    Object hallucinations produce objects absent from the image, and the relative contributions of the vision encoder and language decoder remain an important question.

  • Method

    NoLan is a simple, training-free decoding framework that contrasts multimodal and text-only output distributions to dynamically suppress language priors.

  • Results

    NoLan reduces hallucinations across diverse LVLM benchmarks and architectures, improving POPE accuracy by up to 8.38 and F1 by up to 8.77 over regular decoding.

  • Takeaways & Limitations

    The findings associate object hallucinations primarily with language-decoder priors and show that inference-time contrastive suppression can improve vision-language grounding without external tools.

Abstract

from arXiv · show

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.

1 Introduction

Object hallucinations in LVLMs pose risks because generated objects may not reflect the image, motivating analysis of whether vision or language components are primarily responsible. The paper identifies language-decoder priors as the main source and introduces NoLan, a training-free decoding framework that suppresses them dynamically.

  • Object hallucinations can cause misinformation and misinterpretation, creating risks in robotics, autonomous systems, and healthcare.
  • Analytical experiments indicate that hallucinations mainly stem from language-model priors rather than the vision model.
  • NoLan dynamically suppresses language priors by leveraging output-distribution differences between multimodal and text-only inputs.
  • NoLan is designed as a plug-and-play, training-free approach that requires neither additional training nor external tools.

2 Related work

Prior work addresses LVLM hallucinations through evaluation, training, post-processing, and training-free decoding, but these approaches differ in resource demands and assumptions. NoLan is motivated by a token-specific, dynamically modeled language prior and by evidence that the vision encoder can detect objects in hallucination cases.

  • POPE evaluates object hallucination by converting it into a binary task asking whether a specified object is present in the image.
  • Existing mitigation strategies include alignment, data augmentation, hallucination-specific fine-tuning, post-hoc revision, and external-tool pipelines.
  • Training-free methods such as VCD, M3ID, and VDD refine predictions by contrasting distributions from altered, conditioned, or unconditioned inputs.
  • NoLan models each token with a distinct language prior, unlike methods assuming uniform priors or sequence-length-only dependence.
  • The vision encoder can robustly detect object presence in hallucination samples, supporting analysis beyond the assumption of weak visual signals.

3 Method

The method analyzes whether hallucinations arise from visual detection or language-decoder priors by comparing multimodal and text-only outputs. NoLan then contrasts their logits, using either fixed or dynamically adjusted modulation to suppress language-prior influence during generation.

  • Preliminary experiments: The analysis separates the vision encoder’s object detection role from the language decoder’s text-generation role.
  • Preliminary experiments: The vision-encoder experiment extracts CLIP image representations and tests object presence using text queries formatted as “A photo of a [object].”
  • No-Language-Hallucination Decoding: NoLan computes multimodal logits from image-text inputs and language-prior logits from text-only inputs, then uses their difference as modulation logits.
  • Preliminary experiments: Hallucination cases show smaller divergence between multimodal and text-only output distributions, indicating stronger language-prior influence.
  • No-Language-Hallucination Decoding: The modulation logits are l_Δ = α × (l_m − l_u), and the adjusted distribution adds them to multimodal logits before softmax sampling.
  • No-Language-Hallucination Decoding: NoLan-Base fixes α at 1, whereas NoLan-Plus dynamically adjusts α using symmetric KL divergence and a tanh transformation.

4 Experiments

NoLan is evaluated as a training-free decoding method across multiple LVLMs, benchmarks, and hallucination settings. It consistently improves over regular decoding and often surpasses contrastive baselines on object- and attribute-level hallucinations.

  • Decoding methods: NoLan dynamically combines multimodal and text-only output distributions during autoregressive decoding, with constant or KL-based modulation.The method samples from logits refined by the difference between multimodal and text-only logits.
  • Experimental setup: NoLan is evaluated across LLaVA-1.5, InstructBLIP, and Qwen-VL on POPE, MME, LLaVA-Bench, and additional benchmarks.The evaluation covers random, popular, and adversarial POPE conditions, plus targeted MME hallucination subsets.
  • POPE results: 8.38 accuracy and 8.77 F1 are the largest reported improvements of NoLan over regular decoding across tested LVLMs and POPE conditions.NoLan outperforms regular decoding in every evaluated POPE scenario.
  • POPE results: 5.14 accuracy and 3.17 F1 are the largest NoLan-Plus gains over VCD, with NoLan-Plus outperforming VCD in 88.9% of experiments.NoLan-Base improves over VCD by up to 4.56 accuracy and 2.9 F1, outperforming it in 77.8% of evaluated cases.
  • MME results: NoLan improves MME attribute-level performance across all models and positively affects the object-level Existence metric.NoLan-Plus outperforms regular decoding, VCD, and VDD on most MME hallucination subsets.
  • Qualitative analysis: NoLan-Plus reduces hallucinated objects such as “suitcase” and “truck” while preserving the consistency and richness of generated text in LLaVA-Bench cases.The case studies compare regular decoding with NoLan-Plus outputs.

5 Conclusion and discussion

The paper attributes object hallucinations primarily to language-decoder priors and introduces NoLan, a training-free inference framework that suppresses them using multimodal and text-only distributions. Its two variants operate across pretrained autoregressive LVLMs, with NoLan-Plus dynamically adapting during generation and experiments validating effectiveness across benchmarks and architectures.

  • Conclusion and discussion: Analytical experiments indicate that hallucinations primarily stem from language-model priors rather than the vision model.The paper frames this finding as the basis for its decoding approach.
  • Conclusion and discussion: NoLan is a simple, training-free framework that refines outputs using contrastive distributions from multimodal and text-only inputs without external tools.It operates during inference and targets the influence of language priors.
  • Conclusion and discussion: NoLan-Base uses a constant configuration, whereas NoLan-Plus dynamically adjusts to the output distribution during generation.Both variants are designed for integration with pretrained autoregressive LVLMs.
  • Conclusion and discussion: Extensive experiments across diverse LVLM benchmarks and architectures validate NoLan’s effectiveness in mitigating object hallucinations.The paper presents NoLan as a cost-effective and flexible approach for improving vision-language grounding.

A.1 Theoretical proof of NoLan-Plus

NoLan-Plus relates visual-response dependence to the KL divergence between multimodal and text-only output distributions. Lower divergence indicates lower mutual information and stronger hallucination.

  • Definition of Visual Object Hallucination: Visual Object Hallucination occurs when the response is nearly independent of visual input given the textual prompt.The paper quantifies this dependence with conditional mutual information I(y; v | x).
  • Connecting Conditional Mutual Information to KL Divergence: For fixed prompt x, p_m is the image-text output distribution, whereas p_u is the text-only output distribution.
  • Theoretical proof of NoLan-Plus: Conditional mutual information equals the expected KL divergence D_KL(p_m ∥ p_u) between multimodal and text-only distributions.The equality follows by expanding conditional mutual information into an expectation over visual inputs and responses.
  • Theoretical proof of NoLan-Plus: Lower KL divergence D_KL(p_m ∥ p_u) indicates lower mutual information between visual input and response, leading to stronger hallucination.

A.2 Uncertainty analysis and language prior suppression

The paper evaluates uncertainty while suppressing language priors through a training-free contrastive decoding formulation. NoLan variants reduce entropy across four benchmarks, although post-hoc probability changes can introduce decoding instability.

  • Uncertainty analysis and language prior suppression: NoLan’s training-free contrastive decoding suppresses dominant language priors by reshaping output distributions without additional training.Post-hoc token-probability changes may distort non-target rankings and amplify spurious distributional modes.
  • Uncertainty analysis and language prior suppression: NoLan yields substantially lower entropy than regular decoding and the text-only baseline across POPE, MME, MM-Vet, and LLaVA-Bench.
  • Uncertainty analysis and language prior suppression: NoLan-Plus achieves the lowest uncertainty in all four evaluated benchmark settings.The result suggests that NoLan-Plus maintains distributional stability in most cases while suppressing linguistic bias.

A.3 Correlation study between hallucination and token position

The token-position study estimates hallucination likelihood using mean KL divergence at each generated-token position. Its results generally align with prior findings that visual reliance decreases later in generation.

  • Correlation study between hallucination and token position: The study measures hallucination likelihood at each token position using mean KL divergence across LLaVA-Bench samples.It evaluates LLaVA-1.5 7B, with the first generated token indexed as position 0.
  • Correlation study between hallucination and token position: Farther token positions generally correspond to increased hallucination as generation proceeds.This observation is reported as broadly consistent with M3ID’s finding that reliance on the initial visual prompt decreases over time.

A.4 Ablation study

Ablations examine modulation rates, logit components, model size, and NoLan-Plus variants. Results support using both multimodal and text-only logits and show NoLan’s robustness across scales and architectures.

  • Modulation rate: α = 1 gives NoLan-Base its best or second-best performance and is selected as the default modulation rate.For NoLan-Base, α controls amplification of the modulation distribution; α = 0 reverts to standard decoding.
  • Logit components: Using only multimodal or only text-only logits causes a significant performance drop, confirming the importance of both components in NoLan.NoLan combines regular multimodal logits with unimodal text-only logits.
  • LVLMs’ model size: 78.36 and 78.35 are the InstructBLIP 7B and 13B F1 scores in POPE’s Popular setting, respectively, indicating comparable performance across those model sizes.
  • LVLMs’ model size: NoLan consistently exceeds regular decoding across evaluated model sizes and architectures, with larger-model improvements particularly pronounced.The study evaluates 7B and 13B variants of LLaVA-1.5 and InstructBLIP.
  • Variations of NoLan-Plus: The tanh-based NoLan-Plus variant outperforms the sigmoid variant in most evaluated scenarios.The paper attributes this broader effectiveness to tanh’s faster convergence toward its upper bound under the imposed constraints.

A.5 Benchmarking NoLan against the ICD baseline

The supplementary comparison evaluates ICD against NoLan on InstructBLIP across three datasets and question categories. NoLan variants generally outperform ICD and standard decoding.

  • ICD baseline: ICD contrasts standard and disturbed-instruction output distributions to suppress concepts activated by visual priors.Its disturbance prompt intentionally increases alignment uncertainty before distribution subtraction.
  • Evaluation setup: Table 13 compares Regular, VCD, ICD, and NoLan sampling strategies using accuracy and F1 across multiple datasets and question categories.The table caption identifies NoLan as sampling from the proposed contrastive distribution.
  • Comparison results: NoLan variants consistently outperform ICD across most InstructBLIP settings.Table 13 covers GQA, A-OKVQA, and MSCOCO under random, popular, and adversarial categories.

A.6 Supplementary experiments

Supplementary evaluations test NoLan across open-ended, reasoning, hallucination, grounding, efficiency, and qualitative settings. Across these settings, NoLan variants generally improve performance, reduce hallucinations, and retain practical efficiency.

  • MM-Vet: NoLan-Plus improves MM-Vet performance from 31.1 to 33.3 for LLaVA-1.5 7B and from 36.1 to 38.3 for the 13B model.Most capability integrations also improve, including up to 8.3% for the “Rec” and “Spat” combination.
  • MMHal-BENCH: NoLan-Plus raises MMHal-BENCH overall score from 1.55 to 2.29 and lowers hallucination rate from 76% to 68%.NoLan-Base reaches 1.85, while both variants outperform regular decoding.
  • HallusionBench: NoLan-Plus reaches qAcc 18.68 and aAcc 47.48 on HallusionBench, including hardaAcc 40.70.Both NoLan variants outperform regular decoding across the reported metrics.
  • CircularEval: NoLan-Plus achieves a 65.8 overall score on CircularEval, including CP 77.5, RR 60.2, and LR 38.7.Both NoLan variants outperform regular decoding in overall accuracy and several reasoning dimensions.
  • MathVision: NoLan-Plus achieves 9.84% accuracy on MathVision, with scores of 6.96 in Algebra, 17.78 in Graph Theory, and 13.29 in metric geometry-angle.The results concern visually rich tasks requiring precise perception and subject-specific mathematical reasoning.
  • POPE comparison: NoLan-Plus achieves accuracy 88.80 and F1 86.70 on MSCOCO POPE, exceeding PAI’s F1 85.89 and OPERA’s F1 85.40.The comparison supports contrastive decoding with language-prior suppression against attention-based methods.
  • Qwen-VL series: NoLan-Plus raises F1 from 67.98 to 76.90 on Qwen2-VL-2B and from 85.85 to 88.91 on Qwen2-VL-7B.It also improves Qwen2.5-VL F1 from 86.54 to 90.33 on 3B and from 80.65 to 88.04 on 7B models.
  • Efficiency: NoLan-Base reaches 0.6075 seconds per token and 13.59 GB for 50 tokens, using less time and memory than VCD and VDD.The efficiency evaluation uses LLaVA-v1.5-7B on a Titan RTX 24GB GPU.
Loading 2602.22144v1…