Source-linked AI summary
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, Lidong Bing
TL;DR
LVLMs can generate coherent descriptions containing objects absent from images, creating a reliability problem. The paper proposes training-free Visual Contrastive Decoding, which contrasts outputs from original and distorted images; experiments report reduced hallucinations across LVLM families and benefits on general perception benchmarks.
Problem
LVLM object hallucination generates coherent textual content inconsistent with ground-truth image objects, undermining reliability in practical applications.
Method
Visual Contrastive Decoding contrasts output distributions from original and distorted visual inputs to calibrate reliance on statistical bias and unimodal priors.
Results
Experiments across multiple benchmarks and LVLM families confirm reduced hallucinations and potential improvements in overall perception capabilities.
Takeaways & Limitations
VCD mitigates object hallucinations without additional training or external tools and may apply beyond hallucination-focused evaluation.
Takeaways & Limitations
The study uses basic Gaussian noise for visual uncertainty, focuses on image-text LVLMs, and does not cover emerging video-understanding applications.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.
1. Introduction
LVLMs can produce coherent but visually unsupported object descriptions, undermining reliability. The paper introduces training-free Visual Contrastive Decoding, which contrasts original and distorted visual inputs to reduce hallucinations and improve perception.
- 1. Introduction: Object hallucination produces coherent textual content that is inconsistent with ground-truth objects in the image.The problem affects LVLM reliability and practical applicability.
- 1. Introduction: Visual Contrastive Decoding contrasts output distributions from original and distorted visual inputs to calibrate model outputs.The method targets over-reliance on language priors and statistical bias without additional training or external tools.
- 1. Introduction: VCD addresses statistical bias and unimodal priors, identified as two primary causes of object hallucinations in LVLMs.The paper analyzes how visual uncertainty affects these factors.
- 1. Introduction: Up to +7.4 F1 score on POPE and +18% on MME are reported across LLAVA-1.5, InstructBLIP, and Qwen-VL.These improvements are reported without additional training or external tools.
- 1. Introduction: Benchmarking on MME and LLaVA-Bench indicates that VCD may also benefit general perception capabilities beyond object-hallucination mitigation.The passage presents this as potential applicability beyond hallucination mitigation.
2. Related Work
Prior work addressed object hallucination through alignment, augmentation, datasets, revision, and reinforcement learning, but many approaches require costly training or external models. VCD instead calibrates outputs by contrasting original and distorted visual inputs without external models.
- 2. Related Work: VLM object hallucination refers to plausible outputs containing objects that do not match or appear in the image.Recent LVLM work has also studied hallucination evaluation and detection.
- 2. Related Work: Recent approaches use hallucination-targeted fine-tuning datasets, post-hoc revision, or factually augmented RLHF.These methods add training or revision components to reduce hallucination.
- 2. Related Work: VCD contrasts output distributions from original and distorted visual inputs to calibrate reliance on unimodal priors and statistical bias without external models.This provides a training-free alternative to approaches requiring additional datasets, tuning, or pretrained models.
3. Method
The method analyzes how visual uncertainty amplifies language priors and statistical biases, then introduces Visual Contrastive Decoding to counteract them by contrasting predictions from original and distorted images.
- Visual Uncertainty: A Gaussian noise mask creates progressively distorted visual inputs whose distinguishable features diminish as noise increases.The distortion sequence begins with the original image and approaches Gaussian noise as the number of steps grows.
- Motivation: Visual uncertainty can amplify language priors and superficial object correlations, worsening object hallucinations in LVLMs.The analysis examines both language priors and statistical biases inherited from pretraining data.
- Visual Contrastive Decoding: VCD generates output distributions from original and distorted visual inputs, then contrasts them to reduce predictions driven by hallucination-prone biases.The distorted input is produced by applying predefined Gaussian-noise distortions to the original visual input.
- Visual Contrastive Decoding: The contrastive distribution amplifies differences between original-input and distorted-input predictions, with α controlling the strength of amplification.When α = 0, the method reduces to regular decoding, and the adjusted distribution can support nucleus sampling or beam search.
- Adaptive Plausibility Constraints: An adaptive plausibility constraint truncates low-probability tokens to avoid penalizing valid linguistic outputs associated with distorted inputs.The hyperparameter β controls truncation aggressiveness; larger β keeps only higher-probability tokens, while incorporating the constraint improves straightforward decisions.
4. Experiments
Experiments evaluate VCD across multiple LVLMs and benchmarks, including object and attribute hallucination, general perception, and qualitative captioning. VCD consistently improves hallucination-related and perception performance, while preserving recognition competencies and correcting illustrated hallucinations.
- Experimental Setup: VCD was evaluated on LLaVA-1.5, InstructBLIP, and Qwen-VL, with averaged results and standard deviations reported over five runs on POPE and MME.The evaluation covers three LVLM families using Vicuna 7B- or Qwen 7B-based language decoders.
- POPE Results: POPE results show VCD consistently surpassing regular decoding across random, popular, and adversarial settings, by up to +5.8 accuracy and +7.4 F1.For LLaVA-1.5 and Qwen-VL, F1 gains were predominantly driven by recall improvements of up to 10 points.
- MME Hallucination Subset: MME hallucination-subset results show uniform object-level improvements and overall gains in Color scores, while Position scores remain relatively low with minimal uplift.The subset extends evaluation beyond object existence to attribute-level hallucinations involving color and position.
- MME Full Set: On the MME full set, VCD consistently enhances perception-based tasks while preserving the LVLMs’ original recognition competencies.The reported full-set trajectory for LLaVA-1.5 is representative because all evaluated models exhibit comparable performance trajectories.
- Visual Uncertainty: Increasing visual noise causes LLaVA-1.5 performance on POPE to drop, indicating that greater visual uncertainty is associated with more severe object hallucination.VCD is presented as a correction mechanism that contrasts outputs from original and distorted visual inputs.
- Qualitative Evaluation: Qualitative LLaVA-Bench cases show regular decoding hallucinating co-occurring objects such as “dining table” and “fork,” whereas VCD mitigates these errors while preserving output coherence and informativeness.Open-ended LLaVA-Bench evaluation also reports consistent VCD accuracy improvements and more detailed responses.
5. Conclusion and Limitation
The paper finds that visual uncertainty amplifies statistical biases and language-prior reliance, contributing to hallucinations, and introduces training-free VCD to reduce them. It also identifies Gaussian noise as a limitation and points toward finer distortions and video applications for future work.
- Conclusion: VCD uses contrastive distributions to calibrate LVLM outputs without additional training or external tools.The method contrasts outputs from original and distorted visual inputs.
- Conclusion: Experiments across multiple benchmarks and LVLM families confirm VCD reduces hallucinations and may improve overall perception capabilities.
- Limitation: The study uses basic Gaussian noise to introduce visual uncertainty, while object-level blurring may improve outcomes.
- Limitation: The evaluation focuses on image-and-text LVLMs and excludes emerging video-understanding applications.
- Limitation: Future work includes diverse image distortions and extending VCD to a broader range of LVLMs.
A. Detailed Experimental Settings
The experiments fix γ, α, and β across setups, while varying the total noise steps T by benchmark.
- Experimental Settings: γ, α, and β are fixed at 0.1, 1, and 0.1, respectively, in all experimental setups.
- Experimental Settings: T is set to 500 for MME and LLaVA-Bench experiments and 999 for POPE evaluations.
B. Ablation Studies
The ablation studies use fixed default hyperparameters and LLaVA-1.5 as the representative baseline for tuning experiments.
- Ablation Studies: The ablation default configuration sets α, β, and δ to 1, 0.1, and 500, respectively.
- Ablation Studies: LLaVA-1.5 is used as the representative LVLM baseline across hyperparameter-tuning experiments.
B.1. Effect of Total Noise Steps T
The noise-level ablation evaluates LLaVA-1.5 on MME subsets for hallucination, perception, and recognition, revealing different sensitivity patterns across tasks.
- B.1. Effect of Total Noise Steps T: Figure 4 varies noise level δ on LLaVA-1.5 using MME’s hallucination, perception, and recognition subsets.The hallucination subset covers Existence, Count, Position, and Color tasks.
- B.1. Effect of Total Noise Steps T: The hallucination subset is strongly sensitive to δ, with optimal noise levels associated with substantially improved overall scores.
- B.1. Effect of Total Noise Steps T: For perception tasks, δ > 500 yields consistent improvements from VCD.
- B.1. Effect of Total Noise Steps T: VCD maintains steady recognition-task performance across the tested noise values.
B.2. Effect of α in Visual Contrastive Decoding
The ablation studies examine how α controls amplification between original and distorted-input output distributions, finding stable improvements over regular decoding across tested settings.
- Effect of α: α modulates the amplification level between output distributions from original and distorted visual inputs.The study is reported in Table 5 and referenced through Equation 3.
- Effect of α: Minimal variance in aggregate scores across three MME subsets appears as α ranges from 0.25 to 1.0.The results indicate stable behavior across this range.
- Effect of α: Uniform improvement over regular decoding is observed across α values from 0.25 to 1.0.This supports the consistency of the contrastive decoding strategy across the tested settings.
- Related decoding constraint: β controls the adaptive plausible constraint, with larger β producing more aggressive truncation that retains only high-probability tokens.The β ablation reports that β = 0, corresponding to no constraint, produces suboptimal performance.
B.4. Effect of Different Sampling Strategies
VCD is evaluated across multiple sampling strategies and larger LVLM variants, with improvements reported across sampling choices while larger parameter counts do not inherently resolve hallucination issues.
- Sampling strategies: VCD is tested with direct, Top P, Top K, greedy, and temperature-normalized Top K sampling on POPE-Random using LLaVA-1.5.The additional settings include p = 0.9, k = 50, and k = 50 with temperature 1.5/0.7.
- Sampling strategies: VCD improves performance irrespective of the sampling strategy, demonstrating consistency across the evaluated decoding choices.The reported consistency includes enhanced general performance capabilities of LVLMs.
- Model scale: The 7B and 13B LLaVA-1.5 and InstructBLIP variants show comparable POPE performance across settings.For LLaVA-1.5 in the Random setting, the reported F1 scores are 81.33 for 7B and 81.49 for 13B.
- Model scale: Increasing model parameters does not inherently resolve hallucination issues in the evaluated larger-model comparisons.The comparison covers 7B and 13B variants of LLaVA-1.5 and InstructBLIP; Qwen-VL lacks larger variants.
C. Detailed Experimental Results on MME
On MME, VCD improves perception-related performance while preserving stable recognition scores, and additional case studies illustrate hallucination corrections and broader perception and recognition enhancements.
- Perception: VCD uniformly improves the perceptual competencies of three LVLM baselines on MME perception-related tasks.The paper attributes this improvement to reducing statistical biases and language priors, favoring visual information.
- Recognition: VCD alleviates hallucination issues and augments perceptual capabilities without compromising inherent reasoning abilities, as recognition scores remain stable.These results concern recognition-related tasks within the MME benchmark.
- Case studies: Additional LLaVA-Bench examples show hallucination corrections and enhancements in general perception and recognition abilities across different LVLMs.Figure 7 presents correction examples, while Figure 8 presents perception and recognition enhancements.
- Open-ended generation: GPT-4V is used to evaluate the accuracy and detailedness of LVLM responses in open-ended generation.The evaluation configuration is described in Table 11, with an illustrative case in Figure 9.