PaperlayerView exact revision on arXiv

Source-linked AI summary

HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, Jiawei Zhou

arXiv:2403.00425v2cs.CVcs.AIcs.LG

TL;DR

LVLMs continue to produce object hallucinations, while existing mitigation methods incompletely cover hallucination types and may require extra models, processing, or data. HALC uses local adaptive focal-contrast grounding and global matching-based beam search to reduce hallucinations while preserving generation quality. Experiments report state-of-the-art performance across four benchmarks, with integration requiring no additional training or data.

  • Problem

    Existing LVLM hallucination mitigation methods do not fully eliminate object hallucinations and often require external LVLMs, repeated processing, or additional data.

  • Method

    HALC combines token-level adaptive focal-contrast grounding with global matching-based beam search to use fine-grained visual information during decoding.

  • Results

    HALC achieves state-of-the-art object-hallucination reduction while preserving sequence-generation quality across four benchmarks.

  • Takeaways & Limitations

    HALC is a conveniently integrable decoding module for existing LVLMs that requires no additional training or data.

  • Takeaways & Limitations

    HALC trades additional computation time for correcting token hallucinations, with parallelization potentially limiting overhead to roughly twice regular decoding time at HALC steps.

Abstract

from arXiv · show

While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH). We introduce HALC, a novel decoding algorithm designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal visual information in vision-language tasks and operates on both local and global contexts simultaneously. Specifically, HALC integrates a robust auto-focal grounding mechanism (locally) to correct hallucinated tokens on the fly, and a specialized beam search algorithm (globally) to significantly reduce OH while preserving text generation quality. Additionally, HALC can be integrated into any LVLMs as a plug-and-play module without extra training. Extensive experimental studies demonstrate the effectiveness of HALC in reducing OH, outperforming state-of-the-arts across four benchmarks.

1 Introduction

Object hallucination remains a persistent LVLM problem, and existing mitigation methods incompletely address its types while often requiring external models, repeated processing, or additional data. HALC targets these limitations with a plug-and-play decoding strategy operating across local and global visual contexts.

  • Object hallucination includes erroneous object descriptions and can involve existence, attribute, or relationship errors.
  • Existing approaches primarily target object existence hallucinations while assuming autoregressive decoding will correct attribute and relationship errors.
  • Prior methods may require external LVLMs, repeated processing, or additional data, complicating adaptation to existing LVLMs and restricting use cases.
  • HALC addresses all three hallucination types through adaptive focal-contrast grounding locally and matching-based beam search globally.
  • HALC is a plug-and-play decoding algorithm that reduces object hallucination while preserving generation quality and supports unified evaluation across LVLMs, benchmarks, and metrics.

2 Related Work

Related work characterizes object hallucination and its evaluation while showing that existing mitigation strategies remain limited in coverage and adaptability. HALC differs by reducing all three hallucination types without extra LVLMs, training, or data.

  • Object hallucination covers nonexistent objects, misleading attributes, and incorrect relationships in generated content.
  • CHAIR evaluates hallucination specifically, whereas POPE converts hallucination assessment into binary classification using precision, recall, and accuracy.
  • Existing post-hoc, self-correction, and decoding approaches can require powerful external LVLMs or additional data, limiting their adaptability.
  • HALC reduces object hallucination without extra LVLMs, training, or data and addresses both local and global contexts.
  • The authors provide an open-source platform integrating HALC and other methods across LVLM backbones and evaluation metrics.

3 Background and Motivation

Object hallucination occurs when generated text conflicts with the image, partly because autoregressive decoding increasingly relies on textual history over visual input. The paper motivates token-wise fine-grained visual grounding, supported by oracle analyses showing substantial hallucination reduction from optimal contexts.

  • Object hallucination is defined as generated text conflicting with the input image while the goal is faithful, high-quality text generation.
  • Hallucinations can arise from training-data co-occurrence biases, visual uncertainty, and limited ability to discern context and facts accurately.
  • Autoregressive decoding progressively increases reliance on the query and generated history while reducing reliance on visual input.
  • 3.3 Fine-grained Visual Knowledge Reduces OH: The proposed insight is to identify a token-wise optimal visual context that provides informative grounding for each object-related token.
  • 3.3 Fine-grained Visual Knowledge Reduces OH: Over 84.5% of observed hallucinations were eliminated in most cases when decoding used optimal visual contexts in the oracle pilot study.

4 Methodology

HALC dynamically grounds object-related tokens using sampled visual contexts, contrasting their token distributions to correct hallucinations during generation. It combines local focal-contrast decoding with global beam selection based on visual matching.

  • Token-level processing: HALC processes potentially hallucination-prone tokens according to syntactic category, covering existence, attribute, and relationship hallucinations.Nouns, adjectives, adverbs, numbers, verbs, pronouns, and prepositions trigger HALC processing without assuming the token is actually hallucinated.
  • Visual context retrieval: A zero-shot detector retrieves a token-specific visual context window from the image, including width, height, and center coordinates.Detectors such as Grounding DINO or OWLv2 provide visual references for object names, attributes, and relationships.
  • FOV sampling: HALC samples exponentially expanding, overlapping fields of view around the detected context to search for a more informative visual grounding.The sampled FOVs increase image coverage quickly while preserving the initial detection as a reference.
  • Dynamic visual context selection: It feeds sampled FOVs and the ongoing response into the model, then selects the top m pairs with the largest Jensen-Shannon divergence between token distributions.The divergence identifies FOV pairs whose visual contexts most discrepantly influence decoding probabilities.
  • Contrastive decoding and beam search: Bi-directional contrastive decoding redistributes logits from each selected FOV pair, while beam search retains k responses with the best global visual matching scores.The two contrast directions avoid assuming which randomly sampled FOV is the expert; the overview illustrates correction of “surfboard” to “clock.”
  • Efficiency: HALC trades additional computation time for token-level hallucination correction, with parallelization potentially reducing decoding overhead to roughly twice regular decoding time at HALC steps.HALC does not run at every decoding step, but visual grounding introduces additional overhead.

5 Theoretical Analysis on FOV Sampling

Theoretical analysis models FOV sampling around an imperfect initial detection and establishes conditions under which sampled contexts approximate an optimal visual context. The guarantee covers normal and exponential sampling, with approximation improving as the sample count grows.

  • Setup: HALC seeks an underlying optimal visual context v* within the image by sampling FOVs conditioned on an initial detector grounding vd.The initial grounding may be inaccurate or outside the valid neighborhood, motivating the sampling analysis.
  • Assumptions: Theorem 5.1 assumes a tolerable neighborhood around v* in which decoding remains robust and uses a symmetric discrepancy measure such as Jensen-Shannon divergence or total variation distance.The neighborhood defines a range of visual contexts that can yield correct predictions.
  • Normal sampling: For normal sampling πg(·|vd) ∼ N(vd, σ^2I), the minimum deviation from optimal token probabilities is bounded by a constant depending on the neighborhood, detector perturbation, and sampling variance.The bound’s upper limit approaches δ as n →∞.
  • Exponential sampling: For exponential sampling under matched center and aspect-ratio conditions, the minimum deviation is likewise bounded by a constant depending on ε, v*, vd, and λ.The corresponding upper bound also approaches δ as n →∞.
  • Guarantee: The theorem formally guarantees that at least one of n sampled FOVs achieves a good approximation of v* under the stated sampling assumptions.This supports the robustness of FOV sampling even when the initial detector grounding is perturbed beyond the tolerable neighborhood.

6 Experiments

HALC is evaluated against decoding baselines and OH-specific SOTAs across image captioning, POPE-style assessment, MME subsets, and qualitative LLaVA-Bench comparisons. It consistently reduces hallucination while maintaining generation quality, including for longer outputs.

  • Experimental setup: HALC is compared with greedy decoding, beam search, DoLa, OPERA, VCD, Woodpecker, and LURE using a unified implementation.Experiments use MiniGPT-4 V2, LLaVA-1.5, and mPLUG-Owl2, except Woodpecker and LURE, which use additional language models.
  • CHAIR: CHAIR evaluates hallucination in image captions through sentence-level and object-level measures, alongside BLEU for captioning quality.Lower CHAIRS and CHAIRI indicate less hallucination, while higher BLEU generally indicates better captioning quality.
  • CHAIR: HALC consistently outperforms existing methods on CHAIR with substantially lower variability, while Woodpecker reports the highest BLEU under an unfairly stronger ChatGPT-based comparison.The CHAIR experiments use 500 randomly sampled MSCOCO validation images and average five random-seed runs.
  • Longer responses: HALC maintains a low hallucination-object ratio as the number of generated objects increases, unlike the other compared methods.Figure 4 examines 100 randomly sampled images and tracks generated and hallucinated objects as response length increases.
  • OPOPE: OPOPE replaces live POPE interactions with offline caption checks and uses F-beta with β = 0.2 because false negatives are less reliable offline.Results are averaged across random, popular, and adversarial sampling methods.
  • MME: On four MME OH-critical subsets, HALC exceeds other methods by +10.7% for existence, +18.3% for position, +19.4% for color, and +20.2% for count.All methods use MiniGPT-4 as the LVLM backbone, and HALC significantly outperforms competitors on each sub-task.

7 Analysis and Ablation Studies

The ablations show that HALC benefits from detector-grounded FOV initialization, a 0.6 expanding ratio, and carefully controlled beam search, while scoring-model choices matter less than random selection.

  • Sampling initialization: Detector-grounded FOV initialization outperforms random, center, and original-image initialization, providing an effective starting point for conditional FOV sampling.Both random and center initialization also outperform using the original image as visual input.
  • Expanding ratio: An expanding ratio of 0.6 is optimal; smaller ratios increase FOV samples and larger ratios reduce image-region granularity.The authors associate these settings with greater global beam-search difficulty and potentially more severe hallucinations, respectively.
  • Beam size: Increasing beam size initially improves HALC, but beam sizes of at least two increase FOV samples, hallucination rates, and performance variance.The global search becomes more challenged when selecting the optimal visual context from more samples.
  • Scoring methods: BLIP and CLIP scoring produce similar performance variations, and both outperform random beam selection.The comparison evaluates these scoring models as methods for ranking beams.

8 Conclusion

HALC is a decoding algorithm for reducing object hallucination in LVLMs through local visual grounding and global beam search. Experiments report reduced hallucination, preserved generation quality, state-of-the-art performance, and integration without additional training or data.

  • Conclusion: HALC combines adaptive focal-contrast grounding with specialized beam search to reduce hallucinated tokens while preserving sequence generation quality.The method operates at both local and global levels to use fine-grained visual information and promote visually matched generations.
  • Conclusion: HALC achieves state-of-the-art object-hallucination reduction and can be integrated into existing LVLMs without additional training or data.The paper also presents a benchmarking tool for comparing available object-hallucination reduction strategies.

A Proof of Robust Certification of FOV Sampling in Theorem 5.1

The proof analyzes whether FOV sampling conditioned on detector output can approximate an unknown optimal visual context in decoding-probability space. Under stated assumptions, increasing the number of samples tightens the deviation bound, while selecting the best sample remains outside the proof.

  • Assumptions and objective: HALC models the optimal visual context as a width, height, and center-point tuple whose token distribution is compared using a symmetric discrepancy measure.The analysis assumes prediction robustness within an ϵ-neighborhood around the optimum.
  • Sampling rationale: Detector output may lie outside the tolerable neighborhood, so HALC samples FOVs conditioned on that detection to seek a context closer to the optimum.Directly using inaccurate detector output can produce unpredictable distribution deviation and wrong token predictions.
  • Sampling objective: The minimum deviation hπ(v∗, n) is defined as the smallest discrepancy between the optimal token distribution and those from n sampled FOVs.This formalizes the search for a sample whose decoding probabilities approximate the optimum.
  • Proof structure: If at least one sample enters the ϵ-neighborhood, its decoding deviation is bounded by δ, yielding hπ(v∗, n) ≤ δ + P(¬A).The bound separates the event of sampling near the optimum from its complement.
  • Sampling distributions: For normal sampling around detector output, the analysis assumes independent variance σ2 across width, height, and center components; exponential expansion assumes aligned center and aspect ratio.The exponential process samples FOVs sharing the detector’s center and aspect ratio while changing scale by λ.
  • Guarantee and scope: Under the stated conditions, more samples tighten the bound and provide a robust way to approximate the optimum, but the proof does not analyze how HALC selects the best sample.HALC performs that selection through FOV-pair contrast, whereas the theorem establishes only the existence of a suitable sample.

B Time Cost Analysis

HALC adds decoding overhead because it applies visual-context processing to selected tokens, but parallel FOV decoding substantially reduces that cost. With the paper’s settings, the expected overhead ranges from about 1.35x to 2.4x of greedy decoding.

  • Overhead sources: HALC applies its extra visual-context processing to approximately 35% of generated words, focusing on tokens such as nouns, adjectives, numbers, verbs, and pronouns.POS tagging identifies the tokens that pass through HALC decoding.
  • Overhead sources: For each HALC-processed token, the detector initializes FOV sampling and n sampled FOVs are decoded by the LVLM.The per-token cost includes detector time and n LVLM decoding passes.
  • Sequential cost: With n = 4 and relatively small detector cost, HALC is expected to require around 2.4x the normal greedy decoding time.This estimate follows the paper’s time-cost formulation.
  • Parallelized cost: Parallelizing independent extra-FOV decoding reduces the expected cost to approximately 1.35x greedy decoding, or 1.7x when detector time equals decoding time.The 1.7x figure is presented as a worst-case estimate for detector cost.

C.1 Experimental Setups

The experiments compare HALC with standard, contrastive, and object-hallucination-specific baselines under a unified evaluation setup. They use multiple LVLM backbones, standardized benchmarks, reproduced baseline implementations, and controlled HALC sampling and search settings.

  • Baselines: The evaluation compares greedy decoding, beam search, DoLa, OPERA, VCD, Woodpecker, and LURE using a unified implementation.Baseline code and settings are described consistently across methods.
  • HALC settings: HALC uses exponential-expansion FOV sampling with λ = 0.6, n = 4 sampled FOVs, and tunable JSD buffer and beam-size parameters.The number of sampled FOVs is fixed across experiments, while larger search settings increase runtime.
  • Implementation: The experiments report HALC hyperparameters in Table 8 and baseline-specific settings for DoLa, OPERA, and VCD in Tables 9–11.Woodpecker and LURE are reproduced using their published implementations and hyperparameters.
  • Evaluation platform: The authors release a unified object-hallucination benchmark covering CHAIR, POPE, OPOPE, linguistic-quality metrics, and MME scores.The benchmark provides standardized access to multiple LVLMs and VQA tasks.
  • Pilot evaluation: A pilot study uses four MME sub-tasks with 30 images each and reconstructs offline-POPE-style prompts for detailed-caption evaluation.The corresponding decoder comparison is reported in Table 12.

E POPE Results

HALC is evaluated on original POPE tasks and qualitative LLaVA-Bench comparisons across three LVLM backbones. The reported POPE results show consistently stronger performance than the compared methods across sampling settings and hallucination types.

  • Evaluation procedure: For original POPE evaluation, HALC uses the complete query and initial yes/no answer to ground the detector’s focal area.This adapts HALC’s grounding process to the binary-answer POPE format.
  • POPE results: HALC outperforms the other four methods by a large margin in accuracy, precision, recall, and F1 Score across random, popular, and adversarial POPE tasks.The comparison covers all three POPE sampling strategies and all three types of POPE VQA tasks.
  • LLaVA-Bench comparison: LLaVA-Bench qualitatively compares HALC with other OH-reduction methods using mPLUG-Owl2, MiniGPT-4, and LLaVA backbones.Figures 6–8 show the corresponding outputs, with red text marking object hallucinations.
Loading 2403.00425v2…