PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

Kairong Yu, Zixin Zhu, Le Yu, Hongwei Wang

arXiv:2608.28058v1cs.CVcs.AI

TL;DR

LVLM hallucinations are linked here to underexplored instability in internal representations during autoregressive generation, including degradation across layers and drift across decoding steps. DAC is a training-free inference-time method that detects divergence and selectively compensates hidden states through layer-wise and sequential corrections, consistently reducing hallucinations while maintaining strong overall performance across nine benchmarks and multiple backbones. Its scope remains limited to settings studied so far, with broader modalities, models, domains, and adaptive thresholds left for future work.

  • Problem

    LVLM hallucinations produce responses inconsistent with multimodal inputs, while internal representation dynamics during autoregressive generation remain underexplored.

  • Method

    DAC detects hidden-state divergence and selectively applies lightweight compensation through Layer-wise Semantic Compensation and Sequential Semantic Correction.

  • Results

    DAC consistently reduces hallucinations while maintaining strong overall performance across nine hallucination-focused and general-purpose multimodal benchmarks and multiple LVLM backbones.

  • Takeaways & Limitations

    Dynamic hidden-state compensation is presented as a practical direction for improving the reliability of LVLM generation with negligible inference overhead.

  • Takeaways & Limitations

    DAC remains to be studied on closed-source models, video or multi-image inputs, long-context multimodal generation, and highly specialized domains, while fixed thresholds and coefficients may limit adaptability.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose \emph{Dynamic Alignment Compensation} (DAC), a training-free inference-time method that detects representation divergence and selectively applies lightweight residual compensation. DAC combines Layer-wise Semantic Compensation to mitigate inter-layer degradation with Sequential Semantic Correction to constrain temporal drift. Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.

1 Introduction

LVLM hallucinations remain a reliability challenge, while existing mitigations often require external resources or intervene at output and attention levels. DAC instead targets hidden-state instabilities during inference with training-free, selective compensation and reduces hallucinations across benchmarks.

  • LVLMs support broad multimodal applications but can produce responses inconsistent with the input, creating reliability risks in high-stakes scenarios.The paper highlights medical diagnosis and autonomous driving as examples of high-stakes settings.
  • Existing mitigation methods use retrieval, additional supervision, or inference-time output calibration and attention regulation, often requiring extra resources.Training-free methods intervene during inference without updating model parameters.
  • The paper identifies inter-layer degradation and temporal drift as two inference-time representation instabilities in LVLM generation.These instabilities are examined through hidden-state dynamics during autoregressive inference.
  • DAC combines Layer-wise Semantic Compensation for inter-layer degradation with Sequential Semantic Correction for temporal drift.
  • DAC is a training-free decoding method that detects hidden-state divergence and applies lightweight residual compensation only when needed.Its design targets both layer and temporal dimensions.
  • Across nine hallucination-focused and general-purpose multimodal benchmarks and multiple LVLM backbones, DAC consistently reduces hallucinations while maintaining strong overall performance.The evaluation covers both hallucination-specific and general-purpose settings.

2 Related Work

LVLM research spans multimodal architectures, hallucination diagnosis, and mitigation methods that intervene through supervision, attention, logits, or representations. DAC is positioned within training-free inference-time mitigation while extending evaluation across model families and scales.

  • Large Vision-Language Models: LVLMs couple visual encoders with autoregressive language models through cross-modal interfaces to process multimodal inputs.Representative systems include BLIP-2 and LLaVA, followed by newer families with broader capabilities.
  • Large Vision-Language Models: The paper evaluates DAC on LLaVA-1.5, Qwen-VL, Qwen2.5-VL, Qwen3-VL, and LLaVA-NeXT backbones spanning multiple model families, generations, and scales.
  • Hallucination in LVLMs: LVLM hallucinations are fluent, plausible responses inconsistent with multimodal context, especially in open-ended generation where local errors may accumulate autoregressively.
  • Hallucination in LVLMs: Prior diagnostic accounts associate hallucinations with imperfect cross-modal grounding, object co-occurrence bias, and overreliance on language priors.
  • Hallucination in LVLMs: Supervision-based mitigation can be effective but typically requires additional annotations, auxiliary models, or costly training.
  • Hallucination in LVLMs: Training-free mitigation methods intervene during inference through attention adjustment, logits or decoding calibration, and representation-level strategies.

3 Background and Motivation

During autoregressive LVLM inference, multimodal context is propagated through hidden states across decoder layers and decoding steps. The paper links hallucinations to larger inter-layer degradation and temporal drift, motivating dynamic compensation.

  • 3.1 Autoregressive Inference in LVLMs: An LVLM forms an initial multimodal sequence by combining visual tokens produced from the image with tokenized text tokens from the query.
  • 3.1 Autoregressive Inference in LVLMs: At decoding step t, the decoder processes Xt = [X0, y<t], which appends previously generated tokens to the initial multimodal sequence.
  • 3.1 Autoregressive Inference in LVLMs: Hidden states are updated layer by layer, and the final-layer hidden state at the current decoding position determines the next-token distribution.
  • 3.1 Autoregressive Inference in LVLMs: Representation instability during layer-wise propagation or across decoding steps can perturb next-token prediction and increase hallucinated-generation risk.
  • 3.2 Representation Instability: Hallucinated generation shows more frequent and stronger high-JS regions over image and text tokens, concentrated mainly in middle-to-late decoder layers.
  • 3.2 Representation Instability: Hallucinated object tokens exhibit larger temporal representation drift than non-hallucinated object tokens, also concentrated in middle-to-late layers.

4 Methodology

DAC stabilizes autoregressive hidden-state propagation in two dimensions: across adjacent decoder layers and across successive decoding steps. It selectively compensates high-divergence tokens, then applies temporal correction using a layer-wise cache.

  • 4 Methodology: DAC combines Layer-wise Semantic Compensation (LSC) for inter-layer divergence with Sequential Semantic Correction (SSC) for step-wise representation drift.LSC addresses instability across adjacent layers, while SSC constrains drift across generation steps.
  • 4 Methodology: LSC normalizes adjacent-layer hidden states and computes token-level inter-layer divergence using feature-wise softmax representations.For normalized vectors, the divergence is computed from their midpoint distribution.
  • 4 Methodology: LSC compensates only tokens whose inter-layer divergence exceeds threshold γ, using α to control compensation strength; lower-divergence tokens remain unchanged.The compensated state is used as the layer output.
  • 4 Methodology: SSC compares the current-token hidden state with the corrected state cached from the previous decoding step to measure temporal drift.The cache is maintained layer-wise after LSC.
  • 4 Methodology: When temporal drift exceeds its threshold, SSC moves the current state toward the cached state with correction strength β.The corrected state is written back and stored for the next step; SSC is skipped when no previous cache exists.
  • 4 Methodology: DAC applies LSC before SSC at every decoding step, so temporal correction operates on layer-stabilized hidden states.The complete inference pipeline is illustrated in Fig. 3 and summarized in Algorithm 1.

5 Experiments

DAC is evaluated across hallucination-focused and general-purpose benchmarks, recent LVLM backbones, component ablations, representation diagnostics, and inference efficiency. It consistently reduces hallucinations and preserves or improves multimodal performance with limited overhead.

  • Experiment Setup: DAC is evaluated across nine multimodal benchmarks spanning hallucination-focused and general-purpose tasks.The evaluation includes POPE, CHAIR, HallusionBench, MME, MM-Vet, MMBench, MMBench-CN, TextVQA, and LLaVA-Bench.
  • Results on General-Purpose Benchmarks: DAC improves general-purpose benchmark performance across LLaVA-1.5 and Qwen-VL-Chat while preserving OCR and cross-lingual capabilities.On MME, overall scores increase by 120.04 points on LLaVA-1.5 and 78.87 points on Qwen-VL-Chat; improvements also appear on MM-Vet, MMBench, TextVQA, and MMBench-CN.
  • Results on Hallucination Benchmarks: DAC reduces object hallucination on POPE, CHAIR, and HallusionBench across complementary evaluation protocols.On LLaVA-1.5, POPE accuracy/F1 rises from 79.83/79.29 to 87.32/86.96, CHAIR Cs/Ci falls from 47.6/13.3 to 28.2/7.3, and HallusionBench qACC/aACC rises from 8.13/41.5 to 17.05/45.5.
  • Generalization Across Recent LVLM Backbones and Scales: DAC generalizes across newer LVLM families and parameter scales, improving reported general-purpose metrics and achieving best or tied-best results in Table 3.LLaVA-Bench gains are 4.23, 5.33, 16.58, and 4.60 points across the four evaluated backbones.
  • Ablation Studies: Combining LSC and SSC yields the best MM-Vet ablation score, increasing the total from 31.1 to 33.9.SSC mainly benefits OCR and spatial reasoning, while LSC provides broader gains; together they produce complementary benefits.
  • Representation Diagnostics: Hallucinated tokens exhibit stronger inter-layer and temporal JS divergence, especially in middle-to-late decoder layers.The diagnostics show late-layer peaks and consistently larger temporal divergence for hallucinated object tokens than for non-hallucinated tokens.
  • Inference Efficiency: DAC adds little inference cost, increasing latency from 36.8 to 37.1 ms/token while maintaining peak memory at 13,797 MB.For 80-token generation, DAC takes 2975.45 ms and is substantially faster than ICD, VCD, and OPERA.

6 Conclusion

DAC is a training-free inference-time method that stabilizes hidden-state dynamics to mitigate LVLM hallucinations. Across nine multimodal benchmarks and multiple backbones, it reduces hallucinations while maintaining strong overall performance.

  • DAC detects inter-layer representation degradation and temporal representation drift during LVLM generation.It stabilizes hidden-state dynamics through lightweight compensation.
  • DAC applies Layer-wise Semantic Compensation and Sequential Semantic Correction to compensate for the two representation instabilities.The two components address inter-layer degradation and temporal drift, respectively.
  • DAC consistently reduces hallucinations while maintaining strong overall performance across nine hallucination-focused and general-purpose multimodal benchmarks.The evaluation spans multiple LVLM backbones and model scales.
  • DAC’s two components are complementary and the method incurs negligible inference overhead.
  • Dynamic hidden-state compensation is presented as a practical direction for improving the reliability of LVLM generation.

7 Limitations

The evaluation focuses on training-free hallucination mitigation for image-based LVLMs, leaving several broader settings and design choices for future study.

  • DAC is evaluated on image-based LVLMs, while its effectiveness on closed-source models, video or multi-image inputs, long-context generation, and specialized domains remains unstudied.
  • DAC uses feature-wise softmax and Jensen-Shannon divergence as lightweight instability signals, although other distance measures or adaptive criteria may improve robustness.
  • Fixed thresholds and compensation coefficients are used across datasets and models, leaving input-adaptive parameter selection for future work.
  • DAC reduces hallucination risk but does not guarantee factual correctness, so high-stakes applications still require verification and safety assessment.

A Benchmarks and Metrics

The benchmark suite combines hallucination-focused evaluations with general multimodal capability tests. These benchmarks measure object hallucination, visual grounding, image-context reasoning, perception, cognition, instruction following, OCR, and open-ended reasoning.

  • Hallucination-focused benchmarks: The hallucination-focused suite includes CHAIR, POPE, and HallusionBench, which assess object hallucination, visual grounding, and image-context reasoning.
  • Hallucination-focused benchmarks: CHAIR compares generated object mentions with ground-truth objects and reports object-instance and sentence-level hallucination scores.Lower CHAIR scores indicate fewer hallucinations.
  • Hallucination-focused benchmarks: POPE is a binary VQA-style benchmark that asks whether specified objects appear in images under random, popular, and adversarial sampling settings.
  • Hallucination-focused benchmarks: HallusionBench evaluates language hallucination and visual illusion using 346 images and 1,129 expert-written questions.
  • General-purpose benchmarks: MME evaluates multimodal perception and cognition, including visual recognition, OCR, commonsense reasoning, calculation, translation, and code-related reasoning.
  • General-purpose benchmarks: MMBench and MMBench-CN use unified multiple-choice VQA protocols to assess visual understanding, reasoning, cross-lingual grounding, and instruction following.
  • General-purpose benchmarks: MM-Vet assesses open-ended multimodal reasoning across diverse capability dimensions.
  • General-purpose benchmarks: LLaVA-Bench measures open-ended visual instruction following, while TextVQA measures reading and reasoning over scene text using answer-matching accuracy.

B Length-Normalized CHAIR Analysis

The length-normalized CHAIR analysis tests whether DAC’s hallucination reduction is merely caused by shorter captions. DAC lowers hallucinated object rates after controlling for generation length.

  • The analysis evaluates whether DAC’s lower CHAIR scores are merely caused by slightly shorter captions.
  • HObj@100µ measures hallucinated object mentions per 100 generated tokens at the corpus level, while HObj@100M averages sample-level rates.
  • 1.1206 to 0.5797: DAC decreases HObj@100µ on LLaVA-1.5.HObj@100µ is the corpus-level length-normalized hallucinated-object rate.
  • 1.0314 to 0.6569: DAC decreases HObj@100M on LLaVA-1.5.HObj@100M averages the normalized hallucinated-object rate across samples.
  • The reductions indicate that DAC more effectively suppresses unsupported object mentions after controlling for generation length.

C Backbones, Baselines, and Evaluation Protocol

The evaluation spans multiple open-source LVLM backbones, training-free baselines, and nine multimodal benchmarks covering hallucination and general capabilities. Protocols and evaluator settings are standardized to make comparisons reflect decoding behavior.

  • Backbones: Experiments use LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen3-VL-4B, and LLaVA-NeXT-7B.These backbones cover different model families, generations, and scales.
  • Baselines: DAC is compared with greedy decoding and training-free methods operating at logits, attention, and hidden-state levels.The baselines include VCD, ICD, OPERA, and MemVR, with additional reported results from AGLA, DeGF, VISTA, and AIR when available.
  • Baselines: OPERA is omitted on Qwen-based backbones because its implementation is incompatible with the Qwen evaluation pipeline.The incompatibility concerns older PyTorch and Transformers versions.
  • Evaluation protocol: Evaluation covers nine benchmarks spanning hallucination detection, multimodal perception, reasoning, OCR, cross-lingual understanding, and open-ended instruction following.The benchmarks are POPE, CHAIR, HallusionBench, MME, MM-Vet, MMBench, MMBench-CN, TextVQA, and LLaVA-Bench.
  • Evaluation protocol: POPE uses Random, Popular, and Adversarial splits, while CHAIR uses MSCOCO Val2014 images and a fixed image-description prompt.These settings support object-hallucination measurement under standardized protocols.
  • Evaluation protocol: Open-ended evaluations use official judging pipelines, with identical evaluator settings across methods within each benchmark.MM-Vet uses its official online evaluator, while LLaVA-Bench and HallusionBench use GPT-based judging protocols.

D Reproducibility

The reproducibility setup fixes prompts, decoding conditions, baseline configurations, hardware, and DAC hyperparameters, then illustrates outcomes with concrete examples. DAC’s algorithm detects representation instability across layers and decoding steps before applying compensation.

  • Evaluation setup: Evaluations use official annotation prompts with only chat-template formatting adapted, and introduce no DAC-specific prompt engineering.The benchmark protocols are followed unless otherwise specified.
  • Evaluation setup: All methods share the same image subsets for CHAIR, with three random-seed sets averaged, while official evaluators are used for MM-Vet, LLaVA-Bench, and HallusionBench.GPT-based judging settings are identical across compared methods.
  • Baseline implementation: Reimplemented baselines follow official implementations and recommended settings, including visual contrastive decoding for VCD and configured contrastive decoding for ICD.VCD uses original and distorted visual inputs; ICD uses λ = 1 and αICD = 0.1.
  • Baseline implementation: OPERA uses official beam search, whereas MemVR uses official visual retracing with fixed layer and entropy settings.OPERA is omitted on Qwen backbones because of software incompatibility.
  • DAC configuration: DAC uses greedy decoding with fixed hyperparameters γ = 0.5, α = 0.2, τ = 0.3, and β = 0.3 across all datasets.No dataset-specific hyperparameter tuning is performed.
  • DAC algorithm: DAC processes visual and textual tokens autoregressively, compares feature-wise-softmax states across layers and time, and applies residual compensation when divergence exceeds thresholds.The algorithm maintains temporal caches and decodes each generated token after the layer-wise and sequential checks.
  • Capability examples: In four capability examples, DAC changes LLaVA’s answer to match the ground truth in each reported case.The examples cover Python-code output, cake cutting, parallelogram area, and movie origin questions.
Loading 2608.28058v1…