PaperlayerView exact revision on arXiv

Source-linked AI summary

DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editing

Jae-Ho Lee, Jeong-Eun Lee, Gyeong-Moon Park

arXiv:2609.36440v1cs.CV

TL;DR

Object hallucination persists in LVLMs, while teacher-forced editing may not capture autoregressive decisions or multimodal interactions. DARE combines textual and image contrasts with autoregressive-aware signals, reducing hallucinations across benchmarks while preserving multimodal perception and inference efficiency.

  • Problem

    Teacher-forced editing directions may not fully reflect autoregressive token-selection dynamics, and text-only directions may affect visual-token interactions.

    Source: §1 · p. 3

  • Method

    DARE is a training-free weight-editing framework combining teacher-forced textual contrasts, autoregressive-aware signals from decoding, and controlled image contrasts.

    Source: §1 · pp. 3–4

  • Results

    DARE consistently reduces object hallucinations across multiple LVLM benchmarks while preserving multimodal perception capability and inference efficiency.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The reported results show hallucination reduction alongside retained multimodal perception capability and inference efficiency.

    Source: §1 · p. 4

    Text-only editing directions may correlate with components supporting visual-token interactions and accompany reduced visual-token attention across layers.

    Source: §1 · p. 3

Abstract

from arXiv · show

Large vision-language models (LVLMs) have recently achieved remarkable progress across multimodal tasks, yet object hallucination remains a persistent challenge where models generate descriptions inconsistent with the visual input. Recent work mitigates hallucinations through training-free representation editing, typically by constructing hallucination-related directions from teacher-forcing (TF) contrasts between hallucinated and truthful responses. However, LVLMs operate through autoregressive (AR) decoding during generation, raising the question of whether TF-based analysis fully reflects the generation dynamics that lead to hallucinated outputs. In this paper, we analyze the relationship between TF-based editing and AR generation behavior and find that TF-based editing alone may be insufficient to capture both decoding dynamics and multimodal interactions associated with hallucinations. To address this limitation, we propose DARE (Dual-path Auto-Regressive-aware Editing), a hybrid hallucination editing framework that integrates two complementary contrast pathways: textual contrasts and image contrasts, together with autoregressive-aware representation signals. Specifically, DARE constructs hallucination editing directions from (1) TF-based textual contrasts, (2) AR-aware representation transitions during decoding, and (3) controlled visual differences between paired images. Extensive experiments on multiple LVLM hallucination benchmarks demonstrate that DARE consistently reduces object hallucinations while preserving multimodal perception capability and inference efficiency. Our implementation code is available at https://github.com/KU-VGI/DARE.

1 Introduction

Object hallucinations make LVLM descriptions inconsistent with visual input and reduce reliability. DARE combines textual and image contrasts with autoregressive-aware signals, reducing hallucinations while preserving multimodal perception and inference efficiency.

1 Introduction

  • Object hallucination occurs when generated descriptions conflict with objects in the visual input, reducing LVLM reliability and limiting real-world deployment.Prior representation-editing approaches construct hallucination-related directions by contrasting truthful and hallucinated responses under teacher forcing.Source: §1 · pp. 1–2
  • Under teacher forcing, truthful tokens often remain highly ranked on hallucinated captions, while editing frequently barely changes truthful–hallucinated logit margins during autoregressive decoding.The truthful token remains within the top five with high probability, but small margin changes often leave the hallucinated token selected.Source: §1 · pp. 2–3
  • Text-only editing directions are accompanied by reduced attention to visual tokens across layers, suggesting a potential collateral effect on visual pathways.The authors note that multimodal representations entangle textual and visual-token interactions.Source: §1 · p. 3
  • DARE combines teacher-forced textual contrasts, autoregressive-aware decoding signals, and controlled visual differences between paired images.The framework integrates textual and AR-aware directions into an Augmented HalluSpace and complements them with Visual-Difference Editing.Source: §1 · pp. 3–4
  • DARE consistently reduces object hallucinations across multiple LVLM benchmarks while preserving multimodal perception capability and inference efficiency.Its image contrasts use controlled visual differences between paired images alongside textual and autoregressive-aware signals.Source: §1 · pp. 3–4

2 Related Work

Prior work mitigates object hallucination through training-stage methods and inference-stage interventions, including latent steering and weight editing. The paper also frames a gap between teacher-forced editing analyses and autoregressive generation, where token selection depends on relative scores.

Mitigating Object Hallucination in LVLMs

  • Object-hallucination mitigation spans training-stage objectives or datasets and inference-stage decoding, attention intervention, and weight-editing approaches.Training-stage methods require extensive computational resources and time.Source: §2 · p. 4
  • Latent steering adds precomputed stable directions at inference time, whereas weight editing projects weights onto the null space of hallucination-related components.The paper positions its weight refinement as a way to preserve multimodal dependencies more effectively.Source: §2 · p. 5

Teacher-forcing and Auto-regressive Generation Gap

  • Teacher forcing uses ground-truth tokens, unlike autoregressive generation, where models rely on their own outputs and select among candidates by relative scores.The implications of this mismatch for hallucinated generation in LVLMs remain unclear.Source: §2 · p. 5

3 Method

DARE combines teacher-forced textual contrasts, autoregressive representation transitions, and controlled visual contrasts to construct hallucination-editing directions. It refines TF and AR directions into an augmented space and applies null-space editing to selected model projections.

3 Method

  • DARE combines TF-based textual contrasts, AR-aware signals, and controlled image contrasts to identify hallucination-related representation directions.The TF and AR directions form an Augmented HalluSpace, complemented by Visual-Difference Editing.Source: §3 · p. 5

3.1 Preliminary: TF-based HalluSpace Editing

  • TF-based HalluSpace editing contrasts truthful and hallucinated responses under teacher forcing, then uses null-space projection to remove associated representation components without additional training.Because hallucinations arise during autoregressive decoding, this TF-only approach motivates adding decoding-time signals.Source: §3.1 · p. 6

3.2 AR-aware HalluSpace Construction

  • AR-aware directions come from hidden-state transitions in a short window preceding the first hallucinated object token during autoregressive decoding.The transition vectors are collected across hallucination events and decomposed to obtain feature-space directions.Source: §3.2 · pp. 6–7

3.3 Augmented HalluSpace Editing

  • DARE residualizes AR directions against the TF subspace, then combines them with TF directions into an orthonormal Augmented HalluSpace for null-space editing of FFN down-projection weights.Residualization preserves AR information not already explained by TF directions.Source: §3.3 · pp. 7–8

3.4 Visual-Difference Editing

  • Visual-Difference Editing contrasts paired images with controlled object removals under the same teacher-forced text, isolating representation changes induced by visual input.The resulting visual-difference directions are used for null-space editing of attention output projection weights.Source: §3.4 · pp. 8–9

4 Experiments

Experiments across LVLMs and hallucination benchmarks evaluate DARE’s effectiveness, generality, multimodal capability, and efficiency. Ablations and further analyses examine its components, editing settings, and effects on visual attention.

4.1 Experimental Setup

  • The experiments use official checkpoints of LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2 without additional fine-tuning, evaluating CHAIR, POPE, and MME.The primary analysis focuses on LLaVA-1.5; CHAIR and POPE use MSCOCO images, while MME assesses broader multimodal abilities.Source: §4.1 · pp. 9–10

4.2 Experimental Results

  • DARE achieves the lowest hallucination scores across evaluated LVLMs, reducing LLaVA-1.5 CHAIRS from 24.32 to 14.93 and CHAIRI from 8.91 to 6.07.The results also show improvements over the base model and prior mitigation methods, with similar trends on MiniGPT-4 and mPLUG-Owl2.Source: Table 1 · p. 10, §4.2 · pp. 10–11
  • On POPE, DARE consistently improves accuracy, precision, and F1 across evaluated LVLMs, extending gains from caption generation to object-existence questions.Because POPE tests yes/no claims about object presence, the authors interpret these improvements as reduced false-positive object predictions.Source: Table 2 · p. 11, §4.2 · p. 11
  • DARE maintains competitive or improved MME performance against the base model and strong mitigation baselines while reducing hallucinations.MME spans perception and reasoning skills including object existence, counting, position, color, OCR, and common-sense reasoning.Source: §4.2 · pp. 11–12, Figure 3 · p. 12

4.3 Ablation Studies and Further Analysis

  • The best AR-aware settings edit higher layers (16–31), use rank k = 2, and use a temporal window of 3.The authors describe the AR hallucination signals as low-dimensional and the relevant transitions as concentrated shortly before the hallucinated token.Source: Table 4 · p. 12, §4.3 · pp. 12–13
  • Each DARE component reduces both CHAIR metrics, AR-aware editing is the strongest single component, and combining all three yields the best overall result.TF+AR and AR+visual-difference outperform single components; the full combination reduces LLaVA-1.5 CHAIRS from 24.32 to 14.93 and CHAIRI from 8.91 to 6.07.Source: §4.3 · p. 12, Table 3 · p. 12
  • DARE improves over the base model and Nullu on CHAIR and POPE across evaluated Qwen3-VL and Gemma-3 model families and scales.The evaluated variants include Qwen3-VL-4B, Qwen3-VL-8B, Gemma-3-4B, and Gemma-3-12B.Source: Table 5 · p. 13, §4.3 · p. 13
  • DARE reduces hallucination with nearly the same generation speed as efficient editing methods and no inference-time overhead compared with Nullu.Unlike decoding-based methods, which require extra decoding operations and lower throughput, DARE edits weights once and uses the original inference pipeline.Source: §4.3 · p. 13, Figure 4 · p. 14

5 Conclusion

The conclusion presents DARE as a training-free editing framework combining teacher-forced textual contrasts, autoregressive signals, and visual contrasts. The paper reports reduced hallucinations alongside competitive multimodal performance and inference efficiency.

5 Conclusion

  • DARE edits the FFN down-projection with an Augmented HalluSpace combining TF and AR directions, and edits the attention output projection using visual differences.The visual-difference editing is intended to preserve visually grounded information flow.Source: §5 · p. 14
  • DARE consistently reduces hallucinations while maintaining competitive multimodal performance and inference efficiency on standard LVLM hallucination benchmarks.The conclusion attributes the absence of decoding-time overhead to editing performed once.Source: §5 · p. 14
  • Additional evaluations report gains across recent LVLM families and model scales, rather than a single backbone or parameter size.Source: §5 · p. 14
Loading 2609.36440v1…