Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation

1. Introduction

• Prior training-free mitigation methods adjust attention to image tokens but overlook the visual influence of instruction tokens during decoding.Supervised fine-tuning approaches require high-quality datasets and significant training costs.Source: §1 · p. 2
• Instruction tokens can repeatedly direct attention to misleading image regions, limiting broader contextual perception and accounting for up to 46.7% of total hallucinations.The paper names this pattern Attention Hijacking and calls the responsible instruction tokens Attention Hijackers.Source: §1 · p. 2
• The paper poses two questions: how to identify Attention Hijackers among instruction tokens and how to isolate their influence so models can explore visual information appropriately.Its detection approach assesses the visual information instruction tokens contribute to subsequently generated tokens.Source: §1 · p. 2
• AID detects hijackers using instruction-driven visual salience, masks their visual attention, then rebalances instruction- and image-driven salience to avoid excessive masking.The final stage is called Re-Disentanglement.Source: §1 · p. 2
• AID is a training-free strategy that detects Attention Hijackers and disentangles their influence to reduce hallucinations without additional training or external tools.The paper reports significant improvements in alleviating hallucinations through comprehensive experiments.Source: §1 · p. 2

2. Related Work

• Object hallucination refers to generating plausible but mismatched or missing objects in images.Prior work addresses it through contrastive learning, ROI feature fusion, and data augmentation, while applying these techniques to autoregressive LVLMs is challenging.Source: §2 · pp. 2–3
• The Figure 3 illustration contrasts an Attention Hijacker's visual-attention distribution with the image token's, showing greater similarity to the hallucination token for the hijacker.The caption describes consistently higher similarity values for the hijacker.Source: Figure 3 · p. 3
• LVLM hallucination research includes evaluation and detection methods, refined fine-tuning datasets, post-hoc revisors, and factually augmented RLHF.CHAIR evaluates caption accuracy, while POPE evaluates binary object recognition.Source: §2 · p. 3
• Attention-based methods address visual attention deficits by amplifying relevant image-token weights, filtering image information, or addressing long-term decay in Rotary Position Encoding.These approaches focus on visual elements, whereas the paper identifies disruptive instruction tokens as an overlooked influence.Source: §2 · p. 3

3.1. Problem Formulation

• Each decoder attention head combines value vectors using attention weights, with each row representing a token’s weights during feature mixing.The heads use their own queries, keys, and values.Source: §3.1 · p. 3

3.2. Analysis of Attention Hijacking

• Hijacker attention remains similar to hallucinated-token attention across generation and can exceed image-token similarity, with KV-cache reuse sustaining focus on misleading regions.The paper links this pattern to ViT attention concentrated on background regions associated with high-norm outlier tokens.Source: §3.2 · pp. 3–4

3.3. Attention Hijackers Detection

• AID detects hijackers by tracking instruction-driven visual information through image-token self-interaction, image–instruction interaction, and transmission to generated tokens.Image-token visual information evolves across layers; instruction-token visual information accumulates directly from image tokens and indirectly from previous instruction tokens.Source: §3.3 · pp. 4–5

3.4. Attention Disentanglement

• Attention Disentanglement cuts off visual information from identified hijackers while retaining standard interactions for non-hijacking instruction tokens.The cutoff includes image tokens and non-hijacking instruction tokens carrying image information.Source: §3.4 · p. 5

3.5. Re-Disentanglement

• Re-Disentanglement compares instruction-driven with image-driven visual salience, removing hijacker influence only when the instruction token’s impact decreases and preserving it otherwise.This selective comparison is intended to avoid disrupting useful visual interactions through broad masking.Source: §3.5 · pp. 5–6

4.1. Experiment Setup

• Experiments covered five LVLM families and five benchmarks, comparing AID with greedy decoding, beam search, DoLa, OPERA, and VCD.The setup used beam search with beam size 3 for baselines and implemented AID with Hugging Face Transformers.Source: §4.1 · pp. 6–7

4.2. Main Results

• AID consistently surpassed state-of-the-art methods on most POPE settings and achieved the lowest CHAIR hallucination rates at sentence and instance levels while preserving recall.On MME, it reduced object- and attribute-level hallucinations; the authors note that few-token binary outputs constrain methods relying on longer sequences and special entity handling.Source: §4.2 · p. 7
• AID achieved significant overall performance improvements on general-purpose benchmarks, particularly OCR and spatial reasoning, with an average increase of 6.7%.MMBench results are described as highly competitive with state-of-the-art methods.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 6 · p. 8

4.3. Ablation Studies

• Masking the entire visual area improved CHAIR scores above baseline, while lower visual regions accumulated more information in self-interacting image regions.The study progressively masked visual areas from top to bottom using LLaVA-1.5-7B and CHAIR.Source: §4.3 · p. 8, Figure 7 · p. 8
• AHD reduced CHAIRs and CHAIRI by 5.2% and 1.5%, respectively, and masking two hijackers performed better than masking three.Random masking increased hallucinations; AHD+RD reduced CHAIRs and CHAIRI by 8.2% and 2.4% while maintaining recall, whereas AHD+linear probing added little over AHD with two masked hijackers.Source: §4.3 · p. 8, Table 5 · p. 8

4.4. Visualization Results

• In a captioning case study, AID described train colors and people’s positions without irrelevant details, unlike LLaVA-1.5’s hallucinated scene details.The authors report that AID improves accuracy and reduces hallucinations, especially in complex scenes.Source: §4.4 · pp. 8–9, Figure 8 · p. 8

• AID detects and disentangles Attention Hijackers to restore proper attention, but focuses on image understanding rather than factual accuracy.The conclusion suggests integrating retrieval-augmented generation in future work to help mitigate this limitation.Source: §5 · p. 9