Source-linked AI summary
Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
Shuran Ma, JiaLe Li, Yuxin Dong, Shan Zheng, Qingyun Jiang, Xiang Chen, Qi Zhu, Deyi Ji, Yifan Yang, Jianfeng Pan, Yu Tian, Xue Yang
TL;DR
LVLM hallucinations arise in the interaction between visual evidence and language generation, while existing training-free methods often emphasize visual evidence and leave dynamic use of other contexts underexplored. AIMS adaptively coordinates visual, prefilled-text, and generated contexts during decoding; across benchmarks it reduces object hallucination while maintaining multimodal capabilities.
Problem
Training-free mitigation often emphasizes contrastive decoding or visual enhancement, leaving open whether LVLMs can dynamically regulate multiple context sources to suppress hallucinations.
Source: §1 · pp. 1–2Method
AIMS is a training-free framework that uses query-context affinity to adaptively coordinate visual, prefilled-text, and generated contexts during decoding.
Source: §1 · p. 2Results
AIMS achieves up to a 27.1% relative reduction in CHAIR CS across four benchmarks while preserving general multimodal capabilities.
Source: §1 · p. 2Takeaways & Limitations
LVLMs’ visual attention varies by generation stage, and textual contexts can also contribute corrective signals, supporting adaptive multi-context coordination over uniform visual enhancement.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.
1 Introduction
The introduction frames LVLM hallucination as a problem of interaction between visual evidence and language generation, and asks whether models can dynamically regulate multiple context sources. It presents AIMS as a training-free approach motivated by stage-dependent visual attention and the corrective role of textual contexts.
1 Introduction
- Existing training-free approaches emphasize contrastive decoding or visual enhancement, leaving textual prompts and generation history underexplored.This motivates asking whether LVLMs can regulate different context sources dynamically to suppress hallucinations.Source: §1 · p. 2
- LVLMs’ visual attention peaks during object grounding and declines during semantic summarization, while adaptive steering and textual contexts can aid hallucination mitigation.The introduction reports that adaptive steering outperforms uniform visual amplification and that prefilled text and generation history provide corrective signals.Source: §1 · p. 2
- Across four benchmarks, AIMS achieves up to a 27.1% relative reduction in CHAIR CS while preserving general multimodal capabilities.It coordinates visual, prefilled-text, and previously generated contexts during decoding.Source: §1 · p. 2
- AIMS uses query-context affinity to coordinate visual, prefilled-text, and generated contexts, unlike methods that amplify one information source uniformly.The introduction also reports experiments across representative LVLMs and four benchmarks, with reduced hallucination and preserved generation quality.Source: §1 · pp. 2–3
2 Related Work
Related work describes the evolution of LVLM architectures and surveys approaches to mitigating object hallucination, including training-based and training-free interventions.
2.1 LARGE VISION-LANGUAGE MODELS
- LVLM architectures evolved from LLaMA-based systems with MLP or Q-Former visual projection toward backbones designed for denser multimodal semantics and higher-resolution interaction.The section cites LLaVA and MiniGPT as early examples, and Qwen-VL, InternLM-XComposer, and InternVL as later developments.Source: §2.1 · p. 3
2.2 MITIGATING HALLUCINATIONS IN LVLMS
- Hallucination mitigation ranges from alignment improvements and supervised fine-tuning to inference-time attention interventions and contrastive decoding.Attention interventions often add inference overhead; contrastive methods can be unstable because of noise from contrasting conditions.Source: §2.2 · p. 3
3 Preliminary Study
The preliminary study examines visual-attention dynamics, adaptive versus fixed visual steering, and the contributions of multiple context branches on CHAIR using Qwen2.5-VL-3B and 500 MSCOCO examples.
3 Preliminary Study
- Adaptive-V records CS 40.2 while retaining 292.41 tokens, compared with fixed steering at α = 0.15, which records CS 39.0 and 199.70 tokens.Adaptive-V also reduces CI to 8.33, suggesting that visual evidence should be reinforced according to the model’s vision-attending tendency rather than uniformly amplified.Source: Table 1 · p. 4, §3 · p. 4
- P + G reduces CS from 56.00 to 47.62 without visual reinforcement, while V + P + G yields the lowest CS among branch combinations at 46.00.G alone achieves a CI of 8.80, motivating coordination of multiple context branches rather than vision-only enhancement.Source: Table 1 · p. 4, §3 · p. 4
- Visual attention peaks around T1 ∈ {4, 5} as generation begins grounding objects and attributes, then declines toward summarization at T2.The pattern appears across five sequence-length bins despite substantial differences in sequence length.Source: §3 · p. 4
4 method
AIMS builds compact prototypes from visual, prefilled-text, and generated-token key caches, then uses query-dependent affinities to coordinate those sources. It combines the weighted prototypes into a steering target that updates the current query before standard attention.
4 method
- AIMS mean-pools cached keys by token source to form prototypes for vision, prefilled text, and generated context.The prototypes use the model’s existing query–key space and require no additional encoding or auxiliary forward passes.Source: §4 · p. 5
- Query–context affinity determines each source’s contribution, with weights adapting across sources, attention heads, and decoding timesteps.AIMS uses the current query as a signal of which context is relevant at that generation step.Source: §4 · p. 5
- Visual and prefilled-text prototypes remain fixed during decoding, while the generated-context prototype uses the latest token’s key as a lightweight summary of prior generation.That key has already contextualized preceding tokens through causal self-attention.Source: §4 · p. 6
- AIMS combines affinity-weighted prototypes into one steering target, updates the current query, and passes it through the original attention operation.It does not separately reweight the context branches; the key-value cache and original attention computation remain unchanged, with no additional forward pass.Source: §4 · p. 6
5 experiment
Experiments across three LVLMs, four benchmarks, and multiple decoding strategies show that AIMS reduces hallucination while preserving multimodal quality. Ablations support coordinating multiple context branches adaptively, with low reported computational overhead.
5.1 EXPERIMENTAL SETTINGS
- Evaluations cover three LVLMs, three decoding settings, and four benchmarks, with metrics tailored to object hallucination, factual faithfulness, and general capabilities.The models are LLaVA-1.5-7B, Qwen2.5-VL-3B, and Qwen3.5-9B; decoding includes greedy, five-beam search, and nucleus sampling.Source: §5.1 · p. 6
5.2 EXPERIMENTAL RESULTS
- AIMS achieves the lowest CHAIR CS in all nine model-decoding settings and lower AMBER-G Hal than VTI in all six settings, with comparable Cover.The AMBER-G results also report reduced Cog across Qwen2.5-VL-3B decoding strategies, without a substantial coverage trade-off.Source: Table 2 · p. 7, §5.2 · pp. 7–8, Table 3 · p. 7
- AIMS achieves the highest FaithScore and sentence-level FaithScore across all three decoding strategies on Qwen2.5-VL-3B.These atomic-fact verification results provide evidence of improved faithfulness for fine-grained visual facts in free-form responses.Source: Table 4 · p. 8, §5.2 · p. 8
- AIMS largely preserves general multimodal capabilities while substantially reducing hallucinations, achieving a better balance than VTI and MemVR.The comparison covers Tables 2–5; VTI shows noticeable MME degradation on the two Qwen models, while MemVR provides more limited hallucination reduction.Source: Table 5 · p. 8, §5.2 · p. 8
5.3 ABLATION AND ANALYSIS
- Affinity-adaptive weighting yields lower CS and CI than fixed steering at comparable F1, and results remain robust across steering-strength and temperature configurations.Strong fixed steering at α = 0.1 substantially reduces hallucination but severely degrades F1.Source: Table 7 · p. 9, §5.3 · p. 9, Table 8 · p. 10
- Steering-weight variability differs by branch: vision varies broadly across heads in Layers 31–33, while prefilled text and generated context show distinct layer patterns.These patterns motivate fine-grained affinity-adaptive multi-source steering across decoding timesteps.Source: Figure 6 · p. 9, §5.3 · p. 10
- Combining all three context branches achieves the best CHAIR and F1 results, while AIMS reduces hallucination with different affinity-weighting functions.Visual-plus-text and visual-plus-generation-history configurations exhibit trade-offs across metrics.Source: Figure 5 · p. 9, §5.3 · p. 9, Table 6 · p. 9
- On a fixed subset of 10 CHAIR images, AIMS adds nearly no peak memory over baseline and has the second-lowest inference latency, close to MemVR.The paper reports this as a favorable trade-off between effectiveness and computational efficiency.Source: §5.3 · p. 10
6 Conclusion
The conclusion presents AIMS as a training-free approach to coordinating multiple contexts during decoding. Experiments across LVLM architectures and decoding strategies show reduced object hallucination while preserving general-purpose multimodal capabilities.
6 Conclusion
- AIMS constructs compact context prototypes and dynamically coordinates their contributions during decoding, reducing object hallucination while preserving strong general-purpose multimodal capabilities across architectures and decoding strategies.The conclusion frames this approach as a shift from visual-centric intervention toward adaptive multi-context coordination.Source: §6 · p. 10
B.1 Alternative Affinity Weighting Functions
This section compares alternative functions for weighting query–context affinities. The alternatives use Euclidean distance, nonlinear cosine similarity, or KL divergence to determine the weights.
B.1 Alternative Affinity Weighting Functions
- The section evaluates three alternative affinity-weighting functions and reports their results in Table 6.Source: §B.1 · p. 14
- RBF weighting assigns larger weights to contextual prototypes closer to the query in Euclidean distance.The section also notes that each attention head has dimensionality d_h.Source: §B.1 · p. 14
- NCS weighting transforms the normalized query–prototype inner product with softplus and sigmoid.Source: §B.1 · p. 14
- KL-based weighting first transforms the query and contextual prototype into probability distributions, then determines the weight from their KL divergence.Source: §B.1 · p. 14
B.2 Case Studies
The case studies qualitatively compare Baseline and AIMS outputs across LVLMs and benchmarks. They include examples of hallucinated image descriptions and AIMS responses described as correct on varied recognition and reasoning tasks.
B.2 Case Studies
- The case studies span CHAIR examples with Qwen3.5-9B and Qwen2.5-VL-3B, an AMBER-G example, and an MME example with Qwen3.5-9B using nucleus sampling.Source: §B.2 · p. 14
- Two detailed image descriptions contain identified hallucinations: a beige-coated person is said to hold a suitcase rather than a bag, and people outside are incorrectly described as carrying umbrellas.Source: §B.2 · pp. 15–16
- AIMS is described as correctly identifying object colors, recognizing a landmark, verifying artwork attribution, and performing code reasoning.Source: §B.2 · p. 20
- Other AIMS examples are described as correctly recognizing a backpack, translating text, reading logo text, interpreting a traffic signal, identifying a director, and verifying an actor's name.Source: §B.2 · p. 20
- Further examples describe AIMS as correctly counting pizza pieces, recognizing a scene, and solving an arithmetic problem.Source: §B.2 · p. 20