Source-linked AI summary
Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
Aditi Sarker, Rafi Ibn Sultan, Hui Zhu, Dongxiao Zhu, Prashant Khanduri
TL;DR
LVLM hallucinations are associated with unstable representations under semantics-preserving perturbations, while existing stability remedies intervene during inference. INFUSE instead learns stable visual and textual representations during fine-tuning, aligns them across modalities, and reduces hallucination across multiple benchmarks without inference-time overhead.
Problem
LVLMs can generate fluent descriptions unsupported by visual evidence, and their embeddings vary substantially under controlled perturbations while hallucination rates rise with that variability.
Method
INFUSE stabilizes visual and textual representations around perturbation-averaged and truthful-response anchors, then applies bidirectional cross-modal alignment and supervised generation loss.
Results
Across LLaVA-1.5-7B, LLaVA-1.6-7B, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63%, improves several hallucination benchmarks, preserves VQA-v2 and TextVQA, and adds no inference-time overhead.
Takeaways & Limitations
The results support implicit feature stabilization as a practical training principle for reducing hallucination while retaining visual-language task performance.
Takeaways & Limitations
The evaluation covers single-image, single-turn settings at the 7-8B scale, leaving multi-image, multi-turn, and larger-model settings open.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings; hallucination rates rise together with this variability. Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query. We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment. Our framework, INFUSE, first stabilizes visual and textual representations around perturbation-averaged and ground-truth anchors, then aligns the stabilized representations across modalities with bidirectional contrastive objectives. We prove that the anchor's root-mean-square deviation from the perturbation-mean representation shrinks at rate $1/\sqrt{K}$ in the number of views, and that under a Lipschitz decoder, this bounds how much any perturbation can change the model's hallucination behavior. On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to each base model, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves VQA-v2 and TextVQA, all with no inference-time overhead.
1 Introduction
LVLM hallucinations track instability in visual representations under image perturbations, motivating INFUSE’s implicit, training-time stabilization and subsequent cross-modal alignment. Across three backbones, INFUSE substantially reduces hallucination while preserving visual-language task performance without inference-time overhead.
- LVLMs produce fluent responses unsupported by visual evidence, a failure documented across POPE, AMBER, MMHal-Bench, and HallusionBench.
- Feature instability and hallucination: Embedding variance and hallucination rates both increase with perturbation strength, while perturbation-invariant training reduces both before cross-modal alignment.The association alone is not treated as causal; direct intervention provides evidence that stability is an actionable training target.
- Implicit vs. explicit stabilization: INFUSE builds perturbation invariance into model weights during fine-tuning instead of applying latent steering or constrained decoding at inference.This avoids per-query intervention while changing the underlying representation space during training.
- Our approach: INFUSE stabilizes visual and textual representations around perturbation-averaged and truthful-response anchors, then aligns them bidirectionally across modalities.Stage 1 precedes Stage 2 so alignment operates on lower-variance representations; the visual anchor’s estimation error contracts at rate 1/√K.
- 46-63%: INFUSE reduces AMBER CHAIR relative to corresponding base models across LLaVA-1.5-7B, LLaVA-1.6-7B, and Qwen3-VL-8B-Instruct.It also improves ObjHal, MMHal, HallusionBench, and POPE, preserves or improves VQA-v2 and TextVQA, and adds nothing at inference.
2 Related Work
Related work divides hallucination mitigation into inference-time interventions and training-based improvements to supervision, preference alignment, or cross-modal learning. INFUSE differs by directly learning perturbation-stable representations during training and eliminating per-query intervention.
- Hallucination mitigation methods include inference-time and training-time families addressing persistent LVLM hallucination.
- Explicit stabilization: Inference-time methods refine decoding, verify responses, intervene on attention, or correct outputs without updating model parameters.
- Explicit stabilization: INFUSE learns stable representations during training, outperforms VTI on adversarial POPE on LLaVA-1.5, and avoids per-query intervention cost.
- Implicit mitigation: Training-based methods improve supervision, preference alignment, or image-text correspondence but do not explicitly learn perturbation-invariant representations.
3 The Proposed INFUSE Framework
INFUSE uses two-stage training to stabilize visual and textual representations before aligning them across modalities, with theory linking view averaging to output stability. Its design combines perturbation-averaged and truthful-response anchors with contrastive alignment and supervised generation objectives.
- Framework overview: INFUSE first learns stable representations within each modality, then aligns those stabilized representations across modalities while retaining a standard generation objective.The ordering is intended to make cross-modal matching reflect semantics rather than appearance noise.
- Stage 1: Intra-modal stabilization: Perturbation-averaged visual anchors provide lower-variance targets, while lagged embeddings are stop-gradient references from the previous optimization step.The anchor averages embeddings from controlled views; the lagged reference captures the pre-update representation.
- Stage 1: Intra-modal stabilization: Repulsion from the lagged embedding prevents projector stagnation by requiring movement toward the low-variance anchor and away from the pre-update state.This repulsive term serves as an anti-collapse and escape mechanism rather than a semantic judgment about the previous representation.
- Stage 1: Intra-modal stabilization: Text stabilization pulls generated-response embeddings toward human-corrected truthful responses and away from hallucinated-caption embeddings.Gradients shape the decoder hidden states of the model’s generated token sequence.
- 3.2 Theoretical Justification: Theoretical analysis shows the anchor’s root-mean-square deviation from the perturbation-mean representation contracts at rate 1/sqrt(K), while a Lipschitz decoder bounds perturbation-induced changes in hallucination behavior.The bound limits output-probability shifts; it does not itself guarantee lower hallucination rates.
- 3.3 Cross-modal alignment: Stage 2 jointly optimizes bidirectional visual-text contrastive losses and supervised generation loss to close the representation gap without architectural modification.The cross-modal objectives use image-caption pairs and unpaired-sample negatives, while generation loss preserves fluency and downstream ability.
- Implementation and ablation: Using K = 100 masked views and training each stage for 5 epochs, the combined visual and textual stabilization branches yield lower hallucination rates than either branch alone.The reported implementation uses temperature τ = 0.07 for all contrastive objectives.
4 Experimental Evaluation
INFUSE is evaluated across three LVLM backbones using hallucination, fine-grained grounding, general understanding, ablation, cost, sensitivity, and qualitative analyses. It consistently reduces hallucinations, preserves general vision-language performance, and achieves these results with lower reported training cost and no inference-time intervention.
- Hallucination benchmarks: Across all three backbones, INFUSE improves nearly all hallucination metrics, including AMBER CHAIR reductions of 46%, 63%, and 51% relative to the corresponding base models.The AMBER reductions are from 7.8 to 4.2 on LLaVA-1.5, 8.3 to 3.1 on LLaVA-1.6, and 5.9 to 2.9 on Qwen3-VL.
- Fine-grained MMHal analysis: INFUSE achieves the highest overall GPT-4 score of 3.87 with LLaVA-1.6 and the lowest hallucination rate of 15.12% with Qwen3-VL on fine-grained MMHal evaluation.The Qwen3-VL model also performs best in seven of eight fine-grained categories.
- Fine-grained discriminative analysis: INFUSE improves both attribute- and relation-level AMBER scores on LLaVA-1.6 and Qwen3-VL, while CHiP reduces LLaVA-1.6 relation-level F1 from 69.6 to 62.0.The results indicate that final-token anchors do not prevent gains on these finer-grained grounding measures.
- Comparison with explicit stabilization: On adversarial POPE, INFUSE improves LLaVA-1.5 accuracy from 82.57% to 83.56% over VTI and reaches 90.40% accuracy with Qwen3-VL.The Qwen3-VL model achieves an overall F1 of 90.15.
- General vision-language understanding: Across all three backbones, INFUSE matches or improves the corresponding base models on VQA-v2 and TextVQA, achieving the best performance on LLaVA-1.6 and Qwen3-VL.These results indicate that hallucination mitigation does not reduce the evaluated general multimodal capabilities.
- Ablation and training cost: Combining visual-to-visual and textual-to-textual stabilization yields the lowest hallucination rates, while INFUSE uses 6.2 GPU-hours versus CHiP’s 12.5 GPU-hours in the reported training-cost comparison.The cost comparison uses INFUSE’s 2K training set and CHiP’s official 5K+ recipe, rather than the matched controlled comparison.
5 Limitations
INFUSE’s evaluation is bounded to sentence-level embeddings and single-image, single-turn 7–8B settings. The theory controls perturbation-induced output variation, while the reason stabilization lowers absolute hallucination rates remains empirically established rather than theoretically explained.
- INFUSE operates on sentence-level final-token embeddings, leaving token- and region-level stabilization as a natural extension.
- The evaluation covers single-image, single-turn interactions at the 7–8B scale; multi-image, multi-turn, and larger-model settings remain open.
- Proposition 3.1 controls perturbation-induced output variation, but why stabilization lowers absolute hallucination rates remains an open theoretical question.
6 Conclusion
INFUSE implements implicit feature stabilization by anchoring each modality and then aligning stabilized representations across modalities. Across three backbones, it reduces hallucination while preserving VQA-v2 and TextVQA performance, with no inference-time cost.
- INFUSE stabilizes each modality with perturbation-averaged and ground-truth anchors, then applies bidirectional contrastive alignment across modalities.
- 46–63% AMBER CHAIR reduction is reported across three backbones relative to corresponding base models.
- INFUSE preserves VQA-v2 and TextVQA performance while adding no inference-time overhead.
- The method is reported as practical because training costs 6.2 GPU-hours on 2K samples and deployment is free of extra computation.
A Algorithms for INFUSE
INFUSE uses two training stages: intra-modal stabilization constructs visual and textual anchors, then inter-modal alignment grounds the stabilized representations. The implementation trains modality-specific objectives before applying cross-modal contrastive and generation losses.
- Stage [S1] performs intra-modal stabilization, while [S2] performs inter-modal alignment on the stabilized representations.
- [S1] Intra-modal stabilization: [S1] trains on 2,000 RLHF-V triplets containing an image, a ground-truth caption, and a hallucinated rejected caption.
- [S1] Intra-modal stabilization: Visual anchors average K = 100 masked views, and the clean-image embedding is pulled toward this anchor while being repelled from a lagged embedding.
- [S1] Intra-modal stabilization: Textual stability pulls generated-response embeddings toward truthful anchors and away from hallucinated-caption embeddings across the batch.
- [S1] Intra-modal stabilization: The [S1] objective combines visual and textual stability losses and updates the projector and decoder parameters with AdamW.
- Embedding stability under perturbations: [S1] shifts embedding-variance distributions toward lower values, indicating more stable perturbation-invariant visual representations.
- [S2] Inter-modal alignment: After [S1], [S2] uses bidirectional InfoNCE objectives to bring paired visual and textual embeddings together, separate unpaired combinations, and preserve linguistic quality with generation loss.
- Effect of the complete two-stage framework: The complete framework reduces hallucination from approximately 49% to 24% on AMBER and 43% to 16% on MMHal for LLaVA-1.6-7B.
B Training and Data Details
Training uses frozen vision encoders with lightweight projector and LoRA updates across LLaVA and Qwen3-VL configurations. Evaluation spans five hallucination benchmarks, and increasing the number of perturbation views improves mitigation on LLaVA-1.6.
- Training configuration: LLaVA training updates the projector and rank-16 LoRA adapters in the last four Vicuna layers while keeping the CLIP ViT-L/336 encoder frozen.
- Qwen3-VL training configuration: Qwen3-VL training likewise freezes the vision encoder and uses rank-16 LoRA adapters with learning rate 2×10^-5 across both stages.
- Sensitivity to perturbation views: K increasing from 10 to 100 reduces CHAIR from 4.2 to 3.1 and hallucination rate from 29.5% to 24.3% on LLaVA-1.6.
- Evaluation settings: The evaluation covers AMBER, Object HalBench, MMHal-Bench, POPE, and HallusionBench under deterministic decoding.
- Evaluation settings: Figure 7 compares baseline and complete INFUSE models on AMBER and MMHal, with lower hallucination values preferred.
C Robustness Evaluation
INFUSE is evaluated under test-time visual corruptions to measure whether its learned representations remain reliable without changing the model. Performance is best preserved under rotation and cropping, while masking and Gaussian noise cause larger degradation.
- INFUSE + LLaVA-1.5 is tested with masking, Gaussian noise, cropping, and rotation applied to input images while model weights remain fixed.Masking removes 30% or 60% of patches; other perturbations alter pixels, spatial windows, or image orientation.
- Rotation and cropping preserve HallusionBench and AMBER performance best among the tested perturbations.
- Masking and Gaussian noise degrade semantic consistency more severely, producing visible drops in HallusionBench qA and increased AMBER hallucination.
- Perturbations preserving global structure are better tolerated than heavy occlusion or random noise.
D Additional Quantitative Results
Additional evaluations show that INFUSE reduces fine-grained hallucination while preserving response length and strong grounding-related performance. On ObjHal, its gains extend to both response-level and object-level hallucination.
- INFUSE maintains output lengths comparable to DPO, CHiP, and the base model, indicating reductions are not caused by truncating responses.
- INFUSE + Qwen3-VL achieves the lowest response-level hallucination at 4.13 and object-level hallucination at 3.2 on ObjHal.
- CHiP achieves the highest response correctness at 95.08 and object correctness at 96.79 on ObjHal.
E Representative Hallucination Examples
Representative MMHal-Bench and HallusionBench cases show base models hallucinating counts, geometric relations, rankings, charts, and maps, whereas INFUSE gives more visually grounded answers. The examples span five forms of visual reasoning.
- Across five representative cases, INFUSE produces more conservative, visually grounded responses than base models that hallucinate unsupported visual content.
- Object counting: INFUSE correctly identifies five visible oysters, avoiding the base models’ hallucinated count of six.
- Geometric reasoning: INFUSE answers No when asked whether a triangle hypotenuse x = 5 exceeds 5, unlike the base models’ incorrect Yes.
- Table interpretation: INFUSE answers Yes that China leads the shown Olympic medal ranking, matching the visible table entries.
- Chart comprehension: INFUSE answers No when asked whether France leads the import-value chart, matching Switzerland’s longest bar.
- Map-based spatial comparison: INFUSE answers Yes that Texas is larger than California, matching the map’s visible geographic sizes.