PaperlayerView exact revision on arXiv

Source-linked AI summary

RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models

Canjie Liu, Jiawen Kang, Jinbo Wen, Zishao Zhong

arXiv:2609.02731v1cs.CVcs.AI

TL;DR

LVLMs remain vulnerable to visual hallucinations, and existing remedies can require extra data, training, retrieval, or iterative decoding. RVSD combines semantics-directed token sparsification with on-demand semantic-space visual retrieval in a single pass, achieving state-of-the-art hallucination mitigation with improved efficiency and long-context robustness. Its retrieval gate still depends on a predefined entropy threshold, and hallucinations are reduced rather than eliminated.

  • Problem

    LVLMs can produce responses that contradict or lack grounding in visual content, while existing mitigation approaches incur data, training, retrieval, or decoding overhead.

  • Method

    RVSD is a training-free, plug-and-play framework that selectively sparsifies visual tokens and retrieves deferred visual evidence on demand through SSVR in one decoding pass.

  • Results

    RVSD achieves state-of-the-art visual-hallucination mitigation across five benchmarks and three LVLM backbones while improving decoding efficiency and maintaining long-context robustness.

  • Takeaways & Limitations

    RVSD makes visual-token pruning reversible by retaining deferred tokens for retrieval when visual grounding becomes insufficient during generation.

  • Takeaways & Limitations

    SSVR relies on a predefined entropy threshold, and RVSD substantially reduces but does not eliminate visual hallucinations, motivating verification or human oversight in safety-critical applications.

Abstract

from arXiv · show

Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \textbf{RVSD} (\underline{R}etrieval \underline{V}ision \underline{S}parse \underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\footnote{https://github.com/canjie-liu/RVSD}

1 Introduction

LVLMs are capable across vision-language tasks but remain vulnerable to visual hallucinations, while existing mitigation methods trade data, retrieval, or decoding cost against visual grounding. RVSD addresses this tension with semantics-directed sparsification and on-demand retrieval in a single inference pass.

  • LVLMs generate responses that can contradict or lack grounding in visual content, compromising reliability in real-world applications.
  • Existing mitigation paradigms rely on additional supervision, external knowledge bases, or iterative inference, introducing data, storage, retrieval, or latency costs.
  • Sparse decoders can irreversibly discard semantically critical visual tokens, creating a sparsification-hallucination paradox as generation length increases.
  • RVSD is a training-free, plug-and-play framework that combines semantics-directed token sparsification with on-demand visual compensation in one inference pass.
  • SSVR retrieves relevant visual evidence from a deferred memory bank when grounding becomes insufficient, recalibrating hallucinated output distributions.
  • RVSD achieves state-of-the-art performance across benchmarks while improving decoding efficiency and remaining robust under long-context generation.

2 Related Work

Prior work mitigates visual hallucinations through alignment, decoding interventions, retrieval, or token sparsification, but these approaches retain efficiency or grounding limitations. RVSD combines targeted visual-token selection with semantic-space retrieval in a single decoding pass.

  • LVLM research builds on vision-language pretraining, instruction fine-tuning, and preference alignment to improve multimodal generation.
  • Training-centric methods add supervision or preference optimization, while decoding-time methods use contrastive or attention-guided interventions to reduce hallucinations.
  • RVSD retrieves deferred visual tokens directly in semantic space without an external image store, detector, or draft decoding pass.
  • General-purpose sparse decoders prune visual tokens with modality-agnostic criteria primarily designed for acceleration, while newer methods explore hallucination-aware sparsification.
  • RVSD performs visual-token sparsification and compensation within the same decoding pass without additional training or iterative decoding.

3 Problem Formulation

The problem formulation treats visual hallucination as generation that contradicts or lacks support from the input image. RVSD seeks to suppress such unfaithful output while preserving response quality with minimal inference overhead.

  • For image-grounded generation, an input image is encoded into aligned visual tokens that are processed with a textual query and generated prefix.
  • At generation step t, the model uses unnormalized logits from its multimodal language model to predict the next token.
  • A response exhibits visual hallucinations when it contradicts or is unsupported by the input image.
  • VH mitigation aims to suppress unfaithful generations while preserving overall response quality, and RVSD targets this objective during inference with minimal computational overhead.

4 Methodology

RVSD uses a single-pass pipeline that selects semantically relevant visual tokens for normal decoding and retrieves deferred evidence when visual grounding becomes insufficient. Its two components—semantics-directed sparsity and semantic-space retrieval—reduce attention computation while preserving access to fine-grained visual information.

  • Single Forward Pass: RVSD partitions visual tokens into an active sparse set for standard decoding and a deferred set that remains retrievable as visual memory.The active and deferred sets are disjoint and together cover the visual token bank.
  • Semantics-Directed Token Selection: The method scores visual tokens by aggregating cross-modal attention from salient generated text rather than using modality-agnostic pruning.Major textual tokens are selected with TopK, and attention is aggregated over a chosen layer subset.
  • Semantics-Directed Token Selection: RVSD constructs the active set with TopK selection and defers the remaining tokens, reducing per-layer attention complexity from O((t + Nv)^2) to O((t + k)^2) when k ≪ Nv.The budget k controls the number of retained visual tokens.
  • Semantic-Space Visual Retrieval: When dynamic sparsity may discard later-needed evidence, SSVR treats deferred tokens as a memory bank and retrieves relevant visual evidence using the current decoder state as a semantic query.The retrieval branch is activated on demand rather than restoring all pruned features.
  • Semantic-Space Visual Retrieval: A single retrieval call is triggered when normalized predictive uncertainty exceeds a predefined entropy threshold within the intermediate-layer scanning window.The trigger indicates insufficient visual grounding and a need for re-anchoring.
  • Semantic-Space Visual Retrieval: The retrieval query assigns normalized relevance weights to deferred visual tokens, selects the top-kc candidates, and aggregates them into a query-conditioned visual representation.The candidate count is kc = min(K, |Pt|), with K as an upper bound.
  • Lightweight Injection: A transient adapter injects retrieved visual evidence into the residual stream using amplitude alignment and an injection ratio that balances original and retrieved signals.The adapter is released after each retrieval step, enabling parameter-free, on-demand visual retrieval without persistent overhead.

5 Experiments

RVSD is evaluated across hallucination, general capability, ablation, and efficiency settings on three LVLM backbones. It consistently improves hallucination-related performance while preserving or improving multimodal competence and reducing computational costs.

  • POPE: RVSD achieves the best POPE performance across all splits on LLaVA-1.5, LLaVA-NEXT, and Qwen-VL.Average Accuracy/F1 rises from 80.5/81.6 to 86.2/85.0 on LLaVA-1.5, from 82.4/80.6 to 87.6/86.6 on LLaVA-NEXT, and from 83.7/81.7 to 85.3/83.5 on Qwen-VL.
  • MM-Vet: RVSD improves MM-Vet total scores and every reported capability dimension over vanilla decoding on all three backbones.Total scores increase from 30.0 to 33.1 on LLaVA-1.5, 42.1 to 42.7 on LLaVA-NEXT, and 36.7 to 38.4 on Qwen-VL.
  • MME: RVSD obtains the highest MME hallucination-subset total on each backbone, reaching 651.7 on LLaVA-1.5, 668.3 on LLaVA-NEXT, and 593.4 on Qwen-VL.The strongest gains occur on Color and Position for the LLaVA models, while Qwen-VL gains primarily on Existence.
  • CHAIR and AMBER: RVSD delivers the best discrimination and AMBER scores while reducing object-level hallucinations during long-form caption generation.It achieves the lowest CHAIRI on both LLaVA backbones and the lowest CHAIRS on Qwen-VL; baseline improvements are limited or inconsistent.
  • Ablation Studies: Ablations show that both semantics-directed sparse selection and SSVR are necessary, with removing SSVR causing the largest performance drop.Without SSVR, Cognition falls from 371.79 to 348.21; removing sparse selection reduces cognition by 14.29 despite slightly improving perception.
  • Inference Efficiency: RVSD maintains vanilla latency at 1.00× while reducing memory by 2.2%, TFLOPs by 15.6%, and TTFT by 20.2%.VCD and M3ID incur over 2× latency, while VTI increases latency to 1.09×.

6 Conclusion

RVSD combines semantics-directed token selection with SSVR in a single decoding pass to address the sparsification-hallucination paradox. Experiments show strong VH mitigation, efficiency, and long-context robustness, while future work targets video understanding.

  • RVSD integrates semantics-directed token selection with SSVR in a single decoding pass to address the sparsification-hallucination paradox.
  • RVSD achieves state-of-the-art VH mitigation while improving decoding efficiency and remaining robust under long-context generation.
  • Future work will extend RVSD to long-form video understanding, where temporal dependencies and richer modality interactions may exacerbate the sparsification-hallucination paradox.

Limitations

RVSD’s current limitations concern its entropy-threshold retrieval gate, simple dot-product retrieval, and evaluation scope. The authors identify adaptive gating, stronger retrieval mechanisms, and broader model coverage as directions for improvement.

  • SSVR relies on a predefined entropy threshold to activate retrieval, which can cause unnecessary retrieval calls on simpler queries.
  • RVSD currently uses simple dot-product similarity to retrieve evidence from the deferred visual memory bank.The authors suggest learned semantic projections or multi-head cross-modal attention for fine-grained tasks.
  • The evaluation spans two architectural families, but the authors plan to test larger-scale and more heterogeneous LVLMs to further assess generality.

Ethical Considerations

RVSD is presented as an inference-only approach intended to improve LVLM trustworthiness and safer multimodal deployment. However, the authors caution that hallucinations remain and recommend verification or human oversight in safety-critical settings.

  • RVSD aims to improve LVLM trustworthiness by mitigating visual hallucinations and supporting safer multimodal AI deployment.
  • The framework requires no additional data collection or human annotation and uses publicly available datasets and backbone models under their original licenses.
  • RVSD substantially reduces but does not eliminate visual hallucinations, so downstream verification or human oversight is recommended for safety-critical applications.

A More Experiments and Results

Additional experiments evaluate RVSD across challenging datasets, hallucination benchmarks, model architectures, and inference baselines. The results support cross-dataset robustness, while the evaluation protocol also covers general capability and efficiency considerations.

  • POPE A-OKVQA and GQA: RVSD achieves the best accuracy across all POPE splits on A-OKVQA and GQA for all three backbones, with largest gains on adversarial splits.On LLaVA-1.5, Accuracy/F1 improve by 11.6/5.4 on A-OKVQA and 10.9/3.8 on GQA.
  • Evaluation benchmarks: The additional POPE evaluation uses random, popular, and adversarial splits across MSCOCO, A-OKVQA, and GQA, each containing 3,000 yes/no queries.
  • Evaluation benchmarks: MME measures object-level Existence and Count alongside attribute-level Position and Color using Acc/Acc+, with total score computed from the four subtasks.
  • Evaluation benchmarks: AMBER jointly evaluates discrimination and generative captioning, covering existence-, attribute-, and relation-level hallucinations with emphasis on long-form generation.
  • Evaluation benchmarks: CHAIR evaluates object hallucination in generated captions by matching mentioned objects against MSCOCO annotations and reports two metrics.
  • Baselines and models: RVSD is compared with training-free decoding baselines under the same decoding budget, including VCD, M3ID, AvisC, and VTI.

C Implementation Details

RVSD experiments use standardized decoding and hardware settings, with backbone-specific pruning configurations and MME sensitivity measured relative to a fixed baseline.

  • Hyperparameter Configuration: RVSD uses a unified pruning layer set Ls = 2, 6, 15 across three backbones, progressively pruning the visual-token bank.For LLaVA-1.5-7B, the sparse-set budget is k = 192, approximately one third of the visual tokens.
  • Hyperparameter Configuration: SSVR hyperparameters are tuned per backbone to accommodate differences in visual-token granularity and spatial representation.The tuned parameters include the entropy threshold γ, intermediate-layer window, retrieval bound K, and injection ratio α.
  • Decoding Protocol: All methods are evaluated with nucleus sampling at T = 1.0 and top-p = 0.9, without a repetition penalty, using batch size 1.The maximum generation length varies by benchmark, from 2 tokens for binary tasks to 1,024 for open-ended generation.
  • Evaluation Setup: The MME heatmap reports subtask-level ∆ values relative to the baseline setting α = 0.20 and kc = 128.Here, ∆ = score(cfg) − score(base).
  • Implementation Environment: Experiments run on a single NVIDIA RTX A6000 GPU with PyTorch 2.0 and Hugging Face Transformers 4.40.The environment uses Ubuntu 22.04, CUDA 12.2, and NVIDIA driver 535.183.01.

D Hyperparameter Sensitivity Analysis

Sensitivity analyses show that RVSD is generally robust to retrieval settings, while sparse-token budget and injection ratio have the clearest effects on performance and efficiency.

  • MME Sensitivity: α = 0.10 and kc = 128 form a stable MME operating point after subtask-level analysis.Small α values improve OCR and translation by up to +7.5, whereas α = 0.50 causes broad degradations; varying kc from 32 to 256 has negligible effects.
  • Injection Ratio α: Accuracy/F1 remains stable at 84.7/83.4-84.7/83.5 for α in [0.05, 0.20], but rises to 0.50 causes a mild decrease to 84.4/83.2.The passage attributes this behavior to amplitude alignment, where retrieval re-anchors rather than overwrites the host representation.
  • Retrieval Bound K: Accuracy/F1 varies by less than 0.1 across K ∈ {32, 64, 128, 256}.Irrelevant deferred-memory entries receive near-zero aggregation weights, so larger retrieval bounds add neither noise nor benefit.
  • Entropy Threshold γ: Accuracy remains 84.7 at γ = 0.3 and γ = 0.5, decreasing by only 0.1 at γ = 0.7.Moderate threshold changes do not materially affect which uncertain steps are re-anchored on the POPE adversarial split.
  • Sparse Budget k: The sparse budget is most influential: k = 96 lowers Accuracy/F1 to 83.0/81.1, while k = 288 or 576 raises it to 85.0/84.1.The default k = 192 achieves 84.7/83.5 with lower attention cost than k = 576, defining the efficiency–faithfulness trade-off.
  • Efficiency Analysis: RVSD reduces per-token latency, maximum memory, TFLOPs, and TTFT relative to vanilla decoding, according to Table 4.The reported profile was not achieved by previous decoding-side methods for visual-hallucination mitigation.

H Usage of AI Assistant

The manuscript states that GPT-4o was used for language polishing, while generated outputs were manually reviewed and edited for technical accuracy; the supplied case studies also compare RVSD outputs with baselines.

  • AI Assistance Disclosure: GPT-4o was used exclusively for language polishing, including grammar correction and sentence-level rephrasing.The manuscript states that AI assistance did not involve research ideation, method design, experimental analysis, or result interpretation.
  • AI Assistance Disclosure: Generated outputs were manually reviewed and edited to ensure technical accuracy.This disclosure distinguishes language assistance from the manuscript’s research and interpretation activities.
  • Case Studies: Figures 7 and 8 present qualitative case studies comparing RVSD with the vanilla decoder and representative sparse-decoding baselines.The cases cover diverse scenes and challenging long-form generations.
  • Case Studies: The case studies highlight hallucinated content in red and faithful content grounded in the input image in green.The supplied examples include descriptions of airport scenes and fruit-and-vegetable imagery.
Loading 2609.02731v1…