PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang

arXiv:2511.05017v3cs.CVcs.CL

TL;DR

LVLMs can generate fluent outputs unsupported by visual input, and the paper investigates language-overreliance in their visual-text fusion. VisAlign adds global visual information to textual embeddings to rebalance multimodal representations. The authors report consistent benchmark gains and reduced hallucinations.

  • Problem

    LVLM hallucinations produce fluent outputs that are not grounded in visual input, and the paper examines whether appending visual embeddings to text-pretrained LLMs contributes to language overreliance.

    Source: §1 · pp. 1–2

  • Method

    VisAlign incorporates averaged visual information into textual embeddings at the token level to encourage balanced attention and cross-modal representations.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    VisAlign consistently improves hallucination-benchmark performance, including +9.33% on MMVP-MLLM over baseline and +2.99% POPE accuracy on the Adversarial SEEM A-OKVQA setting.

    Source: §6 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that rebalancing representation-level visual grounding can support more visually grounded outputs and multimodal reasoning.

    Source: §1 · p. 2, §7 · p. 9

    VisAlign uses globally average-pooled visual features; this aggregation may miss finer-grained visual cues that more sophisticated pooling could capture.

    Source: §8 · p. 9

Abstract

from arXiv · show

Hallucinations in Large Vision-Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during multimodal reasoning. A key cause is the model's over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a "bowl of fruit" or "cup of coffee", relying on language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale vision-language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards language-dominant representations. This imbalance skews attention towards the text over visual content, weakening the model's ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. Experimental results across multiple hallucination benchmarks demonstrate that our method significantly reduces hallucinations and fosters more balanced multimodal reasoning. Notably, our approach achieves substantial gains, including +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, up to +3.4% on Merlin, and +3% on the hard-data split of HallusionBench.

1 Introduction

LVLMs can produce fluent descriptions that are not grounded in visual evidence, and the paper examines whether language-biased fusion contributes to these hallucinations. It proposes incorporating visual information into textual embeddings to promote more balanced cross-modal reasoning.

1 Introduction

  • LVLM hallucinations include fabricated or misinterpreted visual content, such as false attributes, relations, and video events.These outputs can undermine reliability, including in safety-critical settings.Source: §1 · pp. 1–2
  • The authors hypothesize that appending visual embeddings to a text-pretrained LLM can create modality imbalance and encourage reliance on linguistic priors.They argue that the text-trained backbone may underuse visual embeddings during fine-tuning.Source: §1 · p. 2
  • Their analysis reports that simple visual-token appending causes models to over-rely on language and underuse visual information, with textual tokens receiving disproportionate self-attention.The study uses Video-LLaVA as its main model and also reports results on LLaVA1.5 and OpenQwen2VL to examine generalization.Source: §1 · p. 2
  • The proposed method integrates visual information into text embeddings at the token level, and the authors report significant gains across multiple hallucination benchmarks.They describe the approach as encouraging balanced attention and cross-modal representations that better ground generation in visual input.Source: §1 · p. 2

2 Related Works

Prior LVLM work integrates visual and textual information through token appending, cross-attention, and representation-level grounding, while hallucination mitigation spans training and inference strategies. VisAlign refines textual embeddings using global visual context before transformer processing.

2 Related Works

  • LVLMs commonly append visual embeddings to pretrained LLM token sequences, while other architectures use cross-attention or fuse vision directly into token embeddings.The paper adopts Video-LLaVA as its primary model and extends results to LLaVA1.5 and Open-Qwen2VL.Source: §2 · p. 2
  • Existing mitigation approaches include hallucination datasets and classifiers, fine-tuning, reinforcement learning, attention calibration, and inference-time decoding or reranking.The cited strategies address hallucinations through different training-time and inference-time mechanisms.Source: §2 · pp. 2–3
  • VisAlign refines textual embeddings with global visual context before transformer processing, shifting alignment to the representation stage without modifying the LLM architecture.Its illustrated design appends averaged visual embeddings to each token embedding, then applies a projection layer.Source: §2 · p. 3, Figure 2 · p. 3
  • Representation-level grounding work includes early fusion, co-attention, multimodal pretraining, feature modulation, and injecting visual features into question representations.The paper positions its work as targeting hallucination in LVLMs through alignment with global visual context.Source: §2 · p. 3

3 Background

Video-LLaVA combines visual embeddings with a pretrained LLM through a projection layer. Its training pipeline first aligns visual embeddings with the LLM input space, then fine-tunes the full model.

3 Background

  • Video-LLaVA uses a frozen LanguageBind visual encoder to extract image or video embeddings and a projection layer to map them into the LLM embedding space.The model uses the pretrained Vicuna-7b LLM as its backbone.Source: §3 · p. 4
  • During pretraining, the visual encoder stays frozen while the projection layer is trained to align visual embeddings with the LLM input space.Visual embeddings V have Nv tokens and dimension dv; the projection layer maps them to the LLM dimension dt.Source: §3 · p. 4
  • During fine-tuning, the full model, including the LLM, is trained end-to-end for reasoning over combined visual and textual inputs.The stated goal is visually grounded generation.Source: §3 · p. 4

4 Evaluating Attention Score Distribution

The attention analysis visualizes interactions between visual and textual tokens across Video-LLaVA’s transformer layers. It finds attention concentrated on text at the sequence boundaries and comparatively low attention to intervening visual tokens.

4 Evaluating Attention Score Distribution

  • The analysis plots key tokens on the horizontal axis and query tokens on the vertical axis, with color intensity indicating attention strength.Cooler colors indicate lower scores, warmer colors and white indicate stronger scores, and nearly black regions indicate zero attention from causal masking.Source: §4 · p. 4
  • Video-LLaVA’s attention is disproportionately concentrated on textual tokens at both sequence ends, with lower attention to the visual tokens between them.The reported pattern appears across layers; the input order is initial text, visual tokens, then remaining text.Source: §4 · p. 4
  • The authors connect the attention imbalance to a text-only pretrained backbone that relies heavily on linguistic priors and insufficiently leverages visual embeddings.They argue this restricts visual-signal integration and is associated with hallucinated outputs that are unsupported by visual input.Source: §4 · p. 4

5 Improving Attention Score Distribution by Refining Textual Embeddings

VisAlign adds global visual context to textual embeddings before they enter the LLM, aiming to encourage more balanced visual-textual attention. Its attention maps show more balanced and structured attention than baseline Video-LLaVA, whose attention is concentrated at textual sequence boundaries.

5 Improving Attention Score Distribution by Refining Textual Embeddings

  • VisAlign refines textual embeddings with visual context before the LLM, aiming to promote balanced attention and reduce reliance on textual priors.The method is representational and adds no visual inputs, training objectives, architectural changes, or external supervision.Source: §5 · p. 4, §5.1 · p. 6
  • VisAlign average-pools projected visual embeddings into a global visual summary, a parameter-free aggregation chosen for stability and minimal computational overhead.The summary is used to incorporate global visual context into textual embeddings.Source: §5 · pp. 4–5
  • The method concatenates the visual summary with text embeddings, projects the fused representations back to the LLM embedding dimension, and appends the grounded text tokens to visual embeddings.The resulting sequence places visually grounded textual embeddings before visual embeddings and the remaining grounded textual tokens.Source: §5 · p. 5
  • VisAlign follows the baseline Video-LLaVA datasets and training strategy, training projection and linear layers with the LLM frozen during pretraining, then fine-tuning the full model end-to-end.End-to-end fine-tuning includes the LLM.Source: §5 · p. 5

5.1 Attention Score Distribution with VisAlign

  • VisAlign produces more balanced, structured attention across visual and textual tokens, while baseline Video-LLaVA concentrates attention at textual sequence boundaries.The VisAlign maps also show sharper, more frequent vertical bands and smoother diagonal gradients; the baseline maps are described as fragmented and noisy.Source: §5.1 · pp. 5–6, Figure 3 · p. 5
  • The authors attribute baseline text over-attention to language tokens being in-distribution for the pretrained LLM while visual tokens are comparatively out-of-distribution.They propose that injecting visual information into every textual embedding encourages early-layer attention adaptation during fine-tuning.Source: §5.1 · p. 6

6 Experiments and Results

Across hallucination benchmarks, VisAlign improves visual grounding and reduces hallucinations; on MME, visually grounded categories improve while knowledge-heavy categories decline slightly.

6 Experiments and Results

  • +9.33% improvement over baseline on MMVP-MLLM, which probes fine-grained visual discrimination under minimal semantic variance.Figure 4 contrasts baseline and VisAlign responses on benchmark examples.Source: §6 · p. 6, Table 1 · p. 6, Figure 4 · p. 7
  • On POPE A-OKVQA, VisAlign improves accuracy by 2.99%, precision by 1.76%, and F1-score by 1.18%.Evaluation uses the challenging Adversarial SEEM setting.Source: §6 · p. 6, Table 1 · p. 6
  • On Merlin, VisAlign consistently outperforms the baseline across positive and negative object cases and two image-sampling strategies.The benchmark evaluates fine-grained object-existence verification.Source: §6 · pp. 6–7, Table 2 · p. 7
  • Mementos results show significant Robotics-domain improvements for object and action hallucinations; Comics and Daily Life gains are more modest.The paper attributes the difference to predictable temporal patterns and clear visual cues in Robotics, versus greater complexity and noisier inputs in the other domains.Source: §6 · p. 7, Table 3 · p. 7
  • On HallusionBench, VisAlign gains approximately 3% on average, with improvements across Visual-Dependent and Visual-Supplement questions, especially Map, OCR, and Table tasks.The benchmark includes a hard split with human-edited images that induce modality conflicts.Source: §6 · p. 8, Figure 5 · p. 8
  • VisAlign consistently improves visual grounding and reduces hallucinations, while MME results show gains in visually grounded categories and slight declines in knowledge-heavy ones.The reported MME categories include Existence, Count, and Color versus Celebrity and Landmark.Source: §6 · p. 8
  • VisAlign consistently improves POPE-AOKVQA performance on both LLaVA 1.5 and Open-Qwen2VL, supporting applicability beyond Video-LLaVA.Table 4 reports the additional-model comparison.Source: §6 · p. 9, Table 4 · p. 9
  • On POPE-AOKVQA, VisAlign raises Video-LLaVA accuracy from 54.5 to 57.09 and F1-score from 68.6 to 69.63; combining it with VCD reaches 58.8 accuracy and 70.04 F1-score.The paper describes the training-time refinement and inference-time method as complementary.Source: §6 · p. 9, Table 5 · p. 9

7 Conclusion

The paper analyzes attention imbalance in LVLMs and finds that popular models overemphasize text, increasing reliance on linguistic priors. It proposes redefining textual embeddings to rebalance attention, reporting reduced hallucinations and more visually grounded outputs.

7 Conclusion

  • VisAlign significantly reduces hallucinations and produces more semantically accurate, visually grounded outputs across challenging benchmarks.The paper reports substantial improvements across multiple hallucination benchmarks.Source: §7 · p. 9
  • Popular LVLMs overemphasize text, increasing reliance on linguistic priors.The paper systematically analyzes attention distributions in LVLMs in relation to hallucinations.Source: §7 · p. 9
  • The method redefines textual embeddings during training to rebalance attention and improve the use of visual information.The paper presents this as a way to encourage more balanced multimodal reasoning.Source: §7 · p. 9

8 Limitations

VisAlign uses globally average-pooled visual features for lightweight, architecture-agnostic representation-level alignment. The authors note possible trade-offs, limits to visual detail, and the need for broader validation.

8 Limitations

  • Global average pooling isolates representation-level alignment, but region-level attention pooling could capture finer visual cues and potentially improve alignment.The current aggregation is global, while the proposed alternative operates over region-level visual tokens.Source: §8 · p. 9
  • Stronger reliance on visual evidence may slightly reduce performance on tasks relying more heavily on world knowledge or linguistic priors.MME shows minor drops in Landmark and Celebrity categories.Source: §8 · p. 9
  • Evaluation across several hallucination benchmarks leaves broader applicability uncertain without validation on newer LVLM architectures and large-scale multimodal benchmarks.The authors identify these evaluations as needed for further insight into general applicability.Source: §8 · p. 9

A.1 Related Works

The related work spans benchmarks for object-level, factual-consistency, fine-grained visual, cross-modal, temporal, and generative hallucinations. Together, these datasets cover varied forms of visual-grounding errors.

Object and factual consistency

  • POPE-AOKVQA and NOPE target object-level hallucinations, while MERLIN evaluates factual consistency through atomic fact decomposition.These benchmarks address different aspects of hallucination evaluation.Source: §A.1 · p. 13

Fine-grained and cross-modal evaluation

  • MMVP-MLLM and HallusionBench probe model behavior under minimal semantic variation and cross-modal conflicts.They examine distinct challenges to visual grounding.Source: §A.1 · p. 13

Temporal and unified evaluation

  • Mementos targets temporal hallucinations, and AMBER evaluates both discriminative and generative hallucinations in a unified benchmark.Together, the benchmarks address varied phenomena, including object, action, attribute, relational, and temporal inconsistencies.Source: §A.1 · p. 13

A.2 Detailed Description of Various Benchamarks used in Evaluating our method

The evaluation benchmarks test distinct forms of grounding: subtle visual distinctions, object presence, factual consistency, temporal reasoning, and conflicts between visual input and parametric memory.

MMVP-MLLM

  • MMVP-MLLM gives credit only when both binary-choice questions for a similar-image pair are answered correctly, testing fine-grained visual grounding.The image pairs have highly similar CLIP embeddings to minimize semantic divergence.Source: §A.2 · p. 13

POPE

  • POPE tests yes/no judgments of object presence, with adversarial absent-object questions probing false affirmations of common objects.The evaluation uses the challenging Adversarial SEEM setting from A-OKVQA, with SEEM-based object detection.Source: §A.2 · p. 13

MERLIN

  • MERLIN tests object existence by comparing original images with edited images from which an entire single-instance object category was removed.The evaluation targets this removed-object subset.Source: §A.2 · p. 13

Mementos

  • Mementos evaluates object and action hallucinations in sequential images across Robotics, Comics, and Daily Life.Its focus includes temporal coherence and object-behavior relationships.Source: §A.2 · p. 13

HallusionBench

  • HallusionBench separates visually dependent questions from questions answerable through world knowledge and includes a hard split with human-edited modality conflicts.Its categories assess how parametric memory affects hallucinations.Source: §A.2 · p. 13

A.3 Results on HallusionBench Benchmark

Across HallusionBench and MME, VisAlign strengthens visual-grounding performance, while comparisons show gains alongside VCD and transfer to LLaVA 1.5 and Open-Qwen2VL.

A.3 Results on HallusionBench Benchmark

  • About 3% average improvement on HallusionBench’s hard subset, with Visual-Supplement gains of 3.12% (Map), 14.81% (OCR), and 6.06% (Table).The hard subset uses human-edited images designed to conflict with common knowledge; examples include interpreting a manipulated map and detecting a falsified medal count.Source: §A.3 · pp. 13–14, Table 6 · p. 14

A.4 Effect of VisAlign on MME, a General LVLM benchmark:

  • On MME, VisAlign lifts Existence (170→190), Count (121.66→131.66), and Color (135→148.33), preserves OCR and Posters, but lowers some specialized categories.VCD causes universal drops or no improvements across MME subcategories, whereas VisAlign’s trade-off favors core grounding tasks.Source: §A.4 · p. 14, Table 7 · p. 16

A.5 Comparison with existing hallucination mitigation approaches

  • VisAlign surpasses VCD alone in accuracy (54.5→57.09) and F1-score (68.6→69.63); combining them reaches 58.8 accuracy and 70.04 F1-score.VCD re-ranks candidate outputs at inference, while VisAlign refines representations during training; their gains persist when combined.Source: §A.5 · p. 15

A.6.1 LLava1.5

  • VisAlign consistently improves performance and reduces hallucinations when integrated into LLaVA 1.5.Source: §A.6.1 · p. 15

A.6.2 Open-Qwen2VL

  • VisAlign also improves Open-Qwen2VL performance and hallucination results, including on a model with stronger native visual grounding.The base checkpoint completed Stage 1 and only 5,000 Stage-2 steps because of compute constraints.Source: §A.6.2 · pp. 15–16

A.7 Quantitative Assessment of Visual Contribution

  • Visual contribution is measured as average attention allocated to visual keys across queries and heads; first-layer estimates on 100 samples favor VisAlign.Deeper-layer tracking was infeasible, and the first text-token block was skipped because its visual contribution is always zero.Source: Table 8 · p. 16, §A.7 · p. 16

A.8 Additional qualitative results:

  • Figure 6 shows VisAlign producing more visually grounded predictions and fewer hallucinations than baseline Video-LLaVA on Mementos.Red highlights hallucinated text and blue marks the corresponding corrections.Source: §A.8 · p. 16, Figure 6 · p. 17
Loading 2511.05017v3…