PaperlayerView exact revision on arXiv

Source-linked AI summary

Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

Xi Xiao, Chen Liu, Chih-Ting Liao, Yunbei Zhang, Qizhen Lan, Yuxiang Wei, Lin Zhao, Janet Wang, Jianyang Gu, Muchao Ye, Tianyang Wang, Hao Xu

arXiv:2606.26387v1cs.CVcs.CLcs.LG

TL;DR

MLLMs may rely on language priors despite encoding relevant visual evidence, contributing to hallucinations unsupported by images. VIGIL uses counterfactual visual alignment to promote visual grounding; it reports consistent gains over state-of-the-art baselines, data efficiency, and emergent spatial grounding.

  • Problem

    MLLMs can encode correct visual evidence yet rely on language priors during generation, while standard preference alignment optimizes text outputs rather than visual grounding.

    Source: §1 · pp. 2–3

  • Method

    VIGIL is a post-training framework that maximizes visual information gain by comparing standard responses with responses in a counterfactual state without visual input.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    VIGIL consistently outperforms state-of-the-art baselines, improves more at larger model scales, and matches competitive baselines with 25% of their preference data.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    VIGIL’s evaluations report emergent spatial grounding without explicit localization supervision, alongside improved performance across architectures and scales.

    Source: §1 · p. 3, §5 · p. 12

    The zero-out masked counterfactual state may not capture nuanced cross-modal conflicts in highly cluttered environments.

    Source: §5 · p. 12

Abstract

from arXiv · show

Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.

1 Introduction

MLLMs can encode relevant visual evidence yet rely on language priors when answering, contributing to hallucinations that lack visual support. VIGIL uses counterfactual visual alignment to promote grounding, with reported gains across scales, lower preference-data use, and emergent spatial grounding.

1 Introduction

  • MLLM hallucination is a visual-grounding failure: generated descriptions can lack support from the physical visual input.The introduction notes that MLLMs can reason over complex visual data, but hallucination remains a barrier to reliability.Source: §1 · p. 2
  • MLLMs may encode correct visual features yet answer incorrectly when linguistic correlations hijack decoding.The paper attributes this visual laziness to strong language priors from text-centric pretraining.Source: §1 · p. 2
  • Standard DPO rewards final-text preferences without constraining answer derivation, potentially reinforcing language-prior shortcuts.VIGIL instead masks visual attention to create a counterfactual blind state and penalizes confidence that persists without visual evidence.Source: §1 · pp. 2–3
  • VIGIL matches competitive baselines using only 25% of their preference data.The framework operates offline and is described as stable and computationally lightweight.Source: §1 · p. 3
  • Across diverse architectures ranging from 7B to 72B parameters, VIGIL consistently improves performance, and gains increase for larger and stronger foundation models across this reported scale range.Source: §1 · p. 3
  • Without explicit bounding-box supervision, VIGIL improves zero-shot referring expression comprehension, indicating emergent spatial grounding.Source: §1 · p. 3

2 Preliminaries

Visual laziness describes MLLMs favoring language priors despite access to relevant visual evidence. The section analyzes how preference optimization can reward such shortcuts and motivates comparing seeing and counterfactual blind states to encourage visual reliance.

2.1 Definition of Visual Laziness

  • Visual laziness is an MLLM’s reliance on text-only priors despite relevant visual clues captured by its visual encoder.Source: §2.1 · p. 3

2.2 Information-Theoretic Analysis of Visual Laziness

  • When a preferred answer can be inferred from text alone, DPO can reinforce language priors without grounding the answer in the image.The resulting shortcut makes visual input statistically redundant and is associated with a degenerate posterior, vanishing visual information gain, and a path along the textual manifold.Source: §2.2 · pp. 3–4
  • Treating visual evidence as redundant can decouple perception from reasoning, as when a model calls a pictured green apple red based on textual associations.The section argues that sample reweighting and architectural complexity do not explicitly encourage reliance on visual evidence.Source: §2.2 · p. 4

2.3 Outcome Rewards vs. Dependency Alignment

  • Outcome-only rewards can reinforce correct answers reached for linguistic rather than visual reasons, so VIGIL compares seeing and counterfactual blind states within a DPO objective.The passage contrasts this with GRPO’s sampled-response rewards, noting that equivalent blind-state supervision would require extra trajectories and compute and memory.Source: §2.3 · pp. 4–5

3 Method

VIGIL anchors MLLM responses to visual evidence by constraining preference optimization to distinguish seeing from counterfactual blind states. Its implementation combines attention masking, hard-negative mining, and adaptive weighting of the visual-grounding constraint.

3 Method

  • VIGIL adds a geometric constraint to preference optimization to anchor MLLM reasoning to visual evidence and maximize information between visual inputs and responses.Source: §3 · p. 5

3.1 Grounding via Information Maximization

  • VIG is the pointwise mutual information between visual input and response conditioned on the instruction; near-zero VIG indicates reliance on language priors.Source: §3.1 · p. 5
  • The constrained objective preserves DPO preference reward while requiring VIG to exceed a threshold, implemented by preferring the same response in the seeing state over the attention-masked blind state.The CVD loss penalizes responses whose likelihood does not fall when visual evidence is removed; the reference policy is evaluated under matching seeing and blind masks.Source: §3.1 · pp. 5–6

3.2 VIGIL Implementation

  • VIGIL creates the blind state by masking text-to-visual attention while leaving the input tensors unchanged, and mines plausible absent details as hard negatives.Source: Figure 3 · p. 6, §3.2 · pp. 6–7

3.3 Unified Objective with Dynamic Gating

  • The dynamic gate reduces CVD regularization for samples with a high seeing–blind VIG gap, balancing visual grounding with semantic alignment in the unified objective.Source: §3.3 · p. 7

3.4 Facing the Sword of Damocles: The Fragile Equilibrium of Grounded Reasoning

  • VIGIL is presented as a geometric intervention that needs neither additional reference models nor full-dataset fine-tuning, while encouraging emergent spatial grounding.Source: §3.4 · p. 7

4 Experiments

VIGIL improves hallucination and reasoning results across model scales and architectures, while its ablations examine the role of visual masking, efficiency, and spatial grounding. The experiments compare it with alignment baselines across hallucination, multimodal reasoning, and text-only benchmarks.

4.1 Implementation Details

  • Experiments use one-epoch full-parameter fine-tuning, with large global batches, β = 0.1 for DPO variants, and attention-mask manipulation for counterfactual inputs.The implementation uses OpenRLHF and reproduces baselines on the same training data with their recommended hyperparameters.Source: §4.1 · pp. 7–8

4.2 Evaluation Benchmarks

  • Evaluation covers POPE, AMBER, and MMHal-Bench for hallucination, MathVista and MMBench for multimodal reasoning, and MMLU and GSM8K for text-only capability.AMBER tests free-form descriptions against fine-grained annotations; the protocol also includes POPE’s adversarial, popular, and random splits.Source: §4.2 · p. 8

4.3 Baselines

  • Baselines span SFT, DPO, SimPO, HA-DPO, DA-DPO, and VCD, covering general alignment, hallucination-specific optimization, and inference-time intervention.VCD contrasts logits from original and distorted images during decoding; VIGIL applies analogous visual contrast during post-training.Source: §4.3 · p. 8

4.4 Main Results

  • VIGIL improves Qwen2.5-VL-7B’s POPEAdv gain from 4.1 percentage points to 5.3 percentage points at 72B, with state-of-the-art results across evaluated metrics.Across architectures, VIGIL also outperforms standard DPO and improves POPEAdv by approximately 4.0 points on LLaVA-OneVision-7B and InternVL2.5-26B.Source: §4.4 · pp. 8–9, Table 1 · p. 9, Table 2 · p. 9

4.5 Ablation Study

  • On high-visual-dependency samples, VIGIL scores 87.5 versus standard DPO’s 78.2; its explicit physical masking yields a 5.3-point gain, compared with 2.9 points from implicit reweighting.In the component ablation, hard negatives add 2.0 percentage points and the Visual Anchor drives a 3.2-point gain.Source: §4.5 · pp. 9–11
  • VIGIL trained on 25% of the dataset matches full-data DA-DPO performance and reduces training wall-clock time by approximately 70%.It remains robust across KL penalty coefficients β ∈[0.05, 0.5], whereas standard DPO often becomes unstable at lower β values.Source: Figure 4 · p. 10, §4.5 · p. 11
  • Without bounding-box training, VIGIL improves zero-shot RefCOCOg performance by 4.3 percentage points.The reported localization improvement is evaluated on RefCOCOg (val).Source: Table 6 · p. 10, §4.5 · p. 11

4.6 Qualitative Analysis

  • Qualitative comparisons show VIGIL correctly reads banner digits, counts coins and apples, and identifies causes of traffic congestion where DPO gives plausible but incorrect answers.The examples cover fine-grained perception, counterfactual counting, and causal state reasoning.Source: §4.6 · p. 11

5 Conclusion

VIGIL uses counterfactual visual decoupling to ground responses in visual evidence rather than language priors. The paper reports stronger benchmark performance, data efficiency, and emergent spatial grounding, while noting a limitation of its zero-out blind state.

5 Conclusion

  • VIGIL consistently outperforms state-of-the-art baselines, shows superior scaling and data efficiency, and exhibits emergent spatial grounding.The conclusion also illustrates more visually specific answers than DPO on fine-grained perception, counterfactual counting, and causal reasoning.Source: §5 · p. 12, Figure 5 · p. 12
  • Zero-out masking may fail to capture nuanced cross-modal conflicts in highly cluttered environments.The conclusion proposes more granular interventions, including object-level spectral filtering and semantic-preserving transformations.Source: §5 · p. 12

A.1 Mechanisms of Hallucination in MLLMs

  • Mechanistic studies describe visual laziness as correct visual features being preserved internally but suppressed by language priors during decoding.Inference-time interventions such as VCD and OPERA can reduce hallucinations, but add latency and require hyperparameter tuning.Source: §A.1 · p. 17

A.2 Alignment and Efficiency in Multimodal Learning

  • VIGIL seeks data efficiency within stable offline DPO by maximizing information gained from each preference pair, rather than using dynamic sampling.The cited alternatives include online RL, which can introduce instability and implementation complexity.Source: §A.2 · p. 17

A.3 Architectural vs. Geometric Solutions

  • VIGIL integrates visual counterfactuals into preference learning as a geometric constraint, using a blind state as a negative anchor without adding parameters.This contrasts with modular approaches that use external components and may compromise end-to-end generalizability.Source: §A.3 · p. 17

A.4 DPO vs. GRPO

For multimodal hallucination mitigation, the paper argues that final-answer rewards alone do not reveal whether responses depend on visual evidence. It favors DPO for matched seeing/blind comparisons and reports that Visual-GRPO underperforms on hallucination benchmarks.

A.4 DPO vs. GRPO

  • Final-answer rewards alone may reinforce visual laziness because a plausible or correct answer can still be driven by language priors rather than visual evidence.The target is correct modality dependence, not only correct reasoning.Source: §A.4 · p. 18
  • DPO supports matched comparisons of the same instruction and candidate response in seeing and blind states, penalizing blind confidence while preserving correct answering when vision is available.This provides a direct, low-variance grounding signal without requiring an external verifier for dense rewards.Source: §A.4 · p. 18
  • Reproducing the seeing/blind contrast with GRPO would require multiple blind variants per sample, increasing forward passes, reward computations, and memory use.The paper therefore describes DPO as making the counterfactual constraint practical at scale.Source: §A.4 · p. 18
  • VIGIL outperforms GRPO-based reward tuning on hallucination mitigation while maintaining comparable reasoning capabilities.The comparison is on Qwen2.5-VL-7B; Visual-GRPO is competitive on MathVista but underperforms on hallucination benchmarks.Source: Table S1 · p. 18, §A.4 · p. 18

B Blind State Construction

VIGIL constructs a matched blind state by blocking text queries from accessing visual key/value positions while preserving the image representation and other attention-mask behavior. The paper reports that attention masking performs best among the compared blind-state construction techniques.

B Blind State Construction

  • The blind path reuses the seeing path's image, visual tokens, projector outputs, text tokens, and positional embeddings, intervening only on attention connectivity.Text-query positions are blocked from attending to visual key/value positions, while the vision encoder remains untouched.Source: B Blind State Construction · p. 18
  • Attention masking achieves the best performance among the compared blind-state construction techniques, while avoiding additional image-processing cost.Image-level perturbations include blacking out, blurring, and shuffling; the black-image variant performs worst.Source: Table S2 · p. 19, B Blind State Construction · p. 19
  • Applying the intervention after visual encoding and projection keeps visual-token distributions unchanged, so the likelihood contrast reflects cross-modal access rather than out-of-distribution image corruption.Causal, padding, and position masks remain identical to the seeing path.Source: B Blind State Construction · p. 19

C Preference Set, Reference Policy and Reproducibility

The reproducibility materials specify the preference pool, matched reference-policy evaluation, and visual chain-of-thought judging protocol. A manual audit reports 92.3% agreement between the GPT-4o judge and human annotations.

C Preference Set, Reference Policy and Reproducibility

  • The frozen reference policy is evaluated in the same seeing or blind state as the policy model, keeping the contrast focused on visual access rather than reference-model mismatch.Both policies use Msee for the seeing term and Mblind for the blind term.Source: C Preference Set, Reference Policy and… · p. 19
  • The preference set contains 120K pairs, and DPO, DA-DPO, and VIGIL use the same pool unless otherwise stated.The pool combines hallucination-specific samples with general multimodal reasoning, OCR/chart, and math/spatial tasks.Source: C Preference Set, Reference Policy and… · p. 20, Table S3 · p. 20
  • For visual CoT consistency, GPT-4o judges whether the first visual premise matches the image and ground-truth object list, not later reasoning or the final answer.When inconsistent, it identifies the offending object, attribute, count, or spatial relation.Source: C Preference Set, Reference Policy and… · p. 20
  • The GPT-4o judge agrees with human annotations on 92.3% of 300 randomly sampled VCC judgments.The automatic judge is used only as an auxiliary grounding diagnostic, not for training.Source: C Preference Set, Reference Policy and… · p. 20

D Maintenance of Text-Only Capabilities

With the image branch disabled, VIGIL retains text-only reasoning performance while improving multimodal reasoning. Changes on text-only benchmarks are within run-to-run variation.

D Maintenance of Text-Only Capabilities

  • VIGIL preserves text-only performance: MMLU and GSM8K changes are less than 0.2 points, while across three random seeds benchmark standard deviations are approximately ±0.3, placing differences within run-to-run variation.Source: D Maintenance of Text-Only Capabilities · p. 20
  • VIGIL consistently improves multimodal reasoning performance on MathVista and MMBench.The authors suggest the CVD objective encourages visual dependence when visual evidence is informative, rather than indiscriminate visual reliance.Source: D Maintenance of Text-Only Capabilities · p. 20
  • The reported text-only changes indicate no measurable capability tax or catastrophic forgetting of text-only reasoning skills.The authors attribute this interpretation to the changes falling within run-to-run variation.Source: D Maintenance of Text-Only Capabilities · p. 20

E Training Efficiency and Computational Resource Analysis

VIGIL is reported to reduce preference-data and training-time requirements while adding little forward-pass computation. Its resource comparison is evaluated on Qwen2.5-VL-7B.

E Training Efficiency and Computational Resource Analysis

  • 25% of the training samples lets VIGIL match the performance of 100% DA-DPO, with a reported 70% decrease in total training wall-clock time.For Qwen2.5-VL-7B, alignment takes 3.5 GPU hours on a single A100 node.Source: E Training Efficiency and… · pp. 20–21
  • Less than 1% of the total forward-backward pass TFLOPs are attributable to the CVD constraint.The counterfactual blind state uses attention masking, avoiding redundant re-processing of high-resolution visual tokens.Source: E Training Efficiency and… · p. 21
  • Unlike online RL methods such as GRPO or AdaptVision, VIGIL uses a stable offline framework without group-relative sample and multiple-reference-model overhead.The authors report that this reduces VRAM requirements and allows full-parameter fine-tuning of 72B-scale models on standard infrastructure without aggressive quantization.Source: E Training Efficiency and… · p. 21
  • VIGIL's resource utilization is compared with standard DPO and its variants on Qwen2.5-VL-7B.Table S5 measures GPU hours as the time needed to reach POPE > 86.0.Source: E Training Efficiency and… · p. 20, Table S5 · p. 21

F Comparison with Inference-Time Intervention using VCD

VCD improves the SFT baseline, but adds little when applied after VIGIL and nearly doubles inference latency. VIGIL alone reaches performance on par with the VCD counterpart at lower latency.

F Comparison with Inference-Time Intervention using VCD

  • VCD substantially improves the SFT baseline, confirming that inference-time visual contrast is an effective hallucination mitigation strategy.The comparison uses the same Qwen2.5-VL-7B SFT checkpoint, benchmark splits, decoding length, and evaluation scripts.Source: F Comparison with Inference-Time… · p. 21
  • VIGIL with standard decoding performs on par with the VCD counterpart at much lower inference latency.Applying VCD after VIGIL yields only minimal effects across benchmarks while almost doubling inference latency.Source: F Comparison with Inference-Time… · p. 21, Table S6 · p. 22

G Information-Theoretic Analysis: Quantifying Causal Grounding

The analysis uses information-theoretic measures to examine whether VIGIL increases visual grounding. On High-VDI samples, VIGIL has higher Visual Information Gain and higher entropy under counterfactual blindness than standard DPO.

G Information-Theoretic Analysis: Quantifying Causal Grounding

  • The analysis frames its evaluation of VIGIL's geometric constraint through Maximum Mutual Information and Rate-Distortion Theory.It aims to quantify a shift from linguistic compression to causal visual grounding.Source: G Information-Theoretic Analysis:… · p. 22
  • On High-VDI samples, VIGIL reaches an average Visual Information Gain of 7.2, compared with 2.1 for standard DPO.The authors describe VIG as empirical Pointwise Mutual Information between the visual manifold and response, conditioned on the instruction.Source: G Information-Theoretic Analysis:… · p. 22
  • Hblind is approximately 1.88 for VIGIL and approximately 0.65 for standard DPO.The reported values contrast higher uncertainty for VIGIL with continued low entropy for DPO when visual evidence is removed; Table S7 reports Qwen2.5-VL-7B on High-VDI.Source: G Information-Theoretic Analysis:… · p. 22, Table S7 · p. 22
  • The authors interpret the findings as evidence that VIGIL alters the model's dependency path rather than merely reweighting samples by difficulty.Their account links penalizing low-entropy predictions in the blind state to reasoning anchored in visual evidence rather than linguistic over-fitting.Source: G Information-Theoretic Analysis:… · p. 22

Related papers

Loading 2606.26387v1…