PaperlayerView exact revision on arXiv

Source-linked AI summary

Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain

Seulbi Lee, Sangheum Hwang

arXiv:2602.17186v2cs.CV

TL;DR

LVLMs often rely on textual priors despite available visual evidence, and existing approaches do not quantify the visual contribution of individual samples or tokens. The paper introduces VIG to measure that contribution and uses it for selective training, reporting improved visual grounding and hallucination robustness with only a fraction of the original supervision, while noting computational and scope limitations.

  • Problem

    LVLMs can produce visually ungrounded responses, while prior approaches lack an explicit measure of how much individual training samples or tokens depend on visual information.

  • Method

    VIG is a perplexity-based metric that measures visual-information contribution through prediction-uncertainty reduction at sample and token levels, guiding selective training toward high-VIG supervision.

  • Results

    VIG correlates with benchmark-level modality dependency, identifies visually grounded tokens, and enables selective training with improved visual grounding and hallucination robustness using only a fraction of the original supervision.

  • Takeaways & Limitations

    Focusing optimization on visually informative samples and tokens can improve vision understanding while reducing reliance on textual priors and language bias.

  • Takeaways & Limitations

    VIG requires additional forward passes for scoring, and the empirical study is primarily demonstrated on LLaVA-1.5 and ShareGPT4V families.

Abstract

from arXiv · show

Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.

1. Introduction

LVLMs can rely on textual priors instead of visual evidence, while multimodal training data mixes weakly grounded and visually dependent samples and tokens. The paper introduces VIG and uses it to prioritize visually informative supervision for improved grounding and reduced language bias.

  • Instruction-tuning data contains samples and tokens with heterogeneous visual dependency, from text-answerable questions to responses requiring fine-grained visual details.
  • LVLMs often ignore salient image content or hallucinate attributes, raising questions about whether their answers actually use visual evidence.
  • Uniformly optimizing visually grounded tokens alongside structural or text-predictable tokens provides no incentive to distinguish visual evidence from linguistic shortcuts.
  • Visual Information Gain (VIG) is a perplexity-based, model-agnostic metric that quantifies visual contribution at both sample and token levels.
  • VIG identifies visually grounded tokens such as colors, spatial relations, and attributes while distinguishing them from tokens driven primarily by textual priors.
  • VIG-guided selective training prioritizes high-VIG samples and tokens, pruning weakly grounded supervision while improving visual grounding and mitigating language bias.

2. Related Work

Prior work addresses language bias through inference-time decoding, attention or architectural changes, and higher-quality instruction data. These approaches motivate methods that improve visual grounding while preserving architectural simplicity and avoiding extra inference overhead.

  • Language bias causes LVLMs to produce visually ungrounded responses by over-relying on textual priors, often learned from noisy multimodal datasets.
  • Training-free methods compare predictions with and without visual input, while other approaches modify attention or architecture to promote visual utilization.
  • Existing approaches highlight the need for effective bias mitigation that maintains architectural simplicity and avoids additional inference overhead.

3. Visual Information Gain

Visual Information Gain (VIG) measures how much visual input reduces prediction uncertainty, from sample-level perplexity changes to token-level loss reductions. Its analyses connect VIG to visual grounding and motivate selective training that concentrates optimization on visually informative data.

  • Definition: VIG is defined as the log-ratio between answer perplexities with and without visual conditioning, where higher values indicate greater visual contribution.The text-only condition uses a blurred image to remove visual cues.
  • Definition: Under deterministic supervision, VIG simplifies because the one-hot target distribution makes both intrinsic entropy terms zero.The resulting quantity reflects the change in model divergence from the ground truth.
  • Token-level analysis: VIG decomposes into average token-wise loss differences, revealing which response tokens depend most strongly on visual information.Each difference compares token cross-entropy with and without visual conditioning.
  • Sample-level analysis: 0.923 VIG for a perfectly aligned image, 0.409 for partial attribute alignment, and −0.520 for a conflicting image demonstrate sensitivity to visual support.These examples use the same question–answer pair while varying only the image.
  • Analysis: VIG aligns with benchmark-level modality dependence and highlights visually salient tokens such as colors, spatial relations, and physical states.Figure 2 contrasts stronger multimodal interaction in COCO, CV-Bench, and POPE with weaker visual dependency in GQA and SQA.
  • Selective training: VIG-guided selective training ranks samples by VIG, retains the top p%, and then updates only tokens exceeding the same threshold.Reusing one threshold avoids additional hyperparameters and focuses optimization on visually informative regions.

4. Experiment

Experiments show that VIG-guided selection improves visual understanding and hallucination robustness while substantially reducing supervision across models, benchmarks, and selection settings.

  • Main Results: 38.45M target tokens suffice for LLaVA-1.5 7B to improve performance on all benchmarks, while LLaVA-1.5 13B improves across all benchmarks with 12.14M tokens.The results suggest larger models can use carefully selected, visually grounded data more effectively.
  • Main Results: 41% fewer active tokens still improves Open-Qwen2VL 2B consistently across vision-understanding and hallucination benchmarks.This result supports generalization across substantially different architecture and training data.
  • Comparison with Existing Methods: VIG improves all reported vision-understanding tasks and reduces hallucination without architectural changes or inference-time overhead.On MMHal, it reduces hallucination while simultaneously improving the overall score.
  • Comparison with Existing Methods: VIG combines additively with VCD, PAI, VAR, and LACING; VIG training plus LACING reaches 37.01 on MMVet and 67.89 on MMBench.VIG operates at the data level, while LACING modifies the architecture.
  • Data Distribution: VIG filtering retains COCO object categories at nearly uniform rates across head, torso, and tail frequency tiers, preserving long-tail concepts.The retained data composition is also selectively improved.
  • Ablation Study: VIG selection outperforms random selection and vanilla training, while combining sample- and token-level selection produces the best result on every metric.Token-level filtering is identified as crucial for maximizing the benefits of VIG-guided training.
  • Selection-Ratio Analysis: Across selection ratios, VIG substantially reduces supervision cost while maintaining or improving performance, including on LLaVAW, CHAIR, and MMHal.At p = 30, 50, and 70, the model uses approximately 5%, 17%, and 65% of total tokens, respectively.

5. Conclusion

The paper introduces VIG to quantify visual contributions at sample and token levels, then uses it to selectively train LVLMs with less supervision and stronger grounding.

  • Conclusion: VIG quantifies how much multimodal samples and tokens benefit from visual input using a perplexity-based measure.It highlights colors, spatial relations, and object attributes while deemphasizing text-predictable tokens.
  • Conclusion: VIG-guided selective training prioritizes high-VIG samples and tokens to improve vision understanding and hallucination robustness with only a fraction of original supervision.The approach is complementary to existing visual grounding strategies and yields additional gains when combined.
  • Conclusion: The authors conclude that explicitly quantifying visual contributions in training data is a promising direction for building LVLMs that more reliably use visual evidence.

Impact Statement

The work targets more faithful visual grounding and reduced hallucination, while emphasizing that broader reliability and safety concerns remain outside its scope.

  • Potential Impact: Prioritizing visually informative samples and tokens aims to reduce textual-prior reliance and mitigate hallucinations in LVLMs.The stated application relevance includes visual assistance and document understanding.
  • Limitations and Safety: VIG-guided training does not address biased or incomplete data, uneven performance across domains or user groups, or overconfidence in high-stakes settings.The paper calls for careful evaluation before deployment in sensitive real-world applications.
  • Evidence of Grounding: High-VIG samples depend strongly on fine-grained visual evidence, whereas low or negative VIG samples are mostly answerable from textual priors or common sense.These qualitative patterns support VIG as a sample- and token-level visual-grounding measure.

B. Details of Benchmarks

The experiments evaluate VIG across visual-understanding and hallucination benchmarks, with qualitative examples and implementation details defining how visual dependence and selection are assessed.

  • Visual Understanding Task: The visual-understanding evaluation spans general perception, multimodal reasoning, foundational 2D/3D skills, and document understanding.The benchmark set includes LLaVAW, MMVet, MMBench, CV-Bench, and DocVQA.
  • Hallucination Evaluation Task: Hallucination evaluation uses POPE for object existence and CHAIR for hallucinated objects in generated captions.MMHal is also used to assess hallucination with GPT-4 grading and reported average score and hallucination rate.
  • Qualitative VIG Examples: Table 7 illustrates VIG as a visual-grounding strength signal: high positive scores indicate strong matches, near-zero scores weak or partial grounding, and negative scores conflicting images.
  • VIG Measurement: VIG compares model loss with and without visual information, using Gaussian blur to remove semantic visual cues from the image.The original and blurred images are illustrated in Fig. C.1.

D.1. Instruction-tuning Datasets and the Threshold τp

This appendix describes the instruction-tuning datasets, VIG computation and training setup, and qualitative examples comparing vanilla and VIG-trained responses.

  • D.1. Instruction-tuning Datasets and the Threshold τp: The LLaVA-1.5, ShareGPT4V, and Open-Qwen2VL experiments use their respective instruction-tuning datasets and specified selection thresholds.LLaVA-1.5 uses Liu et al. (2024b), ShareGPT4V replaces detailed descriptions with ShareGPT4V captions, and Open-Qwen2VL uses a 1M MAmmoTH-VL-10M subset.
  • D.1. Instruction-tuning Datasets and the Threshold τp: The qualitative examples compare user prompts with vanilla and VIG-trained responses for airplane-action and detailed-image-description tasks.The examples include the prompt “Is this airplane taking off or landing?” and multiple painting descriptions.
  • D.1. Instruction-tuning Datasets and the Threshold τp: For a painting of a dog dressed as a woman, VIG training describes Renaissance-style clothing and pose rather than only repeating the generic costume description.The VIG response specifies a hood, cape, and human-like pose.
  • D.1. Instruction-tuning Datasets and the Threshold τp: For a dog portrayed as Mona Lisa, VIG training retains the Mona Lisa reference while describing the bandana, iconic pose, expression, and posture.The vanilla response mentions the bandana and pose but also adds broader scene details; the VIG response emphasizes the visual resemblance.
  • D.1. Instruction-tuning Datasets and the Threshold τp: For another dog painting, the vanilla response assigns a red hat and blue robe, while the VIG response describes a black turban and brown robe.The VIG response also identifies a landscape background, direct gaze, realistic style, and portraiture.
  • D.1. Instruction-tuning Datasets and the Threshold τp: VIG training changes the airplane answer from uncertainty to the concise statement that it is taking off from the runway.The vanilla responses include uncertainty or longer assertions, whereas the VIG-trained response is concise.
  • D.2. Implementation Details: VIG scores are computed with pretrained checkpoints before selecting samples for subsequent instruction tuning.The VIG calculation takes approximately 6 hours on 8 RTX 4090 GPUs, and VIG training uses 8 A100 GPUs with original model hyperparameters.

D.3. Qualitative Results

Qualitative and attention analyses show VIG training producing more visually grounded responses and allocating more attention to visual tokens across model scales and architectures.

  • Qualitative evaluation: VIG-guided selective training is evaluated qualitatively on LLaVA-1.5 7B, LLaVA-1.5 13B, and ShareGPT4V 7B.The analysis examines hallucination mitigation and visual grounding across these models.
  • Object existence: Vanilla LLaVA-1.5 models hallucinate people and a dining table in a dog painting, whereas VIG training yields descriptions faithful to the dog’s solitary presence and pose.The authors attribute the baseline errors to generic descriptions retrieved from learned knowledge and connect the improvement to filtering low-VIG tokens.
  • Attribute recognition: VIG training improves attribute recognition by reducing ShareGPT4V’s tendency to transfer Mona Lisa attributes such as a red hat and blue robe.The baseline recognizes the semantic similarity but follows the original painting’s attributes instead of pixel-level evidence.

E.2. Blind Faith in Text

Under conflicting textual descriptions, VIG training improves robustness to text corruption while preserving clean accuracy and text-only benchmark performance.

  • Text corruption: For LLaVA-1.5 13B, corruption accuracy rises from 36.8% to 42.9%, while normalized performance increases from 45.7 to 53.0.Clean accuracy remains comparable between the vanilla and VIG-trained models.
  • Comparison with VCD: On LLaVA-1.5 7B under VQAv2 text corruption, VIG training gains +12.9 Norm points compared with only +1.7 for VCD.The comparison uses the same corrupted-text setting and evaluates Base, Corruption, and Norm metrics.
  • Comparison with VCD: The authors characterize VIG as addressing language bias during training, whereas VCD applies inference-time logit penalties.The reported comparison is presented as evidence that training changes visual grounding more fundamentally.
  • Text-only benchmarks: After VIG training, LLaVA-1.5 13B and ShareGPT4V retain largely unchanged performance on GSM8K, MMLU, HellaSwag, and TruthfulQA.The small overall differences are interpreted as preserving text comprehension while improving visual grounding.

E.4. Impact of VIG-based Filtering on Data Distribution

VIG-based filtering preserves long-tail concepts, favors visually demanding tasks, and outperforms random data reduction through combined sample- and token-level selection.

  • Long-tail preservation: In the selected 70% of LLaVA-665K, COCO object-category retention is nearly uniform across head, torso, and tail frequency tiers.The analysis tests whether filtering disproportionately removes rare concepts.
  • Selection rationale: VIG selection evaluates visual necessity at the sample level rather than filtering rare words, preserving long-tail concepts and conceptual diversity.Tasks answerable from textual priors or common sense are filtered more aggressively.
  • Selection-level ablation: SS+TS achieves the best results and significantly outperforms the baseline, whereas Random degrades performance across most benchmarks.SS selects top-VIG samples, and SS+TS adds token-level VIG selection.

F.2. Effect of Selection Ratio

The study compares VIG selection ratios on LLaVA-1.5 7B, finding that selective training generally surpasses full-data training while trading efficiency against broad-task coverage. The p = 70 setting provides the reported best balance, with peak POPE and MMHal scores at reduced computational cost.

  • Selection thresholds τp are derived separately for each tested ratio p ∈ {30, 50, 70, 100}.The threshold values are reported in Table F.2.
  • VIG-guided filtering with p < 100 consistently outperforms the full-data baseline on hallucination and open-ended benchmarks.Here, p = 100 is the vanilla model trained on the full instruction-tuning dataset without VIG-based selection.
  • p = 30 uses only ∼5% of tokens but shows minor degradation on broad-coverage tasks such as MMBench.
  • p = 70 achieves the optimal balance, securing peak scores on POPE and MMHal while significantly reducing computational cost.The passage attributes this balance to preserving sufficient diversity for complex reasoning.
  • The ablation tracks retained answer-token totals and effective loss-contributing tokens after sample- and token-level selection.The table also marks whether higher or lower metric values are better and bolds the best performance.
Loading 2602.17186v2…