PaperlayerView exact revision on arXiv

Source-linked AI summary

VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu

arXiv:2603.01195v2cs.CVcs.AI

TL;DR

Multimodal instruction datasets can contain redundant or misaligned samples, while existing selection approaches may not isolate the visual modality’s contribution. VisNec compares losses with and without visual input and selects high-necessity samples within semantic clusters; using 15% of training data, it reports 100.2% on LLaVA-665K and 115.8% on Vision-Flan-186K.

  • Problem

    Large instruction datasets can be costly and contain visually redundant or misaligned samples, while existing selection methods do not explicitly distinguish visual contribution.

    Source: §1 · pp. 2–3

  • Method

    VisNec measures the marginal contribution of visual input by contrasting blind and multimodal losses, then selects high-scoring samples within semantic clusters.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Using 15% of training data across ten benchmarks, VisNec achieves 100.2% on LLaVA-665K and 115.8% on Vision-Flan-186K.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VisNec offers a data-selection approach that the paper reports can recover or surpass full-data performance with a smaller subset.

    Source: §1 · p. 3, §5 · p. 15

    The blind input is outside the MLLM’s training distribution; the authors argue this bias cancels in relative ordering because selection uses intra-cluster ranking rather than absolute thresholds.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our selection not only further reduces data size but also surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust multimodal instruction tuning. Codes and selected subsets will be released upon acceptance.

1 Introduction

Large multimodal instruction datasets can be costly and contain visually redundant or misaligned samples, while common selection methods do not isolate visual contribution. VisNec scores that contribution and combines it with semantic clustering; experiments report near-full or higher performance using a 15% subset.

1 Introduction

  • Large instruction datasets raise training costs and include redundant or misaligned samples that can weaken visual grounding and degrade reasoning.Text-solvable examples may encourage linguistic shortcuts, while inconsistent image-text supervision can amplify hallucinations.Source: §1 · p. 2
  • Existing selection methods use generic importance or diversity signals without explicitly measuring visual contribution, risking linguistically easy or harmful samples.The paper identifies visual necessity—the need for visual input to resolve predictive uncertainty—as its guiding selection principle.Source: §1 · pp. 2–3
  • VisNec contrasts model losses with and without visual input to score each sample’s marginal visual contribution and identify critical, redundant, or misaligned instances.Semantic clustering followed by within-cluster VisNec ranking is intended to preserve task diversity.Source: §1 · p. 3
  • Using only 15% of training data, VisNec reaches 100.2% on LLaVA-665K and 115.8% on Vision-Flan-186K across ten benchmarks.The reported results recover or surpass full-data performance.Source: §1 · p. 3
  • The paper frames visual-modality neglect as a source of pseudo-multimodal samples that reinforce linguistic shortcuts and hinder cross-modal reasoning.It presents VisNec as a model-relative framework for selecting visually indispensable, task-diverse samples.Source: §1 · p. 3

2 Related Work

Prior work addresses multimodal instruction tuning, data quality, and sample selection, but the paper argues that existing signals do not isolate visual informativeness from linguistic difficulty or misalignment.

Multimodal Instruction Tuning

  • Large instruction-tuning datasets support diverse multimodal tasks but require substantial training resources and can contain redundant or misaligned samples.The related-work discussion cites LLaVA [21], InstructBLIP [6], and Qwen-VL [1] as established frameworks.Source: §2 · pp. 3–4

Information-Theoretic Data Quality

  • V-usable information [8] measures uncertainty reduction beyond existing context, while text-only loss signals cannot distinguish visual informativeness from linguistic ambiguity or noise.The paper extends this information-theoretic perspective through counterfactual comparison of blind and multimodal losses.Source: §2 · p. 4

Data Selection for Instruction Tuning

  • Prior selection methods face overhead, external-resource dependence, or language-centric scoring that can favor textual richness over visual grounding.The discussion describes Self-Filter, PreSel [32], CoIDO [41], COINCIDE, XMAS [26], and ICONS as examples of these trade-offs.Source: §2 · p. 4
  • IFD [16] selects by text-only response loss, which can conflate visual complexity with textual ambiguity or annotation noise and reward misleading images.VisNec instead contrasts blind and multimodal losses to reward visual informativeness while penalizing misalignment.Source: §2 · p. 4

3 Methodology

VisNec measures how much visual input reduces response-prediction loss beyond text alone, then filters and ranks samples within semantic clusters. The case study illustrates how negative, near-zero, and high scores correspond to misaligned, redundant, and visually dependent examples.

3 Methodology

  • The standard multimodal training objective minimizes response-token negative log-likelihood, but treats all samples alike despite differences in visual necessity.The passage raises the distinction between samples requiring visual reasoning and those solvable from linguistic priors.Source: §3 · p. 5

3.1 Visual Necessity Score (VisNec)

  • The blind forward pass masks image tokens and their attention, keeping the model and response-token loss positions consistent with the multimodal pass.The blind loss estimates residual uncertainty from linguistic context alone.Source: §3.1 · pp. 5–6
  • VisNec is the blind loss minus multimodal loss: positive scores indicate visual gain, near-zero scores redundancy, and negative scores misalignment or noise.A larger loss difference means visual input reduces predictive uncertainty beyond text alone.Source: §3.1 · pp. 5–6

3.2 Semantic-Aware Stratified Sampling

  • Because visual dependency varies by task, VisNec first clusters questions by semantic intent before selecting high-scoring examples within each cluster.The paper gives geometric reasoning, OCR, and creative generation as examples of distinct task intents.Source: §3.2 · pp. 6–7, Figure 2 · p. 7
  • Samples with non-positive scores are discarded; the remaining samples are ranked by VisNec within their semantic cluster under a budget ratio.This coarse-to-fine selection is designed to retain visual indispensability alongside broad task coverage.Source: §3.2 · p. 7

3.3 Case Study

  • The low-score case contains image-annotation contradictions, while the medium-score example is largely answerable from linguistic priors.Their scores are −1.21 and 0.21, respectively.Source: §3.3 · p. 8, Figure 3 · p. 8
  • The high-score case asks about objects at specified image coordinates, illustrating a question that cannot be answered from linguistic priors alone.The paper concludes that filtering negative scores and prioritizing high scores removes noisy annotations and text-solvable shortcuts while retaining cross-modal reasoning samples.Source: §3.3 · pp. 8–9

4 Experiments and Results

Experiments evaluate VisNec at a 15% data budget across two datasets and ten benchmarks, with additional analyses of transferability, sensitivity, loss components, clustering, and cost. Results include full-data-level or higher performance, including 115.8% relative performance on Vision-Flan-186K.

4.1 Experimental Setup

  • Experiments use LLaVA-665K and Vision-Flan-186K at a 15% sampling ratio, evaluating models across ten multimodal benchmarks under official training configurations.Source: §4.1 · p. 9

4.2 Main Results on Multi-modal Data Selection

  • At 15%, VisNec reaches 100.2% Rel. on LLaVA-665K and 115.8% Rel. on Vision-Flan-186K, exceeding full-data training by 15.8% on Vision-Flan-186K.The LLaVA-665K comparison uses 98K samples; the Vision-Flan-186K comparison uses 28K samples.Source: Table 1 · p. 10, §4.2 · pp. 10–11, Table 2 · p. 11

4.3 Transferability across Architectures and Scales

  • At 15%, Qwen2.5-VL-3B, 7B, and 32B subsets reach 103.8%, 104.0%, and 102.4% of their respective full-data performance.VisNec scoring was applied directly to the Qwen2.5-VL family on LLaVA-665K.Source: §4.3 · p. 11, Table 3 · p. 12

4.4 Parameter Sensitivity Study

  • Across k=10–30, relative performance fluctuates within 1.3%, stays above 98.9%, and supports the default k=20.Source: Figure 4 · p. 12, §4.4 · p. 12
  • At 5%, VisNec reaches 99.1% Rel., peaks at 15% with 100.2% Rel., then dips to 99.2% Rel. at 20%.The study uses 15% as the optimal ratio for peak accuracy with minimal computational cost.Source: Figure 4 · p. 12, §4.4 · pp. 12–13

4.5 Contribution of Loss Components

  • Full VisNec achieves 100.2% Rel., 4.6% above the best single-modality baseline; text-only scores 95.6% versus Random at 94.2%, while multimodal-only reaches 94.0%.Without the text-only contrastive signal, selection can favor visually complex samples with low instructional clarity, introducing noise.Source: §4.5 · p. 13, Table 4 · p. 13

4.6 Contribution of Clustering

  • TopVisNec trails the full framework by 3.2%, indicating that score-only selection without cluster constraints does not maintain balanced task distribution.Source: §4.6 · pp. 13–14, Table 5 · p. 14

4.7 Cost Analysis

  • VisNec reaches 100.2% Rel. at 23.0 H100 GPU-hours versus 76.0 for full-data fine-tuning; selection takes 12.0 hours versus 73.5 for Self-Filter and 55.5 for COINCIDE.Source: §4.7 · p. 14, Table 6 · p. 14

5 Conclusions

VisNec addresses redundancy and misalignment in multimodal instruction-tuning data by measuring the marginal contribution of visual context. Experiments report performance above full-data baselines at lower computational cost, with robustness across hyperparameters and architectures.

5 Conclusions

  • VisNec filters redundant and misaligned samples while preserving task diversity by quantifying visual context’s marginal contribution relative to linguistic priors.The framework targets current multimodal instruction-tuning datasets.Source: §5 · p. 15
  • Experiments report that selected visually indispensable samples outperform full-data baselines at a fraction of the computational cost.The conclusion also reports robustness to hyperparameters and cross-architecture transferability.Source: §5 · p. 15

Related papers

Loading 2603.01195v2…