PaperlayerView exact revision on arXiv

Source-linked AI summary

DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection

Yuyang Hong, Jinhui Guo, Jiaqi Gu, Lubin Fan, Ruixiang Wang, Kun Ding, Yue Wu, Shiming Xiang, Jieping Ye

arXiv:2608.30209v1cs.CV

TL;DR

Visual instruction selection must identify useful subsets from rapidly growing datasets, but existing methods overlook internal multimodal consistency. The paper introduces DIC and DICS to score sample consistency and select diverse subsets under budget constraints. DICS outperforms baselines, including full-data fine-tuning in reported settings, while using substantially less data.

  • Problem

    Existing selection methods overlook consistency among images, instructions, and responses, limiting their assessment of multimodal sample quality.

  • Method

    DIC measures sample-level consistency through VIC and RIC, while DICS combines DIC scores, diversity, and sampling efficiency for budget-aware subset selection.

  • Results

    DICS consistently outperforms state-of-the-art baselines across dataset scales and model architectures, and 25% of the 6M dataset surpasses full-dataset fine-tuning.

  • Takeaways & Limitations

    DICS provides a robust and scalable approach to selecting high-consistency visual instruction subsets under varying data budgets.

  • Takeaways & Limitations

    Some curated samples contain erroneous image-answer alignments, which can affect fine-tuning on flawed data.

Abstract

from arXiv · show

Visual instruction tuning is crucial for advancing the vision-language alignment and instruction-following capabilities of Vision-Language Models (VLMs). However, identifying optimal subsets under a fixed ratio constraint from rapidly expanding datasets remains a significant bottleneck. While existing methods largely depend on distribution diversity or heuristic filtering, they often overlook the internal coherence within individual samples. To bridge this gap, we propose Data Intrinsic Consistency (DIC), a self-scoring metric designed to quantify the sample-level inter-component consistency. DIC consists of two modules: Visual Information Consistency (VIC), evaluating the alignment between visual content and instructions, and Response Information Consistency (RIC), assessing response coherence relative to the instruction. Building upon DIC, we introduce Data Intrinsic Consistency Selection (DICS), an adaptive data selection method that optimizes the trade-off between high intra-sample consistency and global distributional diversity under varying data budgets. Extensive experiments demonstrate that DICS consistently outperforms state-of-the-art methods across diverse dataset scales and model architectures, surpassing full-dataset fine-tuning while using only 25% of the LLaVA-1.5-665K data. We further curate DICS-6M, a 6M-sample multi-modal instruction corpus that enables the largest-scale visual instruction selection study to date; remarkably, DICS reaches 94.52\% of the official InternVL3-8B-Instruct performance using less than 25\% of its reported training data. Code can be seen at https://github.com/cqu-student/DICS

1 Introduction

The paper identifies a gap in visual instruction selection: existing approaches overlook consistency among images, instructions, and responses. It proposes DIC and DICS to select diverse, internally consistent subsets under different data budgets.

  • Existing selection methods use diversity, relationships, importance, or heuristics but neglect intrinsic relationships within individual multimodal samples.
  • DIC scores sample-level consistency through VIC for visual-instruction alignment and RIC for response focus, accuracy, and coherence.
  • DICS jointly considers DIC scores, data diversity, and sampling efficiency, prioritizing high-DIC samples at low ratios and removing redundant low-DIC samples at higher ratios.
  • DICS consistently outperforms state-of-the-art baselines on LLaVA-1.5-665K across sampling ratios, with a 25% subset comparable to full-data fine-tuning.
  • Using only 25% of the 6M dataset yields superior performance to fine-tuning on the full 6M dataset.
  • DICS matches 94.52% of official InternVL3-8B-Instruct performance with less than 25% of its training data, using 5.1M versus 21.7M samples.

2 Related Work

Visual instruction selection methods reduce training costs through representation- or gradient-based filtering, but existing approaches often neglect sample-level multimodal quality. DICS addresses this by measuring visual information contribution while preserving distributional diversity.

  • Visual instruction tuning aligns language models with visual modalities and has developed through data scaling and architectural refinement.
  • Representation-based and gradient-based methods identify compact subsets to reduce redundancy and computational costs.
  • Existing methods largely neglect intrinsic sample quality, while some consistency methods require expensive external LLMs or only model unidirectional alignment.
  • DICS quantifies visual information gain by contrasting image-conditioned and image-free scenarios, distinguishing visual necessity from semantic correlation.

3 Data Intrinsic Consistency

Data Intrinsic Consistency (DIC) measures whether an image, instruction, and response are mutually informative and aligned. It combines visual grounding and response-to-instruction coherence through two information-gain-based scores.

  • Visual Information Consistency: Visual Information Consistency (VIC) measures how much observing the image reduces uncertainty when predicting the response.It compares weighted response-prediction loss with and without the image, then exponentiates the loss difference.
  • Visual Information Consistency: Higher VIC indicates responses grounded in visual content, whereas vision-irrelevant or hallucinatory samples receive lower scores.For multi-turn dialogues, VIC is averaged across turns.
  • Response Information Consistency: Response Information Consistency (RIC) evaluates whether the response helps reconstruct the original instruction given the visual context.It uses an inverse-prediction formulation comparing instruction loss without and with the response.
  • Response Information Consistency: RIC increases when the response improves prediction of instruction tokens conditioned on the image.High RIC scores identify samples with coherent semantic alignment, with scores averaged across dialogue turns.
  • Data Intrinsic Consistency: DIC is the minimum of VIC and RIC, requiring both visual informativeness for response generation and precise response alignment with the instruction.This minimum operation enforces strong consistency across both dimensions.

4 Instruction Data Selection

DICS selects a subset using DIC scores, semantic diversity, and the sampling budget. Its adaptive strategy prioritizes high-consistency samples at low ratios and removes redundant low-consistency samples at high ratios.

  • Selection Principles: DICS selects a dataset subset by balancing high DIC scores, preserved semantic coverage, and adaptability to the sampling ratio.These principles aim to maximize consistency while preventing distribution shift under different budgets.
  • Selection Pipeline: The pipeline first computes and ranks each sample’s DIC score, then performs adaptive sampling with Farthest Point Sampling using the target ratio.The method consists of scoring-and-ranking followed by adaptive selection.
  • Adaptive Sampling: At low selection ratios, DICS prioritizes high-DIC samples while maintaining diversity; at high ratios, it removes redundant samples with low DIC scores.Large-scale selection therefore functions like practical filtering while balancing efficiency and data quality.

5 Experiments

Across datasets, model architectures, sampling ratios, and selection strategies, DICS delivers robust data-efficient visual instruction tuning. Its gains are attributed to combining intrinsic consistency with diversity, with strong results on both standard and 6M-scale evaluations.

  • Main Results: DICS achieves a 1.40% average improvement over full-dataset training across 12 LLaVA-1.5-7B benchmarks.It reports SOTA scores on MMBench-CN (55.03), MMMU (36.11), DocVQA (24.64), InfoVQA (23.11), and POPE (84.91).
  • Sampling-Ratio Robustness: DICS consistently outperforms representative baselines across sampling ratios, peaking at 103.15% relative performance at a 50% sampling ratio.The optimal sampling rate varies by metric and dataset, while higher ratios can introduce more low-DIC samples.
  • Cross-Architecture Generalization: DICS generalizes across architectures, with average performances reported for evaluation on LLaVA-1.5-665K using Qwen2-VL-7B.The study also evaluates the method on InternVL3-8B with the DICS-6M corpus.
  • Cross-Dataset Generalization: At 25% sampling, DICS reaches 99.91% relative performance on Vision-FLAN-186K, outperforming baselines while closely matching full-data fine-tuning.This evaluation tests generalization beyond the LLaVA-1.5-665K distribution.
  • Large-Scale Evaluation: On DICS-6M, an 85% subset outperforms full-dataset training by 3.10%, while less than 25% of the official data reaches 94.52% of official InternVL3-8B-Instruct performance.The comparison uses 5.1M versus 21.7M training samples for the latter result.
  • Ablation Studies: At a 25% LLaVA-1.5-665K budget, Min-based DICS reaches 101.40% relative performance by requiring both VIC and RIC to be high.Removing the weighted token strategy lowers performance from 101.40% to 100.25%, and FPS is adopted as the default selection strategy.
  • Qualitative Analysis: Qualitative attention maps show that DICS provides more precise visual grounding for image-based question answering.The example asks for the horse’s color and has the answer “Brown.”
  • Computational Efficiency: DICS requires inference-time forward passes without gradient updates; selection and reduced training save approximately 1 hour on LLaVA-1.5 and 5 hours on DICS-6M.The reported workflows use 8 A100 GPUs for LLaVA-1.5 and 32 A100 GPUs for DICS-6M.

6 Conclusion

The paper introduces DIC, which measures sample-level consistency between images, instructions, and responses, and DICS, which selects diverse subsets using these scores. Experiments support its effectiveness, while the current consistency evaluation remains limited to image-text data.

  • Conclusion: DIC quantifies sample-level inter-component consistency through Visual Information Consistency and Response Information Consistency.VIC evaluates visual-instruction alignment, while RIC assesses response coherence relative to the instruction.
  • Conclusion: DICS adaptively balances intra-sample consistency with distributional diversity across varying data budgets.It prioritizes high-DIC samples at low ratios and removes low-DIC redundant samples at high ratios.
  • Limitations: The consistency evaluation currently focuses on image-text data and excludes modalities such as video and audio.The authors identify broader multimodal extension as future work.
  • Token Weighting Strategy: The token-weighting strategy assigns weight 1.0 to content words and 0.1 to functional words to emphasize semantic content.spaCy POS tags determine the weights, with content and functional categories defined separately.
  • RIC: RIC uses inverse prediction to penalize generic responses by measuring how well the response helps reconstruct the original instruction.The method conditions reconstruction on visual context, with and without the response, using dedicated prompts.

A.4 Implementation Details of DICS

DICS ranks samples by their minimum VIC and RIC score, then combines ratio-dependent quality filtering with diversity-aware sampling. Its FPS implementation has explicit computational costs, while piecewise sampling reduces overhead.

  • DICS Algorithm: DICS defines each sample’s DIC score as the minimum of its VIC and RIC scores before descending ranking.This ranking supplies the quality ordering used by the adaptive selection procedure.
  • DICS Algorithm: For p < 0.5, DICS samples diversely from a candidate pool of size 2p · |D| containing top-ranked samples.For p > 0.5, it retains a core set of size (2p−1)·|D| and samples diversely from the remainder; p = 0.5 selects the top half.
  • Farthest Point Sampling: Farthest Point Sampling provides diversity-aware selection from candidate embeddings.The algorithm requires a candidate set with embeddings and selects the target subset iteratively.
  • Complexity Analysis: Piecewise sampling significantly reduces computational overhead compared with running FPS on the entire dataset.GPU-batched distance computation can select 100K samples from one million 4096-dimensional embeddings in minutes on one A100 GPU.
  • Complexity Analysis: FPS has time complexity O(k ·M ·d) and space complexity O(M · d) for candidate size M, target size k, and embedding dimension d.Each iteration computes distances across all candidates, while storage includes embeddings and a distance array.

B.1 Datasets

The evaluation uses LLaVA-1.5-665K, Vision-FLAN-186K, and the newly curated DICS-6M corpus to test DICS across scales, distributions, and task types. DICS-6M contains six million instruction samples assembled under the InternVL3 collection strategy.

  • Datasets: The study uses three datasets spanning different data scales, distributions, and task types.The datasets are LLaVA-1.5-665K, Vision-FLAN-186K, and DICS-6M.
  • LLaVA-1.5-665K: LLaVA-1.5-665K aggregates 665,298 samples from conversational, visual-question-answering, OCR, and pure-language sources.Its components include 158K GPT-generated multimodal instructions, academic VQA, OCR-related data, and 40K ShareGPT conversations.
  • Vision-FLAN-186K: Vision-FLAN-186K integrates 191 diverse vision-language tasks to address limited task diversity in existing benchmarks.It differs from LLaVA-1.5-665K by integrating tasks from academic sources rather than predominantly relying on LLM-generated conversational data.
  • DICS-6M: DICS-6M comprises 6 million high-quality instruction samples collected according to the official InternVL3 open-source strategy.The corpus supports large-scale validation of DICS.

B.2 Benchmarks

The benchmark suite evaluates general capability, knowledge and reasoning, document understanding, hallucination, and open-ended dialogue using standardized protocols. The study compares DICS with stochastic, heuristic, and other data-selection strategies across broad multimodal tasks.

  • Benchmark Domains: The evaluation covers five domains: general capability, knowledge and reasoning, document understanding, hallucination, and open-ended dialogue.The suite includes MME, MMBench, MMMU, SQA-I, VizWiz, OCR and document benchmarks, POPE, HallusionBench, and MMVet.
  • General Capability and Reasoning: MME and MMBench assess general capability, while MMMU, SQA-I, and VizWiz test knowledge, scientific reasoning, and real-world visual questions.MME includes perception and cognition subtasks, and MMBench uses circular evaluation.
  • Document Understanding: Document-understanding evaluation includes AI2D, DocVQA, and InfoVQA for diagrams, documents, and infographics.The broader benchmark listing also includes extensive OCR-related tasks.
  • Hallucination: POPE and HallusionBench evaluate object-existence hallucination, visual illusions, and reasoning consistency.These benchmarks target model reliability under hallucination-related conditions.
  • Evaluation Protocol: VLMEvalKit provides standardized evaluation with generation temperature fixed at 0.0, while Qwen3-32B judges selected open-ended tasks.The judge model is used for MM-Vet and open-ended responses in MMBench variants.
  • Selection Baselines: The study compares random selection, length filtering, CLIP-Score, and other data-selection strategies against DICS.The comparison framework groups methods into stochastic sampling, heuristic filtering, and broader selection categories.

C.1 More Results

Across sampling ratios, DICS maintains strong performance and selects subsets concentrated in high-consistency regions while retaining score variance and distributional coverage. Its repeated-run results also show minimal variance.

  • Superiority across Data Scales: 103.15% is DICS’s peak performance at the 50% sampling ratio, while performance remains stable in high-data regimes.At ratios below 25%, DICS leads; above 75%, all methods saturate or slightly decline, but DICS remains more resilient than baselines such as PRISM.
  • Distributions of DIC Scores: DICS-selected 25% data concentrates both VIC and RIC scores in higher ranges while preserving broad coverage across those ranges.The retained variance reflects samples of differing complexity, supporting intrinsic consistency without sacrificing distributional diversity.
  • Statistical Analysis: All reported standard deviations are ≤0.13 across five independent runs, indicating minimal variance in the main experiments.These statistics correspond to the overall results reported for Tables 1 and 2.

C.2 Robustness and Scalability

DICS transfers across selector and target architectures and scales, with detailed evaluations using fixed 25% subsets. The results support architecture-agnostic consistency metrics and weak-to-strong selection.

  • Architecture-Agnostic DIC Metrics: 101.25% relative performance is achieved when LLaVA-1.5-7B-selected data trains the Qwen2-VL-7B target, versus 100.91% for Qwen-selected data.The result is reported as surpassing the full-dataset baseline and supports transfer across selector models.
  • Evaluation Scope: The evaluation includes detailed hyperparameters, 25% Vision-FLAN comparisons, five-run statistics, and ablations on DIC and selection strategies.These materials cover implementation settings, dataset transfer, statistical reporting, and component-level comparisons.
  • Scalability via Weak-to-Strong Generalization: 101.79% relative performance is achieved by training LLaVA-1.5-13B with data selected by the smaller 7B selector.This weak-to-strong result indicates that lightweight models can act as data filters for larger target models, reducing selection-time computation.
  • Architectural Generalizability of DICS: 71.35 is achieved by Qwen2-VL-7B-Base after fine-tuning on only 25% of the LLaVA-665K data selected by DICS, compared with a 51.05 base-model score.The comparison evaluates selected-subset training against the lower-bound base model.

C.3 Ablation Studies and Analysis

Ablations and validation experiments show that DIC quality filtering, VIC–RIC complementarity, strict score fusion, and token weighting contribute to DICS performance and data-quality sensitivity. Additional analyses examine human alignment, attention, and computational scaling.

  • Quality Filtering: 97.02% is obtained by the Diversity Only baseline, the lowest result, showing that diversity without intrinsic quality filtering introduces damaging low-quality samples.The comparison supports combining distributional coverage with sample-level consistency.
  • Complementarity of VIC and RIC: 101.02% is achieved by VIC alone for general capabilities, whereas RIC alone reaches 97.60% in general tasks but performs well on hallucination benchmarks.VIC emphasizes visual-textual fidelity, while RIC supports structural and logical response consistency.
  • Fusion Strategies: 101.40% is achieved by min(VIC, RIC), outperforming simple Sum or Product fusion by requiring both visual alignment and response logic to be high.The minimum operator acts as a strict gate against samples that are strong on only one consistency dimension.
  • Impact of Token Weighting: Removing POS-based token weighting reduces performance from 101.40% to 100.25%, supporting emphasis on content-rich tokens.The weighting uses tokens such as nouns and verbs to assess semantic consistency.
  • Impact of Selection Strategies: All diversity-aware second-stage strategies outperform the full-dataset baseline at 25% sampling, with every result exceeding 100%.The finding indicates that DIC-based filtering remains effective across subsequent diversity-aware selection methods.
  • Controlled Corruption Analysis: All three controlled corruptions produce significant drops in the corresponding scores, demonstrating DIC sensitivity to data-quality degradation.The corruptions include image-text mismatch, injected factual errors or hallucinations, and generic or irrelevant responses.
  • Human and VLM-as-Judge Quality Assessment: DIC correlates with both VLM-judge and human quality scores, with Spearman ρ = 0.20 and ρ = 0.27 respectively, both at p < 0.001.Ratings increase across low-, medium-, and high-DIC quantile groups on a 1–5 quality scale.
  • Qualitative Comparison via Attention Map: DICS concentrates attention on core target objects, whereas the random-selection baseline distributes attention diffusely or toward irrelevant backgrounds.The comparison uses attention maps from LLaVA-1.5 and a model trained on a 15% random subset.
Loading 2608.30209v1…