ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning

Changti Wu; Jiahuai Mao; Yuzhuo Miao; Shijie Lian; Bin Yu; Xiaopeng Lin; Cong Huang; Lei Zhang; Kai Chen

Source-linked AI summary

ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning

Changti Wu, Jiahuai Mao, Yuzhuo Miao, Shijie Lian, Bin Yu, Xiaopeng Lin, Cong Huang, Lei Zhang, Kai Chen

arXiv:2602.11636v1Feb 12, 2026cs.CVcs.AI

TL;DR

Large-scale visual instruction tuning is costly because datasets contain redundant samples, while existing selectors may require training, proxy resources, instruction-agnostic representations, or quadratic pairwise comparisons. ScalSelect uses instruction-conditioned early representations and global dominant-subspace selection with linear-time scalability. Using 16% of the data, it retains over 97.5% of full-data performance and surpasses full-data training in some settings.

Problem
Redundant visual instruction datasets make full-data tuning costly, while existing selection methods can require extra training, proxy resources, instruction-agnostic representations, or quadratic pairwise comparisons.
Method
ScalSelect extracts instruction-conditioned early visual representations and scores samples by their contributions to the dominant low-rank subspace of the full representation space.
Results
Under a 16% selection budget, ScalSelect achieves over 97.50% of full-data performance across two datasets and surpasses full-data training in some settings.
Takeaways & Limitations
ScalSelect provides a practical training-free approach for large-scale multimodal data selection without pairwise comparisons, proxy models, or auxiliary datasets.
Takeaways & Limitations
Prior local selection criteria do not necessarily preserve the overall representation space and incur at least quadratic time complexity.

Abstract

from arXiv · show

Large-scale Visual Instruction Tuning (VIT) has become a key paradigm for advancing the performance of vision-language models (VLMs) across various multimodal tasks. However, training on the large-scale datasets is computationally expensive and inefficient due to redundancy in the data, which motivates the need for multimodal data selection to improve training efficiency. Existing data selection methods for VIT either require costly training or gradient computation. Training-free alternatives often depend on proxy models or datasets, instruction-agnostic representations, and pairwise similarity with quadratic complexity, limiting scalability and representation fidelity. In this work, we propose ScalSelect, a scalable training-free multimodal data selection method with linear-time complexity with respect to the number of samples, eliminating the need for external models or auxiliary datasets. ScalSelect first constructs sample representations by extracting visual features most attended by instruction tokens in the target VLM, capturing instruction-relevant information. It then identifies samples whose representations best approximate the dominant subspace of the full dataset representations, enabling scalable importance scoring without pairwise comparisons. Extensive experiments across multiple VLMs, datasets, and selection budgets demonstrate that ScalSelect achieves over 97.5% of the performance of training on the full dataset using only 16% of the data, and even outperforms full-data training in some settings. The code is available at \href{https://github.com/ChangtiWu/ScalSelect}{ScalSelect}.

1 Introduction

Large-scale visual instruction datasets contain redundancy, making full-data tuning costly and motivating efficient multimodal data selection. ScalSelect addresses this with instruction-aware representations and global subspace preservation, achieving strong performance with a small subset.

Redundant image-instruction datasets can add computational cost while providing limited marginal gains.
Training-based and gradient-based selectors add proxy training, warm-up procedures, or backpropagation overhead.
Many training-free methods use proxy models, auxiliary datasets, or instruction-agnostic visual representations that mix relevant and irrelevant information.
Pairwise selection methods emphasize local relationships but may fail to preserve the global representation space and require at least quadratic time.
ScalSelect constructs instruction-conditioned early representations and selects samples that contribute to the dominant low-rank subspace of the full dataset.
Under a 16% selection budget, ScalSelect achieves over 97.50% of full-data performance on LLaVA-Vicuna-7B across two datasets and surpasses full-data training on some Qwen3-VL experiments.
Its design combines instruction-aware modeling, global subspace-aware selection, linear-time scalability, and no proxy models or additional training.

2 Related Work

Visual instruction tuning extends vision-language models through large-scale image-instruction fine-tuning, while multimodal data selection seeks smaller informative subsets. ScalSelect differs from local similarity-based approaches by preserving the dataset’s global dominant subspace.

Visual instruction tuning fine-tunes vision-language models on large-scale image-instruction pairs for multimodal tasks.
Multimodal data selection reduces training cost by identifying informative subsets while retaining most full-finetuning performance.

3 ScalSelect

ScalSelect selects multimodal training data by combining instruction-conditioned visual representations from the first LLM layer with global dominant-subspace preservation. It ranks samples by their contribution to a low-rank representation of the full dataset rather than by pairwise relationships.

3.1 Instruction-Conditioned Early Representation: ScalSelect constructs sample representations from instruction-conditioned attention and hidden states in the first LLM transformer layer.Visual tokens are selected according to attention received from user instruction tokens, then their first-layer hidden states are averaged.
3.1 Instruction-Conditioned Early Representation: Instruction-conditioned representations capture different visual regions and semantic attributes that can arise from varying textual instructions.The method addresses the instruction-agnostic treatment of visual content used by many prior selection approaches.
3.2 Subspace-Aware Global Selection: ScalSelect selects samples that best approximate the dominant low-rank subspace of the full representation matrix.It centers the representation matrix, applies truncated SVD to its leading components, and uses their statistical leverage scores to rank samples.
3.2 Subspace-Aware Global Selection: The selected subset preserves information relevant to reconstructing the best rank-k approximation of the original representation matrix.The subspace-oriented objective targets global representational structure rather than only local diversity among samples.
3.2 Subspace-Aware Global Selection: The low-rank subspace uses the smallest k whose leading singular values explain 90% of total spectral energy.This threshold is fixed throughout the work for the truncated-SVD approximation.
3.2 Subspace-Aware Global Selection: Samples with the highest statistical leverage scores form the final selected subset.Each score measures a sample row's influence on the best low-rank fit of the centered representation matrix.

4 Scalability Analysis

ScalSelect is analyzed as a scalable selection method whose representation stage requires linear time in the number of samples. Compared with alternatives, it avoids pairwise quadratic comparisons and external proxy resources while retaining linear-time complexity.

Scalability of ScalSelect: The Attention-Conditioned Early Representation stage has O(N) time complexity because each sample requires one forward pass.It uses only first-layer attention scores and hidden states from the target LLM.
Scalability Comparison with Other Methods: ScalSelect provides training-free linear-time selection without relying on pairwise comparisons, proxy models, or auxiliary datasets.This combines the scalability advantage of linear processing with representations derived from the target VLM.
Scalability Comparison with Other Methods: COINCIDE has complexity O(KN+N^2/K) for balanced clusters and still relies on a proxy model.Its complexity remains partly quadratic in N when K is not proportional to N.
Scalability Comparison with Other Methods: PRISM requires O(N^2) time for pairwise sample-similarity computation.This creates a quadratic scalability bottleneck as the dataset grows.
Scalability Comparison with Other Methods: RDS+ achieves linear-time complexity but assumes access to a proxy dataset matching the evaluation distribution.The passage identifies this assumption as potentially unrealistic and a source of distribution bias.

5 Experiments

Experiments evaluate ScalSelect across datasets, models, selection budgets, training settings, and ablations. The results show strong efficiency, generalization, and sensitivity to instruction conditioning, centering, and layer choice.

Experiment Setup: Experiments use LLaVA-V-625K, multiple VLM architectures, diverse multimodal benchmarks, and representative data-selection baselines.LLaVA-V-625K contains 625K multimodal samples spanning VQA, OCR, region-level VQA, and visual conversation; models are fine-tuned for one epoch on 8 NVIDIA H100 GPUs.
Main Results: 99.66% relative performance is achieved with 300K samples, while 400K samples reach 101.16% relative performance versus Full-Finetune.Performance improves as the selection budget increases from 50K to 400K; 50K samples already retain over 95% of Full-Finetune performance on average.
Extended Results: Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct slightly exceed Full-Finetune with 100K selected samples, indicating benefits across model architectures.Both Qwen3-VL models show stronger performance under the 100K budget than LLaVA-Vicuna-7B.
Extended Results: ScalSelect consistently outperforms Random sampling and approaches Full-Finetune across LLaVA-V-625K and LRV-Sub-180K at a 16% selection budget.On LRV-Sub-180K, 29K selected samples attain 97.51% of Full-Finetune performance and surpass Random sampling.
Extended Results: Performance differences remain relatively small between freezing and fine-tuning the projector, supporting application under both visual instruction tuning paradigms.The comparison freezes the visual encoder while either fine-tuning only the LLM or fine-tuning both the LLM and projector.
Ablation and Further Analysis: Removing instruction conditioning degrades performance, while removing centering causes significant degradation and collapses the selected subspace dimensionality to k = 1.Instruction-conditioned cues particularly improve OCRBench; column-wise centering removes the global mean effect and supports a richer subspace.
Ablation and Further Analysis: The first transformer layer gives the best overall performance, whereas the middle layer performs worst and the deep layer improves OCRBench.The deep setting uses the penultimate layer because the final layer is more specialized toward output generation.
Ablation and Further Analysis: Importance scores are highly skewed: a small fraction of samples contributes disproportionately to the dominant subspace, supporting stable selection across budgets.The ranking curve drops sharply at first and then decays gradually across the remaining samples.

6 Conclusion

ScalSelect is a scalable, training-free method that uses instruction-conditioned early representations and subspace-aware global selection to preserve full-dataset structure without pairwise comparisons or auxiliary models. Experiments show it retains most full-data performance with a small data fraction and can surpass full-data training in some settings.

ScalSelect combines instruction-conditioned early representations with subspace-aware global selection for efficient visual instruction tuning.It preserves the dominant structure of the full dataset without pairwise comparisons or auxiliary models.
ScalSelect retains most of full-data training performance using only a small fraction of the data.
ScalSelect surpasses full-data training in some settings.

A.1 Settings of Extracting Representation

The representation-extraction experiments use target-VLM inference with a maximum context length of 4096 tokens and model-specific image-processing settings.

Representation extraction uses target-VLM inference with a maximum context length of 4096 tokens.
LLaVA-Vicuna-7B processes images at a fixed resolution of 336 × 336 following LLaVA-1.5’s default configuration.
Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct resize images so visual tokens do not exceed 576, using 32 × 32 patches.

A.2 Training Hyperparameters

All experiments use eight NVIDIA H100 GPUs with LLaMA-Factory, and the same training configuration is applied across models, datasets, and selection budgets.

All experiments run on 8 × NVIDIA H100 GPUs with 80GB memory using LLaMA-Factory.
The same training configuration is used across all models, datasets, and selection budgets to ensure fair comparison.

A.3 Settings of Baselines

Baseline methods follow the original experimental settings and implementation details reported in their respective papers. The supplied settings also include a training-hyperparameters table.

All baseline methods use the original experimental settings and implementation details from their respective papers.
Training hyperparameters are summarized in Table 8.

B Sensitivity of the Attention-Mass Threshold

ScalSelect’s attention-mass threshold mainly controls visual-token retention, while relative performance stays stable across the tested range. Its subspace-approximation guarantee is presented as a randomized sampling reference, whereas the practical method uses deterministic top-leverage selection.

Attention-Mass Threshold: As τ increases from 0.85 to 0.95, the retained visual-token ratio grows while marginal token-reduction gains diminish.The analysis uses LLaVA-Vicuna-7B on the LLaVA-V-625K dataset.
Attention-Mass Threshold: Overall relative performance remains stable across τ∈{0.85, 0.90, 0.95}, with only minor variations.The results indicate that the threshold affects representation compactness more than performance sensitivity.
Attention-Mass Threshold: τ=0.9 is fixed as the default because it balances visual coverage and representation compactness without strong performance sensitivity.
Subspace Approximation Reference: The relative-error subspace-approximation result is a theoretical reference based on randomized leverage-score sampling in CUR/CX decompositions.The reference uses sufficiently many samples with an O(klogk/ε^2) oversampling rate and gives a 99% probability guarantee.
Deterministic Selection: In practice, ScalSelect selects the highest-scoring samples deterministically using leverage scores, treating this as a well-established heuristic.The sampled rows of the representation matrix correspond to sampled columns after transposition.