PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

Jinhe Bi, Aniri, Zengjie Jin, Yifan Wang, Danqi Yan, Wenke Huang, Xiaowen Ma, Sikuan Yan, Artur Hecker, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, Yunpu Ma

arXiv:2502.12119v5cs.CVcs.AIcs.CL

TL;DR

Growing visual instruction datasets contain redundancy, while existing selection methods impose substantial computational costs that can negate efficiency gains. PRISM addresses this gap by correcting anisotropic visual-feature geometry through implicit re-centering, reducing training time by 70% while outperforming full-dataset fine-tuning.

  • Problem

    Existing data-selection methods for visual instruction tuning are computationally costly, creating a need for selection that is both effective at identifying high-quality data and efficient to deploy.

  • Method

    PRISM is a training-free framework that uses implicit re-centering to correct visual-feature geometry and enable redundancy detection from the MLLM’s own features.

  • Results

    PRISM reduces training time by 70% while achieving performance superior to full-dataset fine-tuning across evaluated benchmarks.

  • Takeaways & Limitations

    After re-centering, PRISM selects samples with low corpus-wide redundancy scores, producing subsets enriched with semantically diverse and informative data.

  • Takeaways & Limitations

    PRISM focuses on semantic redundancy based on feature correlation and does not detect factual inaccuracies or ethical biases in instruction data.

Abstract

from arXiv · show

Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a \textit{Global Semantic Drift}, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise \textbf{PRISM}, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30\% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7\% relative improvement over the baseline. The code is available for access via \href{https://github.com/bibisbar/PRISM}{this repository}.

1 Introduction

Visual instruction datasets contain redundancy that raises tuning costs, while existing selection methods add substantial computational overhead. PRISM addresses this gap by correcting anisotropic visual feature geometry through implicit re-centering and training-free intrinsic selection.

  • Visual instruction datasets increasingly contain low-quality and redundant examples, raising computational costs and potentially degrading final model performance.
  • Existing proxy-based and training-based selectors incur substantial computational costs that can negate their intended efficiency gains.
  • Representation anisotropy gives visual features a non-zero mean and narrow-cone geometry that distorts similarity metrics through Global Semantic Drift.
  • PRISM introduces training-free Intrinsic Selection by implicitly re-centering visual features to correct their geometry and expose intrinsic redundancy signals.
  • 70% training-time reduction accompanies performance superior to models fine-tuned on the complete dataset.

2 Visual Instruction Selection

Visual instruction selection must preserve full-dataset performance while reducing total selection-plus-tuning time. PRISM targets this practical gap by selecting directly from re-centered intrinsic features rather than relying on costly external or training-based signals.

  • A viable selector must preserve performance at least equal to full-dataset fine-tuning.
  • A viable selector must make total selection and subset-tuning time strictly lower than full-dataset tuning time.
  • The Overall Selection Cost score combines performance and computation into one measure of selection overhead.
  • Practical viability requires computational savings without sacrificing final model performance.
  • PRISM avoids proxy-based bias and training overhead by operating directly on the model’s intrinsic, re-centered features.

3 PRISM

PRISM identifies representation anisotropy as the source of global semantic drift that makes raw visual-feature geometry unreliable for data selection. It implicitly re-centers pretrained features, enabling intrinsic, training-free redundancy estimation and semantically diverse pruning.

  • Theoretical Analysis: Anisotropy as a Source of Computational Overhead: Representation anisotropy creates a dominant shared component that masks sample-specific semantics in visual embeddings.The shared component satisfies ∥µ∥2 ≫ E[∥δi∥2], so global drift overwhelms unique semantic residuals.
  • Theoretical Analysis: Anisotropy as a Source of Computational Overhead: Cosine similarity between anisotropic representations is dominated by global drift, obscuring semantic dissimilarity between samples.Theorem 1 attributes this masking to the shared mean vector µ rather than the residual semantic components δi and δj.
  • Theoretical Analysis: Anisotropy as a Source of Computational Overhead: Geometric dissimilarity methods such as Farthest-Point Sampling and distance-based clustering therefore fail to select semantically diverse subsets.Ignoring the geometric flaw forces existing methods toward computationally intensive approximations of data value.
  • PRISM: PRISM restores isotropic geometry through implicit re-centering, making intrinsic visual features a direct signal for semantic redundancy.The method removes the global shift from pretrained MLLM features so their internal variation patterns reflect unique semantic components.
  • Redundancy Identification via Implicit Re-centering of Intrinsic Features: PRISM computes each sample’s corpus-wide redundancy from centered-feature similarities and selects samples below a percentile threshold.The redundancy score is evaluated exactly in O(Nd) time without materializing an N × N matrix, and low-scoring samples are retained.

4 Experiments

PRISM is evaluated across datasets, benchmarks, model families, instruction pools, and ablations. It generally improves performance over full fine-tuning while reducing computational cost, with behavior depending on subset size and design choices.

  • Experiment Setup: PRISM is evaluated on LLaVA-665K and VisionFlan-186K using multimodal benchmarks spanning perception, reasoning, hallucination detection, and conversation.The experiments use LLaVA-1.5-7B as the primary base model and one training epoch with official hyperparameters.
  • Main Results: PRISM outperforms full-dataset fine-tuning by 1.7% relative performance, including MMBench scores of 65.2 versus 64.3 and MM-Vet scores of 32.0 versus 31.1.
  • Main Results: PRISM transfers across five model–encoder combinations, including Vicuna, Phi-2, Qwen2.5, and Llama-3 with CLIP and SigLIP encoders.The reported results indicate that its gains are not specific to the LLaVA architecture.
  • Main Results: PRISM reduces total GPU time by about 70% while occupying the most favorable efficiency–performance region against full fine-tuning and alternative selection methods.The advantage is attributed to its training-free design and aggregate correlation computation.
  • Main Results: At the same 30% selection budget on VisionFlan-186K, PRISM selects 57K samples and reaches 100.9% relative performance versus 94.1% for random selection.This evaluates robustness to a task-diverse instruction pool containing 191 vision-language tasks.
  • In-Depth Model Behavior Analysis: Visual performance increases with larger selected subsets, while textual performance is better preserved under smaller budgets.The sampling-ratio analysis reports that visual performance surpasses full fine-tuning at 30%.
  • Ablation Study: Shallow-layer visual features outperform middle and deep layers, low-correlation subsets yield the highest performance, and averaging image tokens outperforms using the last image token.These ablations support early-layer geometry, correlation-based diversity, and holistic image representations as useful design choices.

5 Conclusion

The paper attributes selection inefficiency to visual representation anisotropy and proposes PRISM, a training-free method using implicit re-centering. PRISM reduces training time by 70% while surpassing full-dataset performance.

  • PRISM traces MLLM data-selection inefficiency to representation anisotropy in visual features.
  • PRISM corrects semantic structure through implicit re-centering without training a selection model.
  • 70% training-time reduction accompanies performance that surpasses full-dataset fine-tuning.

Limitations

PRISM’s limitations concern both its data-quality scope and the assumptions behind its geometry-aware selection. The method targets semantic redundancy through visual-feature correlation and does not address several other quality dimensions.

  • Scope: PRISM focuses on semantic redundancy based on feature correlation rather than factual inaccuracies or ethical biases.The authors identify extending the method to these data-quality dimensions as future work.
  • Selection: PRISM applies a single global threshold that selects the lowest-redundancy 30% of samples across the combined pool.This threshold is intended to produce a data-driven composition, with sources containing more low-correlation samples contributing more examples.
  • Geometry: The method targets the dominant corpus-level mean shift instead of performing full whitening or exact covariance correction.This design prioritizes an efficient selection pipeline over complete isotropization of the representation space.
  • Evaluation: Text-only evaluations indicate that PRISM preserves or improves language knowledge after visual instruction tuning.The largest gain is reported for the 13B Vicuna backbone.

B.4 Robustness on VisionFlan

On VisionFlan-186K, PRISM is tested under a different, task-diverse instruction distribution using the same 30% visual selection budget. It slightly exceeds full-data training while retaining only about one third of the instruction pool.

  • Dataset: VisionFlan-186K contains 191 distinct vision-language tasks and 186K samples, providing a complementary distribution to LLaVA-665K.The evaluation tests whether PRISM remains effective when fine-grained task diversity is preserved.
  • Selection protocol: Using a 30% selection budget, PRISM selects 57K samples from VisionFlan-186K.The budget applies to the visual subset, while text-only samples are retained unchanged in comparable mixtures.
  • Results: 100.9% relative performance is achieved by PRISM, compared with 100.0% for full-data training and 94.1% for random selection.The comparison is based on the aggregate score reported for VisionFlan-186K.
  • Evaluation protocol: The comparison protocol requires the same base model, visual pool, training recipe, epochs, evaluation scripts, and applicable random seed across methods.This supplements the shared rule that only the visual subset is pruned.
  • Metric interpretation: Relative scores for methods with missing entries are averaged only over their available metrics, not over a fully matched benchmark set.The authors state that PRISM and the full-data baseline are evaluated on the complete metric set under the same protocol.

D Model Architecture Details and Generalization Setup

PRISM’s generalization setup spans multiple language-model families, parameter scales, and vision encoders, while separately diagnosing modality-specific representation geometry. The analysis motivates correcting visual features rather than concatenated multimodal features.

  • Model configurations: The evaluation spans Vicuna, Phi-2, Qwen2.5, and Llama-3 language-model families across multiple parameter scales.Qwen2.5 includes 7B Base and Instruct configurations, while Llama-3 uses an 8B backbone.
  • Model configurations: PRISM is evaluated with CLIP ViT-L/14 at 336px and SigLIP-SO400M vision encoders.These encoders represent different cross-modal alignment strategies.
  • Generalization analysis: Each configuration receives both an anisotropy diagnosis and an end-to-end evaluation of PRISM-selected data.The setup is intended to test whether the diagnosis and performance improvements persist across architectures.
  • Modality analysis: Visual features have substantial non-zero mean vectors, whereas text features are comparatively centered and have a flatter singular-value spectrum.This indicates different geometric behavior across the two modalities.
  • Modality analysis: PRISM therefore applies geometric correction to the visual stream instead of a single re-centering operation on concatenated multimodal features.A joint mean could be dominated by visual drift and distort the near-isotropic textual geometry.

E.3 Additional Experiments with Multimodal Features

Additional experiments test whether textual prompts improve PRISM’s selection signal. They find that multimodal-feature selection underperforms the visual-only approach, supporting the focus on visual anisotropy.

  • Experimental design: The additional experiment aggregates image-token and text-prompt-token features before applying the PRISM selection pipeline.It is conducted on LLaVA-1.5-7B.
  • Results: 97.8% relative performance is obtained with multimodal-feature selection, below the 101.7% achieved by visual-only PRISM.Including textual prompts therefore does not improve the reported selection outcome.
  • Interpretation: The authors hypothesize that textual prompts add syntactic-style and template-length variance unrelated to semantic redundancy.They interpret the results as reinforcing visual anisotropy as the dominant factor for intrinsic selection.
  • Efficiency: PRISM’s end-to-end cost-performance comparison reports reduced total time alongside state-of-the-art performance.Figure 11 frames the comparison using GPU hours and final model performance.

F Limitations of Existing Methods and Our Geometric Motivation

Existing selection methods can sacrifice performance, incur prohibitive costs, or introduce proxy bias because they overlook pathological geometry in MLLM visual representations. PRISM addresses this geometric motivation by correcting anisotropy and restoring the usefulness of inexpensive correlation-based selection.

  • Limitations of Existing Methods: Prior selection methods exhibit performance degradation, prohibitive computational cost, and proxy-model bias.These drawbacks can violate performance fidelity and net-efficiency goals while producing subsets misaligned with the target MLLM.
  • Limitations of Existing Methods: Training-based selection may spend more time selecting data than it saves during subsequent tuning.TIVE’s warm-up phase generates gradient signals but can make selection impractical at scale.
  • Geometric Motivation: Anisotropy concentrates embeddings along biased directions, undermining semantic distance and simple geometric selection metrics.The feature space can have low effective rank, with a few singular values capturing most variance.
  • Geometric Motivation: A non-zero visual-feature mean creates a dominant global component that masks sample-specific semantic residuals in cosine similarity.Under ||µ||2 ≫ E[||δi||2], true semantic similarity is relegated to lower-order terms.
  • PRISM’s Geometric Motivation: PRISM introduces intrinsic selection by implicitly re-centering visual features to remove the global non-semantic shift.This correction restores a better-behaved geometry and makes inexpensive correlation-based selection useful without proxy or training-dynamics costs.
  • Geometric Motivation: Anisotropy diagnostics remain consistent across five model–encoder combinations, supporting the generality of the geometric issue.Visual features show substantial non-zero shifts while textual features remain approximately centered.

H.3 Dataset-Mean Centering Sanity Check

The centering sanity check shows that visual-feature anisotropy is substantial rather than a minor offset and persists across architectures and encoders. Dataset-mean centering nearly eliminates the measured global mean in the analyzed OCRVQA features.

  • Centering Sanity Check: The OCRVQA corpus mean has norm ||µ||2 = 50.8456, while dataset-mean centering reduces it to 1.8645 × 10^-6.The analysis uses N = 500 samples and D = 4096 dimensions.
  • Centering Sanity Check: The mean component accounts for 0.9721 of the origin second moment, supporting the dominant-global-component assumption.This directly supports the condition ||µ||2 ≫ E[||δi||2^2] used in the analysis.
  • Cross-Architecture Evidence: Representation anisotropy persists across Vicuna, Phi-2, Qwen, and Llama backbones with CLIP and SigLIP encoders.Llama-3 and Qwen show approximately 10% lower visual Mean(|x|) than Vicuna, but the visual-textual gap remains pronounced.

J Sensitivity Analysis of the Selection Ratio

PRISM remains robust across selection budgets: multimodal performance scales with retained data, while language capability remains high even under extreme pruning. Its efficient implementation uses re-centering and aggregate correlation scoring rather than expensive covariance inversion or pairwise enumeration.

  • Selection-Ratio Sensitivity: At the 30% selection ratio, PRISM-Instruct-250K surpasses the full-dataset baseline on multimodal performance.The analysis describes consistent multimodal scaling with the retained data budget.
  • Selection-Ratio Sensitivity: At a 5% selection ratio, textual performance remains remarkably high relative to the baseline.This suggests that PRISM identifies a compact data core while pruning substantial redundancy.
  • Computational Design: PRISM uses corpus-mean re-centering and Pearson-style normalized inner-product scoring at O(Nd) cost.Mahalanobis scoring would require O(Nd^2) covariance estimation and O(d^3) inversion.
  • Computational Design: PRISM computes exact aggregate average-correlation scores from a global normalized-feature mean without materializing an N × N matrix.This preserves the ranking induced by the complete pairwise objective while avoiding explicit pair enumeration.
  • Qualitative Selection Behavior: High-ranked examples emphasize fine-grained perception and complex contextual reasoning, whereas pruned examples are often trivial or semantically redundant.The contrast is illustrated by the “Asia street” and “Sunny weather” examples.
  • Qualitative Selection Behavior: Pruning trivial samples improves computational-budget use without sacrificing model performance.Such samples are described as offering saturated information and redundant gradients.

M.1 Wall-Clock Runtime Breakdown

PRISM’s aggregate scoring and streaming implementation sharply reduce end-to-end runtime compared with full fine-tuning and training-based selection. Its redundancy score identifies samples aligned with dominant corpus-level similarity regions, so selecting the low-correlation tail targets less duplicated semantics.

  • Runtime Comparison: TIVE requires 87 GPU hours for selection and 101 hours overall, making the pipeline 7.5% slower than full fine-tuning.Its selection phase consumes 92% of the baseline’s entire tuning time.
  • Runtime Comparison: PRISM completes selection and tuning in 29.5 GPU hours, a 71% reduction versus the 94-hour full-finetuning baseline.Selection itself takes 1.5 hours, including feature extraction and correlation computation.
  • Linear-Time Implementation: The global normalized-feature mean is a sufficient statistic that preserves complete-graph score rankings while avoiding explicit pairwise enumeration.Each sample’s score is obtained from its inner product with this aggregate statistic.
  • Linear-Time Implementation: PRISM’s score computation costs O(Nd) after feature extraction and avoids materializing the N × N correlation matrix.Chunked streaming uses O(Bd + d + N) active memory when scores are retained, or O(Bd + d) when streamed.
  • Redundancy-Score Interpretation: A high redundancy score denotes dense corpus-level coverage, while a low score indicates weak affinity to dominant redundant modes.Selecting the low-correlation tail therefore targets novelty, diversity, and reduced over-representation.
  • Redundancy-Score Interpretation: PRISM interprets low correlation within a curated instruction pool as low redundancy rather than arbitrary outlierness.This interpretation is tied to the assumption that the input image–instruction pairs are already valid.
Loading 2502.12119v5…