Source-linked AI summary
Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency
Jianjian Yin, Liulei Li, Tao Chen, Yi Chen, Yazhou Yao, Wenguan Wang
TL;DR
Existing visual foundation model pruning relies on rigid token alignment and single-dataset optimization, limiting robustness and task specificity. Cut-ViT uses gram-anchored subspace constraints with spectral entropy adaptation to produce task-specific subnetworks, achieving superior performance across six tasks and nine datasets with lower pruning costs.
Problem
Existing pruning methods suffer robustness degradation from rigid point-to-point token alignment and task-specificity deficiency from optimizing on a single generic dataset.
Method
Cut-ViT decomposes spatial and channel gram matrices into subspace bases, applies basis-agnostic and residual consistency constraints, and adapts pruning through spectral entropy.
Results
Cut-ViT achieves superior performance across six tasks and nine datasets, using 20.9% of prior pruning time and 45.5% of prior GPU memory.
Takeaways & Limitations
Task-specific gram-subspace pruning can preserve robust DINOv3 representations while generating subnetworks efficiently for downstream applications.
Takeaways & Limitations
The native DINOv3 teacher is frozen, with its basis fixed during optimization while the pruned network basis may rotate.
Abstract
from arXiv · showhide
Pruning visual foundation models has attracted considerable attention. However, existing methods focus on rigid point-to-point token alignment on a single dataset for pruning, suffering from two limitations: i) robustness degradation, and ii) task-specificity deficiency. To address these limitations, we propose a task-specific pruning pipeline, named Cut-ViT. Specifically, we first construct gram anchoring matrices from both spatial and semantic perspectives, and perform the subspace decomposition to extract the corresponding subspace bases. Basis-agnostic and residual constraints are then adopted to align the gram subspaces between the native and pruned DINOv3 models along spatial and channel dimensions, enabling subnetworks to inherit robust feature representations of native DINOv3. Furthermore, we design spectral entropy adaptation, which quantifies the information density of feature manifolds along spatial and channel dimensions, thereby adapting the pruning objective to specific downstream tasks. Experiments show that Cut-ViT requires approximately one minute on a single A100 GPU to obtain subnetworks at various sparsity levels, using only 20.9% of the time and 45.5% of the GPU memory compared with previous methods, while achieving SOTA performance on six tasks across nine datasets.
1 Introduction
Cut-ViT addresses robustness degradation and task-specificity deficiency in training-free pruning by aligning gram-anchored subspaces and adapting pruning to target tasks. It achieves stronger downstream performance with substantially lower pruning-resource requirements.
- Motivation: Existing one-shot pruning methods use rigid token alignment, which can degrade representations by prioritizing exact features over global semantic topology.They also commonly optimize on one generic dataset, producing universal subnetworks that compromise precision for specific downstream applications.
- Method: Cut-ViT decomposes spatial and channel gram matrices into orthonormal bases to capture joint variation and reduce sensitivity to high-frequency noise.Basis-agnostic and residual constraints then align the pruned model’s feature subspace with the native DINOv3 teacher.
- Method: Cut-ViT performs optimization directly on the target dataset, propagating its data distribution through gram anchoring and subspace consistency.This task-specific pipeline is designed to adapt pruning objectives to downstream task properties.
- Results: Cut-ViT outperforms existing training-free pruning methods across six tasks and nine datasets, while out-of-distribution experiments preserve native DINOv3 robustness.Representative gains include 3.4% in (J &F)m on DAVIS-2017, 5.1% in PCK@0.05 on FG3DCar, and 2.8% in mAP on MS COCO.
2 Related Work
Related work positions Cut-ViT within visual foundation model compression and distinguishes it from prior static subspace-pruning approaches. Its stated novelty is dynamic, rotation-invariant subspace alignment during pruning.
- Visual Foundation Model: Visual foundation models learn transferable visual representations but impose substantial computational overhead that complicates deployment on edge devices.Examples include CLIP, SAM, and DINO.
- Model Compression: Model compression research includes quantization, token optimization, and structured pruning as major approaches for reducing computational cost.Quantization reduces weight precision, while token optimization removes or merges redundant tokens.
- Novelty: Cut-ViT differs from prior SVD-based methods by dynamically optimizing rotation-invariant teacher–pruned-model subspace alignment during pruning.This design is presented as ensuring inheritance of robust topological structures.
3 Methodology
Cut-ViT formulates pruning as consistency between native and pruned DINOv3 feature subspaces, using spatial and channel Gram matrices rather than rigid point-wise alignment. Basis-agnostic and residual constraints preserve dominant structure while spectral entropy weights adapt the objective to representational demands.
- Preliminary: Training-free OSP identifies a binary mask for pretrained VFM weights by minimizing the performance gap between pruned and native models.Parameter importance is approximated with an empirical Fisher matrix over calibration samples, and search algorithms locate the subnetwork.
- Gram Anchoring Subspace Decomposition: Gram matrices model second-order feature correlations, and SVD extracts orthonormal bases spanning their dominant subspaces.The feature matrix is unfolded along the target dimension before Gram construction and decomposition.
- Gram Anchoring Subspace Decomposition: Spatial Gram Anchoring captures inter-token topology, whereas Channel Gram Anchoring captures semantic attributes such as texture, style, and class.The spatial matrix is F·F^T in R^{L×L}; the channel matrix is F^T·F in R^{D×D}.
- Basis-Invariant Subspace Consistency: Basis-agnostic consistency aligns teacher and pruned subspaces independently of basis orientation because SVD bases are not unique.The loss maximizes subspace overlap through the Frobenius norm of the affinity matrix and is invariant to orthogonal basis transformations.
- Basis-Invariant Subspace Consistency: The residual constraint projects pruned features onto the native subspace and minimizes energy orthogonal to that subspace.This suppresses redundant information and noise outside the target manifold while basis consistency aligns principal directions.
- Task-specific Pruning via Spectral Entropy Adaptation: Spectral entropy adaptation weights spatial and channel basis losses to match downstream tasks’ representational demands.The weights are derived from normalized singular-value information densities of native spatial and channel Gram matrices.
4 Experiments
Cut-ViT is evaluated across six visual tasks, nine datasets, and sparsity levels from 10% to 30%, with comparisons covering accuracy, generalization, efficiency, ablations, and architecture robustness. It consistently improves over training-free pruning while retaining competitive performance against training-based methods and substantially reducing pruning cost.
- Evaluation Setup: Cut-ViT is evaluated across six downstream tasks and nine datasets at sparsity levels ranging from 10% to 30%.The evaluation covers semantic segmentation, object detection, video object segmentation, semantic matching, depth estimation, and image classification.
- Comparison with SOTA Methods: At 20% sparsity, Cut-ViT improves DAVIS-2017 video object segmentation by over 4.2%, 4.4%, and 4.0% on (J &F)m, Jm, and Fm, respectively.It also outperforms HydraViT while slightly trailing EA-ViT.
- Comparison with SOTA Methods: At 30% sparsity, Cut-ViT improves mAP by 2.8% over training-free object-detection approaches on COCO while trailing EA-ViT by 0.4%.For semantic matching, it improves over training-free methods by 5.1%, 12.1%, and 5.5% on FG3DCar, JODS, and PASCAL, respectively.
- Out of Distribution Generalization: Cut-ViT consistently outperforms training-free pruning on out-of-distribution PASCAL VOC 2012 inference, gaining 1.5% mIoU at 20% sparsity.The models are trained on ADE20K, providing a cross-dataset generalization test.
- Complexity Analysis: 20.9% of SnapViT’s pruning time and 45.5% of its GPU memory are required by Cut-ViT, alongside a 1.7% mIoU improvement.Against EA-ViT, Cut-ViT trails by 0.9% in performance while using 0.44% of the time and 28.2% of the GPU memory.
- Ablation Study: Ablations show that basis-agnostic constraints, residual constraints, task-specific adaptation, and spatial and channel gram matrices complement one another.Spectral entropy weighting is reported as superior to static weighting on object detection, video object segmentation, and semantic matching; the method also generalizes across SAM, DeiT, and CLIP.
5 Conclusion
Cut-ViT addresses robustness degradation and task-specificity deficiency by enforcing spatial and channel subspace consistency and adapting pruning to downstream tasks. Across six tasks and nine datasets, it achieves state-of-the-art performance with minimal computational overhead.
- Cut-ViT constructs spatial and channel gram anchoring matrices and decomposes them to obtain latent-space subspace bases.
- Basis-agnostic and residual constraints enforce consistency between original and pruned DINOv3 subspaces across spatial and channel domains.
- Spectral entropy adaptation quantifies feature-manifold information density to tailor pruning objectives to downstream tasks.
- Cut-ViT achieves state-of-the-art performance across six tasks and nine datasets with minimal computational overhead compared with prior approaches.