Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

1 Introduction

• Visual foundation models, including the DINO series [49], offer strong downstream representations but impose computational overhead that complicates deployment on resource-constrained edge devices.The introduction identifies efficient compression without compromising performance as a research focus.Source: §1 · p. 1
• Existing one-shot pruning methods risk robustness degradation through rigid token matching and task-specificity deficiency through generic-dataset pruning for universal subnetworks.The paper argues that exact token matching can lose global semantic topology, while task-agnostic pruning further compromises precision when model capacity is already reduced for edge deployment.Source: §1 · p. 2
• Cut-ViT decomposes spatial and channel Gram matrices into orthonormal bases, then uses basis-agnostic and residual constraints to align subnetworks with DINOv3’s robust feature subspaces [49].This subspace consistency approach aims to preserve the teacher’s spatial and semantic manifold without overfitting to specific token values.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Cut-ViT optimizes on each target dataset and uses spatial and channel spectral entropy to adapt pruning objectives to data distributions and task-specific information density.Target-distribution information propagates through subspace consistency without ground-truth annotations; entropy accounts for different information needs, such as global semantics in classification and high spatial frequency in segmentation.Source: §1 · p. 3
• Across nine datasets and six downstream tasks, Cut-ViT outperforms existing training-free OSP methods, using 20.9% of inference time and 45.5% of GPU memory.It improves (J &F)m by 3.4% on DAVIS-2017 video object segmentation, PCK@0.05 by 5.1% on FG3DCar semantic matching, and mAP by 2.8% on MS COCO object detection versus prior state-of-the-art OSP approaches. Against training-based pruning, it remains competitive while using 0.44% of pruning time and 28.2% of GPU memory.Source: §1 · p. 3

Visual Foundation Model

• Cut-ViT directly selects VFM encoder parameters to generate subnetworks at varying sparsities in one inference pass, unlike lightweighting methods based on knowledge distillation.VFMs such as CLIP, SAM, and DINO [49] learn transferable representations but have computational overhead that challenges edge deployment.Source: §2 · p. 3

Model Compression

• Quantization, token optimization [2] [70], and structured pruning are major compression directions, but cited structured-pruning methods require intensive training.HydraViT, Scala, and EA-ViT use training-based approaches to produce or select subnetworks.Source: §2 · p. 4

One-shot Pruning without Retraining

• Prior one-shot pruning methods, including SNOWS and SnapViT, use rigid point-to-point token alignment, while Cut-ViT aligns Gram anchors in subspace to preserve native DINOv3 representations.One-shot structured pruning generates networks at varying sparsities in a single inference pass; LAMP and SNIP use layer-adaptive and magnitude-based criteria, respectively.Source: §2 · p. 4

Subspace Alignment

• Unlike prior SVD-based work relying on static subspace construction, Cut-ViT uses rotation-invariant dynamic optimization to align teacher and pruned-model subspaces during pruning.The approach is intended to ensure subnetworks inherit robust topological structures.Source: §2 · p. 4

3.1 Preliminary

• Standard training-free pruning estimates parameter importance with an empirical Fisher matrix over calibration samples, then uses search algorithms to select a subnetwork minimizing the student–teacher performance gap.The traditional distillation objective compares student local-crop features with teacher global-crop features using rigid element-wise losses such as MSE or KL divergence.Source: §3.1 · pp. 4–5

3.2 Gram Anchoring Subspace Decomposition

• SVD of feature Gram matrices extracts spatial bases for inter-token topology and channel bases for global semantic attributes, replacing pointwise matching with subspace consistency.The spatial Gram matrix is F F^⊤, while the channel Gram matrix is F^⊤ F.Source: §3.2 · p. 6

3.3 Basis-Invariant Subspace Consistency

• Basis-agnostic loss aligns the pruned and native top-K subspaces through their overlap, remaining invariant to arbitrary orthogonal changes in basis orientation.The native DINOv3 teacher is frozen, while the pruned model’s basis can rotate during optimization.Source: §3.3 · pp. 6–7
• A residual constraint penalizes pruned features orthogonal to the native teacher subspace, suppressing out-of-manifold noise beyond aligning principal directions.It projects features onto the teacher subspace and minimizes the energy of the orthogonal residual.Source: §3.3 · pp. 7–8

3.4 Task-specific Pruning via Spectral Entropy Adaptation

• 61s v.s. 292s of SnapViT: Cut-ViT prunes using a minimal, unlabeled subset of the target dataset to capture its domain manifold.The paper attributes its lower latency to an efficient design that supports multiple searches.Source: §3.4 · p. 8
• Spectral entropy weights spatial and channel basis losses by their relative information density, adapting pruning to task demands such as global semantics or dense spatial detail.Classification tends toward lower spatial entropy, while dense prediction motivates higher spatial entropy to preserve boundaries and pixel-level semantics.Source: §3.4 · pp. 8–9

4.1 Comparison with SOTA Methods

• At 20% sparsity, video-segmentation gains exceed 4.2%, 4.4%, and 4.0% on (J &F)m, Jm, and Fm; at 30%, semantic-matching gains over training-free methods are 5.1%, 12.1%, and 5.5% on FG3DCar, JODS, and PASCAL.For semantic matching, Cut-ViT also exceeds HydraViT by 0.9%, 0.6%, and 0.3% on those datasets; for video segmentation, it outperforms HydraViT but slightly trails EA-ViT.Source: §4.1 · pp. 10–11, Table 1 · p. 10, Table 2 · p. 10
• At 30% sparsity, Cut-ViT improves COCO mAP by 2.8% over training-free methods; on ADE20K, its mIoU gains are 1.2%, 1.7%, and 0.8% at 30%, 20%, and 10% sparsity.On NYUv2, it outperforms training-free methods in both ARel and δ1 and yields models closer to native DINOv3 when pruning the same number of parameters; it is competitive with training-based methods on depth estimation.Source: §4.1 · p. 11, Table 3 · p. 11, Table 4 · p. 11
• On ImageNet at 10%, 20%, and 30% sparsity, Cut-ViT gains 1.8%, 1.2%, and 0.8% over training-free methods, even without gram anchoring.It achieves competitive performance relative to training-based pruning methods, relying on basis-agnostic and residual constraints applied to CLS features.Source: §4.1 · pp. 11–12, Table 7 · p. 12

4.2 Out of Distribution Generalization

• When models trained on ADE20K are evaluated on PASCAL VOC 2012, Cut-ViT outperforms training-free pruning across sparsity levels, with a 1.5% mIoU gain at 20% sparsity.The evaluation tests whether subnetworks retain the native DINOv3 model’s generalization ability on a different dataset.Source: §4.2 · p. 12, Table 6 · p. 12

4.3 Ablation Study

• Cut-ViT uses 20.9% of SnapViT’s pruning time and 45.5% of its GPU memory while delivering a 1.7% mIoU improvement; versus EA-ViT, it uses 0.44% of the time and 28.2% of the memory, with performance 0.9% lower.The reported comparison indicates reduced pruning overhead alongside competitive performance.Source: §4.3 · pp. 12–13, Table 8 · p. 12
• Ablations show that basis-agnostic, residual, and task-specific constraints complement one another, while spatial and channel gram anchoring align spatial context and channel-wise semantics with DINOv3.The two gram matrices together support capturing both semantic and spatial contextual information.Source: §4.3 · p. 13, Table 9 · p. 13, Table 10 · p. 13
• The analysis identifies K = 192 as an optimal trade-off among information preservation, SVD efficiency, and cross-task performance; spectral entropy weighting is superior on object detection, video segmentation, and semantic matching.Information preservation is assessed with cumulative explained variance ratio (CEVR), and efficiency by the time to obtain principal components from a matrix via SVD.Source: §4.3 · pp. 13–14, Table 11 · p. 13, Table 12 · p. 13
• Pruning on target-task datasets yields only marginal gains for SNOWS and SnapViT, while evaluations on SAM, DeiT, and CLIP show that Cut-ViT generalizes across architectures.Those architecture evaluations cover prompt segmentation, video object segmentation, and open-vocabulary segmentation using 0.1% of SA-1B, DAVIS-2017, and PASCAL-Context, respectively.Source: §4.3 · p. 14, Table 13 · p. 14, Table 14 · p. 14

• Cut-ViT achieves state-of-the-art performance with minimal computational overhead across six tasks and nine datasets.Source: §5 · p. 15
• Gram anchoring matrices and basis-agnostic and residual constraints align spatial and channel subspaces, helping pruned DINOv3 inherit robust foundation-model representations.Source: §5 · p. 15
• Spectral entropy adaptation measures feature-manifold information density across spatial and channel dimensions to adapt pruning objectives to downstream tasks.Source: §5 · p. 15