Source-linked AI summary
MiniViT: Compressing Vision Transformers with Weight Multiplexing
Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, Lu Yuan
TL;DR
Large vision transformers have parameter counts that restrict deployment on resource-limited devices. MiniViT compresses them by multiplexing transformed shared weights and applying weight distillation, achieving strong accuracy and downstream transfer. The paper reports substantial compression with accuracy improvements over original or baseline models, while noting increased computation and degradation at higher compression ratios.
Problem
Large vision transformers consume substantial storage and memory, motivating parameter reduction without compromising performance.
Method
MiniViT combines cross-layer weight sharing, weight transformations, and attention- and hidden-state distillation to compress pretrained vision transformers.
Results
MiniViT compresses pretrained vision transformers by 2 times while improving accuracy over baselines and transferring well to downstream tasks.
Takeaways & Limitations
MiniViT achieves large parameter reductions with accuracy improvements and downstream performance comparable to original models.
Takeaways & Limitations
MiniViT slightly increases computational cost because of weight-transformation blocks and experiences moderate performance degradation as compression increases.
Abstract
from arXiv · showhide
Vision Transformer (ViT) models have recently drawn much attention in computer vision due to their high model capability. However, ViT models suffer from huge number of parameters, restricting their applicability on devices with limited memory. To alleviate this problem, we propose MiniViT, a new compression framework, which achieves parameter reduction in vision transformers while retaining the same performance. The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks. More specifically, we make the weights shared across layers, while imposing a transformation on the weights to increase diversity. Weight distillation over self-attention is also applied to transfer knowledge from large-scale ViT models to weight-multiplexed compact models. Comprehensive experiments demonstrate the efficacy of MiniViT, showing that it can reduce the size of the pre-trained Swin-B transformer by 48\%, while achieving an increase of 1.0\% in Top-1 accuracy on ImageNet. Moreover, using a single-layer of parameters, MiniViT is able to compress DeiT-B by 9.7 times from 86M to 9M parameters, without seriously compromising the performance. Finally, we verify the transferability of MiniViT by reporting its performance on downstream benchmarks. Code and models are available at here.
1. Introduction
Vision transformers offer strong performance but their large parameter counts limit deployment on resource-constrained devices. MiniViT addresses this through weight multiplexing, combining shared weights with transformations and distillation to preserve diversity, stability, accuracy, and downstream transfer.
- Motivation: Large vision transformers require substantial storage and memory, limiting use on edge, IoT, and real-time applications.The paper motivates removing redundant parameters and computational overhead without compromising performance.
- Challenges: Vision-transformer weight sharing is underexplored and can cause unstable training, collapse with more shared layers, and substantial accuracy loss.On Swin-S, fourfold parameter reduction produced a 5.6% accuracy degradation.
- Method: MiniViT multiplexes weights across layers using transformations for the MSA and MLP modules, with separate layer-normalization parameters.These transformations promote parameter diversity and improve training stability.
- Method: Weight distillation transfers information from pretrained models into compact weight-shared models through attention-level and hidden-state distillation.This extends beyond prediction-level distillation by matching internal behavior of the large teacher model.
- Results: MiniViT improves accuracy over baselines, compresses pretrained vision transformers by 2 times, and transfers well to downstream tasks.Mini-Swin-B with 12-layer parameters is 0.8% more accurate than 24-layer Swin-B; Mini-DeiT-B reaches 79.8% Top-1 accuracy with 9M parameters versus DeiT-B's 86M and 81.8%.
2. Background
Vision transformers convert image patches into token sequences and process them with alternating self-attention and MLP blocks. Weight sharing improves parameter efficiency by reusing one transformer block across layers, while standard attention and MLP operations define each block’s computations.
- Vision Transformers: Vision transformers split images into 2D patches, project them into token embeddings, add positional information, and classify the encoded sequence.
- Vision Transformers: A transformer encoder alternates multi-head self-attention and MLP blocks, with layer normalization and residual connections around each block.
- Vision Transformers: Self-attention computes pairwise token affinities from queries and keys, then uses them to form weighted sums of values.
- Vision Transformers: An MLP applies two fully connected layers with an activation function, using an intermediate dimension d′ typically larger than d.
- Weight Sharing: Weight sharing reuses one transformer block and its parameters across all layers, preventing parameter growth with network depth.
3. Method
MiniViT compresses vision transformers by sharing weights across layers while adding lightweight transformations that preserve layer diversity, then uses multi-level distillation to transfer teacher behavior.
- Weight Multiplexing: Weight multiplexing combines shared transformer weights with layer-specific transformations and distillation to improve stability and compressed-model performance.The framework shares weights across multiple blocks, transforms them per layer, and transfers knowledge from pretrained models.
- Weight Transformation: The framework shares MSA and MLP weights within each stage while leaving transformation kernels and normalization layers unshared.Stages are configurable, and compressed layers within each stage must have identical structures and dimensions.
- Weight Transformation: Two linear transformations around self-attention diversify attention matrices and combine information across attention heads.The transformation kernels before and after softmax are layer-specific.
- Weight Transformation: Lightweight depth-wise-convolution transformations diversify MLP outputs using K^2d parameters instead of N^2d parameters, where K << N.The convolutions sparsify and share weights within transformation matrices while introducing locality.
- Weight Distillation: Weight distillation transfers teacher knowledge through prediction-logit, self-attention, and hidden-state objectives.Attention distillation matches relations among queries, keys, and values, while hidden-state distillation matches relation matrices of MLP features.
- Weight Distillation: The final distillation objective combines prediction, attention, and hidden-state losses as L_train = L_pred + βL_attn + γL_hddn.The default weights are β = 1 and γ = 0.1.
4. Experiments
Experiments show that MiniViT’s weight transformation and distillation stabilize and improve weight-shared vision transformers while substantially reducing parameters. Across ImageNet, ablations, efficiency comparisons, and downstream transfer, the method generally preserves or improves accuracy with configurable compression.
- Analysis and Ablation: Weight sharing alone can collapse training and reduce accuracy, whereas weight transformation improves parameter diversity and training stability.The analysis links identical weights across layers to unstable gradients and divergent feature representations.
- Analysis and Ablation: Weight distillation and weight transformation each improve weight-sharing performance, while combining them yields the best accuracy and can outperform the original models.The component analysis evaluates weight sharing, distillation, transformation, and their combination on ImageNet-1K.
- Analysis and Ablation: Sharing every two blocks reduces DeiT-B parameters from 86M to 44M while increasing Top-1 accuracy by 1%.The same setting reduces Swin-T parameters from 28M to 16M with a 1% Top-1 accuracy increase.
- Results on ImageNet: Using 46M parameters, Mini-Swin-B achieves 0.8% higher accuracy than Swin-B on ImageNet-1K, while Mini-DeiT-B achieves a 1.8% improvement with half the parameters.MiniViTs also improve performance on ImageNet-Real and ImageNet-V2 relative to the corresponding original models.
- Results on ImageNet: Mini-DeiT-B achieves 1.0% and 2.5% higher Top-1 accuracy than S2ViTE-B and VTP, respectively, using 44M parameters.Mini-DeiT-B also exceeds AutoFormer-B by 0.8% accuracy with fewer parameters, while Mini-DeiT-Ti remains comparable to AutoFormer-T at 3M parameters.
- Downstream Transfer: On COCO detection, weight sharing causes a 1.6 AP decrease, providing evidence that compression effects extend beyond classification.The supplied passage introduces the downstream detection comparison but does not report the complete MiniViT outcome.
5. Related Work
Related work situates MiniViT among vision-transformer compression, weight-sharing, and knowledge-distillation methods. Existing approaches reduce parameters through pruning or sharing, while distillation transfers information from larger teachers.
- Vision Transformer Compression: Vision-transformer compression methods prune redundant tokens, attention heads, or hidden dimensions, and some combine pruning, skipping, and distillation.The related literature focuses mainly on isomorphic vision transformers.
- Weight Sharing: Weight sharing improves parameter efficiency by preventing parameter counts from growing with network depth.ALBERT is identified as a representative cross-layer parameter-sharing method in NLP.
- Weight Sharing: Universal transformers share weights across positions and time steps, while Deep Equilibrium Models use equilibrium states after weight-shared outputs converge to a fixed point.These methods apply weight sharing to sequence-to-sequence or transformer settings in language tasks.
- Knowledge Distillation: Knowledge distillation uses teacher-student training to reduce model size, but its application to vision transformers remains under-explored.Prior examples include distillation tokens and methods that extract knowledge from teacher transformers.
6. Conclusion
The paper concludes that MiniViT combines weight sharing, transformation, and distillation to reduce vision-transformer parameters while maintaining or improving performance. It also identifies computational overhead and increasing degradation at higher compression ratios as limitations.
- Conclusion: MiniViT combines weight sharing, transformation, and distillation to reduce parameters while achieving better performance than the original models.The framework is presented as a general compression method for vision transformers.
- Limitations: Weight transformation slightly increases computational cost compared with classical weight sharing.The added transformation blocks improve parameter efficiency but introduce this computational overhead.
- Limitations: MiniViT suffers moderate performance degradation as the compression ratio increases.The authors identify improving both parameter and computational efficiency as future work.