Source-linked AI summary
Patch Slimming for Efficient Vision Transformers
Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, Dacheng Tao
TL;DR
Vision transformers deliver strong computer-vision performance but are computationally expensive, limiting deployment on resource-constrained devices. The paper introduces top-down patch slimming, which estimates each patch’s impact on final output features and removes lower-impact patches. Experiments show substantial computation reduction while largely preserving accuracy.
Problem
Vision transformers achieve high performance but their high computational cost hinders deployment on resource-limited devices.
Method
Patch slimming identifies effective patches in the last layer, then guides top-down pruning of earlier layers using estimated patch impact on the final output feature.
Results
Over 45% FLOPs can be reduced with less than 0.4% accuracy loss when FLOP reduction is below 45%.
Takeaways & Limitations
The method produces a compact, pyramid-like vision transformer and significantly reduces computational cost on benchmark datasets.
Takeaways & Limitations
The pruning scheme relies on Lipschitz continuity and preserves patches at corresponding spatial locations across layers to maintain information propagation.
Abstract
from arXiv · showhide
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a top-down paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impact will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.
1. Introduction
Vision transformers achieve strong performance but remain computationally expensive, motivating patch-level compression. Patch slimming removes redundant patches top-down while preserving information needed for final predictions.
- Vision transformers offer high performance across computer-vision tasks but impose high computational costs that hinder deployment on resource-limited devices.
- Existing compression methods mainly target weights, features, knowledge transfer, or CNN components, while few works address vision-transformer acceleration.
- Attention aggregates information across patches, so pruning must preserve inter-layer information flow in ways conventional CNN channel pruning cannot guarantee.
- Patch slimming identifies and removes redundant patches to reduce vision-transformer computation.
- The proposed top-down scheme starts at the last layer, estimates patch importance for the class token, and prunes lower-impact patches while controlling network error.
2. Related work
Prior structured-pruning research removes redundant CNN channels and transformer components such as attention heads or MLP neurons. These approaches motivate pruning, but operate on different structural elements than vision-transformer patches.
- Structure pruning for CNNs: CNN channel pruning accelerates inference by removing entire convolutional filters or kernels from pre-defined architectures.
- Structure pruning for CNNs: Filter redundancy has been identified using sparsity regularization, scaling factors, similarity, and task-related criteria.
- Structure pruning for transformers: Transformer pruning research reduces multi-head-attention heads and multilayer-perceptron neurons, including in NLP models.
3. Patch Slimming for Vision Transformer
Patch slimming represents patch preservation with binary masks and applies the resulting pruning to transformer blocks. Its top-down selection exploits increasing patch redundancy in deeper layers and reduces attention and MLP computation.
- Patch redundancy: Deeper layers exhibit greater patch redundancy, with average within-layer patch similarity exceeding 0.8 in deeper layers.
- Patch slimming formulation: Patch slimming uses a binary mask m_l to indicate which of N patches are preserved in each layer.
- Patch slimming formulation: A pruned transformer block applies attention and MLP operations to masked features, with effective patches selected for computation.
- Computation efficiency: The implementation avoids computation for pruned patches by calculating queries and subsequent operations only for effective patches.
- Computation efficiency: After pruning η% of patches, η% of MLP FLOPs are reduced and η%(2N^2d + 2Nd^2) MSA FLOPs are reduced.
4. Excavating Redundancy via Inverse Pruning
The method prunes vision-transformer patches from the output layer backward, preserving corresponding patches across layers and estimating each patch’s impact on effective final features.
- Top-Down Pruning: One-to-one spatial correspondence across transformer layers requires preserving important patches at the same locations through the network.Self-attention and shortcut connections maintain correspondence between patches in adjacent layers, unlike independently pruned CNN channels.
- Top-Down Pruning: Deeper transformer layers contain more redundant patches because attention repeatedly aggregates information across patches.Average patch similarity increases with depth and can exceed 0.8 in deeper ViT-Base layers.
- Top-Down Pruning: Top-down pruning starts at the output layer and transmits selected effective patches toward earlier layers.All patches preserved in layer l+1 are also preserved in layer l, yielding more patches in shallow layers than deep layers.
- Impact Estimation: For classification, the final-layer mask can retain only the class token while removing other output patches without affecting label prediction.With the class token as the first patch, the mask sets m_L,1 = 1 and m_L,i = 0 for i = 2, 3, · · ·, N.
- Impact Estimation: Patch importance is approximated by each patch’s impact on the effective final-layer output, using attention maps in later layers and current-layer input features.Larger significance scores indicate greater impact on final reconstruction error and are used to prioritize preservation.
- Pruning Procedure: The pruning procedure greedily preserves high-scoring patches until reconstruction error falls below the tolerance, then fine-tunes the reduced block and final architecture.Patches are selected in increments of r′, with fine-tuning after each selection step and additional fine-tuning after masks are fixed.
5. Experiments
Experiments on ImageNet evaluate patch slimming across multiple vision transformer families and ablate its pruning strategy. The method reduces computation while preserving accuracy, with learned top-down pruning outperforming uniform pruning and alternative importance scores.
- Experiments on ImageNet: Experiments evaluate patch slimming on DeiT, T2T-ViT, and LV-ViT models using ImageNet benchmarks and comparisons with pruning baselines.The study includes DeiT-Ti, DeiT-S, DeiT-B, T2T-ViT, and LV-ViT models.
- Experiments on ImageNet: 40.4% FLOPs reduction with a 0.4% accuracy decrease is achieved on T2T-ViT, outperforming the compared PoWER method.This result supports patch-level redundancy across different vision transformer architectures.
- Experiments on ImageNet: LV-ViT-M is reduced from 16.0G to 8.3G FLOPs while retaining 83.7% top-1 accuracy with dynamic patch slimming.The method also outperforms other state-of-the-art transformer models such as Swin Transformer in this comparison.
- Ablation Study: Less than 45% FLOPs reduction causes less than 0.4% accuracy loss on DeiT-S, indicating substantial patch redundancy.Increasing tolerant error permits more patches to be pruned but increases reconstructed feature error.
- Ablation Study: The learned architecture preserves more patches in shallow layers and prunes most patches in deep layers, forming a pyramid-like structure.Uniform pruning at a similar overall rate reaches only 77.2% accuracy, a 2.6% drop.
- Ablation Study: Deeper layers contain more safely removable patches, whereas pruning lower-layer patches produces larger accuracy drops.The paper attributes this depth pattern to progressive attention-based communication among patches.
- Ablation Study: Impact estimation loses less accuracy than random or attention-map-norm scores at the same pruning rate, including 50%.The score propagates reconstruction error from effective output-layer patches to approximate patch significance.
6. Conclusion
The paper accelerates vision transformers by pruning redundant patches in a top-down framework, using their impact on effective output features to guide selection. Experiments validate reduced computational cost and produce a compact pyramid-like transformer.
- The method reduces vision-transformer computation by pruning redundant patches rather than only targeting network channels.Patch slimming reduces the number of patches calculated in the model.
- Patch importance is approximated from each patch’s impact on effective output features.
- Pruning proceeds top-down across layers, producing a compact vision transformer with a pyramid-like architecture.
- Extensive benchmark experiments validate that patch slimming can effectively reduce computational cost.