Source-linked AI summary
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun
TL;DR
Vision transformers incur high cost from dense token interactions, while existing pruning can disrupt spatial structure and typically depends on pre-trained models. Evo-ViT dynamically selects informative and placeholder tokens using global class attention and updates them through slow-fast paths while preserving structure and information flow. On ImageNet classification, it improves DeiT inference throughput by 40%-60% and further accelerates LeViT with comparable performance.
Problem
Dense token modeling is costly, while existing unstructured pruning has incomplete spatial structure and cannot train from scratch.
Method
Evo-ViT uses evolved global class attention for instance-wise token selection and slow-fast updating that preserves all tokens while assigning informative and placeholder tokens different computation paths.
Results
Evo-ViT improves DeiT inference throughput by 40%-60% and further accelerates LeViT while maintaining comparable ImageNet classification performance.
Takeaways & Limitations
Preserving placeholder tokens lets Evo-ViT accelerate both flat and deep-narrow transformers from the beginning of training while retaining spatial structure and information flow.
Takeaways & Limitations
The paper identifies extending Evo-ViT to downstream tasks such as object detection and instance segmentation as future work.
Abstract
from arXiv · showhide
Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) the incomplete spatial structure caused by pruning is not compatible with structured spatial compression that is commonly used in modern deep-narrow transformers; 2) it usually requires a time-consuming pre-training procedure. To tackle the limitations and expand the applicable scenario of token pruning, we present Evo-ViT, a self-motivated slow-fast token evolution approach for vision transformers. Specifically, we conduct unstructured instance-wise token selection by taking advantage of the simple and effective global class attention that is native to vision transformers. Then, we propose to update the selected informative tokens and uninformative tokens with different computation paths, namely, slow-fast updating. Since slow-fast updating mechanism maintains the spatial structure and information flow, Evo-ViT can accelerate vanilla transformers of both flat and deep-narrow structures from the very beginning of the training process. Experimental results demonstrate that our method significantly reduces the computational cost of vision transformers while maintaining comparable performance on image classification.
Introduction
Vision transformers are computationally inefficient because dense long-range token modeling processes uninformative image regions. Evo-ViT preserves spatial structure while dynamically assigning informative and placeholder tokens different update paths, accelerating both flat and deep-narrow transformers from training start.
- Dense long-range dependency modeling is inefficient because images contain large low-level-texture and uninformative-background regions.
- Existing structured compression reduces local spatial redundancy, whereas unstructured pruning can lose spatial structure and cannot generally train from scratch.
- Evo-ViT dynamically distinguishes informative tokens from placeholder tokens while preserving all tokens, spatial structure, and information flow.The method is intended as a generic plugin for flat and deep-narrow ViTs from the beginning of training.
- Evo-ViT uses class-token attention to identify token importance, then updates informative tokens slowly through the full encoder and placeholder tokens quickly through a representative token.
- 40%-60% higher inference throughput is reported for DeiT, while LeViT is further accelerated with comparable performance on ImageNet classification.
Related Work
Prior transformer-efficiency work includes sparse attention, distillation, and pruning, with token pruning chiefly reducing computation in pre-trained flat-structure vision transformers.
- Transformer acceleration methods are categorized into sparse attention, knowledge distillation, and pruning because self-attention and feed-forward networks are computationally costly.
- Patch slimming performs layer-by-layer token reduction in pre-trained vision transformers while controlling feature reconstruction error.
- A lightweight prediction module estimates token importance from current features to guide pruning in pre-trained transformers.
Preliminaries
ViT represents images as embedded patch tokens plus a class token, processes them through stacked MSA-FFN encoders, and uses class attention to summarize patch information for classification.
- ViT reshapes an image into flattened patches, linearly projects them into embeddings, adds a CLS token, and applies positional embeddings to the sequence.
- Each transformer encoder applies multi-head self-attention and a feed-forward network with residual connections and layer normalization.
- The CLS token interacts with patch tokens at every encoder and summarizes them into a global representation for final classification.
- ViT's MSA and FFN costs are O(4NC^2 + 2N^2C) and O(8NC^2), respectively, making token reduction computationally valuable.
Methodology
Evo-ViT selects informative and placeholder tokens with evolved global class attention, then updates them through different computation paths while preserving token structure and information flow. Its training strategy progressively exploits increasingly stable token selection, with the CLS token used only for selection during inference.
- Structure preserving token selection: Evo-ViT divides patch tokens into informative tokens and placeholder tokens using evolved global class attention.The top-k class-attention scores identify informative tokens; the remaining tokens are retained as placeholders rather than dropped.
- Motivation for structure preservation: Figure 4 compares CKA similarity to the final CLS token and Pearson correlation of token features across layers to illustrate shallow-layer pruning difficulty.Panel (a) shows CKA similarity between the final CLS token and token features; panel (b) shows Pearson correlation among token features.
- Structure preserving token selection: Global class attention evolves class attention across layers through residual connections, supporting token selection in the next layer.Only the global class-attention scores of selected informative tokens are updated in layers with token selection.
- Slow-fast token updating: Informative tokens and a representative placeholder token undergo the full MSA and FFN computation, while placeholder tokens are aggregated before updating.The placeholder tokens are aggregated into one representative token using an aggregation function such as a weighted sum.
- Slow-fast token updating: Placeholder tokens are fast-updated from the representative token’s residuals, which serve as a skip connection and keep slow and fast outputs similarly scaled.The expansion function maps the representative residual back to the placeholder-token shape; simple copying is used in the method.
- Training Strategies: A layer-to-stage training schedule exploits increasingly stable token-selection results across layers, while inference uses the CLS token only for token selection.Final average-pooled features are used for classification during inference.
Experiments
Experiments on DeiT and LeViT evaluate Evo-ViT across flat and deep-narrow transformers, comparing accuracy, throughput, module contributions, and token-selection behavior. The method improves efficiency while preserving competitive classification performance, with structure preservation and global attention supporting selection quality.
- Experimental setup: Experiments cover flat DeiT and deep-narrow LeViT transformers on ImageNet-1k classification.The evaluation includes overall comparisons, pruning comparisons, and module ablations.
- Main results: Evo-ViT further accelerates LeViT, although its accuracy degradation is larger because deeper layers contain fewer, less redundant tokens.With 384×384 inputs, LeViT achieves less accuracy degradation and a higher acceleration ratio.
- Ablation analysis: Structure preservation improves selection performance by maintaining complete information flow, while global class attention improves selection consistency across layers.Fast updating contributes less on DeiT than on LeViT, where placeholder tokens provide additional cues.
- Ablation analysis: The evolved global class attention outperforms alternative selection and sub-sampling strategies on both accuracy and efficiency.Restricting all layers to object-focused tokens can degrade performance, possibly because background information assists classification.
- Ablation analysis: A consistent token-keeping ratio across layers achieves the best performance, while selection becomes more stable as training progresses.Early epochs can misclassify informative regions, whereas later training increasingly stabilizes discriminative selection.
Conclusions
Evo-ViT addresses token-pruning limitations with structure-preserving selection and slow-fast updating. Experiments on DeiT and LeViT show significant acceleration with comparable classification performance, while downstream-task extension remains future work.
- Evo-ViT combines structure-preserving token selection with slow-fast updating to address token-pruning limitations in structured transformers and scratch training.
- Experiments on DeiT and LeViT indicate significant acceleration while maintaining comparable classification performance.
- Extending Evo-ViT to object detection and instance segmentation is identified as future work.