PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, Pengtao Xie

arXiv:2202.07800v2cs.CVcs.LG

TL;DR

ViTs process redundant image-token computations because not all tokens are attentive or useful for prediction. EViT identifies attentive tokens using class-token attention and fuses inattentive tokens, yielding faster inference or higher-resolution inputs at comparable cost. Experiments report substantial speedups with small accuracy reductions and improved accuracy at similar throughput.

  • Problem

    ViTs process all image patches despite some tokens being semantically meaningless or distractive and not positively contributing to predictions.

  • Method

    EViT reorganizes tokens between MHSA and FFN by preserving class-token-attentive image tokens and fusing inattentive tokens into one token.

  • Results

    EViT improves the accuracy–efficiency trade-off across DeiT and LV-ViT, including 50% higher DeiT-S inference throughput with Top-1 accuracy reduction within 0.3%.

  • Takeaways & Limitations

    EViT supports either lower-cost inference with similar accuracy or higher-resolution inputs at comparable computational cost without adding parameters.

Abstract

from arXiv · show

Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at https://github.com/youweiliang/evit

1 INTRODUCTION

ViTs face substantial computational burdens, and not all image tokens contribute positively to predictions. EViT reorganizes tokens by preserving attentive ones and fusing inattentive ones, improving efficiency while retaining accuracy.

  • 1 INTRODUCTION: ViT computation becomes burdensome because models require large-scale training and many iterations for effective performance.These demands motivate expediting ViTs for practical use.
  • 1 INTRODUCTION: Image tokens differ in predictive importance: removing unrelated tokens can preserve predictions, whereas removing category-relevant tokens can cause errors.The observation motivates selectively retaining tokens rather than processing all tokens equally.
  • 1 INTRODUCTION: EViT identifies token attentiveness using class-token attention, preserves attentive tokens, and fuses inattentive tokens into one token.Fusion allows gradients to back-propagate through inattentive tokens and supports better attentive-token identification.
  • 1 INTRODUCTION: 50% faster DeiT-S inference accompanies only a 0.3% recognition-accuracy sacrifice on ImageNet with the same input-token count.The method is applied to representative DeiT and LV-ViT models without introducing additional parameters.
  • 1 INTRODUCTION: At the same computational cost, EViT can use more image tokens from higher-resolution inputs to improve recognition performance.This provides a second efficiency–accuracy operating point beyond reducing computation for a fixed input-token count.

2 RELATED WORK

Related work addresses neural-network redundancy and Transformer acceleration through adaptive computation, efficient attention, compressed architectures, and token reduction. These approaches primarily reduce computation through architectural changes or fewer tokens.

  • 2 RELATED WORK: Vision Transformers extend Transformer-based modeling across image classification, detection, segmentation, retrieval, and video understanding.ViT follow-ups modify the architecture for broader high-level vision tasks.
  • 2 RELATED WORK: Adaptive computation and parameter-reduction methods aim to reduce the storage and computational overhead of overparameterized neural networks.The motivation includes deploying deep networks on mobile devices.
  • 2 RELATED WORK: Transformer acceleration methods include more efficient attention mechanisms and compressed Transformer structures.These methods mainly reduce network complexity through artificially designed modules.
  • 2 RELATED WORK: Another ViT acceleration direction reduces the number of image tokens processed by the model.This direction is distinct from changing the attention mechanism or compressing the Transformer structure.

3 TOKEN REORGANIZATIONS

EViT reorganizes ViT image tokens by identifying attentive tokens through class-token attention, preserving top-k tokens, and fusing the rest to reduce computation while retaining information. The method supports token reduction across deeper layers and higher-resolution inputs at comparable computational cost.

  • ViT Overview: The ViT [CLS] token aggregates global image information, and its attention values indicate each token’s contribution to the [CLS] output.The first attention-map row represents attention from [CLS] to all tokens.
  • Attentive Token Identification: Directly removing tokens with the smallest class-token attention can severely deteriorate ImageNet classification accuracy in trained DeiT-S.Table 1 evaluates removal at the 4th, 7th, and 10th layers.
  • Attentive Token Identification: EViT preserves the k tokens with the largest average attention across heads and fuses the remaining tokens into one token.The token keeping rate is κ = k/n, and the fused token is appended to the attentive tokens for subsequent layers.
  • Inattentive Token Fusion: Weighted inattentive-token fusion supplements attentive tokens because background and large-object tokens may still contribute to predictions.The fusion operation has negligible cost compared with ViT’s main computation.
  • Higher-Resolution Inputs: EViT can process more tokens from higher-resolution images while maintaining computational cost comparable to a vanilla ViT.The supplied passages state that Table 5 validates this approach.

4 EXPERIMENTS

Experiments show that EViT accelerates DeiT and LV-ViT while retaining competitive recognition accuracy, and can use higher-resolution inputs at comparable computation. Ablations support inattentive-token fusion, longer training, and oracle guidance as beneficial design factors.

  • Main results: 50% higher inference throughput with DeiT-S at keep rate 0.7, while Top-1 accuracy reduction stays within 0.3% on ImageNet.The result is reported for EViT-DeiT-S trained with a 0.7 keep rate.
  • Ablation studies: EViT generally outperforms versions without inattentive token fusion, indicating that fusion preserves useful information from non-topk tokens.The reported improvement is typically small but consistent in the comparison.
  • Ablation studies: Longer training continues to improve DeiT-S performance in the efficient computation regime.The comparison trains EViT-DeiT-S with a 0.7 keep rate for 450 and 600 epochs.
  • Higher-resolution inputs: 84.7% ImageNet top-1 accuracy is achieved by EViT-LV-ViT-S trained at 224^2 and finetuned at 448^2, 0.3% above LV-ViT-S@384 at basically the same throughput and parameter count.Higher-resolution training and finetuning allow more tokens under comparable computational cost.
  • Comparisons and guidance: EViT with an oracle ViT significantly improves recognition accuracy, while outperforming DynamicViT at the same computational cost with fewer parameters.When trained from scratch, DynamicViT shows a significant accuracy reduction, whereas EViT accuracy continues improving with longer training.
  • Comparisons and guidance: EViT achieves a better accuracy-throughput trade-off than various Vision Transformer models and is complementary to architectural or token-interaction modifications.Figure 4 measures throughput on an NVIDIA A100 using the largest possible batch size for each model.

5 CONCLUSION

EViT reorganizes ViT tokens by retaining highly attended tokens and fusing less informative ones. Across DeiT and LV-ViT, it improves the accuracy-efficiency trade-off, accelerates inference with small accuracy reductions, and supports higher-resolution inputs at similar or lower computation.

  • 5 CONCLUSION: EViT identifies tokens receiving the largest class-token attention and fuses less informative tokens into a new token.The method is applied to DeiT and LV-ViT variants without additional parameters.
  • 5 CONCLUSION: EViT achieves significant inference speedups on DeiT and LV-ViT while keeping recognition-accuracy reductions relatively small.The conclusion reports this pattern across both tested Vision Transformer variants.
  • 5 CONCLUSION: Higher-resolution training lets EViT improve recognition accuracy while maintaining similar or smaller computation costs than the original DeiT and LV-ViT models.The conclusion also reports further trade-off improvement when EViT uses an oracle ViT.

A VISUALIZATION

EViT visualizations on randomly selected ImageNet images show attentive tokens across different categories, with masked regions marking inattentive tokens fused into a new token. The accompanying pseudocode identifies and fuses tokens using class-token attention and a configurable keep rate.

  • Visualization: Figure 5 visualizes EViT-DeiT-S attentive tokens across randomly selected ImageNet images from different categories.Unmasked regions represent attentive tokens, while masked regions represent inattentive tokens.
  • ViT encoder procedure: The pseudocode applies normalization, multi-head self-attention, token reorganization, projection, residual addition, and feed-forward processing within a ViT encoder.EViT is presented as a token reorganization method integrated into ViT training and inference.
  • Attentive token identification: EViT computes class-token attentiveness across attention heads and sorts tokens to select the top-K attentive tokens.The keep count is K = ceil(k * (N - 1)), where k is the token keep rate.
  • Token reorganization: The method separates attentive and inattentive tokens, then forms one fused token from a weighted combination of the inattentive tokens.The reorganized sequence concatenates the class token, attentive tokens, and fused token.

B EXTENDED EXPERIMENTS

Extended experiments evaluate EViT across token-identification strategies, training settings, reorganization choices, resolutions, and attention-mask quality. Results show speedups with small accuracy reductions, benefits from token fusion and later reorganization, and performance dependence on mask quality.

  • DeiT results: 26% higher inference throughput accompanies a Top-1 accuracy reduction within 0.1% for EViT-DeiT-B at a keep rate of 0.8 on ImageNet.The reported result demonstrates substantial acceleration while restricting the accuracy drop to a small range.
  • Token fusion: EViT with inattentive token fusion outperforms vanilla inattentive token removal when training on higher-resolution images, especially in Top-5 accuracy.The improvement is described as relatively small but practical.
  • Attention strategies: Tokens-to-tokens attention performs comparably with [CLS]-to-tokens attention but is less efficient.The tokens-to-tokens strategy lets each token vote, whereas [CLS]-to-tokens uses only the [CLS] token to select attentive tokens.
  • Reorganization locations: Moving token reorganization into shallower layers deteriorates accuracy at similar computational cost, suggesting early attention maps are unreliable for token removal.Reorganization before the third layer causes considerable recognition-accuracy drops despite matched cost.
  • Mask quality: Classification accuracy clearly correlates with attention-mask quality: EViT-topk, EViT-random, and EViT-mink test progressively different mask qualities.The experiment concludes that better-quality masks help improve accuracy.
  • Token evolution: Inattentive tokens identified in a shallow layer tend to remain less preferred in deeper layers, while training gradually shifts them toward less informative image regions.During EViT training, masked regions increasingly focus on backgrounds such as grass.
Loading 2202.07800v2…