PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition

Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, Gao Huang

arXiv:2105.15075v2cs.CVcs.AIcs.LG

TL;DR

Fixed tokenization forces vision Transformers to spend the same attention cost on images with different difficulty. DVT cascades Transformers with increasing token counts, exits when confidence is sufficient, and reuses features and relationships; experiments report improved efficiency across ImageNet and CIFAR.

  • Problem

    Vision Transformers use fixed token counts even though images differ in difficulty, while larger token representations increase self-attention computation quadratically.

  • Method

    DVT cascades Transformers with increasing token counts, terminates inference using prediction confidence, and reuses features and self-attention relationships across components.

  • Results

    DVT significantly improves computational efficiency across ImageNet and CIFAR, reducing T2T-ViT computational cost by 1.6-3.6x without sacrificing accuracy.

  • Takeaways & Limitations

    DVT provides a general, online-adjustable framework for efficient vision Transformer inference under fluctuating computational resources or minimal-power requirements.

Abstract

from arXiv · show

Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of "easy" images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of "hard" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.

1 Introduction

Vision Transformers usually use a fixed token count, creating a trade-off between recognition accuracy and quadratic self-attention cost. DVT instead adapts tokenization per image through confidence-based early termination and computation reuse.

  • Motivation: More image tokens can improve prediction accuracy, but self-attention cost grows quadratically with token number, motivating fixed 14x14 or 16x16 settings.These settings are adopted as an empirical efficiency–effectiveness trade-off.
  • Dynamic Vision Transformer: DVT configures token numbers per image by cascading Transformers with increasing token counts and stopping inference once confidence is sufficient.Easy samples can exit earlier, while harder samples continue to downstream Transformers.
  • Efficient computation: Feature-wise reuse trains downstream models from previously extracted features, while relationship-wise reuse transfers upstream self-attention relationships to refine downstream attention.Both mechanisms target redundant computation across the cascade.
  • Flexibility: DVT supports ViT, DeiT, and T2T-ViT backbones and adjusts computational cost online through its early-termination criterion.The paper highlights fluctuating-resource and low-power scenarios as application settings.
  • Results: 1.6-3.6x lower computational cost is reported for T2T-ViT without sacrificing accuracy, with NVIDIA 2080Ti inference speed consistent with theoretical efficiency.The evaluation covers ImageNet and CIFAR using T2T-ViT and DeiT backbones.

2 Related Work

Prior vision Transformers generally represent images with a fixed number of tokens, while efficient dynamic architectures have mainly been developed for CNNs. DVT extends adaptive computation to vision Transformers with input-conditioned token counts.

  • Vision Transformers: ViT, DeiT, T2T-ViT, and related models represent each image using a fixed number of tokens.The paper presents input-conditioned token selection as a distinct direction.
  • Efficient deep networks: Efficient deep-network research targets computational cost because executed computation affects power consumption, carbon emissions, and latency.Efficient vision Transformer design remains under-explored relative to CNN efficiency research.
  • Dynamic models: Dynamic CNNs use early exits, adaptive depth, layer or channel skipping, and spatial adaptation for efficient recognition.These approaches motivate adaptive computation but are developed for CNNs.
  • Position of DVT: DVT differs from multi-exit CNNs by cascading multiple complete Transformers rather than reducing depth within one network.The comparison identifies the architectural distinction behind DVT’s Transformer-specific design.

3 Dynamic Vision Transformer

DVT adapts token resolution per image by cascading Transformers with increasing token counts and stopping when confidence is sufficient. Feature and relationship reuse preserve upstream computation while later exits provide finer representations for difficult samples.

  • Inference: The cascade allocates computation unevenly across samples by combining coarse, efficient inference with progressively more accurate downstream models.Transformer cost grows quadratically with token number, so later stages trade additional computation for accuracy.
  • Inference: DVT sequentially activates Transformers with increasing token counts, stopping early when a prediction meets its confidence threshold.Easy images can exit with fewer tokens, while difficult images proceed to more expensive, finer representations.
  • Training and architecture: DVT trains all exits to produce correct predictions using a standard cross-entropy objective over their softmax outputs.The framework can be built on existing vision Transformer backbones without changing their basic architecture.
  • Feature reuse: Feature reuse learns layer-wise embeddings from upstream final-layer image tokens and injects them into downstream intermediate tokens.The upstream tokens are reshaped, upsampled, and flattened to match the downstream token layout.
  • Relationship reuse: Relationship reuse refines downstream attention logits with transformed upstream attention relationships gathered across layers and heads.The transformation network uses an MLP and an attention-map upsample operation; logits combine newly generated and reused relationships.
  • Inference policy: Inference thresholds are selected on validation data under a computational budget, balancing recognition accuracy against FLOPs.The thresholds determine whether each sample exits or advances to a higher-token model.

4 Experiments

Experiments evaluate DVT on ImageNet and CIFAR using T2T-ViT and DeiT backbones, including efficiency comparisons, practical speed, ablations, and sample visualizations. Results show substantial computational savings while preserving or improving accuracy.

  • Experimental Setup: DVT experiments use ImageNet and CIFAR-10/100 with T2T-ViT-12, T2T-ViT-14, and DeiT-small backbones.The default three exits represent images with 7x7, 10x10, and 14x14 tokens.
  • Main Results: 82.3% accuracy is achieved with 3.6x less FLOPs than vanilla T2T-ViT on ImageNet.Across 0.5–2 GFLOPs, DVT uses approximately 1.7–1.9x less computation at the same performance.
  • Main Results: DVT improves small-model accuracy by 3.7–6.8% at the same inference speed and accelerates large T2T-ViT models by 1.4–1.7x without sacrificing performance.The practical test uses a two-exit T2T-ViT-12 DVT with 7x7 and 14x14 tokens on an NVIDIA 2080Ti GPU.
  • Ablation Study: Feature and relationship reuse significantly improve second- and third-exit accuracy with at most 6% additional computation.Reuse mechanisms are compatible and jointly improve performance, although reuse slightly reduces first-exit accuracy.
  • Ablation Study: Final upstream tokens and reuse across each downstream layer are important for feature reuse, while downstream layers benefit more from upstream reuse at shallow depths.Relationship-reuse ablations likewise find that downstream layers benefit from flexibly reusing all upstream attention logits.
  • Ablation Study: The confidence-based early-termination criterion outperforms entropy-based and random exiting under comparable computational budgets.Visualization distinguishes easy samples, recognized at the first exit, from hard samples requiring the third exit and finer representations.

5 Conclusion

The conclusion presents DVT as a framework for configuring token counts per image through adaptive cascaded inference. It reports improved computational efficiency both theoretically and empirically.

  • Conclusion: DVT sequentially activates Transformers with increasing token counts until prediction confidence indicates that an appropriate representation has been reached.Feature and relationship reuse mechanisms facilitate computation reuse across the cascade.
  • Conclusion: Experiments indicate that DVT significantly improves the computational efficiency of state-of-the-art vision Transformers theoretically and empirically.

A Training Details

Training details specify implementation fidelity to the backbone models and document additional analyses of reuse locations and upstream relationships.

  • Training Details: DVT follows the official T2T-ViT and DeiT configurations, including their training settings and hyperparameters.The implementation is based on the official backbone code.
  • Additional Analyses: Layer-wise analyses evaluate where feature and relationship reuse is applied and which upstream attention logits are reused.These studies are reported in Tables 11–13.

B Additional Results

Additional results analyze reuse across layers and provide accuracy–efficiency curves for CIFAR and ImageNet. They show that shallow downstream reuse is especially valuable, while additional relationships offer limited gains.

  • Reuse Analysis: Shallow downstream layers benefit more from reusing upstream features and relationships than deep downstream layers.Later layers focus more on refining shallow features than leveraging upstream information.
  • Reuse Analysis: Both shallow and deep upstream attention relationships improve downstream accuracy, but reusing more relationships yields only slight additional gains.The limited improvement is attributed to redundancy among attention logits from different upstream layers.
  • Efficiency Curves: Top-1 accuracy versus GFLOPs curves are reported for CIFAR-10 and CIFAR-100, while ImageNet results use top-1 accuracy versus throughput.The curves correspond to the paper’s reported tables.
Loading 2105.15075v2…