PaperlayerView exact revision on arXiv

Source-linked AI summary

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze

arXiv:2104.01136v2cs.CV

TL;DR

The paper addresses the speed–accuracy trade-off for small and medium vision-transformer models. It proposes LeViT, which combines transformer blocks with convolution-inspired pyramids, efficient patch processing, attention bias, and redesigned MLP blocks. Across the evaluated operating points, LeViT outperforms comparable transformer and convolutional architectures in speed–accuracy trade-offs.

  • Problem

    The paper seeks better performance–accuracy trade-offs for small and medium vision-transformer models, especially higher throughput on GPU, Intel CPU, and ARM hardware.

  • Method

    LeViT combines a multi-stage transformer pyramid, attention-based downsampling, an efficient patch descriptor, per-head translation-invariant attention bias, and redesigned MLP blocks.

  • Results

    LeViT largely outperforms the compared transformer and convolutional architectures across the evaluated speed–accuracy operating points.

  • Takeaways & Limitations

    LeViT provides a strong high-speed accuracy–efficiency trade-off across CPU, GPU, and mobile-oriented hardware settings.

Abstract

from arXiv · show

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers. As a result, we propose LeVIT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. We release the code at https://github.com/facebookresearch/LeViT

1 Introduction

Transformers use residual architectures built from MLP and self-attention blocks, with self-attention combining all pairs of input tokens.

  • Self-attention combines all pairs of input tokens within a residual architecture.
  • Transformer blocks contain two components: Multi-Layer Perceptrons and self-attention layers.
  • The architecture manipulates variable-size sequences of token embeddings.

1 CPU thread 1 GPU

LeViT targets faster inference for small and medium vision-transformer models by combining transformer computation with convolutional design principles, including pyramidal resolution reduction and attention-based downsampling.

  • LeViT targets better inference speed across GPUs, Intel CPUs, and ARM hardware while preserving the vision-transformer framework.The paper emphasizes throughput because high throughput corresponds to better energy efficiency.
  • LeViT replaces a uniform transformer structure with a convolution-inspired pyramid that uses pooling to reduce activation-map resolution.This design follows the staged resolution reductions used in classical convolutional architectures.
  • The architecture uses attention as a downsampling mechanism to shrink spatial resolution and computational width.
  • LeViT introduces a computationally efficient patch descriptor that reduces features in the first layers.
  • LeViT replaces ViT positional embeddings with a learned, per-head translation-invariant attention bias.

2 Related work

Prior vision-transformer work established strong image-classification performance but relied on substantial data, while hybrid designs began incorporating convolutional structure and positional information.

  • Convolutional networks: EfficientNet represents convolutional efforts to optimize accuracy and efficiency under FLOPs constraints.
  • Transformers: ViT achieved state-of-the-art image-classification performance, especially with large-scale pre-training.DeiT later targeted competitive ImageNet-only training and smaller high-throughput models.
  • Transformers: ViT requires strong pre-training, augmentation, or regularization because it has less built-in local structure than convolutional networks.
  • Positional encoding: Positional encoding methods provide spatial order information that self-attention otherwise lacks.Prior approaches include absolute, parametric, Fourier-based, and relative positional encodings.
  • Hybrid architectures: Hybrid vision architectures combine transformer attention with convolutional or other modules to capture complementary feature relationships.

3 Motivation

The motivation for LeViT comes from convolution-like behavior already present in vision transformers and from experiments showing that convolutional stages can improve runtime-controlled training and accuracy.

  • 3.1 Convolutions in the ViT architecture: ViT’s patch extractor is a 16x16 convolution with stride 16, and its attention embeddings can therefore behave as convolutional functions of the input.
  • 3.1 Convolutions in the ViT architecture: DeiT attention heads learn patterns resembling convolutional filters, including low- and high-frequency structures and Gabor-like patterns.
  • 3.1 Convolutions in the ViT architecture: Training can produce convolution-like filters despite transformers lacking an explicit convolutional inductive bias.
  • 3.2 Preliminary experiment: grafting: Grafted ResNet-50 and DeiT-Small models combine convolutional stages with transformer layers under similar runtimes.The experiment varies convolutional stages and transformer layers while controlling runtime.
  • 3.2 Preliminary experiment: grafting: Grafted architectures outperform DeiT and ResNet-50 alone, with two ResNet stages giving the smallest parameter count and best accuracy in the experiment.The evaluation uses 300-epoch training, ImageNet top-1 validation accuracy, and single-GPU images-per-second speed.
  • 3.2 Preliminary experiment: grafting: Grafted models converge like convolutional networks early in training and later adopt a DeiT-S-like convergence rate.The authors hypothesize that convolutional layers learn low-level representations efficiently through strong inductive biases such as translation invariance.
  • 3.2 Preliminary experiment: grafting: Runtime-controlled results motivate placing convolutional stages below transformers and merging the two architectures more closely.Most processing remains in the transformer stack for the most accurate grafted variants.

4 Model

LeViT adapts transformer representations and design principles from convolutional networks to build a multi-stage, compute-optimized architecture. Its components combine shrinking activation maps, efficient patch extraction, attention bias, and redesigned attention-MLP blocks.

  • Design principles: LeViT treats intermediate transformer token embeddings as C × H × W activation maps, allowing convolutional operations such as pooling and convolutions.This provides a compatible representation for importing convolutional design principles into DeiT-like transformers.
  • Patch embedding: Four 3×3 stride-2 convolutions reduce the input from (3, 224, 224) to (256, 14, 14) with 184 MFLOPs in LeViT-256.The channel progression is 3, 32, 64, 128, 256; the corresponding first 10 ResNet-18 layers require 1042 MFLOPs.
  • Output and variants: LeViT removes the classification token, replaces it with average pooling, uses batch normalization and Hardswish, and supports model variants with different stage sizes.LeViT models are identified by the number of channels entering the first transformer stage, such as LeViT-256.
  • Multi-resolution pyramid: LeViT integrates convolutional pyramid stages into a transformer, decreasing activation-map resolution while increasing channels during processing.Its stages retain residual structures with alternating MLP and attention blocks.
  • Downsampling: Shrinking attention downsamples (C, H, W) to (C′, H/2, W/2) with C′ > C, using subsampling before Q and no residual connection.The number of heads is set to C/D to help prevent information loss after the scale change.
  • Attention bias: LeViT adds a learned, per-head translation-invariant attention bias to each attention map instead of relying on ViT’s positional embedding.Each head has H × W parameters for pixel offsets, and symmetrized coordinate differences encourage flip invariance.
  • Efficient blocks: The attention-MLP blocks reduce compute through smaller key matrices, Hardswish-activated attention outputs, and 1×1-convolution-based MLP blocks.Keys use D ∈ {16, 32} while V has 2D channels, and the attention activation is framed as a convolutional bottleneck.

5 Experiments

LeViT experiments evaluate speed–accuracy trade-offs across ImageNet validation, multiple hardware platforms, and comparisons with efficient convolutional and transformer baselines. Results favor LeViT across operating points, while ablations support its multi-stage design choices.

  • Evaluation setup: Experiments train and evaluate models on ImageNet-2012 using PyTorch, without exploring additional training data.
  • Evaluation setup: Raw inference timings complement FLOP counts because activations and other operations can have substantial runtime costs, especially with few channels.
  • Evaluation setup: Timings cover a Volta GPU, an Intel Xeon CPU, and an ARM Graviton2 CPU, with batch settings reflecting GPU throughput and single-thread CPU inference.
  • Speed–accuracy trade-offs: LeViT-384 matches DeiT-Small accuracy with half the FLOPs, while LeViT-128S matches DeiT-Tiny with 4× fewer FLOPs.
  • Speed–accuracy trade-offs: LeViT-192 and LeViT-256 match EfficientNet B2 and B3 accuracies while running 5× and 7× faster on CPU, respectively.
  • State-of-the-art comparisons: Compared at similar accuracy, Token-to-token ViT uses around 5× more FLOPs than LeViT-384, while Bottleneck Transformers and Visual Transformers are about 5× slower than LeViT-192.
  • State-of-the-art comparisons: PiT remains 1.2× to 2.4× slower than LeViT, and alternative test sets preserve LeViT’s speed–accuracy advantages despite small accuracy differences.
  • Ablations: Ablations show that removing the pyramid, PatchConv, BatchNorm, or hard distillation degrades accuracy, supporting the importance of LeViT’s controlled design choices.

6 Conclusion

The paper concludes that LeViT combines transformer accuracy with convolution-inspired efficiency for high-speed inference. Its speed results from controlled architectural choices, while its accuracy largely stems from DeiT-style training techniques.

  • LeViT is reported as 1.5 to 5 times faster than comparable efficient neural networks at similar precision for datacenter and mobile feature-extraction settings.
  • The authors identify DeiT-style training as the main source of LeViT’s accuracy and carefully controlled design choices as the source of its speed.

A.1 Block timings

LeViT and DeiT blocks at the same 14×14 resolution have comparable runtime, despite LeViT being wider. LeViT reallocates computation across attention and MLP components.

  • At 14×14 resolution, LeViT and DeiT blocks have comparable run times, although LeViT is 33% wider (C = 256 vs C = 192).Stage 1 is the most expensive part of LeViT-256; later stages cost less because resolution decreases.
  • LeViT spends less time on attention QKT but more time on the subsequent matrix product AV.
  • LeViT spends less time on the MLP despite its larger width because its expansion factor is halved from four to two.

A.2 More details on our ablation

The ablations modify LeViT’s pyramid structure and block width while preserving computational cost, enabling tests of these architectural choices.

  • The ablation experiments provide additional details for the study reported in Section 5.6 and Table 4.
  • A1 – without pyramid shape: Without the pyramid shape, the three stages are replaced by a single depth-11 stage at 14×14 resolution while preserving the FLOP count.The modified configuration uses D = 19, N = 3, and C = 2ND = 114.
  • A6 – without wider blocks: Without wider blocks, LeViT-128S is modified to use more traditional blocks while preserving the number of FLOPs.The altered design uses equal Q, K, V dimensions and an MLP expansion ratio of 4.
  • Table 6 reports component timings for LeViT on one Intel Xeon E5-2698 CPU core with batch size 1.

B Visualizations: attention bias

LeViT’s attention bias is visualized as a two-dimensional map of relative-position preferences. Different heads exhibit uniform, local, directional, or periodic patterns.

  • Attention bias values represent the attention between pixels at specific relative positions and enter the softmax computation.Values as low as -20 can suppress attention between two pixels.
  • Some heads are uniform, while others specialize in nearby pixels, including directional patterns over vertically or horizontally adjacent pixels.
  • Head 1 of Stage 2, block 4 shows a period-2 pattern that may relate to the following subsampling filter.
  • Figure 6 visualizes attention bias for several blocks of a trained LeViT-256 model, centered on the upper-left pixel of the map.Higher values appear yellow and lower values dark blue, with values ranging from -20 to 7.
Loading 2104.01136v2…