PaperlayerView exact revision on arXiv

Source-linked AI summary

CvT: Introducing Convolutions to Vision Transformers

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang

arXiv:2103.15808v1cs.CV

TL;DR

Vision Transformers can lack CNNs’ useful spatial inductive biases and may underperform on smaller datasets. CvT addresses this by adding hierarchical convolutional token embeddings and convolutional attention projections, while retaining Transformer capabilities. The resulting model achieves strong performance with computational efficiency and can omit positional embeddings without performance degradation.

  • Problem

    Vision Transformers may underperform similarly sized CNNs on smaller datasets because they lack CNN-like local spatial structure and related invariances.

  • Method

    CvT introduces hierarchical multi-stage Transformers with convolutional token embeddings and replaces position-wise attention projections with depth-wise separable convolutional projections.

  • Results

    CvT achieves superior performance while maintaining computational efficiency, and removing positional embeddings does not degrade its performance.

  • Takeaways & Limitations

    Built-in local context lets CvT simplify the architecture by omitting positional embeddings and potentially accommodate variable-resolution vision inputs.

Abstract

from arXiv · show

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{https://github.com/leoxiaobin/CvT}.

1. Introduction

CvT addresses Vision Transformers’ weaker small-data performance by introducing convolutional structure while retaining Transformer modeling. Its hierarchical design combines CNN-like efficiency and spatial inductive biases with dynamic attention, and achieves strong ImageNet results without positional embeddings.

  • Motivation: Vision Transformers can underperform similarly sized CNNs on smaller datasets because they lack CNN-like local spatial structure and invariances.CNNs use local receptive fields, shared weights, and spatial subsampling to capture correlated neighboring pixels and support shift, scale, and distortion invariance.
  • Method: CvT introduces convolutions into ViT through hierarchical Transformer stages and convolutional token embeddings.Each stage applies overlapping convolution to a 2D-reshaped token map, followed by layer normalization, while progressively reducing sequence length.
  • Method: CvT combines CNN properties—local receptive fields, shared weights, and spatial subsampling—with Transformer benefits including dynamic attention and global context.The architecture is designed to preserve computational and memory efficiency through convolution-aware processing.
  • Results: CvT improves performance over CNN- and Transformer-based models while using fewer FLOPs and parameters, including at ImageNet-22k pretraining scale.The passage reports state-of-the-art performance on ImageNet-1k and at larger-scale ImageNet-22k pretraining.
  • Results: Removing positional embeddings does not degrade CvT performance and enables simpler handling of variable-resolution inputs.The reported design removes positional embeddings by default, supporting vision tasks requiring variable input resolution.

2. Related Work

Prior vision Transformers established strong large-scale image recognition but motivated designs that better capture local context and handle spatial variation. CvT instead introduces convolutional token embeddings and convolutional projections within a hierarchical Transformer to combine local and global modeling efficiently.

  • Vision Transformers: ViT showed that a pure Transformer can achieve state-of-the-art image classification when trained on sufficiently large datasets.ViT represents images as fixed-length non-overlapping patch tokens and applies standard Transformer layers.
  • CvT Positioning: CvT aims to combine CNN local dependencies with Transformer global dependencies for efficient image classification.Its design introduces image-domain inductive biases through convolutions rather than relying solely on positional encoding changes.
  • Concurrent Designs: Concurrent methods modify positional encodings or token processing to improve local context modeling and support arbitrary-sized inputs.Examples include conditional positional encodings in CPVT and nested pixel- and patch-level Transformers in TNT.
  • CNNs with Self-Attention: Earlier CNN self-attention work primarily adds global or adaptive attention mechanisms to convolutional models to capture long-range or local relationships.These approaches differ in direction from CvT’s introduction of convolutional operations into a vision Transformer.
  • CvT Positioning: CvT differs from prior convolution-augmented Transformers by applying convolutions to token embedding and attention projection while using a hierarchical multi-stage backbone.The convolutional projection replaces the position-wise linear projection in attention, and the hierarchy supports varied token-map resolutions.

3. Convolutional vision Transformer

CvT builds a multi-stage Transformer using convolutional token embeddings and convolutional projections, combining local spatial modeling with efficient attention computation. The design progressively changes token resolution and feature dimension while preserving a Transformer backbone.

  • Architecture: CvT uses three stages, each beginning with convolutional token embedding that applies overlapping convolution to a 2D token map before layer normalization.The embedding reshapes flattened sequences back to spatial grids and supports hierarchical processing.
  • Architecture: Convolutional token embedding progressively decreases sequence length while increasing token feature dimension across stages.This lets tokens represent more complex visual patterns over larger spatial footprints, similar to CNN feature layers.
  • Convolutional Projection: Convolutional Projection replaces position-wise linear projection in MHSA with depth-wise separable convolutions, adding local spatial context.Tokens are reshaped into a 2D map, projected by convolution, and flattened again for subsequent processing.
  • Efficiency: Depth-wise separable convolution adds s^2C parameters and O(s^2CT) FLOPs, compared with s^2C^2 parameters and O(s^2C^2T) for standard convolution.The added cost is described as negligible relative to total model parameters and FLOPs.
  • Efficiency: The projection reduces MHSA computation by subsampling key and value tokens with stride 2 while leaving query stride unchanged.This reduces the number of key and value tokens 4 times and correspondingly reduces later attention computation.
  • Positional Embeddings: CvT models local spatial relationships through convolutional embeddings and projections, allowing position embeddings to be removed without hurting performance.The paper connects this design to simpler vision models supporting variable input resolution.

4. Experiments

CvT is evaluated on ImageNet classification, transfer tasks, and component ablations. The experiments show strong accuracy-efficiency trade-offs, successful transfer, and benefits from convolutional token embedding and projection designs.

  • Comparison to state of the art: CvT-21 reaches 82.5% ImageNet Top-1 accuracy, 0.5% above DeiT-B with 63% fewer parameters and 60% fewer FLOPs.
  • Comparison to state of the art: CvT-13 reaches 81.6% ImageNet Top-1 accuracy, exceeding PVT-Small, T2T-ViTt-14, and TNT-S by 1.7%, 0.8%, and 0.2%, respectively.
  • Comparison to state of the art: CvT-13 surpasses ResNet-152 by 3.2% on ImageNet Top-1 accuracy using 20M parameters and 4.5G FLOPs, while ResNet-151 has three times its parameters.
  • Comparison to state of the art: CvT-W24* pretrained on ImageNet-22k reaches 87.7% Top-1 accuracy on ImageNet and exceeds ViT-L/16 by 2.5% with similar parameters and FLOPs.
  • Transfer to downstream tasks: CvT-W24 achieves the best performance across all considered downstream tasks after ImageNet-22k pretraining, including comparison with the larger BiT-R152x4.
  • Ablation Study: Replacing patch embedding with Convolutional Token Embedding reduces performance by 0.8%, while the convolutional design improves spatial modeling without position embedding.

5. Conclusion

The paper concludes that introducing convolutions throughout a multi-stage Vision Transformer combines Transformer and CNN benefits, achieving strong performance with computational efficiency.

  • CvT merges the benefits of Transformers and CNNs for image recognition by introducing convolutions into the Vision Transformer architecture.
  • Convolutional token embedding, convolutional projection, and a convolution-enabled multi-stage design support CvT’s performance and computational efficiency.
  • Extensive experiments demonstrate superior performance while maintaining computational efficiency.
Loading 2103.15808v1…