Incorporating Convolution Designs into Visual Transformers

Kun Yuan; Shaopeng Guo; Ziwei Liu; Aojun Zhou; Fengwei Yu; Wei Wu

Source-linked AI summary

Incorporating Convolution Designs into Visual Transformers

Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, Wei Wu

arXiv:2103.11816v2Mar 22, 2021cs.CV

TL;DR

Pure visual Transformers can depend on large training datasets or extra CNN supervision, limiting their use when data or computing resources are constrained. CeiT addresses these limitations by combining convolutional feature extraction and locality with Transformer long-range dependencies, and reports strong performance across ImageNet and downstream tasks with faster convergence.

Problem
Pure visual Transformers require large training datasets or extra CNN teachers to achieve competitive performance, limiting application with restricted data or computing resources.
Method
CeiT combines CNN-based low-level feature extraction and locality with Transformer long-range dependency modeling through I2T, LeFF, and LCA.
Results
CeiT reports effectiveness and generalization ability on ImageNet and seven downstream tasks, without large training data or extra CNN teachers, and better convergence with 3× fewer training iterations.
Takeaways & Limitations
Incorporating convolutional designs provides a new perspective for more effective visual Transformers while reducing reported training cost.

Abstract

from arXiv · show

Motivated by the success of Transformers in natural language processing (NLP) tasks, there emerge some attempts (e.g., ViT and DeiT) to apply Transformers to the vision domain. However, pure Transformer architectures often require a large amount of training data or extra supervision to obtain comparable performance with convolutional neural networks (CNNs). To overcome these limitations, we analyze the potential drawbacks when directly borrowing Transformer architectures from NLP. Then we propose a new \textbf{Convolution-enhanced image Transformer (CeiT)} which combines the advantages of CNNs in extracting low-level features, strengthening locality, and the advantages of Transformers in establishing long-range dependencies. Three modifications are made to the original Transformer: \textbf{1)} instead of the straightforward tokenization from raw input images, we design an \textbf{Image-to-Tokens (I2T)} module that extracts patches from generated low-level features; \textbf{2)} the feed-froward network in each encoder block is replaced with a \textbf{Locally-enhanced Feed-Forward (LeFF)} layer that promotes the correlation among neighboring tokens in the spatial dimension; \textbf{3)} a \textbf{Layer-wise Class token Attention (LCA)} is attached at the top of the Transformer that utilizes the multi-level representations. Experimental results on ImageNet and seven downstream tasks show the effectiveness and generalization ability of CeiT compared with previous Transformers and state-of-the-art CNNs, without requiring a large amount of training data and extra CNN teachers. Besides, CeiT models also demonstrate better convergence with $3\times$ fewer training iterations, which can reduce the training cost significantly\footnote{Code and models will be released upon acceptance.}.

1. Introduction

The introduction identifies data and locality limitations in pure visual Transformers and proposes CeiT, which incorporates convolutional designs while retaining long-range dependency modeling. CeiT is reported to perform effectively across image and downstream tasks and to converge with fewer training iterations.

Motivation: ViT relies on very large datasets, while DeiT requires a trained CNN teacher, creating data, computation, and teacher-selection limitations.ViT depends on JFT-300M, whereas DeiT uses CNN-based knowledge distillation on ImageNet.
Motivation: Pure Transformers overlook convolutional inductive biases, including low-level feature extraction and spatial locality among neighboring image elements.Direct raw-image tokenization makes low-level structures difficult to extract, while self-attention emphasizes long-range dependencies.
CeiT: CeiT combines CNN-based low-level feature extraction and locality with Transformer-based long-range dependency modeling.The architecture introduces I2T, LeFF, and LCA relative to vanilla ViT.
Results: CeiT reports effectiveness and generalization ability compared with previous Transformers and state-of-the-art CNNs on ImageNet and seven downstream tasks.The introduction presents this as an experimental contribution without requiring a large training dataset or extra CNN teachers.
Results: 3× fewer training iterations yield better convergence for CeiT models than pure Transformer models, reducing training cost significantly.The reported comparison is summarized through Figure 1.

2. Related Work

Related work covers pure vision Transformers and hybrid convolution-attention models, while CeiT introduces convolutional designs into Transformer building blocks. Its tokenization comparison contrasts raw-image patches with patches extracted from convolutional low-level features.

Transformer in Vision: ViT reshapes images into 16 × 16 patches and achieves comparable CNN performance mainly when trained on very large datasets.The related-work discussion presents ViT as a standard Transformer adapted for image classification.
Hybrid Models of Convolution and Self-attention: Hybrid models introduce attention into CNNs or replace convolutional layers with self-attention to capture long-range dependencies.Examples include non-local layers, criss-cross attention, and self-attention-based replacements of convolutional layers.
Tokenization: I2T compares with raw-image tokenization by extracting patches from low-level features generated by a convolutional stem.The figure contrasts the raw-input approach above with the I2T approach below.
Hybrid Models of Convolution and Self-attention: CeiT differs by incorporating convolutional designs into Transformer building blocks to inherit CNN inductive biases through a more elaborate architecture.This positions CeiT among hybrid approaches while emphasizing its design at the Transformer-block level.

3. Methodology

CeiT incorporates convolutional designs into ViT to address limited low-level feature extraction, locality, and multi-level representation use while retaining global self-attention. Its I2T, LeFF, and LCA modules provide this integration with small added computational cost and better convergence.

3. Methodology: CeiT combines CNN locality and low-level extraction with Transformer long-range dependencies through three modifications to ViT.The methodology covers I2T, LeFF, and LCA as the principal architectural changes.
3.1. Revisiting Vision Transformer: ViT directly tokenizes large raw-image patches, making low-level features difficult to capture and large kernels harder to optimize.The paper identifies these limitations as contributing to greater data or iteration requirements.
3.2. Image-to-Tokens with Low-level Features: CeiT replaces straightforward ViT tokenization with I2T, which extracts smaller patches from convolutionally generated low-level feature maps.I2T uses a convolutional layer and max-pooling layer; BatchNorm after convolution benefits training.
3.3. Locally-Enhanced Feed-Forward Network: LeFF replaces each encoder FFN while preserving MSA, restoring expanded patch tokens spatially so depth-wise convolution enhances correlations among neighboring tokens.The tokens are then flattened, projected back to the initial dimension, and concatenated with the class token.
3.4. Layer-wise Class-Token Attention: LCA applies attention to class tokens from different layers instead of using only the final-layer class token as the representation.It receives a sequence of class tokens and computes unidirectional similarities centered on the last-layer class token.

4. Experiments

Experiments evaluate CeiT on ImageNet and seven downstream tasks against CNN and Transformer baselines, including ablations of its modules and convergence comparisons. CeiT achieves strong accuracy, transfer performance, and faster convergence under the reported settings.

CeiT vs CNNs: CeiT-T reaches 76.4% Top-1 accuracy on ImageNet, close to ResNet-50 while using 3× fewer FLOPs and 4× fewer parameters.The comparison uses CeiT-T and ResNet-50 models.
CeiT vs CNNs: 82.0% Top-1 accuracy is achieved by CeiT-S at a similar model size to ResNet-50, exceeding ResNet-50 by 5.3 percentage points.CeiT-S also surpasses the reported ResNet-152 and RegNetY-8GF comparisons.
CeiT vs ViT/DeiT/T2T/PVT: CeiT models outperform DeiT baselines and several other Transformer models, including a 4.2-point Top-1 advantage for CeiT-T over DeiT-T.The passage also reports higher results for CeiT-S than DeiT-S and DeiT-B, plus comparisons with T2T-ViT and PVT.
CeiT vs DeiT-Teacher: CeiT-T surpasses DeiT-T-Teacher by 1.9% Top-1 accuracy, while CeiT-S exceeds DeiT-S-Teacher by 0.8% without additional CNN supervision.DeiT-Teacher uses an extra CNN model for supervision, whereas CeiT uses only ground-truth supervision.
Transfer learning: CeiT-S transfers effectively across seven downstream benchmarks, outperforming DeiT-B on most datasets with fewer parameters and FLOPs.CeiT-S↑384 achieves state-of-the-art results on most reported downstream datasets and comparable results with EfficientNet-B7.
Ablation studies: Ablations show that BatchNorm benefits I2T and LeFF, 3 × 3 LeFF is selected for its parameter-accuracy trade-off, and LCA improves accuracy from 72.2% to 72.8%.The LeFF ablation reports gains for 3 × 3 and 5 × 5 kernels, with BatchNorm improving accuracy by up to 2.2% Top-1.
Convergence: CeiT models obtain comparable results with 100 training epochs, whereas DeiT models require 300 epochs for comparable performance.The reported convergence comparison uses 3× fewer training epochs for CeiT.

5. Conclusion

CeiT combines convolutional designs with Transformer modeling and achieves state-of-the-art performance across ImageNet and downstream tasks. It also converges with fewer training iterations, reducing training cost without large datasets or extra CNN teachers.

CeiT combines CNN advantages in low-level feature extraction and locality with Transformer advantages in establishing long-range dependencies.
CeiT achieves state-of-the-art performance on ImageNet and various downstream tasks without requiring large amounts of training data or extra CNN teachers.
3× fewer training iterations yield better convergence than pure Transformers and significantly reduce training cost.