ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

Stéphane d'Ascoli; Hugo Touvron; Matthew Leavitt; Ari Morcos; Giulio Biroli; Levent Sagun

Source-linked AI summary

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, Levent Sagun

arXiv:2103.10697v2Mar 19, 2021cs.CVcs.LGstat.ML

TL;DR

The paper asks whether convolutional sample efficiency and Transformer flexibility can be combined without inheriting their limitations. It introduces GPSA, which starts with convolution-like locality but lets attention heads adjust the balance between positional and content information. The resulting ConViT outperforms DeiT on ImageNet and improves sample efficiency, while the paper analyzes how locality develops and is escaped.

Problem
The paper asks whether convolutional inductive biases can improve ViT learning without imposing the limitations of hard locality constraints.
Method
GPSA initializes attention heads to mimic convolutional locality and uses learnable gating parameters to regulate positional versus content information.
Results
ConViT outperforms DeiT while offering much improved sample efficiency.
Takeaways & Limitations
Soft inductive biases can combine architectural priors with expressive power, improving trainability and sample efficiency without increasing model size or requiring tuning.
Takeaways & Limitations
The paper identifies future investigation of whether initializing attention layers as pre-trained convolutions could further reduce sample complexity.

Abstract

from arXiv · show

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.

1. Introduction

CNNs provide sample-efficient learning through hard locality and weight-sharing constraints, while ViTs offer greater flexibility but typically depend on extensive data or distillation. ConViT introduces a soft convolutional bias that can adaptively preserve locality or recover expressivity.

Motivation: CNNs improve sample efficiency through hard locality and weight-sharing constraints, but these constraints can limit flexibility when data is plentiful.The paper frames this as a trade-off between a higher performance floor and a potentially lower ceiling.
Motivation: ViTs use self-attention across image patches and can match or exceed CNN performance, but require large-scale pre-training or convolutional-teacher distillation.DeiT avoids supplementary pre-training by relying on knowledge distillation from a convolutional teacher.
Contribution: ConViT addresses this trade-off by softly introducing convolutional inductive bias while allowing each attention layer to decide whether to behave convolutionally.The approach aims to retain convolutional benefits without imposing their limitations.
Contribution: GPSA can be initialized as a convolutional layer, while per-head gating parameters let attention recover expressivity beyond locality.The gating mechanism regulates the relative use of positional and content information.
Results and analysis: ConViT outperforms DeiT while providing much improved sample efficiency, and the paper analyzes locality in vanilla self-attention and GPSA through measurements and ablations.The authors also release an open-source implementation and pretrained models.

2. Background

The paper presents self-attention as a content-based mechanism and positional self-attention as a way to incorporate relative patch locations. With suitable positional centers and locality strengths, multi-head positional attention can reproduce convolution-like attention maps.

Multi-head self-attention: Self-attention matches query embeddings with key embeddings to produce attention scores representing semantic relevance between sequence elements.In ViTs, queries and keys are linear projections of 16 × 16 image-patch embeddings.
Multi-head self-attention: Multi-head self-attention processes several heads in parallel, each learning different interdependencies while preserving the input sequence length and embedding dimension.The supplied passage describes the layer as mapping L embeddings of dimension Demb to the same shape.
Positional self-attention: Vanilla self-attention is position-agnostic, whereas positional self-attention incorporates relative patch positions through trainable embeddings and distance-dependent encodings.ViT instead injects positional information into patch embeddings before the first layer.
Convolutional interpretation: A positional-attention head centers its attention at a relative offset, while its locality strength controls whether attention is concentrated near that center or spread over a larger area.Large αh concentrates attention at the center; small αh produces a broader map.
Convolutional interpretation: Setting head centers to convolutional-kernel offsets and using large locality strengths yields strictly convolutional attention maps.This establishes positional self-attention as a generalized convolutional mechanism.
ConViT architecture: ConViT replaces some ViT self-attention layers with GPSA layers and typically places 10 GPSA layers before 2 vanilla self-attention layers.Because GPSA uses positional attention, the class token is concatenated after the final GPSA layer.

3. Approach

ConViT replaces selected ViT self-attention layers with GPSA layers initialized to mimic convolution while allowing heads to adjust their reliance on positional versus content information. This design improves ImageNet performance and sample efficiency relative to DeiT, while addressing parameterization and training issues in positional attention.

GPSA design: GPSA initializes attention heads in a convolutional configuration, then lets each head decide whether to remain local or recover broader expressivity.A learnable gating parameter controls the relative importance of positional and content information.
Adaptive attention span: GPSA avoids the quadratic positional-encoding parameter count of standard positional self-attention by fixing relative encodings and learning compact positional embeddings.The learned embeddings determine each head’s attention center and span, with Dpos much smaller than Dh.
Positional gating: GPSA combines content and positional attention after separate softmax operations, preventing one term from overwhelming the other during convolutional initialization.The resulting positive attention matrix is normalized into a probability distribution.
ConViT architecture: ConViT replaces the first 10 ViT blocks with GPSA layers while retaining absolute positional embeddings for a fair comparison.The model uses relative positional encodings in GPSA and relies much less on absolute embeddings.
ImageNet performance: ConViT outperforms comparable DeiT models at matched size, computational cost, and even throughput, including 82.2% top-1 accuracy for ConViT-S+ against DeiT-B.Hard distillation raises hard-distilled ConViT-S+ to 82.9% top-1 accuracy, matching hard-distilled DeiT-B with half the parameters.
Sample efficiency: With only 10% of ImageNet-1k training data, ConViT reaches 59.5% top-1 accuracy versus 46.5% for DeiT-S.The sample-efficiency experiment subsamples each class while increasing epochs so the total number of presented images remains constant.

4. Investigating the role of locality

The paper measures how attention moves between local and long-range configurations, finding that vanilla self-attention becomes local in lower layers while GPSA progressively escapes its convolutional initialization. Ablations show that GPSA, gating, and convolutional initialization improve training and are especially complementary with limited data.

Measuring locality: The nonlocality metric weights distances from each query patch by attention scores, with larger values indicating attention farther from the query.It is averaged over query patches and attention heads to characterize each layer.
Vanilla self-attention: DeiT attention becomes more local early in training, then upper layers regain nonlocality while lower layers remain local.This pattern indicates that lower layers retain local behavior while upper layers capture long-range dependencies.
GPSA dynamics: GPSA layers begin strongly local but increase nonlocality during training, ending below DeiT and preserving information from their convolutional initialization.Unlike DeiT, their final nonlocality does not increase monotonically across layers.
GPSA dynamics: The average positional-attention gate decreases during training: layers 6–10 largely ignore position, while early layers retain positional information.Larger ConViT models retain positional information through deeper layers, reaching layer 6 in ConViT-B versus layer 4 in ConViT-Ti.
Ablations and training efficiency: After 20 epochs, nine GPSA layers nearly double test accuracy, showing a substantial early-training advantage from convolutional initialization.The experiment uses ConViT-S trained on the first 100 ImageNet classes.
Ablations and training efficiency: In the low-data regime, GPSA alone improves accuracy by +6.8, while gating and convolutional initialization add +2.3 and +2.8, reaching +5.1 together.The reported gains illustrate complementarity between the components.
Ablations and training efficiency: A frozen, convolutionally initialized ConViT nearly matches DeiT in the full-data regime and exceeds it by +6.5 in the low-data regime.Randomly initialized frozen GPSA instead reduces full-data performance by -5.4 compared with DeiT.

5. Conclusion and perspectives

The paper argues that soft convolutional priors can combine architectural inductive bias with Transformer expressivity. Its proposed recipe improves trainability and sample efficiency without increasing model size or requiring tuning, while leaving pre-trained convolutional initialization as future work.

Conclusion: Softly letting layers choose between convolutional and attention-like behavior merges architectural priors with expressive power.This differs from interleaving fixed convolutional and self-attention layers.
Conclusion: The resulting recipe improves trainability and sample efficiency without increasing model size or requiring tuning.The conclusion presents these as practical benefits of the approach.
Perspectives: Future work asks whether initializing attention layers from pre-trained convolutions could reduce sample complexity further.The paper specifically raises this question after demonstrating benefits from random convolutional initialization.

A. The importance of positional gating

This ablation examines whether positional gating is necessary in addition to convolutional initialization. Convolutional initialization helps early even without GPSA, but replacing GPSA with vanilla positional self-attention harms final performance.

Ablation: Replacing GPSA with vanilla positional self-attention retains an early convolutional-initialization advantage but deteriorates final performance.The comparison trains ConViT-B and DeiT-B for 300 ImageNet epochs.

B. The effect of distillation

Distillation makes DeiT representations more local and improves its performance, while ConViT still benefits from distillation. The results suggest that directly instantiating a soft convolutional bias adds benefits beyond transferring that bias through a convolutional teacher.

Locality: Hard distillation from a RegNet teacher makes DeiT more local by reducing its late-training nonlocality increase.The distilled model retains the early nonlocality drop but becomes more local at the end of training.
Performance: ConViT benefits from hard distillation, although less than DeiT, suggesting overlap between transferred and built-in convolutional inductive biases.The comparison uses RegNet-16GF distillation for 300 ImageNet epochs.
Interpretation: ConViT's improvement with hard distillation shows that directly instantiating soft inductive biases can add benefits beyond indirect transfer from a convolutional teacher.This conclusion is stated for the combination of soft inductive bias and distillation.

C. Further performance results

Further experiments show that ConViT maintains strong performance across dataset settings and that its design ingredients affect learning dynamics.

C. Further performance results: ConViT+ and DeiT+ were compared throughout training on CIFAR100, ImageNet, and subsampled ImageNet.Figure 11 reports top-1 accuracy trajectories and relative improvement between the models’ best accuracies.
C. Further performance results: 82.1% top-1 accuracy was reached on CIFAR100 by a 10M-parameter ConViT after extended training.The models used rescaled 224 × 224 images and an increased epoch count to mimic ImageNet training.
C. Further performance results: The experiments ablated GPSA-layer count, gating parameters, and convolutional initialization to assess their effects on learning dynamics.These ingredients were studied alongside the presence of GPSA layers.

D. Effect of model size

Larger ConViT and DeiT models develop more non-local attention, while larger ConViTs retain positional attention across more early layers.

D. Effect of model size: Larger DeiT-B and ConViT-B models reach significantly higher nonlocality than their tiny counterparts.Nonlocality measures how far attention heads look from the query pixel during ImageNet training.
D. Effect of model size: The average positional-attention weight decreases over training, shifting attention toward image content.This pattern is qualitatively similar across tiny, small, and base models.
D. Effect of model size: At the end of training, positional attention remains in the first 4 ConViT-Ti layers, 5 ConViT-S layers, and 6 ConViT-B layers.The layer counts refer to layers whose average gating parameter remains below one.

E. Attention maps

Attention-map analyses show that vanilla DeiT attention becomes locally concentrated in early layers, whereas ConViT heads diversify between positional and content-focused operations.

E. Attention maps: After training, DeiT first-layer heads focus on the query pixel and its immediate surroundings, while tenth-layer heads capture long-range dependencies.Before training, averaged attention maps exhibit an essentially random structure.
E. Attention maps: ConViT attention maps perform substantially different operations across heads and layers, unlike the more similar DeiT maps.The comparisons cover tiny and small models trained on ImageNet.
E. Attention maps: In a second ConViT layer, some heads remain local while others focus on content; in the last layer, all heads ignore positional information.The figures encode gating values from position-focused to content-focused attention.

F. Further ablations

Further ablations examine the roles of absolute positional embeddings and positional versus content attention in ConViT and DeiT.

F. Further ablations: Masking absolute positional embeddings at test time mildly affects ConViT accuracy but significantly affects DeiT accuracy.ConViT already contains relative positional information in its GPSA layers.
F. Further ablations: In early GPSA layers, masking positional or content attention affects performance similarly, indicating that both information types are useful.The comparison manually sets the gating parameter to remove either content or positional attention.
F. Further ablations: In late GPSA layers, masking content attention kills performance, whereas masking positional attention does not.This confirms that content information is more crucial in the last GPSA layers.