PaperlayerView exact revision on arXiv

Source-linked AI summary

Visformer: The Vision-friendly Transformer

Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, Qi Tian

arXiv:2104.12533v5cs.CV

TL;DR

Vision Transformers can overfit or depend strongly on data and training strategy when data is limited. This paper analyzes the gap with an eight-step transition from DeiT-S to ResNet-50 and proposes Visformer, which outperforms both under comparable complexity.

  • Problem

    Vision Transformers show over-fitting and weaker performance than convolutional models under limited data or moderate augmentation, despite stronger performance with large datasets.

  • Method

    The paper performs an eight-step transition from DeiT-S to ResNet-50, analyzes the resulting architectural observations, and uses them to design Visformer.

  • Results

    Visformer-S outperforms DeiT-S and ResNet-50 by 2.12% and 3.46%, respectively, under comparable complexity in the elite setting.

  • Takeaways & Limitations

    Visformer maintains promising performance when transferred to compact models and evaluated on small datasets.

Abstract

from arXiv · show

The past year has witnessed the rapid development of applying the Transformer module to vision problems. While some researchers have demonstrated that Transformer-based models enjoy a favorable ability of fitting data, there are still growing number of evidences showing that these models suffer over-fitting especially when the training data is limited. This paper offers an empirical study by performing step-by-step operations to gradually transit a Transformer-based model to a convolution-based model. The results we obtain during the transition process deliver useful messages for improving visual recognition. Based on these observations, we propose a new architecture named Visformer, which is abbreviated from the `Vision-friendly Transformer'. With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy, and the advantage becomes more significant when the model complexity is lower or the training set is smaller. The code is available at https://github.com/danczs/Visformer.

I. INTRODUCTION

The paper studies why vision Transformers and convolutional models differ under training conditions, then uses a gradual transition to design Visformer with stronger performance across settings.

  • Vision Transformers can overfit with limited training data and perform worse than convolutional models under moderate augmentation.
  • DeiT-S has higher elite performance, whereas ResNet-50 is more robust under the base setting, exposing different performance bounds.Changing DeiT-S from elite to base causes a 10%+ accuracy drop, while ResNet-50 gains only 1.3% under elite training.
  • The study identifies architectural differences by applying eight step-by-step operations that transition DeiT-S toward ResNet-50.The operations include global pooling, step-wise patch embeddings, stage-wise design, batch normalization, 3 × 3 convolutions, removing position embeddings, replacing self-attention, and adjusting network shape.
  • Visformer-S outperforms DeiT-S and ResNet-50 by 2.12% and 3.46%, respectively, under the elite setting with comparable model complexity.
  • Visformer retains strong performance under reduced data and model size, including more than 6% improvement of Visformer-Ti over DeiT-Ti.Visformer-S also performs better than ResNet-50 with 10% labels and 10% classes.
  • The paper presents Visformer as a model combining properties of Transformer-based and convolution-based networks to achieve satisfying lower-bound and upper-bound performance.

II. RELATED WORK

Related work contrasts convolution’s local-pattern modeling with attention’s long-range interactions, while noting computational and data-related challenges for vision Transformers.

  • Convolution captures repeatable local patterns with small kernels, but faraway visual contexts communicate less directly.This limitation has encouraged deeper networks and attention-based approaches.
  • Pure attention vision models apply self-attention to image patches but face high computational cost on realistic full-sized images.
  • Hybrid vision models combine convolution and self-attention, including non-local networks designed to capture global dependencies.
  • The paper differs from prior hybrid approaches by exploring the full process of converting a Transformer into a convolutional neural network.
  • Scaling studies report benefits from greater depth, compute, data, augmentation, and regularization for vision Transformers.
  • Downstream vision tasks use larger input resolutions, making quadratic self-attention complexity a critical challenge.Sliding windows and additional information-exchange pipelines are widely used to address this issue.

III. METHODOLOGY

The methodology compares convolutional and Transformer layers and evaluates representative models under base and elite ImageNet training protocols, using a transition process to analyze their differences.

  • The study uses convolution and Transformer as the two representative layer types within residual-block networks.Convolution models local patterns, whereas Transformer self-attention formulates relationships between distant tokens.
  • DeiT-S and ResNet-50 represent Transformer-based and convolution-based models with differences in token handling, downsampling, and stage organization.
  • DeiT-S reports 80.1% accuracy versus 78.7% for ResNet-50, but uses more than three times as many epochs and stronger augmentation.
  • ImageNet evaluation uses 1K classes, 1.28M training images, and 50K testing images.
  • The base setting uses shorter training and moderate augmentation, while the elite setting uses longer training and stronger augmentation tailored to Transformers.
  • Figure 1 summarizes the eight-step transition from DeiT-S to ResNet-50, including changes from self-attention toward convolution.

C. The transition from DeiT-S to ResNet-50

The transition from DeiT-S to ResNet-50 progressively replaces Transformer-specific components with convolution-friendly designs and examines how these changes affect base and elite performance.

  • The transition contains eight steps and reports base performance, elite performance, and model statistics for each intermediate network.
  • Global average pooling replaces the classification token because convolutional models obtain classification features by pooling across the spatial dimension.
  • The first transition also converts token sequences into feature maps, with patch embedding and linear layers corresponding to convolutional operations.
  • Replacing the classification token with global average pooling substantially improves base performance while slightly improving elite performance.Global average pooling alone raises base performance from 64.17% to 69.44%.

2) Replacing patch flattening with step-wise patch embedding:

The paper replaces large patch flattening with factorized, step-wise embeddings that preserve within-patch position information and improve transition-network performance. This modification adds about 4% FLOPs, while subsequent stage-wise design further improves base performance but reduces elite performance.

  • Step-wise patch embedding: Step-wise embeddings encode position information within patches into features, enabling more efficient pattern learning.This addresses the loss of within-patch position information caused by directly flattening large patches.
  • Step-wise patch embedding: Step-wise small patch embeddings factorize the original large patch embedding into sequential convolutional embeddings and down-sampling stages.The design uses 7×7, 4×4, and 2×2 convolutions with corresponding strides, while increasing channels after embedding.
  • Step-wise patch embedding: About 4% extra FLOPs accompany the step-wise embedding transition, which significantly improves both base and elite performance.
  • Stage-wise design: Stage-wise blocks are assigned to 28×28, 14×14, and 7×7 feature resolutions, with reduced dimensions before attention at 28×28.
  • Stage-wise design: Stage-wise organization further improves base performance but markedly decreases elite performance in the transition network.The study attributes the base-performance gain to leveraging local image priors and reports that self-attention struggles at very large resolutions.

4) Replacing LayerNorm with BatchNorm:

The transition study replaces LayerNorm with BatchNorm and examines how normalization and spatial convolutions affect base and elite performance. BatchNorm improves performance in the mixed model but can cause convergence problems when added to a pure self-attention network.

  • Normalization: LayerNorm is inherited from NLP-oriented Transformers, whereas convolutional models commonly use BatchNorm to stabilize training.LayerNorm is batch-size independent, while BatchNorm can perform better with an appropriate batch size.
  • Normalization: Replacing LayerNorm with BatchNorm improves both base and elite performance of the network.
  • Normalization: Adding BatchNorm to Net2 causes convergence problems, but BatchNorm remains reliable for advancing performance in the mixed model.
  • Spatial convolutions: Adding 3×3 convolutions between 1×1 convolutions converts feed-forward MLP blocks into bottleneck blocks that relate nearby feature-map tokens.These convolutions model local neighborhoods rather than the global relations targeted by self-attention.
  • Spatial convolutions: 77.37% base performance becomes comparable with ResNet-50 at 77.43%, while elite performance decreases by 0.82% after adding 3×3 convolutions.
  • Position embedding: Removing position embeddings changes the transition model's elite performance by only 0.29%, versus a 3.95% decrease for DeiT-S.The paper attributes this difference to spatial position information preserved by feature maps and spatial convolutions.

7) Replacing self-attention with feed-forward:

Replacing self-attention with feed-forward layers yields a pure convolutional network, but this transition exposes different trade-offs between base and elite performance. The observations motivate Visformer's selective use of self-attention, bottleneck blocks, and stage-wise design.

  • Self-attention replacement: Replacing self-attention with feed-forward layers produces a pure convolutional network composed of bottleneck blocks while keeping FLOPs unchanged.
  • Self-attention replacement: The resulting pure convolutional network performs much worse in both base and elite settings, indicating self-attention supports higher elite performance.The passage states that self-attention is not responsible for poor base performance in ViT or DeiT.
  • Architecture differences: The transition network differs from ResNet-50 in shape, normalization placement, and down-sampling strategy, not only in its use of self-attention.
  • Architecture differences: Base performance improves after transition, whereas ResNet-50 has worse elite performance, confirming inconsistencies between the two performance settings in convolutional networks.
  • Stage-wise attention: Self-attentions in lower-resolution stages are more important, while replacing attention in the first stage has almost no performance effect.The study conjectures that large token counts make relations harder to learn at higher resolutions.
  • Stage-wise convolutions: Bottleneck blocks improve elite performance only in the first stage; 3×3 convolutions are unnecessary in later low-resolution stages when attention is global.
  • Visformer design: Visformer uses stage-wise design, self-attention only in the last two stages, first-stage bottleneck blocks, and grouped 3×3 convolutions.
  • Visformer design: Visformer-Ti halves multi-head-attention dimensionality and requires around one-quarter of Visformer-S's computational cost.

E. VisformerV2: optimizing the architecture configuration

VisformerV2 addresses untuned architectural configurations while also introducing a score-normalization method that avoids FP16 overflow without degrading performance or adding substantial runtime.

  • Architecture configuration: The original Visformer used an insufficiently tuned stage configuration, depth, and width, motivating systematic architecture experiments and VisformerV2.The original stage configuration was, and depth and width were also not polished.
  • Half-precision stability: Attention score generation can overflow in Transformers and Visformer when computations use 16-bit precision.Large Q and K matrices produce dot products that can overflow easily in FP16.
  • Half-precision stability: Normalizing the attention score with d effectively avoids overflow during score computation without degrading network performance.Pre-normalizing Q and K alone still sometimes overflowed because the dot products remained at risk.
  • Half-precision stability: Compared with PB-Relax, the proposed overflow solution requires no hyper-parameter tuning and introduces nearly no extra runtime.PB-Relax usually considerably increases runtime, whereas the proposed method nearly does not.

A. The improvements on the upper-bound and lower-bound

Visformer improves performance across training regimes and model scales, with particularly strong advantages over DeiT under the base setting and robustness when ImageNet data is limited.

  • Upper-bound and lower-bound: 14.08% and 10.47% are Visformer-S and Visformer-Ti’s respective advantages over DeiT under the base setting.Under the elite setting, the corresponding advantages are 2.12% and 6.41%.
  • Training with limited data: Visformer remains robust on ImageNet subsets containing 10% or 1% of classes or images, while DeiT-S shows dramatic accuracy drops.All models use the elite setting with 300 epochs; class-subset accuracy could be higher with extended training.
  • Model scale: Visformer-Ti is the best overall tiny model, and its advantage over comparable models slightly increases as reduced channels lower over-fitting risk.The comparison uses ResNet-50-55% with 1.3G FLOPs, similar to Visformer-Ti and DeiT-Ti.

C. Designing VisformerV2

VisformerV2 refines Visformer’s architecture through targeted stage and depth-width experiments, improving classification while retaining competitive efficiency and downstream-task performance.

  • Architecture design: Relative position bias raises Visformer’s result to 82.39%, while adding an early stage helps detection and segmentation despite not improving classification.The final design uses one block in the new stage.
  • Architecture design: Deep-and-narrow stage configurations degrade performance, whereas assigning more blocks to the third stage improves the networks significantly.The tested configurations included {1, 3, 11, 11}, {1, 6, 10, 10}, {1, 3, 18, 3}, and {1, 6, 16, 3}.
  • Efficiency: Deep-and-narrow architectures significantly increase GPU runtime because wide-and-shallow designs parallelize better.The authors compensate by using fewer FLOPs; at larger input resolutions or model scales, runtime becomes more consistent with FLOPs.
  • State-of-the-art comparison: VisformerV2-Ti and VisformerV2-S outperform comparable vision Transformers, with VisformerV2-S using fewer FLOPs and parameters than Visformer-S.At the tiny level, both Visformer-Ti and VisformerV2-Ti outperform other models with similar FLOPs.
  • Efficiency: VisformerV2-S is less runtime-efficient than Visformer-S but remains faster than most vision Transformers.For detection, VisformerV2-S also outperforms Swin-T by a large margin and is as efficient as Swin-T and MSG-T in FPS.

F. COCO Object Detection

Visformer models are adapted for COCO detection by replacing standard self-attention with shifted-window self-attention and evaluated under Mask R-CNN frameworks. VisformerV2-S outperforms Swin-T while maintaining competitive or faster inference efficiency.

  • Model adaptation: For high-resolution detection inputs, standard self-attention is replaced with shifted-window self-attention from Swin.This modification enables Visformer models to be applied to the detection task, with Swin Transformers serving as important baselines.
  • Evaluation setup: The COCO evaluation uses Mask R-CNN and Cascade Mask R-CNN on COCO 2017, with results reported on val2017.The experiments use both 1× and 3× training schedules, and measure FPS on a V100 GPU with batch size 1.
  • Mask R-CNN results: Visformer-S slightly outperforms Swin-T under the Mask R-CNN 1× schedule.Assigning a block to the first stage improves detection results even though classification performance does not improve.
  • Mask R-CNN results: VisformerV2-S outperforms Swin-T by 2.2% and is faster than Visformer-S.Its improved parallelization makes FPS more consistent with FLOPs.
  • Cascade Mask R-CNN results: Under Cascade Mask R-CNN with 3× and multi-scale training, VisformerV2-S outperforms other compared methods.Its FPS is comparable to Swin-T and MSG-T and higher than that of other vision Transformer methods.
  • Overall implication: The COCO results extend Visformer's reported advantage beyond ImageNet classification to object detection and instance segmentation.The paper presents Visformer as outperforming both DeiT-S and ResNet-50 while retaining transfer potential to other tasks.
Loading 2104.12533v5…