PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao

arXiv:2202.10108v2cs.CV

TL;DR

Vision transformers lack intrinsic biases for local visual structures and scale variation, making them dependent on large-scale data and longer training schedules. ViTAE incorporates locality and scale-invariance through convolution-based reduction and normal cells, including a multi-stage ViTAEv2 design, and reports strong results across classification and downstream tasks, including 88.5% ImageNet Top-1 accuracy for a 644M-parameter model.

  • Problem

    Vision transformers lack intrinsic inductive biases for modeling local visual structures and handling objects at different scales, so these biases are learned implicitly from large-scale data.

  • Method

    ViTAE introduces locality and scale-invariance through convolution-based reduction and normal cells, stacked in isotropic and multi-stage architectures.

  • Results

    ViTAE and ViTAEv2 outperform representative vision transformers across classification and downstream vision tasks; ViTAE-H achieves 88.5% Top-1 accuracy on ImageNet validation with 644M parameters.

  • Takeaways & Limitations

    Intrinsic inductive bias remains beneficial when scaling vision transformers to large models and supports performance across image classification, detection, segmentation, and pose estimation.

  • Takeaways & Limitations

    The parallel structure may slow training in dynamic computation graph frameworks because parallel branches can be computed sequentially.

Abstract

from arXiv · show

Vision transformers have shown great potential in various computer vision tasks owing to their strong capability to model long-range dependency using the self-attention mechanism. Nevertheless, they treat an image as a 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance, which is instead learned implicitly from large-scale training data with longer training schedules. In this paper, we propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and can learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. The proposed two kinds of cells are stacked in both isotropic and multi-stage manners to formulate two families of ViTAE models, i.e., the vanilla ViTAE and ViTAEv2. Experiments on the ImageNet dataset as well as downstream tasks on the MS COCO, ADE20K, and AP10K datasets validate the superiority of our models over the baseline transformer models and concurrent works. Besides, we scale up our ViTAE model to 644M parameters and obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 accuracy on ImageNet real validation set, without using extra private data.

1 Introduction

Vision transformers model global dependencies effectively but lack convolution-like inductive biases for local structures and scale variation. ViTAE introduces locality and scale-invariance through reduction and normal cells, and extends them to multi-stage ViTAEv2 for downstream tasks.

  • Motivation: Vision transformers treat images as token sequences but lack intrinsic bias for local visual clues and objects at different scales.They instead learn these biases implicitly from large-scale data and longer training schedules.
  • Method: Reduction cells use pyramid reduction with atrous convolutions at different dilation rates to embed multi-scale context into tokens.Three reduction cells reduce spatial resolution by 1/16 before normal cells learn discriminative features.
  • ViTAEv2: ViTAEv2 rearranges reduction and normal cells into a multi-stage design, producing multi-scale features for downstream vision tasks.This design supports object detection, semantic segmentation, and pose estimation.
  • Method: ViTAE introduces scale-invariance and locality inductive biases through two cell types: reduction cells and normal cells.Reduction cells embed multi-scale context, while normal cells jointly model locality and global dependencies.
  • Results: 88.5% Top-1 classification accuracy is achieved by ViTAE-H with 644M parameters on the ImageNet validation set without extra private data.The introduction also reports improved data efficiency, training efficiency, classification accuracy, and downstream generalization over representative vision transformers.

2 Related Work

Related work identifies locality and scale-invariance as important convolutional inductive biases, while existing transformer hybrids often model locality and global dependency sequentially. ViTAE instead incorporates these biases through parallel convolution and attention modules and extends the design to larger and multi-stage models.

  • Convolutional inductive bias: CNNs provide locality through neighborhood convolutions and scale-invariance through hierarchical and multi-scale feature extraction.Multi-scale features can be formed through inter-layer fusion or parallel convolutions with different receptive fields.
  • Vision transformers: Vision transformers lack these structural biases and therefore learn local structure and scale adaptation from data.The paper investigates encoding locality with convolutions and scale-invariance with multi-scale convolutions using different dilation rates.
  • Prior methods: Prior approaches introduce inductive bias through distillation, convolution-attention stacks, multi-stage structures, or local windows.Sequential structures may ignore global context during locality modeling and locality during global modeling.
  • ViTAE: ViTAE models locality and global dependencies simultaneously with parallel convolution and attention modules within each transformer layer.The modules complement each other for classification and dense prediction tasks.
  • Extensions: The paper extends ViTAE by scaling models to 644M parameters, introducing ViTAEv2, and exploring local window attention for efficiency.ViTAEv2 is reported to outperform state-of-the-art models on image classification and downstream vision tasks.

3 Methodology

ViTAE adapts vision transformers by combining self-attention with convolutional inductive biases for locality and multi-scale representation. Its reduction and normal cells organize downsampling, token formation, and joint local-global modeling in isotropic models.

  • Vision transformer review: Vision transformers split images into patch tokens and process them through sequential MHSA and FFN layers.MHSA forms queries, keys, and values from tokens, while FFN applies token-wise transformations.
  • Model variants: ViTAE stacks reduction and normal cells into isotropic models, with class-token processing and positional encoding preceding final classification.The paper also scales ViTAE variants and describes a masked-image pretraining procedure using reconstructed masked pixels.
  • Reduction cell: ViTAE uses reduction cells to downsample images while embedding multi-scale context and local information into visual tokens.Three reduction cells downsample spatial dimensions by 4×, 2×, and 2×, producing tokens at one-sixteenth the input resolution.
  • Reduction cell: Each reduction cell combines a pyramid reduction branch for multi-scale global context with a parallel convolutional branch for local context before the FFN.The pyramid branch uses convolutions with different dilation rates, while the parallel branch uses stride convolutions with the same downsampling ratio.
  • Reduction cell: Multi-scale convolutions give tokens broader contextual information before MHSA, helping model relationships across scales and reducing information loss during token merging.MHSA then models long-range dependencies over the resulting token sequence.
  • Normal cell: Normal cells omit the pyramid reduction module and further model local and long-range dependencies without changing token length.Removing the pyramid module also reduces computational cost because normal cells are numerous in stacked models.

4.1 Implementation details

The models are trained and evaluated on ImageNet1k using standardized optimization, augmentation, batching, and 300-epoch schedules.

  • ImageNet1k provides about 1.3 million images across 1,000 classes, with 224 × 224 training images.
  • AdamW with cosine learning-rate scheduling is used, while data augmentation follows T2T for fair comparison.
  • ViTAE uses batch size 512, whereas ViTAEv2 uses batch size 1024.
  • All models are trained for 300 epochs using PyTorch and TIMM.

4.2 Comparison with the state-of-the-art

ViTAE and ViTAEv2 are compared with similarly sized CNNs and vision transformers using ImageNet classification metrics, including higher-resolution fine-tuning settings.

  • 75.3% Top-1 accuracy is achieved by ViTAE-T, exceeding ResNet-18 despite ResNet-18 having more parameters.
  • 82.9% real Top-1 accuracy is achieved by ViTAE, comparable to ResNet-50 with four times more parameters.
  • 82.6% Top-1 accuracy is achieved by ViTAEv2-S, improving on ViTAE-S under the multi-stage design.
  • 1.2% absolute Top-1 accuracy improvement is obtained by ViTAE-S when fine-tuned with 384 × 384 images.

4.3 Analysis of the isotropic design of ViTAE

Experiments analyze ViTAE’s efficiency, downstream generalization, architectural components, visual attention, and layer-wise attention distance.

  • Data and training efficiency: ViTAE-T reaches 75.3% Top-1 accuracy after 300 epochs, with convolutional modules separating locality and multi-scale modeling from long-range dependency modeling.
  • Downstream generalization: ViTAE achieves state-of-the-art performance across fine-grained, CIFAR10, and CIFAR100 downstream classification tasks.
  • Ablation study: Early fusion with batch normalization in NC achieves 69.9% Top-1 accuracy among the tested settings.
  • Ablation study: 72.6% Top-1 accuracy is achieved when reduction and normal cells are combined, while PCM alone in RC raises accuracy to 71.7%.
  • Visual inspection: Grad-CAM shows ViTAE covers single and multiple targets more precisely, attends less to background, and handles different object scales better than T2T-ViT.
  • Attention-distance analysis: PCM allows shallow transformer layers to focus more on long-range dependencies, while deep-layer attention distances are nearly unchanged versus T2T-ViT-7.

4.4 Analysis of the scaled up ViTAE models

Scaled-up ViTAE models are evaluated with supervised and self-supervised pretraining, data-efficient fine-tuning, and convolutional-kernel ablations on ImageNet.

  • Training setup: Scaled-up models are pretrained for 1,600 epochs with MAE and then fine-tuned on labeled ImageNet1K data.
  • Image classification: 0.4 Top-1 accuracy improvement is obtained by ViTAE-B over ViT-B, while ViTAE-L gains 0.3 performance points.
  • Few-shot classification: With only 10% fine-tuning data, scaled-up ViTAE models obtain comparable or better performance than small CNNs and transformers trained with 100% data.
  • Image classification: 91.2% ImageNet Real Top-1 accuracy is achieved by ViTAE-H trained only on ImageNet1K.
  • Kernel-size ablation: Kernel-size ablations compare convolution branches during pretraining, including no branch, 1 × 1 kernels, and later padding to 3 × 3.

4.5 Analysis of the multi-stage design ViTAEv2

ViTAEv2 uses a multi-stage arrangement of reduction and normal cells to provide multi-scale features for downstream vision tasks. Its attention design balances accuracy, memory, speed, and performance across classification, detection, segmentation, and pose estimation.

  • Multi-stage design: ViTAEv2 rearranges reduction and normal cells across stages so multi-scale features are available for downstream tasks.Several normal cells follow each reduction module rather than being concentrated only at the final stage.
  • Attention design: The chosen W,W,F,F attention configuration provides the best trade-off between classification performance and computational cost across image resolutions.Here, W denotes local window attention and F denotes vanilla attention.
  • Attention design: Removing window shifting and relative position encoding changes accuracy by about 0.1%, motivating their exclusion from the default design.The default model therefore keeps the implementation simpler.
  • Classification and inference: ViTAEv2-S outperforms ViT-Small by 2.7% Top-1 accuracy and Swin by 1.3% absolute Top-1 accuracy, while remaining competitive in inference speed.Compared with Swin, ViTAEv2-S is about 10%∼20% slower in inference, but it outperforms T2T-ViT-24 in both performance and inference speed.
  • Downstream tasks: Across downstream tasks, ViTAEv2 achieves 50.6 AP b and 43.6 AP m after 12 epochs, and 45.0 mIoU for semantic segmentation with 10M fewer parameters.The reported downstream evaluations cover object detection, instance segmentation, semantic segmentation, and animal pose estimation.

4.6 Robustness

The paper connects ViTAE’s parallel convolution and attention structure to a possible robustness benefit, but states that the theoretical explanation remains to be established.

  • Robustness: ViTAE’s parallel PCM and attention modules extract local and global features in a complementary manner, motivating investigation of robustness to adversarial attacks.The passage frames adversarial robustness as an open empirical question rather than reporting a result.
  • Robustness: The theoretical foundation explaining why inductive bias may improve vision-transformer robustness remains an open issue.The paper does not provide that theoretical explanation in the supplied discussion.

5 Discussions

The discussion attributes ViTAE’s performance and efficiency to collaboration between convolutional and transformer components, while noting a framework-dependent training trade-off.

  • Discussion: ViTAE achieves fast convergence and high data efficiency, with attention-distance analysis indicating complementary roles for transformer and convolution layers.The transformer models long-range dependencies, while convolution layers focus on local features.
  • Discussion: The parallel structure may slow training in dynamic computation graph frameworks because parallel branches can be computed sequentially.Static computation graph frameworks are identified as one way to mitigate this issue.

6 Conclusion

The paper concludes that locality and scale-invariance inductive biases improve ViTAE and ViTAEv2 across classification and downstream vision tasks. It also identifies further inductive biases as a direction for future work.

  • Conclusion: ViTAE and ViTAEv2 incorporate locality and scale-invariance through reduction and normal cells, achieving superior performance and data efficiency.The multi-stage ViTAEv2 shows gains across classification accuracy, data efficiency, and downstream generalization.
  • Conclusion: ViTAEv2 obtains state-of-the-art performance on image classification, object detection, semantic segmentation, and pose estimation.The conclusion presents this as the broad outcome of the multi-stage design.
  • Conclusion: Future work may explore additional intrinsic or learnable inductive biases, including viewpoint invariance.The authors frame this as a way to investigate further improvements and understand intrinsic versus learned inductive bias.
Loading 2202.10108v2…