PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Yufei Xu, Qiming Zhang, Jing Zhang, Dacheng Tao

arXiv:2106.03348v4cs.CV

TL;DR

Vision transformers lack intrinsic biases for local structures and scale variation, often learning them from data. ViTAE incorporates convolution-inspired locality and scale-invariance through parallel convolution-attention cells and multi-scale token embedding, outperforming representative vision transformers across several evaluation dimensions.

  • Problem

    Vision transformers lack intrinsic inductive biases for modeling local visual structures and objects at different scales, requiring large-scale data to learn them implicitly.

  • Method

    ViTAE uses reduction and normal cells with multi-scale convolutional token embedding and parallel convolution-attention processing to model locality and long-range dependencies.

  • Results

    75.3% and 82.0% top-1 accuracy are achieved on ImageNet with 4.8M and 23.6M parameters, while outperforming representative vision transformers across multiple evaluation dimensions.

  • Takeaways & Limitations

    ViTAE provides a simple transformer architecture that incorporates locality and scale-invariance and shows broad gains in classification, efficiency, and downstream generalization.

  • Takeaways & Limitations

    ViTAE was not scaled or trained on large datasets such as ImageNet-21K and JFT-300M because of computational resource constraints.

Abstract

from arXiv · show

Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at GitHub.

1 Introduction

ViTAE addresses vision transformers’ lack of intrinsic locality and scale-invariance inductive biases by incorporating convolution-inspired structures that jointly model local and long-range features. It achieves 75.3% and 82.0% ImageNet top-1 accuracy with 4.8M and 23.6M parameters, respectively.

  • Vision transformers lack intrinsic inductive biases for modeling local structures and handling objects at various scales, instead learning these biases implicitly from large-scale data.
  • CNNs motivate ViTAE because their convolutions naturally provide locality and scale-invariance biases, extracting local features while supporting objects at different scales.
  • ViTAE uses reduction cells to embed multi-scale image context into tokens and normal cells to jointly model locality and global dependencies through parallel modules and feature fusion.
  • ViTAE outperforms representative vision transformers in classification accuracy, data efficiency, training efficiency, and downstream-task generalization.
  • ViTAE achieves 75.3% and 82.0% top-1 accuracy on ImageNet with 4.8M and 23.6M parameters, respectively.

2 Related Work

CNNs provide intrinsic locality bias through convolutional receptive fields, while ViT lacks such intrinsic bias and instead learns it implicitly from large-scale data. Subsequent transformer-based vision works simplify model structures and learn intrinsic biases directly from large-scale datasets.

  • CNNs: CNNs extract local features from neighboring pixels within kernel-defined receptive fields, giving them an intrinsic inductive bias for modeling locality.CNNs have driven breakthroughs in image classification and downstream computer vision tasks [94] [26] [95] [87].
  • Vision Transformers: ViT [19] pioneered pure transformers for vision and achieves promising results, but lacks intrinsic bias for local visual structures and learns it implicitly from large amounts of data.Follow-up works simplify model structures with fewer intrinsic biases and learn them directly from large-scale data [74] [75] [22] [18] [20].

3 Methodology

ViTAE introduces convolution-derived intrinsic locality and scale-invariance inductive biases through reduction cells (RCs) and normal cells (NCs). RCs embed multi-scale and local context while progressively downsampling images, and NCs further model locality and long-range dependencies with the same parallel attention–convolution structure but without pyramid reduction.

  • Architecture: ViTAE stacks three reduction cells and several normal cells, using RCs to embed multi-scale context and locality before NCs model token-level locality and long-range dependencies.The architecture uses parallel multi-head self-attention and convolutional modules followed by a feed-forward network; RCs additionally include a pyramid reduction module.
  • Reduction Cell: Three RCs progressively downsample the input by 4×, 2×, and 2×, producing feature maps of size [H/16, W/16, D] that are flattened into tokens for subsequent NCs.The resulting tokens are combined with a randomly initialized class token and positional encodings before entering the NC stack.
  • Reduction Cell: Each reduction cell combines a pyramid reduction module with dilated stride convolutions for multi-scale context and a parallel convolutional module for local context.The PRM concatenates convolutional features produced with predefined dilation and reduction ratios, while the PCM uses stacked convolutions and matching spatial downsampling.
  • Reduction Cell: The fused RC tokens carry both local and multi-scale context by design, providing intrinsic locality and scale-invariance inductive biases before feed-forward processing.After fusion, tokens pass through the FFN, are reshaped back to feature maps, and continue through the reduction-cell hierarchy.
  • Normal Cell: Normal cells retain the parallel attention–convolution design but omit the pyramid reduction module because the feature maps are already spatially reduced after the RCs.All NCs share an identical architecture with different learnable weights, and the class token is fixed during inference.

4 Experiments

Experiments on ImageNet and downstream datasets show that ViTAE outperforms comparable CNN and transformer baselines while improving data and training efficiency. Ablations and qualitative analyses support its convolution-derived locality and scale-robustness benefits.

  • ImageNet comparison: ViTAE-T achieves 75.3% Top-1 accuracy on ImageNet, outperforming ResNet-18 despite using more parameters in the baseline.The comparison includes CNNs, transformers with learned inductive bias, and transformers with introduced intrinsic inductive bias.
  • Data and training efficiency: With only 1/7 as many epochs, ViTAE-T achieves better Cifar10 classification performance than a model with 86M parameters while using 4.8M parameters.Training from scratch on Cifar10 and Cifar100 further demonstrates ViTAE’s data efficiency.
  • Downstream generalization: After fine-tuning, ViTAE achieves state-of-the-art performance on most evaluated fine-grained and Cifar datasets using comparable or fewer parameters.The downstream tasks include Flowers, Cars, Pets, iNaturalist19, Cifar10, and Cifar100.
  • Analysis: Using PCM lets ViTAE transformer layers focus more on long-range dependencies, especially in shallow layers, while convolutions model locality.The attention-distance analysis compares ViTAE-T with T2T-ViT-7 and supports the proposed divide-and-conquer design.
  • Qualitative analysis: Grad-CAM shows that ViTAE attends more precisely to single or multiple targets, less to background, and handles small, middle, and large birds across scales.The qualitative comparison is against T2T-ViT.

5 Limitation and discussion

ViTAE incorporates two inductive biases through reduction and normal cells, achieving strong ImageNet performance with fast convergence and high data efficiency, but remains untested at scale on very large datasets because of computational constraints.

  • 5 Limitation and discussion: ViTAE combines reduction and normal cells to incorporate two types of inductive biases into transformers.The two cells collaborate within the proposed model.
  • 5 Limitation and discussion: ViTAE achieves impressive ImageNet performance with fast convergence and high data efficiency.
  • 5 Limitation and discussion: Computational resource constraints prevented scaling ViTAE or training it on large datasets such as ImageNet-21K and JFT-300M [30].Its behavior on such large-scale training remains unclear.

6 Conclusion

ViTAE incorporates locality and scale-invariance inductive biases into transformers through reduction and normal cells, yielding a simple yet effective vision transformer. Extensive experiments show advantages over representative vision transformers in accuracy, data efficiency, training efficiency, and generalization.

  • 6 Conclusion: ViTAE redesigns transformer blocks with reduction and normal cells that introduce locality and scale-invariance intrinsic inductive biases.These cells form the proposed ViTAE vision transformer architecture.
  • 6 Conclusion: Extensive experiments show that ViTAE outperforms representative vision transformers in classification accuracy, data efficiency, training efficiency, and generalization.

A Appendix · A.1 Results of other ViTAE variants

Appendix experiments compare additional ViTAE variants on ImageNet, including compact models that outperform larger transformer methods and ResNet-50. Stage-wise variants are also introduced to better support downstream vision tasks and evaluate RC and NC modules.

  • A.1 Results of other ViTAE variants: 75.3% Top-1 accuracy on ImageNet with 4.8M parameters lets ViTAE-T outperform transformer methods exceeding 5M parameters.The comparison is reported for additional ViTAE variants in Table 6.
  • A.1 Results of other ViTAE variants: 77.9% Top-1 accuracy with 6.5M parameters gives ViTAE a 1.2% absolute improvement over ResNet-50 while using three-quarters fewer parameters.These results are reported on ImageNet.
  • A. Appendix: The appendix designs three additional ViTAE variants to enable fair comparisons with other methods.Their results are presented in Table 6.
  • A.1 Results of other ViTAE variants: The classic vision transformer design is described as poorly suited to downstream detection, segmentation, and pose-estimation tasks.The passage motivates adapting ViTAE’s architecture for these applications.
  • A.1 Results of other ViTAE variants: Stage-wise ViTAE variants are designed because stage-wise architectures better adapt to popular vision backbones for downstream tasks.The variants are shown in Table 7.
  • A.1 Results of other ViTAE variants: These stage-wise variants further explore the proposed RC and NC modules, with their classification performances summarized in Table 6.“NC Arrangement” denotes the number of NCs arranged after each stage, though the supplied passage truncates the explanation.

A.2 Performance on downstream tasks

The proposed ViTAE models are further evaluated across four downstream computer-vision tasks: detection, segmentation, pose estimation, and video object segmentation.

  • A.2 Performance on downstream tasks: ViTAE is validated on downstream computer-vision tasks beyond its primary evaluation.The evaluation covers detection, segmentation, pose estimation, and video object segmentation.
  • A.2 Performance on downstream tasks: The downstream evaluation includes object detection and image segmentation.
  • A.2 Performance on downstream tasks: It also covers pose estimation and video object segmentation.

A.3 Object detection

The object detection and instance segmentation evaluation fine-tunes ViTAE within Mask RCNN and Cascade RCNN [5] on COCO 2017 under Swin [47]’s training setting, comparing it with ResNet [26] and transformer backbones.

  • A.3 Object detection: ViTAE is evaluated for object detection and instance segmentation using Mask RCNN and Cascade RCNN [5] on COCO 2017, with multi-scale training and AdamW matching Swin [47].The dataset contains 118K training, 5K validation, and 20K test-dev images; comparisons include the classic CNN backbone ResNet [26] and transformer structures.

A.4 Semantic segmentation … A.10 More visual results.

Across downstream vision tasks, ViTAE is evaluated under standard task-specific frameworks and shows parameter-efficient gains or improved generalization. Additional analyses indicate that its inductive-bias modules support data and training efficiency, encode location information without explicit position embedding, and improve visual target coverage amid complex backgrounds.

  • A.4 Semantic segmentation: ViTAE is evaluated for semantic segmentation on ADE20K using UperNet, 160K training iterations, and mmsegmentation’s default settings.ADE20K contains 150 semantic categories, with 20K training images and 2K validation images; the evaluation follows Swin’s training and testing setting.
  • A.5 Pose estimation: On COCO pose estimation, the ViTAE-based model achieves an absolute 2% mAP gain over ResNet models with 7M fewer parameters.The experiment uses the simple baseline framework and follows mmpose’s default settings.
  • A.6 Video object segmentation: ViTAE’s intrinsic inductive bias improves backbone generalization across downstream tasks, including video object segmentation.The STM framework with a ViTAE-T-Stage backbone is evaluated on Davis-2016 and Davis-2017 under STM’s training and testing settings.
  • A.7 More comparisons of data efficiency and training efficiency.: With inductive bias, T2T-ViT achieves better performance than DeiT using less data and fewer training epochs on ImageNet.The comparison trains DeiT with 20%, 60%, and 100% of the data for 100 epochs, and with all data for 100, 200, and 300 epochs.
  • A.8 Analysis of position embedding: Removing position embedding does not downgrade ViTAE’s performance, indicating that its PCM and PRM modules help encode location information.The ViTAE model without position embedding is trained for 300 epochs and evaluated against variants with different position embeddings.
  • A.9 More Ablation Studies: Adding a third identical convolution layer to PCM downgrades performance by 3%, showing that the proposed cell structure is sensitive to normalization and layer design.This ablation adds another batch normalization layer so PCM contains three identical convolution layers, as depicted in Figure 6 and evaluated in Table 10.
  • A.10 More visual results.: Grad-CAM visualizations show that ViTAE covers targets more precisely and suppresses noise from complex backgrounds compared with T2T-ViT.The visual results support better adaptation to targets under different situations.
Loading 2106.03348v4…