PaperlayerView exact revision on arXiv

Source-linked AI summary

Bottleneck Transformers for Visual Recognition

Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani

arXiv:2101.11605v2cs.CVcs.AIcs.LG

TL;DR

Vision recognition needs long-range dependency modeling, but global self-attention is costly at high image resolutions. BoTNet replaces late ResNet spatial convolutions with global MHSA in a hybrid backbone, achieving strong segmentation and classification results while retaining a simple design.

  • Problem

    Detection and segmentation require long-range dependencies, while self-attention introduces quadratic memory and computation costs at their larger image resolutions.

  • Method

    BoTNet uses convolutions for downsampling and abstract feature extraction, then replaces the final three ResNet bottleneck spatial convolutions with global MHSA.

  • Results

    BoTNet improves instance segmentation and detection results and reaches 44.4% mask AP, while classification models reach 84.7% top-1 accuracy and are up to 1.64x faster than EfficientNet on TPU-v3.

  • Takeaways & Limitations

    ResNet bottleneck blocks with MHSA can be viewed as Transformer blocks, making BoTNet a simple hybrid backbone for visual recognition.

  • Takeaways & Limitations

    Global self-attention remains constrained by quadratic memory and computation, especially at the larger resolutions used for detection and segmentation.

Abstract

from arXiv · show

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in compute time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision

1. Introduction

BoTNet addresses the need for long-range visual dependencies by combining convolutional feature extraction with global self-attention in ResNet bottlenecks. Replacing only the final three spatial convolutions improves segmentation and classification results with limited overhead.

  • Motivation: Global dependencies matter for detection and segmentation because convolution captures local information and requires stacked layers to aggregate broader context.An explicit non-local mechanism may model these dependencies without requiring as many layers.
  • Conceptual framing: BoT blocks can be viewed as Transformer blocks with a bottleneck structure, linking MHSA-based ResNet blocks and Transformer architectures.This interpretation differs only through minor choices such as residual connections and normalization layers.
  • Instance segmentation: Replacing the final three 3 × 3 convolutions with MHSA improves COCO mask AP by 1.2% over a canonical ResNet-50 Mask R-CNN baseline.The comparison uses no hyperparameter differences and incurs minimal training and inference overhead.
  • Instance segmentation: 44.4% mask AP is achieved on COCO instance segmentation without extra architectural or training modifications, with gains reported across configurations and backbones.The reported improvements include +2.4 Mask AP and +2.6 Box AP on small objects.
  • Image classification: 84.7% top-1 accuracy is achieved on ImageNet, with BoTNet models up to 1.64x faster in compute time than EfficientNet on TPU-v3.The classification adaptation follows scaling strategies after smaller-scale training showed limited gains.

2. Related Work

The related-work discussion positions BoTNet as a hybrid backbone that combines convolution and self-attention, while distinguishing it from Transformer-based frameworks outside the backbone and from Non-Local designs.

  • Architecture taxonomy: BoTNet is a hybrid architecture using both convolutions and self-attention within the backbone, unlike pure attention models and frameworks that place attention outside it.The taxonomy contrasts BoTNet with ViT-like pure attention models and DETR-like outside-backbone Transformer use.
  • Transformer connection: ResNet bottleneck blocks with MHSA can be interpreted as Transformer blocks with a bottleneck structure, which the paper names Bottleneck Transformer blocks.The paper presents this as a conceptual relationship rather than a new block construction.
  • DETR connection: DETR uses Transformer blocks outside the backbone to replace region proposals and non-maximal suppression, whereas BoTNet supplies a Transformer-like backbone for R-CNN systems.The experiments use Mask R-CNN and Faster R-CNN, leaving integration with DETR for future work.
  • Non-Local connection: BoTNet differs from Non-Local networks by replacing existing convolutional blocks rather than inserting additional blocks into the ResNet backbone.MHSA also adds multiple heads, value projection, and position encodings, while using the ResNet bottleneck channel factor.

3. Method

BoTNet replaces late-stage ResNet spatial convolutions with global MHSA, using relative position encodings and operating attention on downsampled featuremaps to control cost.

  • BoTNet-50: BoTNet-50 uses MHSA in c5, operating on 64 × 64 features in the first block and 32 × 32 features in the remaining two for 1024 × 1024 inputs.This places global attention after convolutional downsampling.
  • Efficiency: BoT50 has 1.2x more multiply-adds, 1.3x training-throughput overhead, and 1.2x fewer parameters than R50.The reported comparison is for the TPU-v3 steptime and architecture described in Table 1.
  • BoTNet backbone: BoTNet replaces the final three spatial 3 × 3 convolutions in a ResNet with global all2all MHSA layers over 2D featuremaps.The ResNet stages c2–c5 use strides 4, 8, 16, and 32 relative to the input image.
  • MHSA layer: MHSA computes attention logits as qkT + qrT and uses four heads with split relative position encodings for height and width.The figure defines q, k, and r as query, key, and position encodings, respectively.
  • Efficiency: Global self-attention has O(n^2d) memory and computation, so BoTNet uses convolutions for downsampling before attention on smaller featuremaps.This hybrid design targets high-resolution detection and segmentation settings.

4. Experiments

The experiments evaluate ResNet-50 and BoTNet-50 backbones for COCO instance segmentation under matched training settings. BoTNet-50 improves over ResNet-50 except under the shortest schedule, with longer training generally needed to reveal its gains.

  • Experimental setup: COCO experiments use standard box and mask AP metrics with shared training infrastructure and matched backbone hyperparameters.The evaluation uses COCO train and validation splits, TPU-based Detectron infrastructure, and FPN with ResNet-50 or BoTNet-50 backbones.
  • Training schedules: BoTNet-50 is a significant improvement over ResNet-50 except under the 1x, 12-epoch schedule.Both backbones use the same hyperparameters across 1x, 2x, 3x, and 6x schedules.
  • Training schedules: 72 epochs produce smaller gains than 36 epochs under the default scale jitter.The authors address this reduction with more aggressive multi-scale jitter in the following experiment.

4.2. Scale Jitter helps BoTNet more than ResNet

Scale jitter strengthens BoTNet’s advantage over ResNet-50 in the 72-epoch setting. The gains are larger with aggressive jitter, suggesting self-attention benefits more from additional multi-scale augmentation.

  • Scale jitter results: +2.1% APbb and +1.7% APmk favor BoT50 over R50 with scale jitter [0.5, 2.0].Both models are trained for 72 epochs at 1024 × 1024 resolution.
  • Scale jitter results: +2.2% APbb and +1.6% APmk are reported for BoT50 with scale jitter [0.1, 2.0].This setting also uses the 6x, 72-epoch schedule.
  • Interpretation: BoTNet benefits more from extra multi-scale jitter than the pure-convolution ResNet baseline.The comparison is made across the reported jitter settings under otherwise matched training conditions.

4.3. Relative Position Encodings Boost Performance

Relative position and content interactions both improve BoTNet’s detection and segmentation metrics, with relative position contributing more individually and combined gains being additive.

  • Ablation results: 0.6% APbb and 0.6% APmk come from standalone content-content interaction qkT over the R50 baseline.The ablation compares content-content and content-position interactions in MHSA.
  • Combined interactions: Combining qkT and qrT yields additive gains of 1.5% APbb and 1.2% APmk.Absolute position encodings provide less gain than relative position encodings.

4.4. BoTNet improves backbones in ResNet Family

BoTNet functions as a drop-in replacement across ResNet-50, ResNet-101, and ResNet-152 backbones. Replacing three spatial convolutions with global attention can outperform deeper convolutional alternatives on reported COCO metrics.

  • Backbone scaling: BoTNet is applicable as a drop-in replacement for ResNet-50, ResNet-101, and ResNet-152 backbones.The comparison uses the canonical 36-epoch schedule, 1024 × 1024 images, and multi-scale jitter [0.8, 1.25].
  • Comparison with deeper ResNets: +0.3% APbb and +0.5% APmk make BoT50 better than R101.BoT50 is also competitive with R152 on APmk.
  • Comparison with deeper ResNets: Replacing three convolutions with all2all attention improves metrics more than adding 50 convolutional layers and is competitive with adding 100.The comparison supports the hypothesis that attention captures long-range dependencies more effectively than stacking convolutional layers.

4.5. BoTNet scales well with larger images

BoTNet benefits from larger training images and achieves strong COCO instance-segmentation results without architectural bells and whistles. Its compute efficiency also compares favorably with deeper ResNet baselines.

  • 49.7% APbb and 44.4% APmk are achieved by BoT200 trained with 1280 × 1280 images.The model uses 72 epochs and multi-scale jitter of [0.1, 2.0].
  • BoTNet benefits from larger training images across R50, R101, and R152 backbones.BoTNet trained at 1024 × 1024 also outperforms baseline ResNet trained at 1280 × 1280.
  • BoT200’s 49.7% APbb and 44.4% APmk surpass the previous best published single-model, single-scale ResNeSt result of 44.2% APmk.The result is reported without modified FPN or Cascade R-CNN.
  • BoT50 is better than R101 and competitive with R152 despite performing significantly less computation.BoT50 uses 121 × 109 M.Adds, compared with 162.99 × 109 B M.Adds for R101 and 240.56 × 109 M.Adds for R152.

4.6. Comparison with Non-Local Neural Networks

BoTNet compares favorably with Non-Local networks by reducing parameters while delivering stronger detection and segmentation gains in the reported ablations.

  • +1.5 APbb and +1.2 APmk are obtained by BoT-R50, which replaces blocks rather than adding new ones.This performs as well as adding another BoT block and better than adding one additional Non-Local block.
  • Non-Local operations add model parameters, whereas BoTNet reduces them.The comparison is reported for insertions in the c4 stack of a ResNet backbone.

4.7. Image Classification on ImageNet

Under canonical ImageNet training, BoTNet-50 provides little accuracy gain over ResNet-50, but scaled and better-trained BoTNet variants achieve stronger results. At larger scales, BoTNet reaches 84.7% top-1 accuracy with a reported 1.64x speedup over EfficientNet.

  • 4.7.1 BoTNet-S1 architecture: BoTNet-50 does not provide significant gains over ResNet-50 under canonical ImageNet training.Both use 100 epochs, batch size 1024, weight decay 1e-4, standard ResNet augmentation, and a cosine learning-rate schedule.
  • 4.7.2 Evaluation in the standard training setting: 0.9% improvement is achieved by BoT-S1-50 over R50 in the regular 100-epoch setting, at the cost of more computation.The comparison is made alongside BoT50 and R50.
  • 4.7.3 Evaluation in the improved training setting: +1.4% and +0.6% gains are reported for BoT-S1-50 and BoT50, respectively, over R50 under improved training.The setting uses 200 epochs, batch size 4096, RandAugment, weight decay 8e-5, and label smoothing of 0.1.
  • 4.7.4 Scaling BoTNets: 84.7% top-1 accuracy is achieved by BoTNet T7, matching B7-RA with a 1.64x speedup in efficiency.BoTNets scale better beyond 83% top-1 accuracy, while T3 and T4 do not outperform SENets.
  • 4.7.4 Scaling BoTNets: BoTNets perform better than DeiT-384, supporting the reported advantage of combining convolutions with self-attention over pure attention models on ImageNet-1K.The comparison includes EfficientNet and DeiT under strong augmentation, regularization, and long training schedules.

5. Conclusion

The paper identifies several directions for extending and comparing self-attention-based vision backbones, including new tasks, self-supervised learning, larger datasets, and alternative mechanisms.

  • Future work includes applying self-attention to keypoint detection and 3D shape prediction.
  • Self-attention architectures for self-supervised computer vision learning and larger datasets such as JFT, YFCC, and Instagram remain open directions.
  • Comparing with and incorporating alternatives to self-attention, including lambda-layers, is identified as an important future direction.
Loading 2101.11605v2…