PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Plain Vision Transformer Backbones for Object Detection

Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He

arXiv:2203.16527v2cs.CV

TL;DR

Object detection typically adapts backbones to hierarchical, multi-scale processing, creating a question for the original single-scale ViT. This paper fine-tunes plain ViT backbones with a simple single-map pyramid and sparse cross-window propagation, achieving competitive results including 61.3 AP_box on COCO with ImageNet-1K MAE pre-training.

  • Problem

    Plain ViT backbones are single-scale and non-hierarchical, raising challenges for multi-scale and high-resolution object detection while hierarchical redesigns couple pre-training to detection needs.

  • Method

    ViTDet minimally adapts plain ViT backbones during fine-tuning with a simple pyramid, non-shifted window attention, and a few cross-window propagation blocks.

  • Results

    61.3 AP_box is reported on COCO with a plain ViT-Huge backbone using only ImageNet-1K MAE pre-training without labels, while competing with hierarchical-backbone detectors.

  • Takeaways & Limitations

    Plain-backbone detection can preserve separation between general-purpose pre-training and task-specific fine-tuning while remaining competitive across detector frameworks and datasets.

  • Takeaways & Limitations

    The study uses standard detection-specific components and leaves exploration of fewer inductive biases in detection heads for future work.

Abstract

from arXiv · show

We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors. Code for ViTDet is available in Detectron2.

1 Introduction

The paper explores plain, non-hierarchical ViT backbones for object detection, preserving separation between pre-training and detection fine-tuning. Minimal adaptations—a simple pyramid and limited cross-window propagation—make this approach competitive with hierarchical detectors.

  • Motivation: Plain ViT maintains a single-scale feature map, creating challenges for multi-scale detection and high-resolution inputs.Hierarchical redesigns address these challenges but abandon the original plain-backbone architecture.
  • Approach: ViTDet builds a simple feature pyramid from only the backbone’s last feature map, without FPN or a hierarchical backbone.The pyramid uses a large-stride feature map and is introduced only during fine-tuning.
  • Approach: Window attention without shifting is sufficient when a small number of cross-window blocks propagates information.The propagation blocks can use global attention or convolutions.
  • Results: 61.3 AP_box is achieved on COCO with a plain ViT-Huge backbone using only ImageNet-1K MAE pre-training without labels.The detector also shows competitive results on LVIS and across multiple detector frameworks.
  • Results: MAE-pre-trained ViTDet can outperform or compete with hierarchical-backbone detectors, especially at larger model sizes.The paper positions this result as evidence that plain-backbone detection is promising.
  • Implications: The approach keeps detection-specific designs in fine-tuning, supporting future ViT advances that are not constrained by hierarchy.The authors connect this decoupling to block designs, self-supervised learning, and scaling.

2 Related Work

Prior detection research largely used ConvNet or hierarchical Transformer backbones, while plain-backbone studies explored whether original ViT architectures could support detection without detection-specific pre-training design.

  • Object detector backbones: Object detectors commonly combine task-agnostic pre-trained backbones with task-specific necks and heads.Examples include RoI operations, RPNs or anchors, and FPNs.
  • Object detector backbones: FPN extends single-scale detection by combining features from multiple stages of hierarchical ConvNet backbones.It uses lateral and top-down connections to integrate higher-resolution and lower-resolution features.
  • Transformer backbones: Hierarchical Transformers such as Swin, MViT, PVT, and PiT inherit hierarchical or translation-equivariant designs that ease adaptation to detection.These designs include convolutions, pooling, and sliding windows.
  • Plain-backbone detectors: UViT studies plain ViT backbones for detection but modifies architecture during pre-training, unlike this work’s task-agnostic approach.The paper emphasizes compatibility with existing and future ViT improvements.
  • Plain-backbone detectors: Unlike UViT’s single-scale detector heads, this method builds a simple pyramid on the single-scale backbone without constraining neck or head design.The study uses pre-trained plain backbones while retaining flexible detector components.
  • Object detection methodologies: The paper frames plain versus hierarchical backbones as an additional object-detection design dimension alongside established methodology choices.Examples include two-stage versus one-stage and anchor-based versus anchor-free detectors.

3 Method

The method adapts an original plain ViT for detection only during fine-tuning, using a single-map feature pyramid and efficient window attention with sparse cross-window propagation.

  • Design goal: The detector uses standard Mask R-CNN or Cascade Mask R-CNN heads while minimizing modifications to the pre-trained plain backbone.The goal is feasibility rather than introducing new detector components.
  • Simple feature pyramid: Because a plain backbone lacks different-resolution stages, the method uses only its strongest final feature map rather than FPN lateral and top-down connections.The paper reports that simple deconvolutions suffice without lateral connections.
  • Simple feature pyramid: A simple feature pyramid generates multi-scale maps from the final stride-16 feature map using parallel convolutions and deconvolutions.The resulting scales are 1/32, 1/16, 1/8, and 1/4.
  • Simple feature pyramid: The study compares two plain-backbone FPN variants against the proposed simple pyramid: one divides the backbone into stages, and the other uses only the last map.These variants are evaluated as alternatives to the design in Figure 2(c).
  • Backbone adaptation: High-resolution fine-tuning replaces global attention throughout the backbone with regular non-overlapping window attention.The adaptation addresses the memory and speed cost of global attention on high-resolution images.
  • Backbone adaptation: Unlike Swin, the method does not shift windows and instead inserts a few cross-window propagation blocks among subsets of pre-trained blocks.Propagation uses either global self-attention or convolutional residual blocks, with four blocks as the default.
  • Backbone adaptation: These adaptations preserve compatibility with global-self-attention pre-training and avoid redesigning the pre-training architecture.The backbone remains task agnostic while detection-specific structure is added during fine-tuning.
  • Implementation: The implementation uses vanilla ViT-B, ViT-L, and ViT-H backbones with patch size 16 and Mask R-CNN-family detector heads.Training uses 1024×1024 inputs and large-scale jittering.

4 Experiments

Ablations show that a simple pyramid from a single-scale plain ViT map and window attention with limited cross-window propagation are sufficient for strong detection. With MAE pre-training, ViTDet scales competitively against hierarchical backbones while retaining practical fine-tuning adaptations.

  • Feature pyramid design: A simple feature pyramid is sufficient for plain ViT detection, matching FPN variants while avoiding their top-down and lateral connections.The authors attribute this to the importance of having pyramidal feature maps rather than FPN-specific connections.
  • Feature pyramid design: 54.5 AP demonstrates that an aggressively simple pyramid from a single-scale map can nearly match richer pyramid designs.The result uses ViT-L and is 3.3 AP higher than the no-pyramid baseline.
  • Backbone adaptation: Cross-window propagation improves window-attention baselines, with global and convolutional strategies gaining 1.7 and 1.9 points, respectively.A shifted-window variant gains 1.1 points but performs worse than the proposed propagation strategies.
  • Backbone adaptation: Even without backbone propagation, the detector reaches 52.9 AP because the feature pyramid, RPN, and RoI heads also induce cross-window communication.This explains why pure window attention remains reasonably effective as a baseline.
  • Backbone adaptation: Propagation placed in the last four blocks is nearly as good as even placement, whereas placing it only in the first four blocks provides no gain.The findings support maintaining cross-window communication late in the backbone.
  • Backbone adaptation: Four propagation blocks provide a practical trade-off: convolutional propagation increases memory and time by ≤5% with 4% more parameters, while all-block global attention is impractical.Using all 24 blocks gives only a marginal 0.5-point gain over the four-block default and requires special memory optimization.
  • Pre-training: MAE pre-training on ImageNet-1K increases APbox by 3.1 for ViT-B and 4.6 for ViT-L over the compared pre-training settings.The paper reports that MAE provides particularly strong gains for the plain ViT backbone.
  • Comparisons with hierarchical backbones: ViTDet shows better scaling behavior than hierarchical counterparts, including a 2.6 APbox advantage for ViT-H over MViTv2-H.The plain ViT also has better wall-clock performance in the reported ViT-H versus MViTv2-H comparison.

5 Conclusion

The exploration demonstrates that plain-backbone detection can preserve independence between general-purpose pre-training and downstream task-specific designs, while benefiting from readily available MAE-pre-trained models.

  • Plain-backbone detection is a promising research direction.
  • The methodology largely maintains independence between general-purpose backbones and downstream task-specific designs.
  • The detector benefits from readily available pre-trained models produced by MAE.

A.1 Additional Ablation Results

Additional experiments show that ViTDet’s main trends persist across backbone size, dataset, detector head, and pre-training setting. The results also indicate favorable scaling behavior and gains from MAE pre-training.

  • ViT-B shows good gains from multiple propagation strategies compared with using no propagation.This matches the observations reported for ViT-L.
  • The supplementary measurements add FLOPs, parameter counts, and inference time for the results plotted in Figure 3.
  • MAE pre-training produces large gains over supervised pre-training on LVIS.
  • IN-21K supervised pre-training gains more on LVIS because it significantly improves rare-category detection.
  • RetinaNet experiments show similar trends and better scaling behavior for the plain-backbone detector than for the compared hierarchical-backbone configurations.The comparison uses Swin and MViTv2 backbones without hyper-parameter tuning; ViT-H gains +3.4 APb in the reported example, whose sentence is truncated in the supplied passage.

A.2 Implementation Details

The implementation uses a simple pyramid and standard detection frameworks with shared implementation choices for plain and hierarchical backbones. Experiments specify COCO and LVIS training settings, ViT patch adaptation, propagation configurations, and supplementary measurement conventions.

  • Architectures: The feature pyramid uses ViT’s final map at 1/16 resolution, max pooling for 1/32, and deconvolution for 1/8 or 1/4 scales.Each pyramid level then uses 1×1 and 3×3 convolutions with LayerNorm, while the 1/4 case adds LayerNorm and GeLU after the first deconvolution.
  • Architectures: ViTDet is evaluated with Mask R-CNN, Cascade Mask R-CNN, and RetinaNet using shared detection implementation choices for plain and hierarchical backbones.Mask R-CNN and Cascade Mask R-CNN use two hidden RPN convolutions and four hidden RoI-head convolutions, followed by LayerNorm.
  • Architectures: All ViT backbones use 16×16 patches, with ViT-H patch embeddings interpolated from 14×14 after pre-training.
  • Ablations: The ablations vary propagation strategy, cross-window block locations, and the number of global propagation blocks.
  • Training settings: COCO training uses 1024×1024 inputs, large-scale jitter from 0.1 to 2.0, AdamW, warm-up for 250 iterations, and a batch size of 64 across 64 GPUs.
  • Training settings: The experiments search learning rate, weight decay, drop path, and epochs by model size and backbone type, with MAE ViT models also using layer-wise learning-rate decay.The decay gives up to 0.3 AP for MAE-pre-trained ViT models and no observed gain for hierarchical backbones or supervised-pre-trained ViT.
  • Training settings: LVIS fine-tuning runs for 100 epochs with specified ViT-L/H learning rates, repeat-factor sampling, a 0.02 score threshold, and up to 300 detections per image.
  • Pre-training: Hierarchical-backbone MAE uses encoder mask tokens, unlike plain ViT’s efficient token-skipping strategy, resulting in slower training.The paper states that extending the token-skipping strategy to hierarchical backbones is beyond its scope.
Loading 2203.16527v2…