Source-linked AI summary
Benchmarking Detection Transfer Learning with Vision Transformers
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, Ross Girshick
TL;DR
The paper addresses missing transfer-learning benchmarks for standard ViTs in object detection by adapting them to Mask R-CNN and comparing five initializations on COCO. Masking-based methods achieve gains over supervised and prior self-supervised methods, with larger improvements as model size increases.
Problem
Object detection transfer benchmarks for standard ViTs were not established because of memory, architectural, and training challenges.
Method
The paper develops techniques for using standard ViTs as Mask R-CNN backbones and evaluates five initializations on COCO.
Results
Masking-based methods BEiT and MAE gain up to 4% absolute box AP over supervised and prior self-supervised methods, with gains increasing by model size.
Takeaways & Limitations
Masking-based initialization provides convincing COCO transfer-learning improvements, while supervised and MoCo v3 do not show the same scaling behavior.
Takeaways & Limitations
The results may improve with longer training and more complex formulas, while exhaustive drop-path tuning is computationally impractical.
Abstract
from arXiv · showhide
Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed. The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive. These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models. In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN. These tools facilitate the primary goal of our study: we compare five ViT initializations, including recent state-of-the-art self-supervised learning methods, supervised initialization, and a strong random initialization baseline. Our results show that recent masking-based unsupervised learning methods may, for the first time, provide convincing transfer learning improvements on COCO, increasing box AP up to 4% (absolute) over supervised and prior self-supervised pre-training methods. Moreover, these masking-based initializations scale better, with the improvement growing as model size increases.
1. Introduction
The paper addresses the lack of established object-detection transfer benchmarks for standard ViTs by creating a COCO Mask R-CNN protocol and overcoming integration challenges.
- Unsupervised pre-training is evaluated by downstream metrics against supervised pre-training and random initialization baselines.
- Object detection was not explored in pioneering ViT training work because integrating ViTs into common detectors such as Mask R-CNN was challenging.
- The paper establishes a COCO protocol for evaluating ViT transfer learning on object detection and instance segmentation with Mask R-CNN.
- ViT detection benchmarking requires addressing memory demands, single-scale versus multi-scale architecture, and training formulae across initializations.
- The practices are intended as a blueprint for comparing pre-training methods on advanced ViT derivatives, with code planned for Detectron2.
2. Approach
The approach adapts standard ViTs for Mask R-CNN by resolving feature-pyramid compatibility and quadratic attention costs, while using upgraded components and a shared training protocol.
- Mask R-CNN is chosen as a relatively simple, widely used framework that provides compelling though not entirely state-of-the-art results.
- The detector uses upgraded Mask R-CNN modules and a training formula with normalization, large-scale jitter, and drop path regularization.
- The approach addresses ViT’s single-scale outputs and high memory and runtime costs when integrating with an FPN-based detector.
- Four resolution-modifying modules upsample or downsample intermediate ViT feature maps at equally spaced d/4-block intervals for FPN integration.
- Windowed self-attention reduces complexity from global quadratic computation by computing attention independently within non-overlapping local windows.
- The same formula trains pre-trained models for up to 100 epochs and scratch models for up to 400 epochs because convergence is slower without pre-training.
- Hyperparameter tuning uses practical shortcuts: learning rate and weight decay are tuned separately from drop path, only on ViT-B, and on shorter schedules.
- Results are tuned and reported on the same COCO 2017 validation split, which the authors identify as technically non-best-practice.
3. Initialization Methods
The study compares random, supervised, contrastive self-supervised, and masking-based ViT initializations, while documenting pre-training differences that complicate direct matching.
- The benchmark compares five initialization methods: random, supervised, MoCo v3, BEiT, and MAE.
- Random initialization uses no pre-training, whereas supervised initialization uses ImageNet-1k images and labels.
- MoCo v3 uses unsupervised ImageNet-1k weights, while BEiT and MAE use unsupervised ImageNet-1k pre-training with different training procedures.
- The comparison retains each method’s default pre-training epoch count because methods may differ in benefits from longer training and per-epoch computational cost.
- BEiT’s relative position biases and layer scale require corresponding parameterization during detection fine-tuning, unlike the other initializations.
- BEiT additionally uses a DALL·E dVAE trained on approximately 250 million proprietary and undisclosed images, creating an incompletely understood data difference.
4. Experiments and Analysis
The experiments compare ViT initializations and detector designs on COCO, showing faster convergence for pre-trained models and stronger transfer from masking-based methods, especially at larger scales.
- Comparing Initializations: 50.7 vs. 49.3 APbox: training from scratch outperforms supervised IN1k initialization by up to 1.4 APbox.The results show that supervised pre-training is not always stronger than random initialization.
- Comparing Initializations: MoCo v3 underperforms random initialization on APbox and performs similarly to supervised initialization.
- Comparing Initializations: 50.3 vs. 48.9 APbox: BEiT and MAE outperform random initialization by up to 1.4 APbox for ViT-B.
- Comparing Initializations: 53.3 vs. 49.3 APbox: BEiT and MAE outperform supervised initialization by up to 4.0 APbox for ViT-L.For ViT-L, the masking-based methods also outperform random initialization by up to 2.6 APbox.
- Comparing Initializations: Pre-training accelerates COCO fine-tuning by roughly 4× compared to random initialization.Pre-trained initializations typically overfit by 100 epochs, whereas random initialization typically requires 400 epochs.
- Ablations and Analysis: Multi-scale FPN increases APbox by approximately 1.3–1.7 while increasing training and inference time by approximately 5% and 10%, respectively.Its memory overhead is below 1%.
- Ablations and Analysis: Including relative position biases may improve APbox by approximately 0.2–0.3 points but increases training time, inference time, and memory by roughly 25%, 15%, and 15%.The authors describe this positional-information choice as a reasonably fair comparison but note its non-trivial overhead.
- Ablations and Analysis: MAE pre-training continues producing large APbox gains through 800 epochs, with only a small +0.2 improvement from 800 to 1600 epochs.
5. Conclusion
The paper makes standard ViT backbones practical for Mask R-CNN and benchmarks five initialization methods on COCO. Masking-based methods outperform supervised and random initialization, with gains increasing for larger models.
- Training techniques enable standard ViT models to serve practically as Mask R-CNN backbones while maintaining acceptable training memory and time.These techniques also yield effective training formulae for benchmarking five ViT initialization methods.
- Masking-based methods BEiT and MAE outperform both supervised and random initialization, with gains increasing as model size increases.This scaling behavior is not observed with supervised or MoCo v3 initialization.