Source-linked AI summary
You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu
TL;DR
The paper asks whether a canonical ViT can perform 2D object detection from a pure sequence-to-sequence perspective with minimal spatial knowledge. YOLOS answers by minimally modifying ViT with detection tokens and set-based matching, and finds promising COCO transfer from ImageNet-1k pre-training. The study also identifies sensitivity to pre-training and unresolved scaling issues.
Problem
The paper asks whether ViT can transfer from image-level recognition to the more challenging object- and region-level task of 2D object detection.
Method
YOLOS minimally modifies canonical ViT by replacing [CLS] with [DET] tokens and classification loss with bipartite matching for sequence-based object detection.
Results
YOLOS demonstrates that 2D object detection can be accomplished in a pure sequence-to-sequence manner using fixed-sized image patches and minimal 2D inductive biases.
Takeaways & Limitations
YOLOS indicates the versatility and transferability of Transformer representations from image recognition to downstream object detection.
Takeaways & Limitations
YOLOS still requires 150 epochs of transfer learning, and its detection results are far from saturating, leaving substantial room to improve pre-trained representations.
Abstract
from arXiv · showhide
Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS. Code and pre-trained models are available at https://github.com/hustvl/YOLOS.
1 Introduction
YOLOS asks whether a canonical, pre-trained ViT can transfer from image recognition to 2D object detection without substantial spatial inductive bias. It answers with a minimally modified sequence-to-sequence detector and studies transfer sensitivity to pre-training and scaling.
- Motivation: YOLOS addresses whether ViT can transfer from image-level recognition to more challenging object- and region-level tasks such as object detection.
- Related work: ViT-FRCNN still relies on CNNs, 2D feature-map reinterpretation, region-wise pooling, and region-based CNN components for object detection.
- YOLOS: YOLOS replaces ViT’s [CLS] token and classification loss with [DET] tokens and bipartite matching loss for sequence-based set prediction.
- Contributions: YOLOS uses ImageNet-1k as its sole pre-training dataset and achieves competitive COCO detection with minimal modifications.
- Contributions: YOLOS demonstrates that 2D object detection can use fixed-size non-overlapping image patches in a pure sequence-to-sequence manner with minimal 2D inductive biases.
- Contributions: Detection results are sensitive to the ViT pre-training scheme, and performance remains far from saturating, making YOLOS a benchmark for pre-training strategies.
2 You Only Look at One Sequence
YOLOS keeps the ViT encoder and compact prediction head while adding detection tokens and set-based matching. Its design deliberately avoids region-specific spatial machinery and uses global token interactions.
- Design principles: YOLOS follows the original ViT and uses a compact detector head, prioritizing an intentionally simple setup over peak detection performance.
- Input sequence: The model appends one hundred randomly initialized [DET] tokens to image-patch tokens and trains them with bipartite matching loss.
- Stem: The ViT stem flattens the image into a sequence of non-overlapping patches and linearly projects each patch into D-dimensional [PATCH] tokens.
- Stem: Learnable 1D positional embeddings are added to all N+100 input tokens, producing the sequence consumed by the YOLOS Transformer encoder.
- Body: [PATCH] and [DET] tokens interact globally through a stack of Transformer encoder layers, each combining multi-head self-attention and an MLP.
- Detector heads: Classification and bounding-box regression each use an MLP with separate parameters and two hidden layers with intermediate ReLU activations.
- Detection token: Randomly initialized [DET] tokens act as object-representation proxies, while matching predictions to ground-truth objects avoids using input 2D structure during label assignment.
- Inductive bias: YOLOS omits pyramidal hierarchies, local spatial attention, region pooling, and non-1 × 1 convolutions to minimize added 2D inductive biases.
3 Experiments
Experiments show that YOLOS transfers canonical ViT representations to COCO detection, while exposing strong sensitivity to pre-training and scaling choices. The resulting detector is competitive in some settings, but larger models and CNN-style scaling do not translate consistently to transfer performance.
- 3.1 Setup: ImageNet-1k pre-training saves total theoretical forward-pass computation compared with training YOLOS from scratch, although scratch models trained for hundreds of epochs still lag behind.This finding holds for both tiny and small models under the stated transfer-learning paradigms.
- 3.2 The Effects of Pre-training: 200 versus 300 pre-training epochs have size-dependent effects: YOLOS-Ti benefits from 300 epochs, whereas YOLOS-S transfers comparably after 200 epochs.For YOLOS-Ti, even 300 fine-tuning epochs do not close the gap caused by shorter pre-training.
- 3.2 The Effects of Pre-training: ∼1 AP improvement is obtained for both tiny and small models with transformer-specific distillation using a CNN teacher during pre-training.The authors attribute part of the improvement to helping ViT adapt to COCO.
- 3.2 The Effects of Pre-training: 800-epoch DINO pre-training performs on par with 300-epoch DeiT label-supervised pre-training for YOLOS-S, while MoCo-v3 transfer is less satisfactory.The paper describes this as preliminary evidence for self-supervised pre-training potential, while noting a remaining gap for MoCo-v3.
- 3.2 The Effects of Pre-training: YOLOS is more sensitive to pre-training schemes and less saturated on COCO than common image-recognition transfer benchmarks, motivating its use as a ViT transfer-learning benchmark.The proposed benchmark covers both label-supervised and self-supervised pre-training strategies.
- 3.3 Pre-training and Transfer Learning Performance of Different Scaled Models: dwr scaling gives the strongest pre-training accuracy, but CNN-style scaling is inconsistent after transfer: dwr reaches similar box AP to simple width scaling on COCO.The paper notes that CNN scaling recipes omit ViT spatial-attention computation, limiting their explanatory power.
- 3.4 Comparisons with CNN-based Object Detectors: YOLOS-Ti is competitive with tiny CNN detectors in AP, FLOPs, and FPS, but larger width-scaled YOLOS models become less competitive against DETR.YOLOS-S is 0.8 AP lower than a similar-sized DETR, and YOLOS-B does not beat DETR despite over 2× the parameters and FLOPs.
- 3.4 Comparisons with CNN-based Object Detectors: 42.0 AP demonstrates that a BERT-Base-derived, 12-layer, 768-channel Transformer can transfer to COCO detection with minimal modifications.The result is presented as evidence of Transformer versatility and generality rather than as a performance-optimized detector.
4 Related Work
Prior work combines Transformer representations with CNN-based detection components, while this paper studies vanilla ViT transfer and pre-training for object detection.
- ViT-FRCNN uses pre-trained ViT features within Faster R-CNN but retains CNNs and region-wise pooling operations.
- UP-DETR studies unsupervised pre-training tailored to the DETR Transformer encoder and decoder.
- The paper focuses on the characteristics of pre-trained vanilla ViT in object detection, which are rare in existing literature.
- This work examines vanilla ViT pre-trained on ImageNet-1k and transferred to COCO object detection using different strategies.
5 Discussion
The discussion favors task-agnostic vanilla ViT transfer with minimal task-specific changes, while acknowledging substantial adaptation cost and unsaturated detection performance.
- Recent ViT variants add pyramidal hierarchies and locality to improve dense recognition tasks, including object detection.
- The authors instead pursue NLP-style pre-training of task-agnostic vanilla ViT followed by adaptation to downstream vision tasks.
- YOLOS still requires 150 epochs of transfer learning, and its detection results remain far from saturating.
- The authors encourage greater focus on general visual representation learning for task-agnostic vanilla Transformer models.
6 Conclusion
The paper demonstrates that vanilla ViT can transfer from ImageNet-1k recognition to COCO detection through a pure sequence-to-sequence formulation with minimal additional 2D inductive bias.
- Vanilla ViT pre-trained on mid-sized ImageNet-1k is evaluated for transfer to the more challenging COCO object detection benchmark.
- 2D object detection can be accomplished in a pure sequence-to-sequence manner with minimal additional inductive biases.
- The promising COCO performance suggests Transformer versatility and generality across downstream tasks.
Position Embedding (PE) of YOLOS
YOLOS uses two positional-embedding strategies: Type-I adds multiple intermediate embeddings, while Type-II uses one larger interpolated embedding and is lighter and more promising.
- PE adaptation: Changing input resolutions and aspect ratios requires positional-embedding adjustment, while on-the-fly 2D interpolation may introduce inductive biases.
- PE strategies: Type-I adds randomly initialized 1D learnable positional embeddings at every intermediate Transformer layer, while Type-II uses a larger interpolated positional embedding.
- PE strategies: Type-I uses more positional embeddings, whereas Type-II uses larger positional embeddings.
- Type-I PE: Type-I is used for small- and base-sized models, with intermediate embeddings trained from scratch during fine-tuning.
- Type-I PE: Intermediate positional embeddings are expensive in model size, with about 1/3 of model weights devoted solely to positional information.
- PE adaptation: Large-scale jittering can avoid positional-embedding interpolation but usually requires longer training schedules because cropping may weaken supervision.
- PE scope: 2D positional embeddings are not considered in this work.
- Type-II PE: Type-II uses one sufficiently large first-layer positional embedding and omits intermediate embeddings as redundant.
Self-attention Maps of YOLOS
YOLOS self-attention visualizations show that attention heads can focus on varied patterns and locations, while corresponding [DET] token behavior differs across models even at equal AP.
- Figures 4 and 5 visualize [DET] token self-attention maps alongside the corresponding predictions for two YOLOS-S models.
- Different self-attention heads focus on different patterns and locations, with some visualizations interpretable and others not.
- The compared 200-epoch and 300-epoch ImageNet-1k pre-trained YOLOS-S models achieve the same AP of 36.1.
- For a given predicted object, the corresponding [DET] token and attention-map patterns are usually different across models.