PaperlayerView exact revision on arXiv

Source-linked AI summary

UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen

arXiv:2011.09094v3cs.CV

TL;DR

DETR’s transformer requires large-scale data and long training schedules, while existing visual pretext tasks do not directly fit its localization role. UP-DETR pre-trains the transformer by predicting randomly cropped query patches, with frozen-backbone and multi-query mechanisms. The authors report gains in convergence and performance across object detection, one-shot detection, and panoptic segmentation.

  • Problem

    DETR requires large-scale training data and an extremely long schedule, while existing unsupervised visual pretext tasks do not directly suit its localization-focused transformer.

    Source: §1 · pp. 1–2

  • Method

    UP-DETR pre-trains DETR’s transformer to locate randomly cropped image patches, freezing the CNN backbone and using attention masks for multi-query localization.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Results

    UP-DETR improves DETR’s performance and convergence in object detection, one-shot detection, and panoptic segmentation, with higher precision and faster convergence reported on PASCAL VOC.

    Source: §1 · p. 2, §5 · pp. 9–10

  • Takeaways & Limitations

    The paper concludes that transformer pre-training is useful across object-detection training-data scales and presents a unified perspective for object detection and one-shot detection.

    Source: §5 · pp. 9–10

    UP-DETR with a frozen CNN and feature reconstruction relies heavily on a pre-trained CNN, leaving pre-training the whole DETR model from scratch as a future direction.

    Source: §4.5.2 · p. 8

Abstract

from arXiv · show

DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture. However, trained with scratch transformers, DETR needs large-scale training data and an extreme long training schedule even on COCO dataset. Inspired by the great success of pre-training transformers in natural language processing, we propose a novel pretext task named random query patch detection in Unsupervised Pre-training DETR (UP-DETR). Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder. The model is pre-trained to detect these query patches from the input image. During the pre-training, we address two critical issues: multi-task learning and multi-query localization. (1) To trade off classification and localization preferences in the pretext task, we find that freezing the CNN backbone is the prerequisite for the success of pre-training transformers. (2) To perform multi-query localization, we develop UP-DETR with multi-query patch detection with attention mask. Besides, UP-DETR also provides a unified perspective for fine-tuning object detection and one-shot detection tasks. In our experiments, UP-DETR significantly boosts the performance of DETR with faster convergence and higher average precision on object detection, one-shot detection and panoptic segmentation. Code and pre-training models: https://github.com/dddzg/up-detr.

1 INTRODUCTION

DETR’s transformer encoder-decoder faces long training schedules and data demands, while existing unsupervised pretext tasks do not directly address its localization-focused role. UP-DETR pre-trains the transformer to localize randomly cropped image patches, addressing classification-localization trade-offs and multi-query localization.

1 INTRODUCTION

  • DETR needs large-scale data and extremely long training schedules, and performs poorly on PASCAL VOC with its insufficient training data and fewer instances.The paper also notes that DETR’s transformer is not pre-trained, unlike its CNN backbone.Source: §1 · p. 1
  • Existing visual pretext tasks target feature discrimination rather than the spatial localization learning needed to predict DETR’s detection boxes.The authors identify architectural and feature-preference differences as reasons those tasks cannot directly pre-train DETR’s transformer.Source: §1 · pp. 1–2
  • Random query patch detection pre-trains the transformer without human annotations by cropping image patches and predicting their bounding boxes in the source image.UP-DETR extends this to multiple patches using an attention mask and object-query shuffle.Source: §1 · p. 2
  • Freezing the CNN backbone is identified as the most important step for preserving feature discrimination during pre-training, alongside patch feature reconstruction.These choices address the trade-off between classification and localization preferences.Source: §1 · p. 2
  • UP-DETR converges faster and achieves higher AP than DETR on PASCAL VOC, and outperforms DETR on VOC and COCO with the same architecture.It also transfers with state-of-the-art performance on one-shot detection and panoptic segmentation.Source: Figure 1 · p. 1, §1 · p. 2

2 RELATED WORK

Related work situates DETR among detection methods, reviews unsupervised pretext tasks, and distinguishes DETR’s decoder from vision-transformer designs. It motivates a localization-oriented pretext task for DETR rather than directly applying backbone pre-training methods.

2.1 Object Detection

  • DETR formulates detection as direct set prediction, avoiding hand-crafted positive-negative sample assignment; detection methods also face differing classification and localization feature preferences.Prior work is cited as linking performance gaps to sample selection and describing domain misalignment between classification and localization.Source: §2.1 · p. 2

2.2 Unsupervised Pre-training

  • Unsupervised pre-training learns intermediate representations through invented pretext tasks, including instance-based discrimination and clustering-based approaches.Instance-based methods vary in negative-sample size or use parametric instance classification, while clustering can be offline or online.Source: §2.2 · p. 2

2.3 Vision Transformer

  • ViT applies transformers to image patches for classification, whereas DETR’s decoder cross-attention and self-attention serve box set prediction, so backbone pre-training tasks do not directly transfer.Figure 2 contrasts single-query patch pre-training with multi-query patches assigned across object queries using shuffle and an attention mask.Source: §2.3 · p. 2, Figure 2 · p. 3

3 UP-DETR

UP-DETR pre-trains DETR’s transformer to localize randomly cropped image patches, then fine-tunes the full model on labeled detection tasks. Its design includes a frozen CNN during transformer pre-training, multi-query localization, and a shared approach to object and one-shot detection.

3 UP-DETR

  • UP-DETR unsupervisedly pre-trains the transformer and then fine-tunes the entire model with labeled data for downstream detection tasks.The described tasks include object detection and one-shot detection.Source: §3 · p. 3

3.1 Pre-training

  • Random query patch detection uses randomly cropped patches as decoder queries and trains the decoder to predict their locations in the input image.The CNN extracts features from both the input image and query patches, with shared CNN parameters.Source: §3.1 · p. 3
  • For M query patches and N object queries, UP-DETR assigns N/M object queries to each patch to support multi-query localization.The default pre-training setup uses N=100 object queries and M=10 query patches.Source: §3.1 · p. 3

3.1.1 Frozen Pre-training Backbone

  • Freezing the CNN during transformer pre-training addresses poor convergence when both are trained from scratch and provides stable backbone parameters.The authors say stable parameters benefit and accelerate transformer pre-training.Source: §3.1.1 · p. 4

3.1.2 Tricks

  • Feature reconstruction loss is intended to preserve CNN feature discrimination during localization-focused pre-training and slightly improves fine-tuning performance.The loss is mean squared error between the ℓ2-normalized CNN patch feature and its reconstruction.Source: §3.1.2 · p. 4
  • An attention mask limits interactions between object queries assigned to different patches, reflecting the patches’ independent localization targets.The mask also slightly lowers loss in the reported experiments.Source: §3.1.2 · p. 4

3.2 Fine-tuning

  • Object detection and one-shot detection share the CNN, transformer encoder-decoder, and object queries; their decoder inputs differ.For one-shot detection, a shared-CNN query feature is added to object queries, and predictions use binary query-match classification rather than category classification.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

4 EXPERIMENTS

Experiments pre-train UP-DETR on unlabeled ImageNet and fine-tune on PASCAL VOC and COCO. The setup uses a ResNet-50 model and evaluates VOC detection under short and long fine-tuning schedules.

Setup

  • UP-DETR is pre-trained on unlabeled ImageNet and fine-tuned on PASCAL VOC and COCO using a 41.3M-parameter ResNet-50 model.The model has six transformer encoder and six decoder layers, width 256, and eight attention heads.Source: §4 · p. 5

PASCAL VOC

  • Object detection results are reported for models trained on PASCAL VOC trainval07+12 and evaluated on test2007; DETR and UP-DETR use R50, while Faster R-CNN uses R50-C4.Bracketed values indicate gaps compared to DETR with the same training schedule.Source: Table 1 · p. 5

Gap between pre-training and fine-tuning

  • Pre-training and fine-tuning differ in image content, image size, and supervision, and the paper states that this gap cannot be eliminated.The authors describe ImageNet images as mostly containing one object and being smaller than multi-object COCO images.Source: §4 · p. 5

4.1 PASCAL VOC Object Detection

On PASCAL VOC’s limited training data, UP-DETR improves DETR’s detection performance and convergence. It remains slightly behind Faster R-CNN on AP50 and AP75.

4.1 PASCAL VOC Object Detection

  • On PASCAL VOC, UP-DETR’s pre-trained transformer improves over DETR’s randomly initialized transformer by +6.2 AP, +5.2 AP50, and +7.5 AP75 at 150 epochs, and +3.1 AP, +2.1 AP50, and +3.7 AP75 at 300 epochs.Source: §4.1 · p. 5
  • Both DETR and UP-DETR remain slightly behind Faster R-CNN on AP50 and AP75, possibly because of different feature-map ratios and NMS postprocessing.The passage notes that NMS lowers AP but slightly improves AP50.Source: §4.1 · p. 6
  • 56.1 AP after 150 epochs for UP-DETR versus 54.1 AP for DETR after 300 epochs; DETR does not catch up even with longer training.The learning curves show faster UP-DETR convergence, with a large AP improvement after the learning-rate reduction.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 COCO Object Detection

On COCO, UP-DETR outperforms DETR across short and long schedules and across object sizes, with faster convergence. Its 300-epoch result also exceeds Faster R-CNN’s 9× schedule on AP, while APS remains a relative weakness.

4.2 COCO Object Detection

  • 43.1 AP after 300 epochs, 1.0 AP above DETR (SwAV CNN) and 1.1 AP above Faster R-CNN (R50-FPN) (9× schedule).At 150 epochs, UP-DETR exceeds DETR by 0.8 AP and is comparable to Faster R-CNN (R50-FPN) (3× schedule).Source: §4.2 · p. 6
  • UP-DETR outperforms DETR on small, medium, and large objects at both training schedules, but slightly trails Faster R-CNN on APS because it lacks an FPN-like architecture.The COCO evaluation uses train2017 and val2017, with about 118k training images.Source: §4.2 · p. 6
  • UP-DETR leads DETR at both 150 and 300 epochs and converges faster, with an approximately 1.0 AP lead remaining after the learning-rate reduction.The improvement is more noticeable before the learning-rate reduction.Source: §4.2 · p. 7

4.3 One-Shot Detection

UP-DETR is evaluated as a downstream approach to one-shot detection, where query patches guide detection of instances from the same class. It improves DETR on both seen and unseen classes.

4.3 One-Shot Detection

  • One-shot detection seeks instances matching a query patch even when its class label was absent from training, and is treated as downstream fine-tuning for UP-DETR.The task uses query patches as decoder input.Source: §4.3 · p. 7
  • +22.8 AP50 on seen classes and +15.8 AP50 on unseen classes versus DETR; UP-DETR also beats compared methods by at least +7.9 and +4.0 AP50, respectively.Evaluation averages AP scores over five query patches per target image, with each existing class evaluated.Source: §4.3 · p. 7
  • Table 5 reports a PASCAL VOC ablation comparing single-query patch with multi-query patch pre-training; its caption gives gaps relative to DETR at the same schedule.The supplied caption does not state the ablation values.Source: Table 5 · p. 7

4.4 Panoptic Segmentation

UP-DETR improves DETR’s panoptic segmentation performance on COCO with a ResNet-50 backbone. The reported comparison shows gains in PQ, PQth, and APseg.

4.4 Panoptic Segmentation

  • Fine-tuning uses COCO panoptic annotations for box annotations over 300 epochs, then freezes DETR’s weights and trains the mask head for 25 epochs.The mask head is added on top of the decoder outputs.Source: §4.4 · p. 7
  • On panoptic segmentation with a ResNet-50 backbone, UP-DETR outperforms DETR3 by +0.4 PQ, +0.5 PQth, and +1.4 APseg.Source: §4.4 · p. 7
  • Table 4 compares state-of-the-art panoptic segmentation methods using a ResNet-50 backbone, and UP-DETR outperforms DETR3 by 0.4 PQ, 0.5 PQth, and 1.4 APseg.Source: §4.4 · p. 7

4.5 Ablations

The ablations compare single- and multi-query patch pretraining, test the roles of a frozen CNN and feature reconstruction, and examine attention masking. They report stronger fine-tuning results with multiple queries and a frozen CNN, faster early convergence with feature reconstruction, and a training-loss advantage for attention masking later in training.

4.5 Ablations

  • The ablations use 15 pretraining epochs and 150 fine-tuning epochs on PASCAL VOC, so their AP results are lower than those in Section 4.1.The learning rate is multiplied by 0.1 at epoch 10 during the ablation training.Source: §4.5 · p. 7

4.5.2 Frozen CNN and Feature Reconstruction

  • UP-DETR with feature reconstruction converges faster early in training, and the authors describe patch detection as complementary to contrastive learning.Patch detection targets spatial localization with position embeddings, while contrastive learning targets instance or cluster classification.Source: Figure 5 · p. 8, §4.5.2 · p. 8

4.5.1 Single-query patch vs. Multi-query patches

  • UP-DETR with M = 10 outperforms M = 1 on fine-tuning and both beat DETR across AP metrics; the authors adopt M = 10 for VOC and COCO.The comparison uses single-query (M = 1) and multi-query (M = 10) pretraining.Source: §4.5.1 · pp. 7–8

4.5.2 Frozen CNN and Feature Reconstruction

  • Without a frozen CNN, pretraining models (a) and (c) perform worse than DETR, while models (b) and (d) perform better than the others.The ablation varies CNN freezing and feature reconstruction; all models use an ImageNet-pretrained CNN. Table 6 reports AP50 after 150 epochs on PASCAL VOC.Source: §4.5.2 · p. 8, Table 6 · p. 8

4.5.3 Attention Mask

  • Without an attention mask, loss is lower early; as training converges, the masked model achieves lower loss by excluding unrelated query patches.The authors attribute the early difference to optimal bipartite matching and greater query attention without the mask, which can mismatch query patches and ground truth.Source: Figure 6 · p. 8, §4.5.3 · p. 8

4.5.2 Frozen CNN and Feature Reconstruction

  • UP-DETR with a frozen CNN and feature reconstruction heavily relies on a pretrained CNN model.The authors identify further investigation of joint patch detection and contrastive learning to pretrain DETR from scratch as a promising direction.Source: §4.5.2 · p. 8

4.6 Visualization

The visualizations show that pretrained UP-DETR localizes queried patches without annotations and can be fine-tuned for both class-based object detection and query-based one-shot detection. The two tasks differ in their decoder inputs and prediction labels.

4.6.1 Pre-training

  • UP-DETR localizes bounding boxes for given query patches and suppresses duplicate boxes after pretraining with random query patch detection.The visualization uses manually cropped and augmented object patches as queries; the authors characterize the procedure as unsupervised one-shot detection or template matching.Source: §4.6.1 · p. 9

4.6.2 Fine-tuning

  • Changing the transformer decoder input supports both class-based object detection and one-shot detection, which predicts matches to a query image with 0/1 labels.Object detection predicts classes present in the training set; one-shot detection predicts semantically similar objects to the query image.Source: §4.6.2 · p. 9

5 CONCLUSION

The paper introduces random query patch detection to unsupervisedly pretrain DETR's transformer and reports gains on object detection and one-shot detection. It also identifies the separation of CNN and transformer pretraining as an open direction for unified end-to-end pretraining and broader downstream use.

5 CONCLUSION

  • UP-DETR uses random query patch detection to unsupervisedly pretrain the DETR transformer, outperforming DETR on PASCAL VOC and COCO, including with a long COCO training schedule.The authors report higher precision and faster convergence on PASCAL VOC.Source: §5 · p. 9
  • UP-DETR provides a unified perspective for one-shot detection and significantly boosts performance on that task.Source: §5 · p. 10
  • CNN and transformer pretraining remain separate; the authors propose unified end-to-end pretraining and application to tasks such as few-shot detection and tracking as future directions.Source: §5 · p. 10

Related papers

Loading 2011.09094v3…