PaperlayerView exact revision on arXiv

Source-linked AI summary

DETReg: Unsupervised Pretraining with Region Priors for Object Detection

Amir Bar, Xin Wang, Vadim Kantorov, Colorado J Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, Amir Globerson

arXiv:2106.04550v5cs.CV

TL;DR

Self-supervised object-detection pretraining often initializes only the backbone, leaving localization and region-embedding components unpretrained. DETReg trains the full detector to match unsupervised region proposals and align object embeddings with a self-supervised encoder. It improves over baselines across benchmarks and reports gains in low-data settings.

  • Problem

    Many self-supervised detection approaches pretrain only the backbone, leaving localization and region-embedding components unpretrained.

    Source: §1 · p. 1

  • Method

    DETReg pretrains the full detector to match object localizations from unsupervised region proposals and align object embeddings with a self-supervised image encoder.

    Source: §1 · pp. 1–2

  • Results

    DETReg improves over pretraining baselines across detection benchmarks and low-data settings, including a 4 AP-point gain over backbone-only image-classification pretraining on PASCAL VOC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study reports that unsupervised pretraining can improve downstream performance across two transformer models, three datasets, and many settings while reducing reliance on labeled data.

    Source: §6 · p. 8

    Experiments focus on DETR-related architectures, leaving DETReg’s applicability to traditional detection architectures for future study.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture. Instead, we introduce DETReg, a new self-supervised method that pretrains the entire object detection network, including the object localization and embedding components. During pretraining, DETReg predicts object localizations to match the localizations from an unsupervised region proposal generator and simultaneously aligns the corresponding feature embeddings with embeddings from a self-supervised image encoder. We implement DETReg using the DETR family of detectors and show that it improves over competitive baselines when finetuned on COCO, PASCAL VOC, and Airbus Ship benchmarks. In low-data regimes DETReg achieves improved performance, e.g., when training with only 1% of the labels and in the few-shot learning settings.

1. Introduction

Object detection depends heavily on human annotations, while common self-supervised approaches leave localization and region-embedding components unpretrained. DETReg addresses this by pretraining the full detector to localize and encode objects, with reported gains across benchmarks and low-data settings.

1. Introduction

  • Object detection relies on human-annotated datasets, which are costly to build and sometimes infeasible for privacy-sensitive applications.The introduction names medical imaging and personal photos as examples.Source: §1 · p. 1
  • DETReg pretrains the full detection network to localize and encode objects simultaneously, unlike approaches that pretrain only the backbone or re-identify random regions.The figure contrasts backbone-only SwAV, random-region UP-DETR, and DETReg's object-centric pretraining.Source: §1 · p. 1, Figure 1 · p. 1
  • The method pairs an Object Localization Task with an Object Embedding Task, then finetunes a classification head using a small number of labels.Region proposals provide class-agnostic box supervision, while embeddings from a self-supervised image encoder provide object-visual targets.Source: §1 · p. 2
  • Evaluation covers MS COCO, PASCAL VOC, and Airbus Ship Detection using two state-of-the-art base architectures, with the strongest stated benefit when annotated data are scarce.The paper reports improved performance over challenging baselines, especially with small amounts of annotated data.Source: §1 · p. 2
  • DETReg exceeds backbone-only pretraining by 4 AP points on PASCAL VOC and 1.6 AP points on MS COCO, and improves baselines in low-data and few-shot settings.The reported low-data settings include 1% to 10% of data and 10- and 30-shot learning.Source: §1 · p. 2

2. Related Work

Prior self-supervised approaches often transfer well to image classification but show less significant improvement on instance-level tasks. DETReg differs by pretraining the full detection network and adding weak supervision from object region priors.

2. Related Work

  • Self-supervised representations transfer strongly to image classification, but their improvement is less significant for object detection and instance segmentation.The related-work section situates DETReg among work on transfer learning and instance-level vision tasks.Source: §2 · p. 2
  • Backbone-focused transfer methods do not necessarily teach a detector what and where objects are; DETReg instead pretrains the entire network with region priors.The paper describes region priors as weak supervision with an explicit notion of objects.Source: §2 · p. 2
  • Semi-supervised and few-shot detection methods can be complementary because they can initialize from a pretrained DETReg model without changing their architecture.The paper distinguishes those approaches from DETReg's pretraining role.Source: §2 · p. 2
  • Unlike UP-DETR, which detects and reconstructs random patches, DETReg uses unsupervised region proposals to provide object-oriented weak supervision.The paper notes that DETR and Deformable DETR use supervised ImageNet-pretrained ResNet backbones; it adopts Deformable DETR as its base architecture.Source: §2 · p. 3
  • Selective Search generates proposals by greedily merging superpixels, while Objectness scores regions using visual cues such as saliency, color contrast, edge density, and superpixel straddling.These are examples of proposal-generation approaches discussed in the region-proposals review.Source: §2 · p. 3
  • Region proposal methods can provide high-recall, low-cost but noisy supervision, and DETReg uses Selective Search while allowing other proposal methods.The paper cites Selective Search's recall and prior use in R-CNN and Fast R-CNN.Source: §2 · p. 3

3. DETReg

DETReg formulates pretraining around detection-like tasks, using region proposals for localization and self-supervised image-encoder features for object embeddings. Its predictions are matched to padded pseudo-labels, with losses for classification, boxes, and embeddings.

3. DETReg

  • DETReg pretrains object detectors’ localization and embedding components using detection-like pretext tasks, with the DETR family as its implementation base.The paper notes that other architectures could also plug into DETReg.Source: §3 · p. 3

3.1. Object Localization Task

  • High-recall region proposals supply class-agnostic box targets; DETReg minimizes prediction-to-proposal differences after matching, and considers Top-K, Random-K, and Importance Sampling selection.Region proposals have low precision and no category information; Importance Sampling weights proposal selection by rank using Pr(Xi = 1) ∝ -log(i/n).Source: §3.1 · pp. 3–4

3.2. Object Embedding Task

  • The embedding task trains an added MLP to predict each query’s object descriptor, using L1 loss against embeddings from a pretrained self-supervised encoder.The encoder embeddings are invariant to image transformations, and the paper primarily uses a pretrained SwAV model.Source: §3.2 · p. 4

3.3. DETReg Pretraining

  • Pretraining matches detector outputs to proposal-and-descriptor pseudo-labels with Hungarian bipartite matching, then applies classification loss and box and embedding losses to non-padded targets.The targets are padded when the number of detector queries exceeds the number of proposals; padded entries are marked as background.Source: §3.3 · p. 4

4. Experiments

The experiments evaluate DETReg across full-data, low-data, few-shot, and semi-supervised detection, followed by analyses of its design choices. DETReg improves over comparison pretraining methods across these settings, with ablations examining the role of object-centric proposals and embedding.

4.1. Object Detection in Full Data Regimes

  • DETReg improved COCO AP by 1.4 points over UP-DETR after 150 fine-tuning epochs, and improved over SwAV by 2.5 AP on PASCAL VOC and 1.2 AP on Airbus.After 150 epochs, DETReg also outperformed the 300-epoch supervised variant.Source: §4.1 · p. 5

4.2. Object Detection in Low-Data Regimes

  • With 256 COCO training images, DETReg improves AP by 4.1 points over the supervised variant, compared with 1.1 for SwAV and 0.5 for ReSim.Across the low-data regime, DETReg consistently outperforms other pretraining strategies with Deformable DETR.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3. Few-Shot Object Detection

  • DETReg achieves state-of-the-art 30-shot results against approaches with larger backbones using simple fine-tuning, and improves over supervised pretraining.Evaluation fine-tunes on 60 base classes, then a balanced set of 10 or 30 instances per class, and reports performance on 20 novel classes.Source: Table 3 · p. 6, §4.3 · p. 6
  • Without intermediate training on COCO base classes, DETReg performs competitively and outperforms TFA without that additional base-class supervision.Table 4 evaluates fine-tuning only on the few-shot subsets.Source: §4.3 · p. 6, Table 4 · p. 7

4.4. Semi-supervised Learning

  • On COCO with 1%–10% of labeled data, DETReg outperforms existing pretraining methods, including the supervised-pretraining baseline.The models were pretrained for 50 epochs on unlabeled COCO train2017; results average five runs with different random seeds.Source: Table 5 · p. 7, §4.4 · p. 7

4.5. DETReg Analysis

  • Shuffling proposal locations across images reduces AP by 11.3, while changing the embedding-loss coefficient yields changes of at most 2 AP; freezing the backbone changes performance by 0.3 AP.The ablations use IN100 pretraining and COCO fine-tuning.Source: Table 6 · p. 7, §4.5 · p. 7
  • DETReg variants improve class-agnostic object detection over other pretraining approaches, including Selective Search; the analysis attributes better localization to coupling embedding and localization components.The evaluation considers the top 100 proposals per method on COCO val2017.Source: Table 7 · p. 7, §4.5 · p. 7
  • Selective Search and Edge Boxes perform similarly after DETReg pretraining: AP is 21.8 versus 21.0 with 2% of COCO data, and 36.2 for both with 10%.Visualizations show attention on object edges for coordinate predictions and embeddings in some examples, while one embedding emphasizes background.Source: §4.5 · p. 8, Figure 4 · p. 8

5. Limitations

DETReg’s class-agnostic results remain far behind supervised counterparts, and the experiments focus on DETR-related architectures. Transformer-based object detectors still require significant computational resources to train.

5. Limitations

  • DETReg’s class-agnostic performance remains far behind supervised counterparts.Its localization pretraining uses simple region-proposal methods for class-agnostic bounding-box supervision.Source: §5 · p. 8
  • The experiments focus on DETR-related architectures; applying DETReg to more traditional detectors remains unexplored.The authors identify this as future work.Source: §5 · p. 8
  • Transformer-based object detectors still require significant computational resources to train, despite DETReg improving training time.Source: §5 · p. 8

6. Conclusion

The authors conclude that DETReg improves downstream performance across two transformer models, three datasets, and many settings. They emphasize its use of unlabeled data to reduce reliance on costly labels, while recommending careful validation before deployment.

6. Conclusion

  • DETReg’s unsupervised pretraining improves downstream performance across two transformer models, three datasets, and many settings.Source: §6 · p. 8
  • Using unlabeled data may reduce the need for massive labeled datasets, which can be expensive in fields such as medical imaging.Source: §6 · p. 8
  • The authors recommend careful validation before deployment and anticipate no negative impact specific to DETReg.Source: §6 · p. 8

Supplementary Material

The supplement provides implementation details, full PASCAL VOC results, semi-supervised comparisons, and further analysis of DETReg. Its analyses examine encoder quality, proposal selection, learned slots, and class-agnostic detection.

Supplementary Material

  • The default implementation uses online fast Selective Search with K=30, a frozen SwAV-pretrained ResNet-50, and a SwAV encoder for 128x128 proposal crops.The backbone is fixed during pretraining.Source: Supplementary Material · p. 12

Object Detection in Full Data Regimes

Semi-supervised Learning

  • Table 9 compares limited-label COCO results with semi-supervised methods that use auxiliary losses on unlabeled data, unlike DETReg’s pretraining-stage use.The supplied text describes the comparison scope but does not report its outcomes.Source: Semi-supervised Learning · p. 12, Table 9 · p. 13

DETReg Analysis

  • The supplement also presents further class-agnostic results and examines the Top-K selection policy alongside the analyses reported in the main text.Source: DETReg Analysis · p. 12
  • 27.7 vs 26.7 AP on COCO with 1% data accompanies a 1 AP gain from the improved 800-epoch multi-crop SwAV encoder over the 400-epoch encoder.Source: DETReg Analysis · p. 12
  • With K=30, DETReg’s Top-K selection emphasizes precision over recall, which the authors suggest may suit performance with high-precision proposals.On ImageNet, K=30 proposals typically cover objects and parts.Source: DETReg Analysis · pp. 12–13, Figure 5 · p. 12, Figure 7 · p. 13
  • DETReg slots resemble Deformable DETR slots despite using no human annotations, while supervised slots show greater location variance and more specialization by box shape.Figure 6 visualizes locations and box shapes for ten manually matched random slots.Source: DETReg Analysis · p. 13, Figure 6 · p. 13
  • Qualitative COCO examples indicate DETReg outperforms Selective Search on class-agnostic detection but remains well behind ground-truth-labeled data.Figure 8 shows the top M predictions for each image, where M is the number of annotations.Source: DETReg Analysis · p. 13, Figure 8 · p. 14
Loading 2106.04550v5…