PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Pretraining for Detection via Object-Level Contrastive Learning

Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, Stephen Lin

arXiv:2106.02637v2cs.CV

TL;DR

Image-level self-supervised pretraining is generic but leaves a representation gap for object detection. SoCo addresses this by aligning object representations, detector architecture, and detection invariances with pretraining, achieving state-of-the-art transfer results on COCO and improvements across additional detection benchmarks.

  • Problem

    Image-level pretraining is not specifically aligned with object-level bounding-box representations and detection requirements.

  • Method

    SoCo uses selective-search object proposals, detector modules such as FPN, and object-level translation and scale invariance for contrastive pretraining.

  • Results

    SoCo achieves state-of-the-art transfer performance on COCO and improves over supervised pretraining by +4.3 APbb / +3.0 APmk with Mask R-CNN R50-FPN under a 1× schedule.

  • Takeaways & Limitations

    Aligning pretraining with object detection enables strong transfer across COCO, Pascal VOC, and LVIS detection benchmarks.

Abstract

from arXiv · show

Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning. Such generality for transfer learning, however, sacrifices specificity if we are interested in a certain downstream task. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. In this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-level representations are introduced via selective search bounding boxes as object proposals; 2) the pretraining network architecture incorporates the same dedicated modules used in the detection pipeline (e.g. FPN); 3) the pretraining is equipped with object detection properties such as object-level translation invariance and scale invariance. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-art results for transfer performance on COCO detection using a Mask R-CNN framework. Code is available at https://github.com/hologerry/SoCo.

1 Introduction

SoCo aligns self-supervised pretraining with object detection by learning object-level representations, detector-compatible architectures, and detection-relevant invariances. It achieves state-of-the-art transfer performance on COCO detection.

  • Motivation: Object detection uses bounding boxes, creating a representation gap with image-level pretraining.The gap concerns object-level representations and detection properties such as translation and scale invariance.
  • Method: SoCo uses selective-search object proposals as independent instances for object-level contrastive learning.Augmented views vary object scale and location, and contrastive learning maximizes similarity across views.
  • Method: SoCo pretrains all detector modules, including dedicated components such as FPN and R-CNN heads, rather than only the feature backbone.This architectural alignment initializes all detector layers for downstream transfer.
  • Results: 43.2 APbb / 38.4 APmk on COCO with a 1× schedule improves over supervised pretraining by +4.3 APbb / +3.0 APmk.With a 2× schedule, SoCo reaches 44.3 APbb / 39.6 APmk, improving by +3.0 APbb / +2.3 APmk.

2 Related Work

Self-supervised learning has advanced generic transfer representations, but progress on dense prediction transfer has been limited. Recent work therefore shifts toward pixel- or region-level representations and detection-specific architectures and training schemes.

  • Self-supervised learning: Self-supervised methods use pretext tasks such as colorization, context prediction, inpainting, rotation prediction, and instance discrimination to learn transferable representations.Contrastive instance discrimination maps augmented views of the same instance to similar embeddings.
  • Transfer limitations: Despite improved ImageNet linear evaluation, progress on transfer performance for dense prediction tasks has been limited.This limitation has motivated pretraining methods designed specifically for object detection and semantic segmentation.
  • Dense prediction: Recent approaches shift image-level representations toward pixel-level or region-level representations for dense prediction.The passage identifies this shift as part of the response to limited dense-prediction transfer progress.
  • Detection alignment: Detection research advocates architectures and training schemes invariant to translation and scale transformations.SoCo incorporates these properties during pretraining, which the authors report benefits object detection.

3 Method

SoCo aligns self-supervised pretraining with object detection by using object proposals, detector-specific architecture, and detection-relevant invariances. It performs object-level contrastive learning across views that vary object scale and location.

  • Object proposal generation: SoCo uses selective search proposals and treats each proposal as an independent object-level instance for contrastive pretraining.Proposals are generated offline, and K are randomly selected per training iteration.
  • View construction: Three views vary the scale and location of each object proposal, encouraging translation-invariant and scale-invariant representations.Box jitter further perturbs proposal coordinates and dimensions with r ∈ [-0.1, 0.1] at probability 0.5.
  • Architectural alignment: SoCo aligns pretraining architecture with Mask R-CNN by pretraining the backbone, FPN, RoIAlign pathway, and R-CNN head rather than only the feature backbone.FPN outputs P2–P5, and proposals are assigned to pyramid levels according to their image areas.
  • Object-level contrastive learning: The online and target networks share an architecture, while target weights are updated as an exponential moving average of online weights.The online network uses a projector and predictor; the target network uses only a projector.
  • Object-level contrastive learning: SoCo minimizes a symmetrized object-level contrastive loss using target-network features from one view and online-network features from the other views.The loss is optimized as LSoCo = L + eL for K object proposals.
  • Detection properties: Scale-aware assignment encourages scale-invariant representations by assigning proposals to feature-pyramid levels according to object scale.The design follows Mask R-CNN FPN’s scale-specific pyramid structure.

4 Experiments

SoCo is evaluated across detection and segmentation transfer settings, detector architectures, datasets, and alignment ablations. Results show strong transfer performance and benefits from object-level and architectural alignment.

  • Experimental setup: SoCo uses ImageNet pretraining with detector-aligned architectures and transfers to Mask R-CNN using R50-FPN and R50-C4 backbones.The evaluation uses COCO and Pascal VOC transfer settings, with COCO reporting both 1× and 2× schedules.
  • COCO transfer: 40.9 APbb / 35.3 APmk and 42.0 APbb / 36.3 APmk are obtained on COCO 1× and 2× with Mask R-CNN R50-C4.The reported results are described as state-of-the-art without additional bells and whistles.
  • Pascal VOC transfer: 59.7 APbb on Pascal VOC improves over supervised pretraining by +6.2 APbb using Faster R-CNN with an R50-C4 backbone.This evaluates transfer beyond the COCO benchmark.
  • Alignment ablations: Adding object-level representations improves the baseline by +2.5 APbb / +2.4 APmk, while adding both FPN and an R-CNN head reaches 41.2 APbb / 37.0 APmk.The ablation reports that FPN alone slightly hurts performance, whereas including both detector modules improves transfer.
  • Hyper-parameter and proposal studies: SoCo is sensitive to proposal choice and batch size: random proposals can diverge at K = 4 or K = 8, while larger or smaller batch sizes hurt performance.Selective-search proposals perform better than a single random proposal, and the default batch size is 2048; τ = 0.99 gives the best momentum setting.

5 Conclusion

SoCo aligns self-supervised pretraining with object detection through object-level representations, detector architecture, and detection-specific invariances. It achieves state-of-the-art COCO transfer performance and generalizes across detector types.

  • SoCo treats selective-search object proposals as independent instances to learn object-level visual representations.
  • SoCo aligns pretraining and detection architectures so all detector layers can be well-initialized.
  • SoCo incorporates object-level translation invariance and scale invariance into pretraining.
  • SoCo achieves state-of-the-art transfer performance on COCO detection using Mask R-CNN.
  • Experiments on two-stage and single-stage detectors demonstrate SoCo's generality and extensibility.

A.1 Implementation Details

Selective search uses three main parameters to generate object proposals: scale, σ, and min_size. The implementation uses default values for all three.

  • Selective search has three main parameters: scale, σ, and min_size.Scale controls segment number and size, σ controls Gaussian smoothing, and min_size sets the minimum component size.
  • The implementation uses scale = 500, σ = 0.9, and min_size = 10.

A.2 Visualization

The visualization shows selective-search proposals and their distributions across images. Proposal counts and sizes are summarized with histograms, while examples are randomly sampled for clarity.

  • The selective-search proposals show reasonably good coverage of objects.
  • Selective search proposals are randomly sampled from each image for visualization.

B Linear Evaluation on ImageNet-1K

The ImageNet-1K linear evaluation is provided for reference using only the ResNet-50 backbone. SoCo performs below prior image-level self-supervised methods, motivating future joint image-level and object-level pretraining.

  • ImageNet-1K linear evaluation uses only ResNet-50 backbone weights, with dedicated modules such as FPN removed.
  • The evaluation trains a linear classifier for 100 epochs while freezing backbone parameters and batch statistics.
  • Table 11 compares top-1 and top-5 accuracies on the ImageNet-1K validation set.
  • SoCo performs below previous image-level self-supervised pretraining methods on ImageNet linear evaluation.The authors leave joint image-level and object-level tasks as future exploration to address this gap.

C Data Efficiency

The data-efficiency experiment compares SoCo* fine-tuning with 50% of COCO training data against supervised pretraining using the full dataset. It uses Mask R-CNN with an R50-FPN backbone in the standard COCO 1× setting and reports 2× data efficiency.

  • SoCo* fine-tuning uses a randomly selected 50% of the COCO training data.The result is compared with a model trained on the full COCO training data using supervised pretraining.
  • 2× data efficiency is reported for SoCo* compared with supervised pretraining.
  • The experiment uses Mask R-CNN with an R50-FPN backbone under the standard COCO 1× setting.

D Broader Impact

The paper warns that the method may have unpredictable failures and that their consequences depend on downstream applications. It also cautions against use in scenarios where failures could lead to serious consequences and notes sensitivity to data bias.

  • The method may have unpredictable failures whose consequences depend on downstream applications.
  • The paper advises against using the method where failures could lead to serious consequences.
  • Because the method is data driven, performance may be affected by biases in the data.The paper recommends care during data collection.
Loading 2106.02637v2…