PaperlayerView exact revision on arXiv

Source-linked AI summary

Instance Localization for Self-supervised Detection Pretraining

Ceyuan Yang, Zhirong Wu, Bolei Zhou, Stephen Lin

arXiv:2102.08318v2cs.CV

TL;DR

Self-supervised representations tuned for holistic classification can misalign with object detection, which requires spatial reasoning. The paper introduces instance localization with bounding-box-aware pretraining and reports weaker classification but stronger localization and state-of-the-art transfer results on PASCAL VOC and MSCOCO.

  • Problem

    Classification-oriented self-supervised pretext tasks can overfit to holistic recognition and fail to promote the spatial reasoning required for object detection.

  • Method

    Instance localization pastes foreground crops onto backgrounds and performs contrastive learning on RoI features extracted using bounding boxes, aligning pretraining with detection architecture and localization.

  • Results

    The method weakens holistic image classification but improves patch localization and surpasses state-of-the-art transfer results on PASCAL VOC and MSCOCO object detection.

  • Takeaways & Limitations

    Object-detection transfer can be strengthened by improving task and architectural alignment, especially through spatial localization representations.

  • Takeaways & Limitations

    Pretraining for 800 epochs is computationally expensive and is left for future work.

Abstract

from arXiv · show

Prior research on self-supervised learning has led to considerable progress on image classification, but often with degraded transfer performance on object detection. The objective of this paper is to advance self-supervised pretrained models specifically for object detection. Based on the inherent difference between classification and detection, we propose a new self-supervised pretext task, called instance localization. Image instances are pasted at various locations and scales onto background images. The pretext task is to predict the instance category given the composited images as well as the foreground bounding boxes. We show that integration of bounding boxes into pretraining promotes better task alignment and architecture alignment for transfer learning. In addition, we propose an augmentation method on the bounding boxes to further enhance the feature alignment. As a result, our model becomes weaker at Imagenet semantic classification but stronger at image patch localization, with an overall stronger pretrained model for object detection. Experimental results demonstrate that our approach yields state-of-the-art transfer learning results for object detection on PASCAL VOC and MSCOCO.

1. Introduction

The paper argues that self-supervised pretraining can misalign with object detection when it favors holistic classification and requires architectural repurposing. It proposes instance localization, which incorporates bounding boxes and spatial augmentation to improve detection transfer despite weaker ImageNet classification.

  • Self-supervised representations that improve image classification may fail to transfer their advantage to object detection.
  • Task misalignment arises from architectural changes during detection finetuning and holistic pretext tasks that omit explicit spatial modeling.
  • Instance localization pretrains on cropped foregrounds pasted at varied locations and scales, extracting RoI features with bounding boxes for contrastive learning.
  • Bounding-box pretraining and spatially jittered coordinates align representations with foreground regions and strengthen feature alignment for detection.
  • Across ResNet50-C4 and ResNet50-FPN, instance localization substantially improves transfer on PASCAL VOC and MSCOCO, especially with limited labeled data.

2. Related Work

Related work spans generative, predictive, contrastive, compositional, and self-training approaches to learning from visual data. The paper positions instance localization as a bounding-box-based alternative that models spatial relations between composited images for object-detection transfer.

  • Self-supervised pretext tasks include image reconstruction, colorization, context prediction, and contrastive instance discrimination.
  • Contrastive learning treats each training instance as its own category and relies on augmentations such as cropping, scaling, color jittering, and blurring.
  • Leading contrastive methods can reach 74% ImageNet linear-classification accuracy while transferring worse to object detection than MoCo.
  • Instance localization adds bounding boxes to pretraining so foreground features align with regions, unlike prior patch-level methods that reason within one image.
  • Image compositions copy foreground objects onto backgrounds for supervised segmentation and unsupervised learning, while this work avoids requiring masks or clean contours.
  • Self-training bootstraps pseudo-labels from few labeled examples, but can become vulnerable when the labeled set is scarce.

3. Pretext Task – Instance Localization

Classification favors translation and scale invariance, whereas detection must preserve object size and location information. Instance localization addresses this mismatch by discriminating foreground instances from backgrounds using bounding-box information.

  • Image classification reduces objects across scales and locations to category labels, while detection requires translation and scale equivariance.
  • Common contrastive pretexts enforce invariance between random views, which can overfit to holistic classification and fail to promote spatial reasoning.
  • Instance localization overlays foreground instances onto backgrounds and uses bounding boxes to discriminate the foreground from its background.
  • Solving the pretext task requires first localizing the foreground instance and then extracting its foreground features.

4. Learning Approach

The approach introduces bounding-box representations into contrastive pretraining to align learned features and architecture with object detection. It composes foreground crops with backgrounds, extracts region features, and augments boxes using spatially varied anchors.

  • Instance Localization: Instance localization introduces bounding-box representations into contrastive learning to improve alignment with object detection.The method extends instance discrimination by incorporating spatial information during representation learning.
  • Spatial Modeling: The composition operation pastes a random crop onto a background at a random position and scale, returning the composited image and bounding-box parameters.Different background images are used for the two views to prevent background cues from becoming shortcuts.
  • Region Feature Extraction: RoIAlign uses the bounding boxes to extract foreground features from convolutional maps before contrastive learning.This introduces region-wise representations that more closely mimic detection-time processing.
  • Feature Alignment: Bounding-box training encourages correspondence between convolutional features and foreground regions, helping effective receptive fields match the spatial extent of boxes.The goal is to reduce interference from surrounding background content and facilitate localization.
  • Bounding-Box Augmentation: Spatially jittered boxes are selected from anchors with varied scales, locations, and aspect ratios when their IoU with the ground truth exceeds 0.5.For R50-C4 and R50-FPN, empirically selected aspect-ratio ranges are [1/3, 3/1] and [1/2, 2/1], respectively.
  • Architectural Alignment: Pretraining minimizes architectural discrepancy by incorporating region-wise operations before transfer to detection architectures such as R50-C4 and R50-FPN.R50-C4 inserts RoI operations after the fourth residual block, while R50-FPN applies them across four feature levels with separate negative queues.

5. Experimental Results

Experiments evaluate instance localization across object detection, instance segmentation, localization, semantic classification, and data-limited transfer settings. The method improves detection transfer and localization while trading off linear image classification performance.

  • Main evaluation: Experiments evaluate transfer on PASCAL VOC and MSCOCO using established self-supervised pretrained weights and object detection or segmentation frameworks.COCO experiments use Faster R-CNN or Mask R-CNN with R50-C4 and R50-FPN backbones.
  • COCO transfer: InsLoc exceeds MoCo-v2 by +0.7 and +1.5 AP for R50-C4 and R50-FPN after 200-epoch pretraining, respectively.With 400-epoch pretraining, it reaches state-of-the-art performance and exceeds supervised ImageNet pretraining by +1.8 and +1.7 AP.
  • Ablation study: 2.3% higher linear localization accuracy accompanies 6.0% lower linear classification accuracy than MoCo-v2, while fine-tuned detection improves.The ablation attributes the detection gain mainly to better spatial localization rather than stronger semantics.
  • Ablation study: 41.4 APbb follows architectural alignment, instance localization, and spatially jittered bounding-box augmentation, versus 41.1 APbb without the final augmentation.Architectural alignment alone gives +0.4 APbb, while adding instance localization gives +1.3 APbb over MoCo-v2.
  • Training duration: Longer pretraining consistently improves performance, but extending training to 800 epochs is computationally expensive and left for future work.The reported experiments identify longer pretraining as improving results beyond the 200-epoch setting.
  • Mini COCO: 3.3 APbb and 2.4 APmk improvements over MoCo-v2 on Mini COCO demonstrate stronger transfer with 10% of COCO training data.The corresponding gains over supervised pretraining are 3.1 APbb and 2.3 APmk.

6. Conclusion

The paper concludes that instance localization strengthens object-detection transfer by incorporating bounding boxes into self-supervised pretraining. It reports weaker holistic classification but stronger patch localization, with state-of-the-art results on VOC and COCO and larger gains when labeled data is scarce.

  • Conclusion: Instance localization introduces bounding boxes into self-supervised representation learning to improve alignment with object detection.The conclusion frames task alignment as the basis for stronger detection transfer.
  • Conclusion: The pretrained model is weaker for holistic image classification but stronger for patch localization.This trade-off reflects the method’s detection-focused pretraining objective.
  • Conclusion: The approach improves over MoCo and reaches state-of-the-art transfer results on PASCAL VOC and MSCOCO.The conclusion summarizes the reported detection-transfer outcome across both benchmarks.
  • Conclusion: The method obtains a larger gain when the amount of labeled data is particularly small.This conclusion is stated as a data-regime-dependent transfer advantage.
Loading 2102.08318v2…