Source-linked AI summary
DetCo: Unsupervised Contrastive Learning for Object Detection
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, Ping Luo
TL;DR
Existing self-supervised methods have not consistently supported both image classification and instance-level detection. DetCo combines multi-level supervision with contrastive learning between global images and local patches, achieving strong transfer across detection and related tasks while retaining competitive classification.
Problem
Existing self-supervised methods transfer inadequately to object detection, while detection-focused methods can sacrifice image classification performance.
Method
DetCo applies multi-level supervision and contrastive learning between global images and local patches to learn discriminative, consistent representations.
Results
DetCo achieves state-of-the-art transfer on diverse 2D and 3D instance-level tasks and remains very competitive on image classification.
Takeaways & Limitations
DetCo provides a self-supervised pre-trained model that improves detection transfer without sacrificing image classification relative to contemporary counterparts.
Abstract
from arXiv · showhide
Unsupervised contrastive learning achieves great success in learning image representations with CNN. Unlike most recent methods that focused on improving accuracy of image classification, we present a novel contrastive learning approach, named DetCo, which fully explores the contrasts between global image and local image patches to learn discriminative representations for object detection. DetCo has several appealing benefits. (1) It is carefully designed by investigating the weaknesses of current self-supervised methods, which discard important representations for object detection. (2) DetCo builds hierarchical intermediate contrastive losses between global image and local patches to improve object detection, while maintaining global representations for image recognition. Theoretical analysis shows that the local patches actually remove the contextual information of an image, improving the lower bound of mutual information for better contrastive learning. (3) Extensive experiments on PASCAL VOC, COCO and Cityscapes demonstrate that DetCo not only outperforms state-of-the-art methods on object detection, but also on segmentation, pose estimation, and 3D shape prediction, while it is still competitive on image classification. For example, on PASCAL VOC, DetCo-100ep achieves 57.4 mAP, which is on par with the result of MoCov2-800ep. Moreover, DetCo consistently outperforms supervised method by 1.6/1.2/1.0 AP on Mask RCNN-C4/FPN/RetinaNet with 1x schedule. Code will be released at \href{https://github.com/xieenze/DetCo}{\color{blue}{\tt github.com/xieenze/DetCo}}.
1. Introduction
Self-supervised contrastive learning transfers well to classification but has struggled to reconcile image-level recognition with instance-level detection. DetCo addresses this gap through multi-level supervision and global–local contrastive learning, achieving strong transfer across detection and related tasks while retaining competitive classification.
- Motivation: Contrastive methods have advanced image classification, but existing approaches either transfer poorly to detection or improve detection while sacrificing classification.This creates a challenge in designing one pretext task that supports both image-level and instance-level representations.
- Method: DetCo enforces discriminative representations at multiple backbone stages and consistency between global images and local patches.Multi-level supervision supports multi-scale prediction, while global–local contrastive learning preserves discriminative image- and patch-level features.
- Results: DetCo improves transfer across object detection, segmentation, pose estimation, and other 2D and 3D instance-level tasks while remaining competitive for classification.The reported evaluation spans PASCAL VOC, COCO, and Cityscapes.
- Results: DetCo improves ImageNet classification over DenseCL, InsLoc, and PatchReID by 6.9%, 5.0%, and 4.8%, respectively.It is also reported as slightly better on object detection than these concurrent methods.
- Results: 46.5 AP is achieved by Sparse R-CNN with DetCo on COCO, improving its 45.0 AP baseline by +1.5 AP.The result uses a ResNet-50 backbone and is reported as a new state-of-the-art detection result.
2. Related Work
Prior self-supervised representation learning includes generative, discriminative, contrastive, clustering, bootstrapping, and handcrafted pretext-task approaches. Most were designed for classification, while detection-oriented methods improved detection at the expense of classification, motivating DetCo’s balanced objective.
- Generative methods: Generative methods reconstruct or generate images in pixel space, making them computationally expensive and potentially preserving details unnecessary for high-level representations.They commonly use auto-encoding or adversarial learning.
- Discriminative methods: Contrastive learning brings views of the same image together and separates views from different images in feature space.SimCLR uses augmented images, MoCo maintains negatives with a momentum encoder and memory bank, SwAV uses online clustering, and BYOL bootstraps targets without negative pairs.
- Pretext tasks: Earlier self-supervised methods also used pretext tasks such as relative patch prediction, colorization, inpainting, jigsaw puzzles, super-resolution, and geometric transformations.These tasks were reported as useful for representation learning.
- Detection transfer: Most prior methods emphasize image classification, whereas DenseCL, InsLoc, and PatchReID target detection but transfer poorly to classification.DetCo is positioned as a pretext task intended to support both instance-level detection and image classification.
3. Methods
DetCo modifies a MoCo v2-style pipeline with multi-level supervision and global–local contrastive learning to strengthen representations for instance-level detection while retaining classification transfer.
- DetCo Framework: DetCo combines a backbone, MLP heads, and memory banks, using MoCo v2 settings as its baseline.Its two distinguishing designs are multi-level supervision and global–local contrastive learning.
- Multi-level Supervision: Multi-level supervision applies contrastive losses to Res2, Res3, Res4, and Res5 rather than only Res5, keeping each backbone stage discriminative.The method uses separate layer queues and assigns smaller loss weights to shallow layers than deep layers.
- Multi-level Supervision: Global views are encoded by paired query and momentum key encoders, with non-shared MLP heads producing representations at four backbone stages.The query encoder extracts multi-stage features, while the key encoder is updated by momentum.
- Global and Local Contrastive Learning: The contrastive design includes global↔local and local↔local objectives so image-level and instance-level discrimination mutually improve.The stated outcome is improved detection and classification performance.
- Global and Local Contrastive Learning: Local contrastive learning transforms each image into two nine-patch sets, extracts stage-wise patch features, concatenates them, and maps them through a local MLP head.The local patch head does not share weights with the global MLP head.
4. Experiments
DetCo consistently improves transfer across object detection, dense prediction, pose, and classification tasks, while ablations link these gains to multi-level supervision and global-local contrastive learning.
- Object Detection: 58.2 mAP and 65.0 AP75 make DetCo-800ep state of the art on PASCAL VOC, exceeding supervised pre-training by 4.7 AP and 6.2 AP75.DetCo-100ep nearly matches MoCo v2-800ep, while the larger AP75 gain suggests improved localization.
- Object Detection: DetCo surpasses MoCo v2 and supervised counterparts under severely limited COCO training, including a 3.8 AP gain over MoCo v2 for Mask R-CNN-C4.The setting uses only 12k iterations, approximately one-seventh of a 90k 1x schedule.
- Object Detection: DetCo transfers strongly to semi-supervised detection, exceeding supervised pre-training by 2.3 AP and MoCo v2 by 1.9 AP with 10% of the data.The paper reports consistent gains across 1%, 2%, 5%, and 10% data settings for RetinaNet.
- Ablation Study: Ablations show that multi-level supervision improves detection and shallow-feature discrimination, whereas global-local contrastive learning boosts both classification and detection.Global-local contrastive learning also improves discrimination across backbone stages.
5. Conclusion and Future work
DetCo uses multi-level supervision and global–local contrastive learning to pretrain on unlabeled data for downstream tasks. It achieves state-of-the-art transfer on instance-level detection while retaining competitive image-classification performance.
- DetCo combines multi-level supervision with global and local contrastive learning beyond a strong MoCo v2 baseline.These components are presented as the framework’s defining additions for pretraining on large-scale unlabeled data.
- DetCo provides state-of-the-art transfer performance on VOC and COCO detection and semantic segmentation.The conclusion frames these results as performance on instance-level detection tasks and related dense-prediction tasks.
- DetCo maintains competitive performance on image classification while improving transfer to dense prediction tasks.The reported scope includes VOC and COCO detection, semantic segmentation, and image classification.