Source-linked AI summary
Dense Contrastive Learning for Self-Supervised Visual Pre-Training
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li
TL;DR
Existing self-supervised methods largely learn global image-level representations, leaving a mismatch with dense prediction tasks that require local features. DenseCL performs pixel- or local-feature-level contrastive learning using correspondence between views, and consistently outperforms MoCo-v2 across downstream dense prediction benchmarks.
Problem
Image-level self-supervised pre-training can be sub-optimal for dense prediction because dense tasks require pixel-level classification or regression.
Method
DenseCL uses a dense projection head and correspondence-aware dense contrastive loss to learn local representations across views.
Results
DenseCL consistently outperforms MoCo-v2 across object detection, semantic segmentation, and instance segmentation, including +2.0% AP on PASCAL VOC object detection.
Takeaways & Limitations
DenseCL substantially improves transfer to dense prediction tasks while introducing negligible computation overhead compared with MoCo-v2.
Takeaways & Limitations
Removing global contrastive learning creates a chicken-and-egg problem because incorrect correspondences impede feature learning and poor features impede correct correspondence.
Abstract
from arXiv · showhide
To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning, which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation. Code is available at: https://git.io/AdelaiDet
1. Introduction
Image-level self-supervised pre-training can mismatch dense prediction, which requires local, pixel-level outputs; DenseCL addresses this by learning contrastive representations densely across corresponding views. It reports consistent gains over MoCo-v2 across downstream detection and segmentation tasks.
- Motivation: Dense prediction assigns labels or regressions across an image, whereas conventional self-supervised learning usually optimizes global image-level features.This mismatch motivates a method customized for dense prediction without requiring dense annotations.
- Method: DenseCL uses a dense projection head to preserve spatial information and generate local feature vectors instead of one globally pooled vector.The method performs dense contrastive learning with a fully convolutional network.
- Method: DenseCL assigns each local feature a positive sample through correspondence across views and extends InfoNCE into a dense contrastive loss.Each local query is matched to a corresponding local feature from another view of the same image.
- Contribution: DenseCL is tailored to dense prediction tasks and is designed to close the gap between self-supervised pre-training and downstream dense prediction.The approach targets object detection, semantic segmentation, and related pixel- or instance-level tasks.
- Results: +2.0% AP, +0.9% AP, and +3.0% mIoU are reported over MoCo-v2 for object detection, instance segmentation, and semantic segmentation, respectively.The reported gains cover multiple downstream dense prediction tasks and datasets.
2. Method
DenseCL extends contrastive self-supervised pre-training from global image representations to local features, combining global and dense objectives while matching corresponding regions across augmented views.
- DenseCL Pipeline: DenseCL extends the existing self-supervised framework to dense prediction by adding dense feature maps and a dense projection head.The framework is tailored to local features rather than only whole-view representations.
- DenseCL Pipeline: The encoder uses parallel global and dense projection heads, with identical 1×1 convolution layers replacing global pooling and the MLP in the dense branch.The two heads are trained end-to-end with a joint contrastive loss over global and local features.
- Dense Contrastive Learning: Each local query is matched with a positive key from the corresponding feature location across views, while negative keys come from different images.The dense projection head produces S^2 local queries when the generated feature maps have spatial size S×S.
- Dense Correspondence across Views: Dense correspondence is obtained by matching feature vectors across views through cosine similarity and an argmax operation.The matches can be implemented with matrix operations, introducing negligible latency overhead.
- Dense Correspondence across Views: Removing the global contrastive term creates a chicken-and-egg issue because incorrect correspondence prevents learning good features, while weak features prevent correct correspondence.The authors therefore retain a global term during training and discuss additional solutions for this issue.
3. Experiments
Experiments evaluate DenseCL against MoCo-v2 and other pre-training baselines across dense prediction tasks, ablations, training schedules, and computational overhead. DenseCL consistently improves transfer performance while adding less than 1% training-time overhead, with gains also depending on correspondence matching and training design.
- Experimental Setup: DenseCL is evaluated after pre-training on COCO or ImageNet and fine-tuning for detection and segmentation tasks using mainstream downstream protocols.The experiments cover VOC detection, COCO detection and instance segmentation, VOC semantic segmentation, and Cityscapes semantic segmentation.
- Main Results: 2.0% AP improvement over MoCo-v2 is achieved on PASCAL VOC object detection with COCO pre-training, while ImageNet pre-training yields a 1.7% AP gain.The gains are consistent across three metrics, with larger improvements on AP75 indicating improved localization accuracy.
- Main Results: 1.1% AP and 0.9% AP gains over MoCo-v2 are obtained on COCO object detection and instance segmentation, respectively, with COCO pre-training.With ImageNet pre-training, the corresponding gains are 0.5% AP for object detection and 0.3% AP for instance segmentation.
- Main Results: DenseCL also improves Cityscapes semantic segmentation, with COCO-pretrained DenseCL surpassing supervised ImageNet pre-training by 1.9% mIoU.Cityscapes provides a substantially different urban street-scene benchmark from VOC and COCO.
- Ablation Study: Backbone-feature matching produces the best dense correspondence results, whereas random matching still improves detection but reduces classification performance.The random strategy improves detection by 1.3% AP but lowers classification by 0.9% mAP; matching projection-head outputs brings no clear improvement.
- Efficiency and Training Design: DenseCL adds less than 1% pre-training overhead, while longer schedules improve performance and DenseCL remains at least 2% AP above MoCo-v2.DenseCL is 1 second slower per COCO epoch and 6 seconds slower per ImageNet epoch; a 1600-epoch COCO model reaches 57.2% AP versus 57.0% for 200-epoch ImageNet MoCo-v2.
- Ablation Study: Dense-only contrastive learning fails to converge when λ = 1.0, motivating retention of the global contrastive term during training.The authors attribute this pilot observation to incorrect correspondences from the randomly initialized model early in training.
4. Conclusion
DenseCL is a self-supervised framework designed for dense prediction, using dense pairwise contrastive learning over pixels or local features. It improves performance across multiple dense prediction tasks and datasets.
- DenseCL introduces dense pairwise contrastive learning at the level of pixels or local features.The framework is designed and optimized for dense prediction tasks.
- Dense correspondence visualizations show more high-similarity matches for DenseCL than MoCo-v2, with matches becoming more accurate during training.The visualizations compare correspondence between two views of the same image.
- DenseCL improves object detection, semantic segmentation, and instance segmentation across PASCAL VOC, COCO, and Cityscapes.The reported improvements cover a variety of tasks and datasets.