Source-linked AI summary
Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, Han Hu
TL;DR
Existing unsupervised visual representation learning mainly uses instance-level pretext tasks, which may lack the spatial sensitivity needed for dense pixel prediction. The paper introduces pixel-level contrastive learning and pixel-to-propagation consistency, with PixPro achieving state-of-the-art transfer results across dense-prediction benchmarks. Pixel-level pretraining also applies to backbone and head networks and complements instance-level methods.
Problem
Instance-level self-supervision may produce representations suited to image-level inference but lacking the spatial sensitivity required for dense tasks such as object detection and semantic segmentation.
Method
The paper introduces PixContrast, which contrasts pixel representations, and PixPro, which enforces consistency between regular and pixel-propagated features from corresponding pixels.
Results
PixPro achieves state-of-the-art transfer performance across dense-prediction benchmarks, including 60.2 AP on Pascal VOC, 41.4 / 40.5 mAP on COCO, and 77.2 mIoU on Cityscapes.
Takeaways & Limitations
Pixel-level pretext tasks show strong potential for unsupervised dense representation learning, can initialize downstream head networks, and complement instance-level methods.
Abstract
from arXiv · showhide
Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propagation consistency task that produces better results, even surpassing the state-of-the-art approaches by a large margin. Specifically, it achieves 60.2 AP, 41.4 / 40.5 mAP and 77.2 mIoU when transferred to Pascal VOC object detection (C4), COCO object detection (FPN / C4) and Cityscapes semantic segmentation using a ResNet-50 backbone network, which are 2.6 AP, 0.8 / 1.0 mAP and 1.0 mIoU better than the previous best methods built on instance-level contrastive learning. Moreover, the pixel-level pretext tasks are found to be effective for pre-training not only regular backbone networks but also head networks used for dense downstream tasks, and are complementary to instance-level contrastive methods. These results demonstrate the strong potential of defining pretext tasks at the pixel level, and suggest a new path forward in unsupervised visual representation learning. Code is available at \url{https://github.com/zdaxie/PixPro}.
1. Introduction
The paper argues that instance-level self-supervision may not provide enough spatial sensitivity for dense prediction, and introduces pixel-level pretext tasks to address this gap. PixPro uses pixel-to-propagation consistency and achieves strong transfer performance, while pixel-level pretraining also extends to downstream head networks and complements instance-level methods.
- Motivation: Instance-level comparisons may produce representations suited to image-level inference but lacking spatial sensitivity for object detection and semantic segmentation.The paper identifies pixel-level self-supervised representation learning as a relatively unexplored problem.
- Pixel-level pretext tasks: PixContrast treats each pixel as a class, using features from the same pixel across random crops as positives and features from different pixels as negatives.A contrastive loss is applied to these self-supervised pixel pairs.
- Pixel-level pretext tasks: PixPro encourages consistency between same-pixel features from asymmetric pipelines, with one branch applying pixel propagation to smooth features of similar pixels.Unlike PixContrast, it avoids negative pairs and therefore avoids sensitivity to negative-pair handling.
- Broader implications: Pixel-level pretext tasks can pretrain both backbone and dense-task head networks, while combining them with instance-level methods uses spatial sensitivity and categorization ability together.The combination can share a data loader and backbone encoders during pretraining.
- Transfer results: 60.2 AP, 41.4 / 40.5 mAP, and 77.2 mIoU were achieved on Pascal VOC, COCO, and Cityscapes dense-prediction benchmarks, respectively.These results were 2.6 AP, 0.8 / 1.0 mAP, and 1.0 mIoU better than leading unsupervised or supervised methods.
2. Related Works
The related work is dominated by instance discrimination, while this paper focuses on pixel-level pretext tasks and dense-prediction transfer. It differs from other local or video-based approaches in how pixel correspondences and consistency are constructed.
- Instance discrimination: Instance discrimination treats each image as a single class and learns representations by distinguishing it from all other images.This pretext task became central to unsupervised visual representation learning after strong transfer results from methods such as MoCo.
- Pixel-level learning: This work instead explores pixel-level pretext tasks and evaluates transfer to object detection and semantic segmentation, which received limited attention in prior research.The authors report that pixel-level representation learning can surpass existing instance-level methods.
- Other single-image pretext tasks: Earlier single-image pretext tasks included context prediction, colorization, jigsaw solving, rotation prediction, clustering, and missing-part prediction.Interest in these tasks declined because of inferior performance or greater architectural and training complexity.
- Video and multi-image learning: Video and multi-image approaches form pixel-level tasks from repetitive pixels across views, commonly using weak cycle-consistency checks because cross-image ground-truth correspondences are unknown.This paper instead builds pixel correspondences from different views directly.
- Concurrent work: Concurrent pixel-level methods mostly use contrastive learning, whereas this work additionally advocates a consistency task that avoids carefully tuning negative pairs.The paper presents consistency as an alternative to negative-pair-based pixel learning.
3. Method
The method introduces pixel-level pretext tasks that align representation learning with dense prediction, including pixel contrast and pixel-to-propagation consistency. The latter combines feature propagation with asymmetric consistency, while the framework also supports pre-training dense-task heads and combining pixel- and instance-level objectives.
- Pixel-level Contrastive Learning: PixContrast extends instance discrimination to pixels by matching corresponding pixels across two augmented views and contrasting them against other pixels.Pixels are warped to original-image coordinates, and normalized distances with threshold T = 0.7 determine positive and negative pairs.
- Pixel-to-Propagation Consistency: Pixel-to-propagation consistency uses a propagation module to smooth features and an asymmetric branch design to enforce consistency without negative pairs.One branch produces regular features, while the other propagates features from similar pixels; the regular branch helps retain spatial sensitivity.
- Pixel-to-Propagation Consistency: The propagation module computes smoothed pixel features by aggregating transformed features from similar pixels, with γ controlling similarity sharpness and defaulting to 2.The transformation g can contain l linear layers; l = 1 is used by default because its results are slightly better.
- Pre-training Dense-task Heads: Both PixContrast and PixPro can pre-train backbone and dense-task head networks, improving transfer when annotated data are limited.Pre-training FPN layers and downstream head networks generally improves transfer accuracy.
- Combined with Instance Contrast: Pixel-level and instance-level pretext tasks are complementary: the former supports spatial inference, while the latter supports categorization representations.Adding an instance-level contrastive loss improves COCO FCOS detection by about 1 mAP, and the methods have similar pre-training complexity to their instance-level counterpart.
4. Experiments
Experiments evaluate PixPro across dense prediction benchmarks and ablations, finding strong transfer performance, benefits from architecture-aligned pre-training, and gains in low-label settings.
- Main Transfer Results: PixPro achieves 60.2 AP on Pascal VOC, 41.4 / 40.5 mAP on COCO, and 77.2 mIoU on Cityscapes.These results use ResNet-50-based downstream configurations for object detection and semantic segmentation.
- Main Transfer Results: PixPro outperforms previous best unsupervised methods by 2.6 AP on Pascal VOC, 0.8 / 1.0 mAP on COCO, and 1.0 mIoU on Cityscapes.
- Ablation Study: The hyper-parameter study finds T = 0.7 and γ = 2 effective, while overly smooth or sharp similarity functions harm transfer performance.P3 provides more stable results than C5 for distance-threshold variation.
- Ablation Study: Removing the pixel-propagation module causes model collapse, whereas its smoothness improves transfer by 0.8 AP on Pascal VOC and 1.1 mAP on COCO.The module introduces asymmetry between the propagation and regular branches.
- Ablation Study: PixPro improves semi-supervised COCO detection by +3.9 mAP with 1% labeled data and +2.3 mAP with 10% labeled data.An additional 120-epoch pixel-level pre-training stage adds +0.7 mAP and +0.2 mAP in the 1% and 10% settings.
5. Conclusion
The paper develops pixel-level pretext tasks for dense representations, with PixPro combining pixel-to-propagation consistency and smoothness priors to improve transfer across dense prediction tasks.
- PixPro uses pixel-to-propagation consistency to learn dense representations without processing negative samples.The approach introduces smoothness priors into pixel-level representation learning.
- PixPro reaches 60.2 AP on Pascal VOC and 41.4 / 40.5 mAP on COCO, exceeding previous best methods by 2.6 AP and 0.8 / 1.0 mAP.
- The results demonstrate the potential of pixel-level pretext tasks for unsupervised visual representation learning.The proposed approach is also described as applicable to videos and multi-modality signals.