Source-linked AI summary
Unsupervised Learning of Dense Visual Representations
Pedro O. Pinheiro, Amjad Almahairi, Ryan Y. Benmalek, Florian Golemo, Aaron Courville
TL;DR
Dense prediction tasks need pixel-level representations, whereas conventional contrastive self-supervision mainly learns global image-level features. VADeR addresses this gap with pixel-level contrastive learning across view-induced correspondences, and its unsupervised pretraining transfers effectively across dense tasks, outperforming ImageNet supervised pretraining in different tasks.
Problem
Global contrastive methods learn low-resolution image-level features, while many visual understanding tasks require dense pixel-level representations.
Method
VADeR learns dense representations by contrasting matching pixels across views as positives and non-matching pixels as negatives, using known pixel correspondences.
Results
VADeR’s unsupervised dense pretraining transfers across many visual understanding tasks and outperforms ImageNet supervised pretraining in different tasks.
Takeaways & Limitations
VADeR provides a natural representation for dense prediction and is more effective than global contrastive representations for many downstream visual understanding tasks.
Takeaways & Limitations
The comparison uses a VADeR model with around 2M extra decoder parameters relative to the ResNet-50 MoCo baseline.
Abstract
from arXiv · showhide
Contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning. In general, these methods learn global (image-level) representations that are invariant to different views (i.e., compositions of data augmentation) of the same image. However, many visual understanding tasks require dense (pixel-level) representations. In this paper, we propose View-Agnostic Dense Representation (VADeR) for unsupervised learning of dense representations. VADeR learns pixelwise representations by forcing local features to remain constant over different viewing conditions. Specifically, this is achieved through pixel-level contrastive learning: matching features (that is, features that describes the same location of the scene on different views) should be close in an embedding space, while non-matching features should be apart. VADeR provides a natural representation for dense prediction tasks and transfers well to downstream tasks. Our method outperforms ImageNet supervised pretraining (and strong unsupervised baselines) in multiple dense prediction tasks.
1 Introduction
VADeR addresses the mismatch between global contrastive representations and dense visual understanding by learning pixel-level features invariant across views. Its unsupervised representations transfer effectively across multiple dense prediction tasks, outperforming supervised ImageNet pretraining in reported evaluations.
- Large labeled datasets drive computer-vision progress but are expensive to scale, motivating methods that exploit unlabeled data.
- Global contrastive representations are efficient and invariant to pixel-level changes, but their low resolution is insufficient for dense prediction.
- Dense representations encode pixels and leverage spatial structure for tasks including segmentation, depth, optical flow, keypoints, and correspondence.
- VADeR enforces perceptual constancy by bringing matching local features from different views together and separating non-matching features using pixel correspondences.
- VADeR’s unsupervised dense pretraining transfers across many visual understanding tasks and outperforms ImageNet supervised pretraining in different tasks.
2 Related Work
Prior work spans self-supervised pretext tasks, contrastive learning, and learned dense features. These approaches use signals from image transformations, spatial or temporal structure, correspondence, and handcrafted or supervised representations.
- Self-supervised learning uses intrinsic data structure to define predictive pretext tasks, including reconstruction, colorization, patch prediction, jigsaw solving, clustering, and rotation prediction.
- Contrastive methods distinguish compatible views from the same instance and incompatible views from different instances, with views defined differently across approaches.
- Some contrastive methods compare global and local features or use memory banks to efficiently sample many negative examples.
- Dense visual representations have been learned using handcrafted descriptors, supervised classification features, and dedicated methods for correspondence.
- Unsupervised structured-representation methods exploit 3D-guided consistency and temporal signals such as optical flow, future frames, and cycle-consistency.
3 Method
VADeR learns viewpoint-agnostic pixel embeddings with an encoder-decoder network and a pixelwise contrastive objective. It uses view-induced correspondences for positives, other-image pixels for negatives, and an FPN-based architecture for dense outputs.
- 3 Method: VADeR maximizes per-pixel similarity across different views so semantically similar pixels can map close in a latent space without supervision.
- 3 Method: Views apply stochastic appearance and geometric transformations to images and their pixels, producing corresponding transformed locations.
- 3 Method: Encoder-decoder networks produce d-dimensional embeddings for every pixel, with the objective of making representations invariant across viewpoint pairs.
- 3.1 View-Agnostic Dense Representations: VADeR adapts noise-contrastive estimation to classify different views of the same pixel as compatible and views of different pixels as incompatible.
- 3.1 View-Agnostic Dense Representations: Training uses correspondence maps to form positive matching-pixel pairs, while negative pixels are sampled from other images and a queue.
- 3.2 Implementation Details: The architecture uses a feature pyramid network with a ResNet-50 backbone to produce multiscale dense features, initialized with MoCo encoder weights.
4 Experimental Results
VADeR is evaluated as a transferable dense representation using fixed-feature, fine-tuning, video propagation, and COCO detection benchmarks. Across these experiments, it generally outperforms global-representation baselines, especially on structured pixel-level tasks.
- Evaluation protocols: VADeR features are evaluated through fixed-feature extraction and fine-tuning, with identical models and hyperparameters across methods except learned features or initialization.Fixed features support linear prediction for segmentation and depth; fine-tuning uses the learned features as initialization.
- Feature extraction: VADeR outperforms MoCo in all fixed-feature segmentation and depth tasks and exceeds supervised ImageNet pretraining on one segmentation task and depth prediction.Results are averaged over 5 trials using mIoU for segmentation and RMSE for depth prediction.
- Video instance segmentation: VADeR surpasses recent self-supervised methods on DAVIS-2017 instance mask propagation and achieves results comparable to the current state of the art without video data or specialized architecture.Evaluation uses region similarity J and contour-based accuracy F at 320×320 and 480×480 input resolutions.
- Fine-tuning: Fine-tuning benefits from VADeR increase as labeled data decreases, while at 100% labeled data VADeR matches MoCo statistically and surpasses supervised ImageNet pretraining.The study considers 2%, 5%, 10%, 20%, 50%, and 100% of the dataset for VOC segmentation and NYU-d v2 depth prediction.
- COCO tasks: VADeR consistently outperforms MoCo and supervised ImageNet pretraining on COCO object detection, instance segmentation, and keypoint detection.These experiments use Mask R-CNN with an FPN backbone and standard COCO metrics, averaged over 5 trials.
- Ablation studies: Correct pixel matching improves semantic segmentation over random matching, while different crops provide a considerable advantage for correspondence.Recognition performance changes little between same-view and different-view crops, unlike correspondence performance.
5 Conclusion
VADeR learns unsupervised dense representations through pixel-level contrastive learning and transfers effectively across structured prediction tasks. Its results support dense representation learning for transfer learning and low-data or unsupervised settings.
- Conclusion: VADeR forces representations of matching pixels across views to be close and non-matching features to be far apart using known pixel correspondences.Qualitative examples suggest that semantic grouping can emerge without labels.
- Conclusion: VADeR dense representations are more effective for downstream pixel-level tasks than global counterparts across recognition and geometry problems.Reported tasks include instance and semantic segmentation, object detection, keypoint detection, correspondence, and depth prediction.
- Conclusion: The authors identify unsupervised dense representations as useful for structured problems in transfer learning and unsupervised or low-data regimes.
6 Broader Impact
The work targets dense visual understanding using unlabeled data while emphasizing both societal benefits and risks. It also calls for evaluations aligned with downstream applications and broader benchmark coverage.
- The research focuses on improving image representations for dense prediction tasks, including image segmentation and object detection.
- Unlabeled-data-based image understanding could support pixel-level applications where specialized labeling is difficult, including medical imagery.
- Self-supervised learning may help detect rare, highly impactful incidents, but the same technologies may be misused to violate privacy and freedom of expression.
- The findings emphasize aligning representation learning methods with downstream applications and evaluating them beyond limited test-beds.
- The authors invite standardized benchmarks spanning geometric and semantic image understanding tasks, alongside methods bridging offline and online performance.
A.1 Implementation Details: feature extraction protocol
The feature-extraction protocols adapt model outputs into task-specific dense predictions for semantic segmentation, depth estimation, and video instance segmentation. Each task uses its specified dataset, preprocessing, and evaluation procedure.
- Semantic segmentation: Semantic segmentation converts model outputs with a 1×1 convolution, 4× upsampling, and softmax into per-pixel category probabilities.Evaluation uses PASCAL VOC and Cityscapes validation sets after training on their specified training splits.
- Depth Estimation: Depth estimation adds a 1×1 convolution to predict one value at each location, then applies 4× bilinear upsampling.The protocol uses NYU-depth v2, original image sizes, random horizontal flips, batch size 16, and 30 epochs with Huber loss.
- Video instance segmentation: Video instance segmentation propagates an initial-frame instance mask across DAVIS-2017 validation videos using temporal averaging and k-NN propagation with k=5.
A.2 Implementation Details: fine-tunning protocol
The fine-tuning protocols use a common VADeR architecture for baseline comparisons, converting dense outputs into task predictions for semantic segmentation and depth estimation. Training uses task-specific schedules and optimization settings.
- Semantic segmentation: Semantic-segmentation baselines share VADeR’s architecture and transform 128-dimensional outputs into per-pixel probability maps.The protocol uses the same datasets and preprocessing as feature extraction, with separate batch sizes, epochs, and learning rates for VOC and Cityscapes.
- Depth Estimation: Depth-estimation baselines also share VADeR’s architecture, upsample predictions to the original input size, and minimize Huber loss.Training uses batch size 16 for 30 epochs, learning-rate reductions at epochs 10 and 20, and SGD with specified optimization parameters.
A.3 Fine-tuning results with varying number of training samples
The experiments vary the number of labeled training samples and report results averaged over five trials, with Table 5 presenting the Figure 4 results in tabular form.
- The experiments evaluate performance across varying numbers of labeled training samples.
- Results are averaged over 5 trials and presented in Table 5 as a tabular version of Figure 4.