Source-linked AI summary
Unsupervised Visual Representation Learning by Context Prediction
Carl Doersch, Abhinav Gupta, Alexei A. Efros
TL;DR
Unsupervised learning sought useful representations from large unlabeled image collections, while prior work had not demonstrated this for full-sized real images. The paper learns a ConvNet representation from an image-level objective and reports transfer to object discovery, detection, and geometry estimation, with a significant detection boost over training from scratch.
Problem
Prior unsupervised methods had not been shown to extract useful information from large collections of full-sized, real images, while labeled-data approaches require costly annotation at scale.
Method
The paper learns a ConvNet representation from an image-level objective and applies color preprocessing to prevent chromatic aberration from providing a trivial solution.
Results
The representation supports unsupervised object discovery, significantly boosts PASCAL VOC 2007 detection over learning from scratch, and performs almost equivalently to fully supervised ImageNet pretraining for NYUv2 surface-normal estimation.
Takeaways & Limitations
Instance-level supervision from a single image can generalize across images and support category-level object tasks and non-object-based geometry estimation.
Takeaways & Limitations
Chromatic aberration can let the ConvNet localize patches relative to the lens rather than learn visual context, requiring preprocessing to avoid this trivial solution.
Abstract
from arXiv · showhide
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.
1. Introduction
The paper turns unlabeled image context into supervision by predicting the relative position of paired patches, aiming to learn representations that capture objects and parts. The resulting representation supports visual similarity across images, object discovery, and improved detection performance.
- Unsupervised learning from large collections of full-sized real images lacked a clear objective for deciding what visual information to represent.
- Context prediction provides self-supervision by learning to predict surrounding elements from an individual representation.
- The proposed task samples patch pairs from eight spatial configurations and asks a learner to predict one patch’s position relative to the other.
- The approach uses a ConvNet and hypothesizes that solving relative position requires recognizing objects and their parts.
- The learned representation supports unsupervised object discovery and significantly boosts PASCAL VOC 2007 detection over learning from scratch.
2. Related Work
Related work frames representation learning as constructing useful embeddings through generative models, similarity objectives, or supervised pretext tasks. Earlier image-context and clustering approaches face challenges involving prediction ambiguity, shape information, or complex-image geometry.
- Generative approaches seek concise latent causes for images, but inferring latent structure is computationally intractable for even relatively simple models.
- Embedding-based methods use pretext tasks such as denoising reconstruction to make semantically similar images close in representation space.
- Text context prediction motivates visual context prediction, but image-domain methods must address ambiguity in determining whether region predictions are correct.
- Pixel prediction is harder than word prediction because the same semantic object can produce many different pixel patterns.
- Hand-crafted clustering methods can lose shape information, while contour-based approaches rely on difficult contour extraction in complex images.
- Video-based learning exploits temporal coherence, whereas discriminative patch-mining work emphasizes learning object-part representations before full objects and scenes.
3. Learning Visual Context Prediction
The method trains a late-fusion ConvNet to classify the relative position of two image patches while discouraging low-level shortcuts. Its learned patch features are evaluated through nearest-neighbor clusters and safeguards against chromatic-aberration cues.
- A ConvNet predicts one of eight spatial offsets for a pair of patches sampled from an image.
- The late-fusion architecture processes patches separately with tied weights before combining their representations for pair classification.
- Patch gaps and location jitter reduce shortcuts based on boundary patterns or textures continuing across neighboring patches.
- Chromatic aberration can reveal a patch’s absolute lens location, making relative-position prediction trivial for some images.
- Nearest-neighbor patch clusters compare randomly initialized features, supervised AlexNet features, and features learned from the context-prediction method.
- Color projection or randomly dropping two color channels removes the green-magenta cue used by that shortcut.
4. Experiments
The experiments evaluate the learned representation through nearest-neighbor matching, object detection, geometry estimation, visual data mining, and the relative-position pretext task. Across these tests, unsupervised pre-training improves detection and supports object discovery, while also revealing sensitivity to image layout and chromatic aberration.
- Experimental overview: The experiments test nearest-neighbor similarity, Pascal VOC detection, visual data mining, and the relative-position pretext task.The representation is also evaluated for surface-normal estimation on NYUv2.
- Object Detection: The detection architecture adapts the patch network to 227x227 object proposals by converting fc6 into a convolutional conv6 layer.The architecture reuses layers through pool5 and operates within the R-CNN pipeline.
- Object Detection: 6% MAP: unsupervised pre-training boosts the from-scratch detection result by 6% MAP and beats an AlexNet-style Pascal-trained model by over 5%.The result remains about 8% behind R-CNN pre-trained with ImageNet labels and is reported as the best VOC 2007 result known to the authors without external labels.
- Geometry Estimation: The learned representation performs nearly equivalently to a fully labeled ImageNet model on NYUv2 surface-normal estimation.The authors suggest ImageNet categorization may provide less incentive to model geometry than the spatial-context objective.
- Visual Data Mining: Visual mining on Pascal VOC 2011 discovers additional objects, including monitors, birds, torsos, and plates of food, without image pre-filtering or extra labels.Bird and torso discoveries provide evidence for learned invariances to deformable objects, although the method loses some cluster purity relative to prior work.
- Visual Data Mining: Substantially higher coverage accompanies lower purity in the VOC 2007 purity-coverage evaluation, suggesting increased invariance but fewer highly pure clusters.The evaluation uses 1000 sets of 10 images drawn from images containing six specified object categories.
- Relative Prediction Task: 38.4% accuracy: the relative-position task exceeds 12.5% chance on Pascal VOC 2007, while ImageNet validation accuracy slightly exceeds training accuracy.The reported ImageNet accuracies are 39.5% on training data and 40.3% on validation data.
- Relative Prediction Task: 39.2% accuracy on object-box patches versus 45.6% on cars indicates sensitivity to objects but also to the layout of the surrounding image.The object-box experiment excludes truncated, occluded, difficult, and smaller boxes.