Source-linked AI summary
Data-Efficient Image Recognition with Contrastive Predictive Coding
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, Aaron van den Oord
TL;DR
The paper asks how artificial vision can recognize image categories efficiently with few labels, hypothesizing that predictable representations may support such learning. It improves Contrastive Predictive Coding and uses its representations for classification and object detection, achieving strong ImageNet results and transfer performance that surpasses supervised baselines.
Problem
Artificial image recognition degrades with limited supervision, while humans learn categories from few examples; the paper asks what accounts for this data-efficiency gap.
Method
The paper revisits CPC, an unsupervised objective that learns representations by predicting future spatial or temporal observations from past ones, then applies the encoder to recognition tasks.
Results
CPC representations improve linear ImageNet accuracy, enable strong low-label classification, and achieve 76.6% mAP on PASCAL VOC object detection, surpassing supervised pre-training by 2% absolute.
Takeaways & Limitations
The results support predictable unsupervised representations as a route to more data-efficient image recognition and transfer learning.
Takeaways & Limitations
Comparisons with label-propagation methods are imperfect because those methods assume all unlabeled images belong to one category, while prior work often uses smaller networks.
Abstract
from arXiv · showhide
Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.
1. Introduction
The paper addresses data-inefficient image recognition by improving CPC to learn predictable unsupervised representations, then evaluates their classification and transfer performance. The resulting features improve low-label recognition, linear ImageNet accuracy, and PASCAL VOC object detection.
- Motivation: Deep networks perform poorly with limited supervision, motivating representations that capture natural-scene structure without requiring labels.The paper contrasts machine vision's low-data weakness with human few-shot learning and identifies unsupervised structure learning as a possible approach.
- Approach: CPC learns representations by predicting future observations from past ones across ordered spatial or temporal dimensions.Its generality has enabled applications to speech, language, and images, while its contrastive objective supports predictable representations.
- Contributions: 23% absolute improvement raised linear ImageNet Top-1 accuracy from 48.7% to 71.5%, establishing a new state-of-the-art for the revised CPC implementation.The paper attributes the improvement to revisiting CPC's architecture and training methodology.
- Contributions: 78% Top-5 accuracy with 1% of ImageNet labels improved over raw-pixel networks by 34% and exceeded the previous semi-supervised state-of-the-art by 20%.The resulting classifier also surpassed supervised ones trained with 5× more labels.
- Contributions: +3.2% Top-1 accuracy over supervised ResNets was achieved using the entire ImageNet dataset, showing gains beyond the low-label regime.The paper also reports matching fully supervised classifiers with half as many labels.
- Contributions: 76.6% mAP on PASCAL VOC 2007 object detection surpassed supervised pre-training by 2% absolute after transferring CPC representations.This transfer experiment tests whether the representation generalizes to a new task and dataset.
2. Experimental Setup
The experiments pre-train CPC representations by spatially predicting patch features, then evaluate them with linear classification, few-label deep classification, and transfer to object detection. The protocol uses unlabeled ImageNet images and varies whether the encoder remains fixed or is fine-tuned.
- CPC pre-training: CPC encodes overlapping image patches, aggregates context from patches above each location, and linearly predicts features below it.The contrastive objective identifies the correct future representation among randomly sampled negatives using cross-entropy, avoiding constant-vector solutions.
- Downstream protocol: After pre-training, the encoder represents new observations while the context network and predictors are discarded for downstream classification.The encoder can process full images downstream, and a supervised classifier is trained on its representations.
- Datasets: Unlabeled pre-training always uses the full ImageNet ILSVRC 2012 training set, while evaluation uses task-specific labeled datasets and classifiers.This generic protocol supports comparisons with methods that learn feature extractors differently.
- Linear classification: Linear classification freezes the encoder and trains mean pooling followed by a single linear layer on the entire ImageNet dataset.Training uses the same augmentation as unsupervised learning, with a single crop and no test-time augmentation.
- Efficient classification: Efficient classification tests few-label generalization with a deep classifier trained on random ImageNet subsets ranging from 1% to 100%.The classifier is an 11-block ResNet, and the encoder may additionally be fine-tuned for the supervised objective.
- Transfer learning: Transfer learning evaluates representation generality on PASCAL VOC 2007 object detection using Faster R-CNN and the full 5,011-image labeled dataset.The encoder can be fine-tuned, with color-dropping and scale augmentation used during training.
3. Related Work
Prior work improves data efficiency through unsupervised representation learning or label propagation. CPC belongs to contrastive representation-learning methods, alongside approaches that predict across views, scales, space, or individual instances.
- Data-efficient learning: Representation learning and label propagation are complementary strategies for exploiting abundant unlabeled data.Representation learning learns an unsupervised feature extractor, whereas label propagation constrains the classifier using unlabeled examples.
- Generative approaches: Generative and adversarial models have improved linear classification, but likelihood-based models have not yet generalized to more complex stimuli.Large-scale adversarial implementations have produced corresponding gains in representation quality.
- Self-supervision: Self-supervised methods formulate prediction tasks directly in learned representations rather than reconstructing observations.Examples include predicting spatial layout, color, and other image properties for transfer to classification and detection.
- Contrastive methods: CPC maximizes mutual information between spatially separated latent representations with InfoNCE, while related methods use different prediction relationships.Contrastive Multiview Coding predicts across views, AMDIM predicts across space and layers, and Instance Discrimination separates individual examples.
- Label propagation: Label propagation trains on labeled subsets and assigns labels to unlabeled data through discrete pseudo-labels or continuous entropy-based constraints.These predictions may also be required to remain smooth under specified data deformations.
4. Results
CPC v2 improves representation quality through architectural, normalization, prediction, and augmentation changes, yielding stronger linear classification and data-efficient recognition. It also transfers effectively to object detection, although some comparisons involve different model classes or evaluation sets.
- CPC v1 to CPC v2: Larger capacity and patches improve linear accuracy, with the expanded architecture adding +5% Top-1 and larger patches adding +2%.The architecture expands to ResNet-161, while patches increase from 64×64 to 80×80 pixels.
- CPC v1 to CPC v2: Additional spatial prediction tasks and patch-based augmentation further improve accuracy by using more prediction directions and removing low-level visual cues.Bottom-up predictions add +2%, all four directions add +2.5%, and random color-dropping adds +3% accuracy.
- Efficient image classification: 2–5× less labeled data is sufficient for CPC-feature classifiers to systematically surpass classifiers trained directly on pixels.The comparison covers varying labeled-data regimes and highlights 5× and 2× data-efficiency gains in low- and high-data settings.
- CPC v1 to CPC v2: 71.5% Top-1 ImageNet accuracy marks CPC v2’s linear-classification result, up from 48.7% for original CPC.The reported comparison establishes a 23% absolute improvement and a new state-of-the-art.
- Efficient image classification: 78.3% Top-5 accuracy with 1% of ImageNet labels is a 34% absolute improvement over purely supervised methods.With the entire dataset, the classifier reaches 83.4%/96.5% Top-1/Top-5 accuracy; with 50% of labels it surpasses the fully supervised baseline using all labels.
- Transfer learning: 76.6% mAP on PASCAL VOC 2007 surpasses supervised pre-training by 2% absolute.The transfer setup uses the CPC encoder as a feature extractor for Faster-RCNN object detection.
5. Discussion
CPC improves image classifiers and object detectors with limited labels, while also improving performance at ImageNet scale. Its broader generality suggests potential for self-supervised learning across multimodal tasks and data-limited domains.
- CPC greatly improves classifiers and object detectors when only small amounts of labeled data are available.
- CPC improves performance even when classifiers receive ImageNet-scale labels.
- These results motivate research on settings where data are naturally limited, such as medical imaging and robotics.
- CPC representations transfer across tasks and modalities, including vision, audio, touch, proprioception, and action.
A. Self-supervised pre-training
Self-supervised pre-training processes images as spatial grids of augmented patches, encodes them into latent features, and uses a pixelCNN context to predict future representations for the CPC loss.
- A ResNet-161 encoder maps the patches to a [6,6,4096] latent tensor, which a pixelCNN aggregates into context vectors.
- The CPC objective predicts target representations from pixelCNN context vectors across selected spatial steps and computes cross-entropy loss.
- Images are resized to 300×300 pixels, randomly cropped to 260×260, and divided into a 6×6 grid of 80×80 patches.
- Each patch receives randomized geometric, color, and grayscale transformations before self-supervised training.
- Training uses Adam for 200 epochs with batch size 512 distributed across 32 workers, plus Polyak averaging and gradient clipping.
B. Linear classification
For linear ImageNet classification, the pretrained CPC representation is normalized, mapped to class logits with a 1×1 convolution, and spatially mean-pooled into final predictions.
- Each image produces a 6×6 grid of 4096-dimensional feature vectors using the same encoder as self-supervised pre-training.
- Batch normalization without a scale parameter normalizes the features before classification.
- A 1×1 convolution maps each feature vector to 1000 ImageNet logits, which are spatially mean-pooled into final log probabilities.
- The linear classifier uses the same image-processing pipeline as self-supervised pre-training and is optimized with Adam.
C.1. Purely supervised
The purely supervised comparison evaluates several ResNet-v2 architectures with dropout and tunes regularization separately for each labeled-data subset.
- The study compares ResNet-50, ResNet-101, ResNet-152, and ResNet-200 v2 architectures for supervised classification.
- Larger architectures perform better even when trained with smaller amounts of data.
- A dropout layer is inserted before the final linear classification layer.
- The supervised models use randomly sized crops followed by the self-supervised pre-training transformations.
- Learning rate, weight decay, dropout, and batch size are tuned separately for each labeled-data subset using a separate validation set.
C.2. Semi-supervised with CPC
The semi-supervised classifier uses CPC features from a spatial image grid, follows the supervised baseline’s preprocessing, and trains in fixed-feature then end-to-end fine-tuning stages.
- The CPC encoder produces a 14×14 grid of feature vectors for an 11-block ResNet classifier with 4096-dimensional hidden and 1024-dimensional bottleneck layers.Dropout is inserted after final mean pooling and before the final linear classifier, matching the supervised baseline.
- Image preprocessing follows the same pipeline as the supervised baseline.
- Training first keeps CPC features fixed while optimizing the classifier, then fine-tunes the entire stack.The classifier uses SGD during fine-tuning, while the encoder uses Adam.