Source-linked AI summary
Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
Zhirong Wu, Yuanjun Xiong, Stella Yu, Dahua Lin
TL;DR
The paper asks whether unsupervised features can capture visual similarity among individual instances despite the cost and infeasibility of obtaining task-specific annotations. It learns instance-discriminative embeddings with a non-parametric softmax and noise-contrastive estimation, achieving state-of-the-art unsupervised classification results on ImageNet and Places while generalizing competitively to other tasks.
Problem
The paper asks whether unsupervised learning can capture apparent similarity among individual instances without costly, task-specific annotations.
Method
The method learns instance-discriminative embeddings using a non-parametric softmax approximated with noise-contrastive estimation.
Results
46.5% top-1 accuracy on ImageNet 1K and 41.6% on Places 205, surpassing state-of-the-art unsupervised image-classification methods by a large margin.
Takeaways & Limitations
The learned 128-dimensional representation scales with more data and deeper networks and transfers competitively to semi-supervised learning and object detection.
Takeaways & Limitations
The noise-contrastive formulation assumes a uniform noise distribution and noise samples occurring m times more frequently than data samples.
Abstract
from arXiv · showhide
Neural net classifiers trained on data with annotated class labels can also capture apparent visual similarity among categories without being directed to do so. We study whether this observation can be extended beyond the conventional domain of supervised learning: Can we learn a good feature representation that captures apparent similarity among instances, instead of classes, by merely asking the feature to be discriminative of individual instances? We formulate this intuition as a non-parametric classification problem at the instance-level, and use noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes. Our experimental results demonstrate that, under unsupervised learning settings, our method surpasses the state-of-the-art on ImageNet classification by a large margin. Our method is also remarkable for consistently improving test performance with more training data and better network architectures. By fine-tuning the learned feature, we further obtain competitive results for semi-supervised learning and object detection tasks. Our non-parametric model is highly compact: With 128 features per image, our method requires only 600MB storage for a million images, enabling fast nearest neighbour retrieval at the run time.
1. Introduction
The paper proposes unsupervised instance-level discrimination to learn representations that capture apparent visual similarity without semantic annotations. It addresses the resulting million-class computational challenge with noise-contrastive estimation and evaluates the learned metric through non-parametric retrieval and classification.
- Motivation and formulation: Obtaining fully annotated datasets is often costly or infeasible, motivating increased attention to unsupervised learning.Most successful computer-vision models had previously relied on supervised learning with task-specific annotations.
- Motivation and formulation: Instance-level discrimination extends class-wise supervision by learning features that distinguish individual images while capturing apparent similarity among instances.The approach relies on pure discriminative learning without semantic categories.
- Computational challenge: 1.2-million instead of 1,000 classes makes directly extending softmax infeasible for ImageNet.The paper approximates the full softmax with noise-contrastive estimation and uses proximal regularization to stabilize learning.
- Non-parametric learning: The method stores instance features in a discrete memory bank and performs test-time classification with k-nearest neighbors based on feature distances.This defines a non-parametric metric-learning approach for both training and testing.
- Experimental results: 46.5% top-1 accuracy on ImageNet 1K and 41.6% for Places 205 demonstrate large-margin state-of-the-art performance under unsupervised learning.The method also consistently improves test performance with more training data and better network architectures.
2. Related Works
Prior unsupervised learning work primarily comprises generative models and self-supervised approaches, alongside related metric-learning and exemplar-based methods. Generative models focus on reconstructing data distributions, while self-supervised methods create predictive tasks from internal data structure.
- Unsupervised Learning: Unsupervised learning without human-provided labels mainly comprises generative models and self-supervised approaches.These are identified as the two primary categories of previous work.
- Generative Models: Generative models aim to reconstruct data distributions faithfully, with latent features supporting object recognition.Classical examples include RBMs and autoencoders; GANs and variational autoencoders improve generative qualities and feature learning.
- Self-supervised Learning: Self-supervised learning exploits internal data structure by training models to predict omitted aspects or components of instances.Image tasks include context prediction, object counting, image completion, color recovery, and jigsaw solving.
- Metric Learning: Feature learning can be viewed as metric learning because each representation induces a distance between instances.Metric learning has achieved competitive performance in face recognition and person reidentification, where test classes are disjoint from training classes.
- Exemplar CNN: Exemplar CNN resembles this work but uses a parametric paradigm during training and testing, whereas the proposed method is non-parametric.The passage also describes Exemplar CNN as computationally demanding for large-scale datasets such as ImageNet.
3. Approach
The approach learns an unsupervised embedding by treating each image as its own class and discriminating among individual instances. A non-parametric softmax with a memory bank, NCE approximation, and proximal regularization makes training scalable and stable.
- Instance-level discrimination: The method learns an embedding v = fθ(x) that induces distances between images, aiming to place visually similar instances closer together.The embedding is produced by a deep neural network and defines dθ(x, y) = ∥fθ(x) − fθ(y)∥.
- Instance-level discrimination: Instance-level discrimination treats every image as a distinct class and trains a classifier to distinguish individual instances.This reformulates unsupervised feature learning as instance-level classification rather than classification over annotated categories.
- Non-parametric classifier and memory bank: The non-parametric classifier replaces class prototypes with direct instance comparisons, normalizes features to ∥v∥ = 1, and uses temperature τ to control distribution concentration.A feature memory bank stores representations for all images, avoiding repeated computation and gradient storage for class weight vectors.
- Noise-contrastive estimation: NCE converts the large multi-class problem into binary data-versus-noise discrimination, using uniform noise Pn = 1/n and noise samples m times more frequent than data samples.Both data and noise features are sampled from the non-parametric memory bank, while the normalizing constant is estimated by Monte Carlo approximation.
- Optimization and scalability: O(n) to O(1) per sample: NCE reduces computational complexity while retaining competitive performance.Proximal regularization addresses oscillations caused by visiting each instance class once per epoch and empirically stabilizes training, speeds convergence, and improves representations with negligible extra cost.
4. Experiments
Experiments show that non-parametric instance-level feature learning substantially improves CIFAR-10 and ImageNet classification, while producing compact, scalable representations that generalize to semi-supervised learning and object detection.
- CIFAR-10: On CIFAR-10, non-parametric softmax improves linear-SVM accuracy from 60.3% to 75.4% and kNN accuracy from 63.0% to 80.8%.With m = 1 negative, kNN accuracy drops to 42.5%, demonstrating the importance of the NCE approximation.
- ImageNet classification: On ImageNet, the method achieves 35.6% with AlexNet and linear classification, rising to 54.0% with ResNet-50 versus 31.5% for Exemplar CNN with ResNet-101.These results outperform all reported baselines, including the state-of-the-art.
- ImageNet classification: Nearest-neighbor accuracy reaches 31.3%, 33.9%, 41.0%, and 46.5% with AlexNet, VGG16, ResNet-18, and ResNet-50, respectively.The results are close to linear-classification performance, indicating that the learned representation induces a useful metric.
- Efficiency: The 128-dimensional features encode all 1.28M ImageNet images in about 600 MB, with exhaustive nearest-neighbor search taking 20 ms per image on a Titan X GPU.The compact representation is efficient for storage and retrieval.
- Representation analysis: Testing accuracy continues improving as training proceeds without overfitting, while accuracy increases with larger training sets and plateaus at a 128-dimensional embedding.The embedding-size study varies representations from 32 to 256 dimensions.
- Task generalization: With 1% labeled data, the method outperforms supervised learning from limited labeled data by 10%, while ResNet-50 detection reaches 65.4% mAP.Detection mAP is 48.1% with AlexNet and 60.5% with VGG16, leaving an 11% gap to supervised pretraining at 76.2% mAP.
5. Summary
The paper presents an unsupervised feature-learning method that maximizes distinction between instances through a novel non-parametric softmax formulation. Experiments show strong image-classification performance on ImageNet and Places, using a compact 128-dimensional representation that scales with more data and deeper networks.
- The method learns features by maximizing distinction between instances with a novel non-parametric softmax formulation.Its motivation is the apparent image similarity captured by supervised learning.
- The approach outperforms the state of the art for image classification on ImageNet and Places.
- The learned representation is compact at 128 dimensions and scales well with more data and deeper networks.