Source-linked AI summary
Unsupervised Learning by Predicting Noise
Piotr Bojanowski, Armand Joulin
TL;DR
Supervised annotations enable transferable convolutional features but require substantial labeled data and may introduce bias, while prior unsupervised approaches can be difficult to scale or insufficiently discriminative. The paper fixes low-dimensional noise targets and aligns end-to-end deep features to them, producing representations on par with state-of-the-art unsupervised and self-supervised methods while remaining simple to train and scale.
Problem
Transferable convolutional features depend on massive labeled or weakly supervised datasets, while prior unsupervised methods may not explicitly learn discriminative features and can be difficult to scale.
Method
Noise As Targets (NAT) fixes predefined low-dimensional target representations and aligns them with deep visual features end to end, avoiding feature collapse.
Results
The learned features perform on par with state-of-the-art unsupervised and self-supervised approaches across several image classification problems.
Takeaways & Limitations
NAT provides a simple, fast-to-train unsupervised framework that explicitly targets discriminative features with few assumptions about the input space.
Takeaways & Limitations
The discrete target alternative assumes balanced clusters and a problem structure that is linearly separable into k classes.
Abstract
from arXiv · showhide
Convolutional neural networks provide visual features that perform remarkably well in many computer vision applications. However, training these networks requires significant amounts of supervision. This paper introduces a generic framework to train deep networks, end-to-end, with no supervision. We propose to fix a set of target representations, called Noise As Targets (NAT), and to constrain the deep features to align to them. This domain agnostic approach avoids the standard unsupervised learning issues of trivial solutions and collapsing of features. Thanks to a stochastic batch reassignment strategy and a separable square loss function, it scales to millions of images. The proposed approach produces representations that perform on par with state-of-the-art unsupervised methods on ImageNet and Pascal VOC.
1. Introduction
The paper addresses the difficulty of learning discriminative convolutional features without annotations at the scale required by modern vision models. It proposes fixed noise targets, online assignment, and efficient optimization to train end-to-end, achieving performance on par with leading unsupervised methods.
- Motivation: Convolutional networks produce transferable visual features, but labeled or weakly supervised training can introduce bias and requires massive image databases.The motivation is to learn useful representations without relying on human annotations.
- Motivation: Existing annotation-free methods either exploit input-derived signals or model image distributions, while retrieval and clustering approaches are difficult to scale.Some prior methods also do not explicitly target discriminative features.
- Approach: NAT fixes low-dimensional target representations and aligns end-to-end convolutional features to them, preventing feature collapse while retaining a discriminative objective.The targets are sampled from an uninformative distribution and resemble representations used in clustering.
- Scalability: An online algorithm combines a quadratic loss with a fast Hungarian-algorithm approximation, making training nearly as efficient as standard supervised procedures on massive datasets.The method can reuse optimization procedures designed for supervised training and is demonstrated with AlexNet on ImageNet without supervision.
- Results: The learned features perform on par with state-of-the-art unsupervised and self-supervised approaches across several image-classification problems while being simpler to train and scale.The evaluation follows the transfer-learning setting of Donahue et al.
2. Related work
The related work spans self-supervision, generative and reconstruction-based models, self-organizing maps, and discriminative clustering. NAT differs by mapping images to predefined random targets with a standard convolutional network and end-to-end training.
- Clustering: Clustering methods learn representations using cluster assignments or clustering losses, but earlier approaches were not tested at a scale comparable to supervised convolutional training.The paper trains the convolutional network end-to-end with a loss related to k-means.
- Generative models: Generative models map predefined random variables to images, whereas NAT reverses this direction by mapping images to a predefined random variable.This reversal permits standard convolutional networks and simplifies training.
- Generative models: GAN-based approaches also use discriminative losses, but their visual-feature pipelines may require an added encoder, while NAT directly uses a standard convolutional architecture.The supplied passage identifies the encoder as the mechanism for extracting visual features from GANs.
- Self-supervision: Self-supervised methods derive weak supervision from spatial context, image transformations, or temporal coherence, making them dependent on domain-specific input structure.NAT contrasts with these methods by making few assumptions about the input space.
- Autoencoders: Autoencoders learn features by encoding and decoding images while minimizing reconstruction loss between decoded and original images.Their decoders may be fully connected or deconvolutional networks.
- Self-organizing maps: Self-organizing maps learn low-dimensional representations that preserve topological properties, assigning each datum to a representation vector in winner-takes-all fashion.These representation vectors are conceptually similar to NAT target representations.
- Discriminative clustering: Discriminative clustering methods use losses such as ridge regression to learn clusters for applications including object discovery and video-text alignment.NAT shares this emphasis on discriminative losses while applying it to end-to-end convolutional training.
3. Method
The method learns unsupervised visual features by aligning end-to-end convnet outputs with fixed target representations while preventing feature collapse. Stochastic batch assignments and a square loss make this framework compatible with large-scale optimization.
- Unsupervised learning: The model jointly learns convnet parameters and target vectors for unlabeled images.Features are produced by a parametrized mapping fθ, while target vectors yi are learned alongside θ.
- Unsupervised learning: Fixed target representations prevent all images from collapsing onto one representation.The target matrix is formed from predefined representations and an assignment matrix, diversifying the learned features.
- Unsupervised learning: The assignment strategy restricts each image to one target while preventing target reuse, but small target sets can become problematic.The constraints diversify assignments; the method therefore focuses on cases where k is at least as large as the number n of images.
- Choosing target representations: NAT samples target vectors uniformly on the ℓ2 unit sphere, with representation correlation decreasing as the square root of target dimension increases.The target dimension d directly controls correlation between sampled representations.
- Optimization: Online optimization updates assignments within each batch, reducing assignment complexity to O(b^3) per batch instead of the Hungarian algorithm’s O(n^3).The batch procedure reassigns images only among targets previously assigned within that batch and supports stochastic optimization.
4. Experiments
Experiments evaluate NAT’s design choices and transfer quality on ImageNet and PASCAL VOC. NAT’s features remain competitive with unsupervised and self-supervised methods while using simple, scalable training choices.
- Experimental setup: Experiments evaluate NAT on ImageNet transfer classification and PASCAL VOC 2007 classification and detection.Features are transferred by freezing convolutional layers and retraining the MLP classifier, with task-specific procedures.
- Loss function: Square loss achieves performance similar to softmax loss, supporting its use for NAT training.The square-loss model requires unit-normalized features to avoid exploding gradients.
- Image preprocessing: Grayscale image gradients avoid trivial features without significantly degrading supervised ImageNet classification performance.The experiment indicates that gradient-related information is sufficient for object classification and does not substantially lower the model’s supervised upper-bound performance.
- Target representations: 19% accuracy with discrete targets is significantly below NAT’s best 33.5% on ImageNet transfer classification.The authors suggest that binary vectors create sharp discontinuous distances that are difficult to optimize.
- Assignment updates: Updating assignments every 3 epochs offers a practical trade-off, while test accuracy remains above 30% across permutation frequencies.Higher update frequencies slightly reduce accuracy, possibly because the network overfits to its own output.
- Transfer and qualitative results: NAT captures distinctive global structures and performs on par with self-supervised methods across ImageNet and PASCAL VOC tasks.Its first-layer filters retain edge and orientation information, though they are slightly less sharp than supervised filters; NAT also slightly outperforms BiGAN in reported comparisons.
5. Conclusion
The paper presents NAT, an unsupervised framework that aligns neural-network outputs with low-dimensional noise to learn discriminative features. It achieves performance on par with state-of-the-art unsupervised approaches while remaining simple, fast, and broadly applicable.
- NAT learns discriminative features by aligning neural-network outputs to low-dimensional noise.
- Unlike many unsupervised methods, NAT targets discriminative features rather than surrogate problems such as image denoising or generation.
- NAT makes few assumptions about the input space compared with self-supervised approaches, supporting a domain-agnostic formulation.
- The approach is simple and fast to train, and its features perform on par with state-of-the-art unsupervised learning approaches.
- The study leaves open whether more informative target distributions and alignment methods could improve NAT and connect it more closely to distribution matching.