Source-linked AI summary
Learning Representations by Maximizing Mutual Information Across Views
Philip Bachman, R Devon Hjelm, William Buchwalter
TL;DR
The paper addresses how to learn useful visual representations without manually labeled data. It maximizes mutual information across features from multiple views of a shared context and develops AMDIM, which reaches 68.1% ImageNet accuracy with linear evaluation while outperforming prior and concurrent results. Mixture-based representations also produce segmentation-like behavior, although their sensitivity to hyperparameters limits tuning reported for ImageNet.
Problem
Learning useful representations from unlabeled data remains challenging, while reducing dependence on costly manually annotated corpora remains important.
Method
AMDIM maximizes mutual information across features from independently augmented views and multiple scales, extending local Deep InfoMax with a more powerful encoder and mixture-based representations.
Results
68.1% ImageNet accuracy with linear evaluation beats the best prior result by over 12% and the best concurrent result by 7%.
Takeaways & Limitations
The approach improves prior self-supervised results while remaining computationally practical, and mixture-based representations exhibit segmentation-like behavior as a side-effect.
Takeaways & Limitations
The mixture-based model is more sensitive to hyperparameters and was not tuned for ImageNet.
Abstract
from arXiv · showhide
We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views -- e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https://github.com/Philip-Bachman/amdim-public.
1 Introduction
The paper frames self-supervised representation learning as a way to reduce dependence on costly labeled corpora and proposes maximizing mutual information across multiple views of shared contexts. Its AMDIM model improves benchmark performance, including 68.1% ImageNet linear-evaluation accuracy.
- Self-supervised learning aims to reduce dependence on potentially costly manually annotated corpora.
- The proposed approach maximizes mutual information between features extracted from multiple views of a shared context.
- Multiple views can come from data augmentation or partitioning an image into past and future pixel sets under an autoregressive ordering.
- AMDIM extends local Deep InfoMax with independently augmented inputs, multiscale prediction, and a more powerful encoder.
- 68.1% ImageNet accuracy with linear evaluation beats the best prior result by over 12% and the best concurrent result by 7%.
2 Related Work
Self-supervised learning creates supervised problems procedurally from prior knowledge about useful data structure. Related approaches predict removed spatial or color information, while evaluation must separate representation-learning gains from architecture and training improvements.
- Self-supervised learning relies on procedurally generated supervised problems rather than explicit labels.
- Prior computer-vision methods learn representations by predicting or reconstructing spatial structure or color information removed from images.
- Mutual-information objectives have been applied across views such as color-removed and full-color video, with object tracking emerging as a side-effect in one special case.
- The model uses the contrastive bound from Contrastive Predictive Coding rather than explicit density estimation.
- Evaluation gains can reflect architecture and training improvements, so the paper follows protocols comparing against optimized existing methods.
3 Method Description
AMDIM extends local Deep InfoMax by predicting features across independently augmented views and multiple encoder scales, using a stronger encoder and mixture-based representations. Its mutual-information objective is estimated with NCE, including efficient minibatch computation and augmentation-aware negative sampling.
- AMDIM overview: AMDIM extends local DIM with independently augmented views, simultaneous multiscale prediction, a more powerful encoder, and mixture-based representations.These modifications are introduced as extensions to local DIM’s mutual-information objective.
- Local DIM: Local DIM maximizes mutual information between global antecedent features and spatially indexed local consequent features from an encoder.The objective is defined over feature distributions produced from shared inputs and spatial locations.
- Noise-Contrastive Estimation: NCE trains the encoder to identify each true antecedent–consequent pair among distractor consequents sampled independently from the marginal distribution.The loss is a log-softmax over matching scores, with higher scores indicating more likely positive pairs.
- Data augmentation: For augmented views, AMDIM samples two stochastic augmentations of each input and predicts features from one view using features from the other.The augmentations include random resized crops, color jitter, grayscale conversion, and a random horizontal flip.
- Multiscale mutual information: AMDIM maximizes mutual information across feature scales, using 1-to-5, 1-to-7, and 5-to-5 costs with uniformly sampled feature locations.The expectations can be approximated efficiently with many samples processed in one encoder pass per batch of view pairs.
4 Experiments
AMDIM is evaluated on standard self-supervised representation-learning benchmarks using frozen encoder features and linear or MLP classifiers. It outperforms prior methods across datasets, with especially strong ImageNet and STL10 results, while ablations and visualizations expose the roles of augmentation, multiscale learning, and mixture representations.
- Evaluation protocol: AMDIM is evaluated on CIFAR10, CIFAR100, STL10, ImageNet, and Places205 by training linear and MLP classifiers on frozen encoder features.The protocol trains the encoder without labels, then evaluates classifier accuracy without backpropagating into the encoder.
- Benchmark results: 91.2% and 70.2% accuracy are obtained with linear evaluation on CIFAR10 and CIFAR100, respectively, with corresponding MLP results of 93.1% and 72.8%.These results are described as comparable with slightly older fully supervised models and ahead of other self-supervised feature-learning work.
- Ablations: Data augmentation has the strongest ablation effect by a large margin, followed by stability regularization and multiscale prediction.The ablations use 50-epoch models and compare stronger augmentation, multiscale feature learning, and NCE cost regularization against a basic-augmentation baseline.
- Benchmark results: 94.2% accuracy on STL10 is achieved with linear evaluation, comparing favorably with semi-supervised results that fine-tune the encoder.The cited semi-supervised reference results are 88.8% and 94.4% accuracy.
- Benchmark results: 68.1% accuracy on ImageNet is achieved with linear evaluation, beating the best prior result by over 12% and the best concurrent result by 7%.MLP evaluation reaches 69.5% accuracy, and the model is reported to be smaller than the models producing those comparison results.
- Model behaviour: Mixture-based representations produce segmentation behaviour as a side-effect, but the mixture model is more sensitive to hyperparameters and was not tuned for ImageNet.On STL10, a simple bag-of-features evaluation using mixture features yields roughly a 1% performance boost.
5 Discussion
AMDIM maximizes mutual information across multiple views and improves prior self-supervised results while remaining computationally practical. The approach is presented as extensible across domains, with scalability, architecture compatibility, mixture representations, and regularization identified for future work.
- AMDIM maximizes mutual information between arbitrary features extracted from multiple views of a shared context.The approach is designed to capture natural relations across views and is intended to remain computationally practical.
- The approach extends beyond images to domains including video, audio, and text.The authors specifically suggest using multiple views of local spatio-temporal contexts in video.
- Future work includes improving compatibility with standard architectures and scalability.The authors also identify better infrastructure as part of the scalability agenda.
- Further research is needed on mixture-based representations and the formal and empirical role of regularization in the NCE-based mutual information bound.