Learning Representations by Maximizing Mutual Information Across Views

1 Introduction

• Self-supervised learning aims to reduce dependence on potentially costly manually annotated corpora, while pre-trained representations have supported strong results across NLP and vision tasks.Recent work has questioned the necessity of pre-trained networks for many vision tasks.Source: §1 · p. 1
• Maximizing mutual information between features from multiple views of a shared context encourages capture of higher-level factors, such as objects or events, that broadly affect the context.Views can come from repeated image augmentations or fixed-order past/future pixel partitions, as in Contrastive Predictive Coding [CPC, van den Oord et al., 2018].Source: §1 · pp. 1–2
• The model extends local Deep InfoMax [DIM, Hjelm et al., 2019] with prediction across independently augmented inputs, simultaneous multiple scales, and a more powerful encoder; each modification improves on local DIM.With mixture-based representations, segmentation-like behavior emerges as a natural side-effect.Source: §1 · p. 2
• Over 68% accuracy on ImageNet with linear evaluation beats the best prior result by over 12% and the best concurrent result by 7%.The evaluation follows the protocol described by Kolesnikov et al. [2019].Source: §1 · p. 2
• Over 94% linear-evaluation accuracy on STL10 requires no encoder fine-tuning; 55% accuracy on Places205 using ImageNet data beats the best prior result by 7%.The model also outperforms prior work across the evaluated CIFAR10, CIFAR100, STL10, ImageNet, and Places205 datasets.Source: §1 · p. 2

• Self-supervised learning designs procedural problems from prior knowledge of useful data structure, rather than explicit labels.Source: §2 · p. 2
• Prior computer-vision methods learn representations by predicting removed spatial structure, color, or image rotation.Spatial-structure prediction/reconstruction: Doersch et al. [2015] and Noroozi and Favaro [2016]; removed-color prediction: Zhang et al. [2016]; rotation prediction: Gidaris et al. [2018].Source: §2 · p. 2
• Mutual information across views of a shared context supports identifying contextually related observations among plausible alternatives, extending objectives linked to tracking and scene prediction.Vondrick et al. [2018] observed tracking as a side-effect when full-color features were original video frames; Eslami et al. [2018] studied predicting a held-out scene view from other views.Source: §2 · p. 2
• The model uses CPC’s contrastive bound rather than the explicit density estimation used in related mutual-information bounds.The contrastive bound is analyzed by McAllester and Stratos [2018] and Poole et al. [2019].Source: §2 · p. 3
• Evaluation gains can reflect architecture and training improvements, so the authors follow Kolesnikov et al. [2019] protocols and compare against optimized prior-method results.Kolesnikov et al. [2019] reported large gains from updating architectures and scaling training; Goyal et al. [2019] noted potential shortcomings of standard protocols.Source: §2 · p. 3

3.1 Local DIM

• Local DIM maximizes I(f1(x); f7(x)ij), training global features to predict local features relative to their marginal likelihood and discouraging collapsed representations.The global feature comes from the encoder’s 1 × 1 layer and the local feature from its 7 × 7 layer.Source: §3.1 · p. 3

3.2 Noise-Contrastive Estimation

• NCE implements the mutual-information bound by contrasting joint-distribution positive pairs with independent marginal consequent distractors in a log-softmax over matching scores.The antecedent feature must select its true consequent from the distractors.Source: §3.2 · pp. 3–4

3.3 Efficient NCE Computation

• Dot-product matching makes NCE tractable with |N7| ≫ 10000, but finite-dimensional embeddings limit the rank of representable belief shifts.The method applies nonlinear transforms to features before linear evaluation of these shifts.Source: §3.3 · p. 4

3.4 Data Augmentation

• AMDIM predicts between features from independently augmented copies of each input, using random resized crops, color jitter, grayscale conversion, and horizontal flipping.Negative consequent features are sampled from the augmented-view marginal distribution.Source: §3.4 · p. 4

3.5 Multiscale Mutual Information

• AMDIM maximizes mutual information across 1-to-5, 1-to-7, and 5-to-5 feature scales, uniformly sampling feature locations and estimating costs efficiently in one encoder pass per batch.These are the scales used in the paper’s experiments.Source: §3.5 · p. 5

3.6 Encoder

• The ResNet-based encoder avoids padding and controls receptive-field growth because excessive overlap between positive-pair receptive fields makes prediction too easy and hurts performance.Its blocks use mean pooling and convolutional residual layers, including 1 × 1 layers to limit receptive-field growth.Source: §3.6 · p. 5

3.7 Mixture-Based Representations

• Mixture features are generated from antecedent features by a residual fully connected ReLU network; entropy regularization tempers greedy assignments based on stochastic NCE scores.The assignment distribution is computed using a temperature parameter treated as a hyperparameter.Source: §3.7 · p. 6

4 Experiments

• Evaluation trained encoders without labels, then fit linear and MLP classifiers on frozen f1(x) features and measured accuracy.Benchmarks were CIFAR10, CIFAR100, STL10, ImageNet, and Places205, following the protocol described by Kolesnikov et al. [2019].Source: §4 · p. 6
• AMDIM reached 91.2% linear and 93.1% MLP accuracy on CIFAR10, 70.2% and 72.8% on CIFAR100, and 94.2% linear accuracy on STL10.The CIFAR results were described as comparable to slightly older fully supervised models and ahead of other self-supervised feature-learning work; the STL10 result compared favorably with cited semi-supervised results of 88.8% and 94.4%.Source: §4 · pp. 6–7, Table 2
• 68.1% was AMDIM’s best ImageNet result, and the model outperformed prior and concurrent methods by a large margin.Table 1 compares ImageNet and ImageNet→Places205 transfer using linear evaluation; the 68.1% result is the reference point for shorter ablation runs.Source: Table 1 · p. 7, Table 3 · p. 7
• Ablations found data augmentation had the strongest effect by a wide margin, followed by stability regularization and multiscale prediction.The ablations used 50-epoch models; the ImageNet models were smaller and trained for one third as long as the 68.1% best result, and the strongest results used Fast AutoAugment.Source: Table 3 · p. 7
• Mixture-based representations produced segmentation behavior as a side effect, while a simple bag-of-features evaluation gave roughly a 1% performance boost on STL10.The mixture model was more sensitive to hyperparameters and had not been tuned for ImageNet; visualizations also showed ImageNet retrieval often focused on the main object, while Places205 similarity was more diffuse.Source: §4 · pp. 7–8, Figure 2 · p. 8

5 Discussion

• AMDIM maximizes mutual information between arbitrary features from multiple views of a shared context and improves on prior results while remaining computationally practical.The authors say the approach could extend to video, audio, and text.Source: §5 · p. 8
• The authors expect that modeling natural relations across multiple views of local spatio-temporal video contexts could improve AMDIM.They present video as an example of extending the approach to other domains.Source: §5 · p. 8
• Future research directions include adapting the objective to standard architectures, improving scalability and infrastructure, studying mixture-based representations, and examining regularization in the NCE-based mutual information bound.The authors characterize contrastive self-supervised learning as promising and AMDIM as a particularly effective approach.Source: §5 · p. 8