PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Multiview Coding

Yonglong Tian, Dilip Krishnan, Phillip Isola

arXiv:1906.05849v5cs.CVcs.LG

TL;DR

The paper asks how unsupervised representations can preserve useful factors shared across noisy, incomplete views while discarding view-specific information. It proposes contrastive multiview learning that maximizes mutual information between representations of co-occurring views and generalizes to any number of views. The framework improves with more views, outperforms cross-view prediction, and performs competitively on unsupervised-learning benchmarks.

  • Problem

    The paper addresses how to identify and retain useful shared factors such as semantics, physics, and geometry while discarding view-specific noise in unsupervised representations.

  • Method

    The method contrasts representations of co-occurring views from the same scene, extending mutual-information-based contrastive learning to arbitrary collections of views.

  • Results

    The framework improves representation quality as the number of views increases, achieves competitive benchmark results, and learns stronger representations than cross-view prediction.

  • Takeaways & Limitations

    Multiview contrastive learning provides a view-agnostic way to learn compact representations that capture information shared across sensory channels.

  • Takeaways & Limitations

    The full-graph formulation has combinatorial computational cost in the number of views, despite capturing additional cross-view information and handling missing views naturally.

Abstract

from arXiv · show

Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We investigate the classic hypothesis that a powerful representation is one that models view-invariant factors. We study this hypothesis under the framework of multiview contrastive learning, where we learn a representation that aims to maximize mutual information between different views of the same scene but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. We analyze key properties of the approach that make it work, finding that the contrastive loss outperforms a popular alternative based on cross-view prediction, and that the more views we learn from, the better the resulting representation captures underlying scene semantics. Our approach achieves state-of-the-art results on image and video unsupervised learning benchmarks. Code is released at: http://github.com/HobbitLong/CMC/.

1. Introduction

The paper argues that useful representations should preserve information shared across views while discarding view-specific noise, and develops contrastive multiview learning toward that goal. It extends this framework beyond two views and reports stronger representations as views increase, competitive benchmark performance, and advantages over cross-view prediction.

  • Motivation: Useful representations should retain shared semantic, physical, and geometric information while discarding view-specific nuisance factors.The paper contrasts important properties such as semantics, physics, and geometry with incidental lighting, camera pose, and sensor noise.
  • Method: Contrastive multiview learning brings representations of views from the same scene together and separates representations of different scenes.The method uses co-occurring views of the same scene as supervisory signal and learns compact embeddings.
  • Results: The learned representations achieve competitive results on standard semantic recognition benchmarks and outperform cross-view prediction in direct comparisons.The method is evaluated through transfer to downstream semantic recognition tasks.
  • Method: The framework applies contrastive learning to multiple views while maximizing mutual information between representations of different views.Its core ideas include contrastive learning, mutual information maximization, and deep representation learning.
  • Results: Representation quality improves as the number of training views increases.The paper presents this as an empirical finding and explicitly studies multiview settings beyond two views.

2. Related work

The related work situates unsupervised representation learning among classical, generative, self-supervised, and contrastive approaches. It highlights prior multiview and predictive methods that motivate the paper’s contrastive framework.

  • Classical representation learning: Classical unsupervised methods such as PCA and ICA often focus on low-level data variations rather than downstream semantic tasks.The related-work discussion contrasts these methods with representations designed to support subsequent problem solving.
  • Deep representation learning: Deep representation learning includes Boltzmann machines, autoencoders, variational autoencoders, generative adversarial networks, and autoregressive models.These methods form part of the broader history preceding self-supervised learning.
  • Self-supervised learning: Self-supervised methods learn from related signal pairs such as luminance and chrominance, image patches, vision and sound, or video frames.The appeal of this paradigm is that naturally occurring correspondences provide abundant training data.
  • Learning objectives: Deep representation objectives include reconstruction-based, adversarial, and contrastive losses that exploit relationships among data views.The paper places contrastive learning within this broader set of objective functions.
  • Contrastive learning: CPC, Deep InfoMax, and Instance Discrimination are closely related contrastive methods that learn by contrasting congruent and incongruent representations.CPC uses past and future views and is applicable to sequential data, while the other methods consider alternative view constructions.

3. Method

The method learns compact representations that preserve information shared across multiple views by contrasting matched and mismatched view pairs. It extends this objective to arbitrary numbers of views through core-view and full-graph formulations, while relating the objective to mutual-information estimation.

  • Multiview objective: The framework learns representations that capture information shared between multiple sensory views without supervision.It uses separate view encoders and targets compact representations that discard channel-specific nuisance factors.
  • Predictive versus contrastive learning: Unlike cross-view prediction, contrastive learning measures loss in representation space by contrasting matched and mismatched views.Prediction losses such as L1 and L2 penalize output dimensions independently, potentially missing shared information.
  • Two-view contrastive learning: Contrastive learning compares congruent pairs from the joint distribution with incongruent pairs from the product of marginals.A critic is trained to select the single positive from a set containing k negative samples.
  • Mutual-information connection: Minimizing the contrastive objective maximizes a lower bound on mutual information between encoded views, although the bound can be weak.The bound depends on the number of negative samples, and the paper reports improved representations with more negatives.
  • More than two views: The core-view and full-graph formulations extend the objective to any number of views using selected pairs or all view pairs.The full graph captures information between every pair and can naturally handle missing views, whereas its bivariate score cost is combinatorial in the number of views.
  • Information weighting: Both multiview formulations prioritize information according to how many views share it, favoring factors common to all views over factors affecting fewer views.For example, “presence of dog” is preferred over “depth sensor noise.”

4. Experiments

CMC is evaluated across image, video, and multiview benchmarks, testing transferability, view scaling, and contrastive learning against predictive alternatives. Across these settings, multiview representations improve semantic performance and contrastive learning outperforms predictive learning when task and dataset are unknown.

  • ImageNet: 68.3% top-1 single crop accuracy is achieved on ImageNet with {L, ab} and ResNet50x2 for each view.Switching to {Y, DbDr} adds 0.7% improvement, while RandAugment yields better or comparable results to other state-of-the-art methods.
  • Video: Unifying ventral and dorsal streams during pre-training produces higher downstream action-recognition accuracy than using only one stream.Increasing views from 2 to 3 by adding optical flow provides a boost for UCF-101.
  • Multiview learning: Semantic-labeling performance steadily improves as CMC adds views in the order L, ab, depth, and surface normals.The evaluated metrics are mean IoU and pixel accuracy, and different view-addition orders show a similar pattern.
  • Multiview learning: CMC produces high-quality feature maps despite being unaware of the downstream task, relative to Random and Supervised baselines.The Random baseline is a lower bound, while the end-to-end Supervised baseline serves as an upper bound.
  • Multiview learning: Full-graph CMC representations significantly outperform randomly projected representations and approach fully supervised performance for all investigated views.Each view is evaluated independently for semantic-label prediction, while the full graph contrasts every view with all other views.
  • Predictive versus contrastive learning: Contrastive learning consistently outperforms predictive learning when both the downstream task and dataset are unknown.On STL-10, object-recognition accuracy is close to the supervised method.

5. Conclusion

The paper presents multiview contrastive learning as a framework for unsupervised representations that maximizes mutual information across views and extends to any number of views. Empirical studies show advantages over predictive learning and improvements from additional views.

  • Conclusion: The framework learns unsupervised representations from multiple views by maximizing mutual information between views of the same scene.It is designed to retain shared information while discarding view-specific nuisance factors.
  • NCE approximation: NCE approximates the full N-way softmax objective using noise samples and is computationally faster than the subsampled softmax alternative.The passage notes that NCE may yield slightly worse performance in standard linear evaluation.
  • Contrastive objectives: Patch-based contrastive loss avoids a memory bank and is easy to deploy in parallel training, but usually performs worse than NCE-based contrastive loss.It increases negative pairs by contrasting global and local features within each mini-batch.

C. Proofs

The proofs characterize the optimal contrastive score as a density-ratio estimator and connect contrastive-loss minimization to a lower bound on mutual information. The bound depends on the number of candidate pairs, whose increase improves approximation accuracy but can make optimization harder.

  • Mutual-information bound: Minimizing the contrastive loss maximizes a lower bound on mutual information between the two views.The proof establishes this connection after substituting the optimal density-ratio score into the objective.
  • Score function as density ratio estimator: The optimal score function is proportional to the ratio between the joint distribution and the product of the marginal distributions.This result applies whether the score is defined on raw views or latent representations.
  • Score function as density ratio estimator: The derivation is agnostic to the layer where the score function begins, including raw inputs and latent representations.The subsequent derivation focuses on the latent representations z1 and z2.
  • Mutual-information bound: As the number of candidate pairs k increases, the approximation becomes more accurate, but the optimization problem becomes harder and the loss can also increase.Therefore, increasing k to infinity does not always produce a higher mutual-information lower bound.

D.1. CMC on STL-10

On STL-10, CMC is evaluated against unsupervised representation-learning methods using AlexNet-based features and multiple contrastive training setups. Contrastive methods outperform predictive and adversarial alternatives, while CMC leads the compared contrastive methods.

  • Experimental setup: CMC is evaluated on STL-10 using a small AlexNet and a two-layer MLP to classify representations.The comparison includes conv5 and fc7 feature layers, with a separate strided-crop setup.
  • Experimental setup: The evaluation includes patch-based and NCE-based contrastive pretraining, with linear classifier evaluation for the learned features.The patch-based setup follows the data augmentation and architecture used in DIM for fair comparison.
  • Results: Contrastive methods such as DIM, CPC, and CMC achieve higher classification accuracy than predictive SplitBrain and adversarial BiGAN.This comparison is reported across the evaluated STL-10 settings.
  • Results: CMC significantly outperforms DIM and CPC in all reported STL-10 cases.The paper attributes this hypothesized advantage to modeling cross-view mutual information and discarding view-specific noisy details.
  • Results: CMC shows nearly 8% absolute improvement over SplitBrain for conv5 and 17% improvement for fc5.Predictive methods also show a larger performance drop when moving from conv5 to fc7, whereas contrastive methods are more stable.
  • Scope: The STL-10 experiments use AlexNet as the backbone, and the authors suggest that more powerful networks such as ResNets may further improve representation quality.This bounds the reported architecture choice and identifies a possible extension.

D.2. CMC on ImageNet with AlexNet

On ImageNet, CMC is compared with predictive and contrastive unsupervised methods using AlexNet features and 1000-way linear evaluation. CMC is reported as the best compared method, with performance improving at higher convolutional layers and with more negative samples.

  • Experimental setup: ImageNet evaluation uses 1000-way logistic regression classifiers on feature maps from each layer, with single-crop top-1 accuracy as the metric.The comparison includes standard AlexNet and accounts for the reduced parameter count caused by channel splitting.
  • Experimental setup: The ImageNet setup retrieves 4096 negative pairs from a memory bank for each positive pair in the memory-based CMC model.The effect of varying the number of negatives is studied separately.
  • Results: CMC is reported as the best among the compared predictive and contrastive models on ImageNet.The paper also notes that CMC tends to perform better at higher convolutional layers.
  • Negative samples: Accuracy steadily increases as the number of negative examples m increases in the NCE-based contrastive loss.This ablation uses 100 randomly chosen ImageNet classes and varies m from 64 to 8192 pairs.

D.4. Compatibility with other methods

CMC is evaluated for compatibility with MoCo and PIRL on ImageNet-100 using a common contrastive objective and training recipe. The reported table observes consistent improvement with a ResNet-50 when these methods are combined with CMC.

  • CMC is combined with MoCo by replacing its memory bank with a momentum encoder, and with PIRL by adding a second JigSaw branch.
  • The combinations are pre-trained and linearly evaluated on the same ImageNet-100 subset used above.
  • The experiments use the same contrastive loss objective and training recipe across methods for a head-to-head comparison.
  • Consistent improvement is observed with a ResNet-50 when CMC is combined with MoCo, PIRL, or both.

E. Implementation Details

The implementation details specify architectures, optimizers, learning-rate schedules, temperature settings, and evaluation protocols across image, video, and multiview experiments. These choices include split encoders for comparisons with DIM and CPC, CaffeNet features for video, and patch-based encoders for NYU Depth-V2.

  • For comparisons with DIM and CPC, the architecture is split into two encoders and uses an encoder-and-dot-product, or bilinear, score function.
  • Patch-based contrastive learning uses Adam with learning rate 0.001, β1 = 0.5, β2 = 0.999, and 200 training epochs.
  • NCE-based contrastive learning runs for 320 epochs with initial learning rate 0.03, tenfold decay every 40 epochs after epoch 200, and temperature τ = 0.1.
  • For ImageNet experiments, AlexNet uses SGD for 200 epochs with temperature τ = 0.07, initial learning rate 0.03, tenfold decay, weight decay 10^-4, and momentum.
  • Video experiments use CaffeNet features from fc6 for contrastive learning and apply dropout of 0.5 to alleviate overfitting.
  • On NYU Depth-V2, 128 × 128 patches are encoded with Adam for 3000 epochs, with learning-rate decay after epochs 2000, 2400, and 2800.
Loading 1906.05849v5…