Source-linked AI summary
Representation Learning with Contrastive Predictive Coding
Aaron van den Oord, Yazhe Li, Oriol Vinyals
TL;DR
Unsupervised learning seeks representations that are less specialized to individual tasks and modalities. Contrastive Predictive Coding predicts future observations in a compact latent space using autoregressive modeling and a contrastive loss, achieving strong or state-of-the-art performance across audio, images, natural language, and reinforcement learning.
Problem
Unsupervised learning is needed for robust, generic representations that retain information beyond what individual supervised tasks require.
Method
Contrastive Predictive Coding learns compact latent representations by predicting future observations with autoregressive models and a probabilistic contrastive loss.
Results
CPC achieves strong or state-of-the-art performance across audio, images, natural language, and reinforcement learning, improving vision accuracy by 9% absolute top-1 and 4% absolute top-5.
Takeaways & Limitations
CPC provides a domain-agnostic unsupervised framework for extracting useful high-level representations across substantially different data modalities.
Abstract
from arXiv · showhide
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
1 Introduction
The introduction motivates less task-specific, more general representations because supervised features can omit information irrelevant to their training task, while unsupervised learning still lacks a broadly successful approach. The paper proposes Contrastive Predictive Coding: compact latent embeddings, autoregressive future prediction, and a noise-contrastive loss for learning representations across modalities.
- Motivation: Supervised representation learning has advanced AI but still faces challenges in data efficiency, robustness, and generalization.Layered differentiable models made manually specified features largely redundant and improved performance in real-world applications.
- Motivation: Task-specific supervised features can transfer within related domains yet omit information irrelevant to the original task but useful elsewhere.The passage gives image classification features that may lack color or counting information relevant to image captioning.
- Problem: Unsupervised learning remains without a comparable breakthrough, and the ideal modality-independent representation is still unclear.Learning high-level representations directly from raw observations without additional supervision or specialization remains elusive.
- Proposed approach: Contrastive Predictive Coding compresses high-dimensional inputs into latent embeddings, predicts many future steps with autoregressive models, and trains using Noise-Contrastive Estimation.The proposed setup is illustrated for audio and is also used for images, text, and reinforcement learning.
2 Contrastive Predicting Coding
Contrastive Predictive Coding learns compact representations by predicting future latent observations from autoregressive context while maximizing shared information and discarding local noise. Its InfoNCE objective replaces high-dimensional reconstruction with density-ratio estimation using negative samples, yielding representations for downstream tasks across modalities.
- Motivation: The model preserves shared information between signal parts while discarding local noise by maximizing mutual information between encoded representations.This extracts underlying latent variables common to the inputs and supports prediction over longer time horizons.
- Contrastive prediction: CPC predicts future representations rather than reconstructing observations, modeling a density ratio between future targets and context instead of the full high-dimensional data distribution.This avoids computationally intensive conditional generative modeling and enables comparison with randomly sampled negatives.
- Architecture: CPC encodes observations into latent representations and uses an autoregressive model to summarize past latents into a context representation for future prediction.The encoder produces z_t = g_enc(x_t), while the autoregressive model produces c_t = g_ar(z_≤t).
- InfoNCE objective: InfoNCE classifies one conditional positive sample against N −1 negatives from the marginal proposal distribution, and its optimum estimates p(x_t+k|c_t)/p(x_t+k).The optimal density-ratio estimate is independent of the number of negative samples N −1.
- InfoNCE objective: Minimizing InfoNCE maximizes a mutual-information lower bound, I(x_t+k, c_t) ≥ log(N) − L_N, which tightens as N increases.The mutual-information bound can be evaluated even though it is not required for training.
- Method contribution: CPC combines future prediction with probabilistic contrastive learning to extract slow features that maximize mutual information over long time horizons.Either z_t or c_t can serve downstream tasks, depending on whether additional past context is useful.
3 Experiments
Experiments evaluate CPC representations across speech, images, natural language, and reinforcement learning using classification, qualitative, and learning-speed measures. CPC predicts future latent representations in speech, improves ImageNet classification, performs similarly to skip-thought vectors, and improves agent performance in most tested games.
- Experimental overview: CPC is evaluated in four domains—speech, images, natural language, and reinforcement learning—using representation probes or measurements of agent learning speed.The probes include linear classification, qualitative evaluations, and auxiliary-loss effects on reinforcement-learning agents.
- Speech: CPC speech representations capture both speaker identity and speech content, achieving good linear-classification accuracies close to fully supervised oracle networks.Speaker identity is evaluated over 251 speakers without averaging utterances over time.
- Natural language: Natural-language evaluation finds performance very similar to skip-thought vectors, while avoiding a powerful LSTM word-level decoder and therefore training faster.The comparison is made on evaluation tasks using the same datasets as the other models.
- Reinforcement learning: After training on 1 billion frames, contrastive loss significantly improves agent performance in 4 of 5 games and has no effect in the remaining game.For lasertag_three_opponents_small, the authors attribute the lack of benefit to a reactive task that does not require memory.
4 Conclusion
The paper presents Contrastive Predictive Coding as an unsupervised framework for learning compact latent representations that encode predictions about future observations across multiple domains.
- 4 Conclusion: CPC extracts compact latent representations that encode predictions over future observations.The framework is designed for unsupervised representation learning.
- 4 Conclusion: CPC combines autoregressive modeling, noise-contrastive estimation, and predictive-coding intuitions to learn abstract representations unsupervised.
- 4 Conclusion: The representations were tested on audio, images, natural language, and reinforcement learning.The paper reports strong or state-of-the-art performance across these domains.
A Appendix · A.1 Estimating the Mutual Information with InfoNCE
The appendix explains that optimizing InfoNCE maximizes a mutual-information bound and analyzes its estimator properties, including dependence on the number of samples and instability of direct MINE in easy prediction settings.
- A.1 Estimating the Mutual Information with InfoNCE: Optimizing InfoNCE maximizes the mutual information between c_t and z_t+k, which is bounded by the mutual information between c_t and x_t+k.This establishes the information-theoretic interpretation of the CPC loss.
- A.1 Estimating the Mutual Information with InfoNCE: The optimal scoring function f(x_t+k, c_t) is given by the conditional-to-marginal density ratio p(x_t+k|c_t) / p(x_t+k).The derivation then separates the positive example from the negative examples.
- A.1 Estimating the Mutual Information with InfoNCE: Equation 8 becomes more accurate as N increases, while log(N) − L_N also increases, making large N useful.The passage notes that the statement also holds for scoring functions that produce a higher, worse L_N.
- A.1 Estimating the Mutual Information with InfoNCE: InfoNCE is related to MINE by expressing the scoring function as f(x, c) = e^F(x,c).Under this parameterization, the resulting expression is equivalent to the MINE estimator up to a constant.
- A.1 Estimating the Mutual Information with InfoNCE: The CPC objective maximizes a lower bound on the MINE estimator rather than maximizing the estimator directly.The equivalence is stated up to a constant.
- A.1 Estimating the Mutual Information with InfoNCE: Direct MINE matched InfoNCE performance on nontrivial tasks but became highly unstable when the target was easy to predict from context.The reported easy case involved predicting one future step when the target overlaps with the context.