PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Representation Learning: A Framework and Review

Phuc H. Le-Khac, Graham Healy, Alan F. Smeaton

arXiv:2010.05113v2cs.LGstat.ML

TL;DR

Contrastive learning has expanded across fields without a unified framework for understanding its methods and trade-offs. This paper surveys its history and applications, proposes a general Contrastive Representation Learning framework with component taxonomies, and discusses its inductive biases, limits, and future directions. The review presents contrastive learning as suited to some datasets and to settings where labeled data are unavailable or insufficient, while noting that successful training can require more computation even though pretrained representations may support stronger downstream performance.

  • Problem

    The field spans multiple domains and methods, making its history, novelties, and trade-offs difficult to analyze without a common framework.

  • Method

    The paper proposes a general Contrastive Representation Learning framework, surveys the field, and supplies taxonomies for its components across application domains.

  • Results

    The review presents contrastive learning as applicable to labeled or self-supervised settings, with applications across image, video, text, audio, and other domains.

  • Takeaways & Limitations

    Contrastive representation learning is suited to some datasets and applications where labeled training data are unavailable or insufficient, and can yield rich general-purpose representations for downstream tasks.

Abstract

from arXiv · show

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

I. INTRODUCTION

The paper motivates contrastive representation learning as a way to encode invariant and disentangled properties that are difficult to learn explicitly, and proposes a framework to organize the field. It surveys the approach across histories, methods, domains, applications, limits, and future directions.

  • Motivation: Representation learning maps raw inputs to feature vectors or tensors intended to capture useful abstract concepts for downstream tasks.The paper frames representation quality as important because system performance depends directly on the data representation.
  • Motivation: Contrastive learning offers a simple way to encode invariant and disentangled properties in learned representations.These properties are harder to obtain through deep architectures alone and are often learned implicitly with the task.
  • Contributions: The paper proposes a Contrastive Representation Learning framework that may shift emphasis from architecture-engineering toward data-engineering.Its framework is intended to simplify analysis of contrastive methods and explain their workings.
  • Contributions: The survey provides a taxonomy of framework components and synthesizes contrastive learning’s history and conceptual development across multiple sub-fields.It aims to make methods, novelties, and trade-offs easier to categorize and explain.
  • Scope: The paper connects contrastive learning with other methods and presents applications across language, vision, audio, and additional domains.The stated application coverage includes images, videos, text, audio, and combinations of these inputs.
  • Scope: It analyzes current limits and discusses future research directions for contrastive representation learning.The paper also presents an overview, formal framework, component taxonomy, domain applications, topical discussion, and conclusion.

II. WHAT IS CONTRASTIVE LEARNING ?

Contrastive learning learns representations by comparing similar and dissimilar samples rather than reconstructing individual inputs or relying on human labels. Its central design choice is the similarity and dissimilarity structure used to pull related representations together and push unrelated ones apart.

  • Contrastive representation quality is approximated by how well the learned representation separates similar and dissimilar samples.This provides a proxy objective when representation quality itself has no straightforward direct measure.
  • Contrastive learning compares input samples to learn representations, rather than reconstructing inputs or processing samples independently.The approach is framed as discriminative representation learning based on comparisons among samples.
  • Similar samples are mapped close together, while dissimilar samples are pushed farther apart in the embedding space.Positive pairs are pulled together and negative pairs are pushed apart through contrastive training.
  • Self-supervised contrastive methods define similarity from the data itself, generating pairwise training signals without human labels.The similarity definition functions as a pretext signal for discriminative learning.
  • Additional labels can be incorporated into similarity and dissimilarity definitions, allowing contrastive learning in both supervised and self-supervised settings.The framework defines these distributions at the dataset level rather than for individual samples.
  • Contrastive methods are discriminative models that predict pseudo-labels of similarity or dissimilarity for pairs of inputs.They occupy the discriminative side of the generative-discriminative spectrum.

C. EXAMPLE: INSTANCE DISCRIMINATION

Instance Discrimination learns visual representations by contrasting augmented views of the same image against views from different images. The example instantiates the broader CRL framework with encoders, projection heads, pair sampling, and a contrastive loss.

  • C. EXAMPLE: INSTANCE DISCRIMINATION: Instance Discrimination treats each image as its own class and learns visual representations through a self-supervised pretext task.The method has achieved state-of-the-art transfer-learning results for some downstream computer-vision tasks.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: Two differently augmented views of the same image form a positive pair, while views from different images form negative pairs.Strong augmentations help prevent agreement based only on low-level visual cues.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: An encoder maps each augmented view to a feature vector, and a projection head maps that representation to a lower-dimensional metric embedding.The embeddings are normalized before contrastive loss evaluation.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: A batch of positive and negative embedding pairs is optimized with a contrastive loss that reduces same-pair distance and increases distances between different pairs.InfoNCE and NT-Xent are identified as popular choices for this loss.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: The InfoNCE objective maximizes similarity for the positive pair while minimizing similarity to the negative pairs in the denominator.The denominator contains one positive and K negative pairs from the same minibatch.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: After contrastive training, the projection head is discarded and the encoder supplies features for transfer learning and target-task fine-tuning.A predictor or classifier is combined with the encoder representation on a new dataset.
  • C. EXAMPLE: INSTANCE DISCRIMINATION: The general CRL framework extends this instance-discrimination setup across images, video, audio, text, and supervised, self-supervised, or distillation settings.Its taxonomy organizes the components of contrastive methods across these domains and learning setups.

B. A TAXONOMY OF SIMILARITY

Contrastive similarity can be defined through multiple views, including multisensory signals, transformations, context-instance relationships, sequential coherence, and natural clustering. These choices determine which inputs form positive or negative pairs and what invariances the representation learns.

  • Overview: The many possible notions of similarity and dissimilarity make contrastive methods flexible but complicate a complete taxonomy.The framework therefore examines general principles underlying how these relationships are constructed.
  • Multisensory signals: Multisensory contrast compares representations from different sensors to capture shared context while reducing sensitivity to sensor-specific details.Sensors may share a modality, such as multiple cameras, or use different modalities, such as audio and image.
  • Data transformation: Data transformations create positive pairs by applying different semantic-preserving operations to the same input, while other inputs provide negative pairs.Examples include image augmentation, back-translation for text, source-to-source transformations for code, and masking or warping for audio.
  • Context-instance relationship: Context-instance similarity contrasts global or contextual representations with local or future representations to align parts of an observation with its broader context.Deep InfoMax compares global image features with local feature-map representations, while CPC contrasts past context with future features.
  • Sequential coherence and consistency: Sequential coherence defines nearby observations in a sequence as positive and distant observations as negative, encouraging slowly changing features.This principle applies to sequences such as video frames and supports learning features that remain invariant across continuous views.
  • Natural clustering: Natural clustering assumes that categories occupy separate manifolds, while contrastive learning organizes the embedding space locally by pulling positive pairs together and separating negative pairs.Contrastive and clustering methods encode related assumptions from bottom-up and top-down directions, respectively.

C. A TAXONOMY OF ENCODERS

The framework separates encoders according to how query and key representations are produced and updated. It distinguishes end-to-end, online-offline, and pre-trained encoder arrangements, each with different memory and optimization implications.

  • Framework components: The framework divides contrastive models into a base encoder for general representation learning and a transformation head for efficient similarity measurement.This separation recognizes that representation quality and metric computation serve different purposes.
  • End-to-end encoders: End-to-end encoders update query and key encoders directly through contrastive-loss gradients, but can impose substantial memory requirements.For same-modality queries and keys, the encoders are usually shared to reduce duplication.
  • Online-offline encoders: Online-offline encoders reduce GPU memory demands by updating an offline encoder separately from the online network.The offline network may use a past checkpoint or a momentum-based weighted average of the online encoder.
  • Online-offline encoders: Memory banks and dynamic queues store detached or momentum-encoded representations to provide keys without retaining all representations in GPU memory.MoCo uses an exponentially weighted momentum encoder and replaces the oldest queue entries with the latest batch.
  • Pre-trained encoders: Pre-trained encoders remain fixed and are used to learn a shared representation space in cross-modal learning or knowledge distillation.This approach decouples representation learning across modalities or teacher-student networks.

D. A TAXONOMY OF TRANSFORM HEADS

Transform heads separate similarity-oriented embedding operations from the base encoder’s representation objective. The taxonomy includes projection, contextualisation, and quantisation heads, which support different contrastive designs.

  • Overview: Separating transform heads from base encoders prevents similarity optimization from becoming entangled with the goal of learning a useful representation.The same base representation can be trained with multiple heads for different contrastive objectives.
  • Projection heads: Projection heads bridge vector spaces through linear or nonlinear transformations and can reduce the computational cost of measuring similarity.They may use a linear transformation or a multilayer perceptron after the base encoder.
  • Contextualisation heads: Contextualisation heads aggregate multiple feature vectors into a contextualized embedding for similarity definitions based on context-instance relationships.CPC uses a GRU over previous time steps, while DIM aggregates feature maps into a global vector.
  • Contextualisation heads: A contextualized metric embedding can replace or complement the representation embedding when contextual information benefits the downstream task.Its role differs from projection, which primarily maps the representation into another space.
  • Quantisation heads: Quantisation heads reduce representation complexity by mapping multiple representations to the same representation or to discrete assignments.Examples include wav2vec 2.0’s discrete codebook vectors and SwAV’s soft cluster assignment vectors.

E. A TAXONOMY OF CONTRASTIVE LOSS FUNCTIONS

Contrastive losses compare query and key representations through a scoring function and an optimization objective. The taxonomy covers pair, triplet, probabilistic NCE-based, and mutual-information-based formulations, while noting collapse and computational limitations.

  • Contrastive loss combines a compatibility score between vectors with an objective that encourages desired and discourages undesired relationships.Scores may measure similarity or distance, while the loss determines how comparisons are optimized.
  • Contrastive objectives must prevent representation collapse, either with explicit negative pairs or with implicit assumptions and architectural constraints.The review includes methods without explicit negatives when they still contrast queries with positive keys to learn similarity.
  • Energy-based margin losses: Pair loss minimizes positive-pair distance and pushes negative pairs beyond a margin around the query.Only negative keys within the margin contribute to the loss.
  • Energy-based margin losses: Triplet loss enforces a margin between the query’s positive and negative distances, but pair and triplet losses can converge slowly because they compare few samples.Mining hard negatives is used to increase the usefulness of comparisons.
  • Probabilistic NCE-based losses: Probabilistic NCE-based losses identify a positive key among negatives, but NCE avoids evaluating the expensive normalization constant through proxy binary classification.The proxy task discriminates positive data samples from noise negatives.
  • Mutual information-based losses: Mutual-information maximization is a principled motivation, but maximizing positive-pair mutual information alone does not guarantee successful contrastive learning.The paper reports evidence that contrastive-loss success cannot be attributed to mutual information alone.

IV. DEVELOPMENT OF CONTRASTIVE LEARNING

Contrastive learning developed from invariant-representation and metric-learning ideas in the 1990s into a broad family of losses, sampling strategies, architectures, and applications. Recent progress emphasized instance discrimination, data augmentation, projection heads, clustering, and alternatives to explicit negatives.

  • The core idea of comparing related data points without supervision dates to 1992–1993, including mutual-information and Siamese-network formulations.These works pursued invariant representations or metric learning with shared-weight networks.
  • The 2005 contrastive pair loss established learning an invariant embedding where simple distances support recognition and verification.It replaced nonlinear similarity relationships in the input space with distance comparisons in the learned representation space.
  • Triplet losses extended contrastive learning to relative ranking and were applied to language modeling, image similarity, fine-grained similarity, and representation learning.Their uses spanned both supervised and unsupervised settings.
  • Later methods increased comparisons per query and emphasized hard examples and the quality of training pairs to address slow convergence and instability.These developments targeted limitations of pair and triplet losses.
  • NCE-based methods made probabilistic contrastive learning more efficient by avoiding full normalization, supporting language models and word embeddings.NCE contrasted data with noise distributions, while related negative sampling focused on word representations.
  • Recent visual progress centered on instance discrimination and augmentation-based pretraining, while SimCLR separated representation learning from contrastive optimization with a projection head.Other developments combined clustering with instance discrimination or questioned whether explicit negatives are necessary.

V. APPLICATIONS

The paper surveys contrastive representation learning across multiple data domains and problem areas through the generalized framework introduced in Section III.

  • Applications span multiple data domains and problem topics, analyzed through the paper’s generalized Contrastive Representation Learning framework.The supplied passage introduces the applications section but does not enumerate its domains.

A. LANGUAGE

Contrastive learning has been applied to language representations at word, sentence, cross-lingual, code, and sequence levels. Methods vary in how they define positive and negative pairs, encode context, and use NCE, InfoNCE, or triplet objectives.

  • Word and context representations: Collobert and Weston trained language models by classifying valid context–middle-word triples against triples with randomly replaced middle words using triplet loss.Positive triples came from Wikipedia, while negatives were formed by replacing the middle word with a random word.
  • Word and context representations: NCE and negative sampling reduce the computational burden of vocabulary-wide softmax objectives, with negative sampling prioritizing useful word representations over probabilistic properties.NCE uses sampled noise words, whereas negative sampling gives up NCE’s probabilistic interpretation.
  • Transformer-based representations: BERT and XLNet were interpreted through mutual-information and similarity-pair perspectives, motivating BERT-NCE and InfoWord variants.InfoWord contrasts global sentence representations with local representations of masked n-grams.
  • Sentence representations: Sentence-level methods use contextual encoders with nearby or semantically related sentences as positives and unrelated sentences as negatives.Quick-Thought uses context-window sentences, while SentenceBERT uses same-paragraph sentence pairs.
  • Sentence representations: CERT creates positive sentence pairs through back-translation and combines a BERT encoder with InfoNCE.The back-translation process produces different sentences intended to preserve meaning.
  • Cross-lingual and program representations: Cross-lingual and program-representation methods extend contrastive learning beyond ordinary monolingual sentences using parallel data or augmentation-generated equivalent code.InfoXML combines monolingual and cross-lingual token and sentence information, while code representations use transformer token averages with InfoNCE.

B. VISION

Vision research applies contrastive learning across metric learning, instance discrimination, temporal coherence, clustering, and supervised settings to learn invariant and transferable representations. These methods differ in how they define positive and negative pairs, encode context, and prevent representation collapse.

  • Metric learning: Contrastive loss minimizes distances within same-class pairs and maximizes distances between different-class pairs, encouraging invariant mappings for irrelevant input variation.The approach was introduced for metric learning problems with high intra-class and low inter-class variance.
  • Temporal and contextual learning: Temporal and context-instance methods define similarity through sequential coherence or predictive relationships rather than only augmented views of the same image.TCN uses simultaneous views or nearby frames as positives and distant frames as negatives, while CPC maximizes mutual information between past context and future instances.
  • Instance discrimination: Instance discrimination treats each image as its own exemplar class and uses memory banks or strong augmentations to obtain many negative or varied positive pairs.Memory banks decouple the number of negatives from batch size, while methods such as SimCLR emphasize strong and varied augmentations.
  • Video representation learning: Video Deep InfoMax combines image transformations, temporal coherence, and global-local feature correspondence to learn spatio-temporal video representations.Its representations were learned for downstream video tasks.
  • Clustering and supervision: Clustering and supervised variants extend contrastive learning beyond instance-level self-supervision by assigning augmented views to clusters or using class labels to define positives and negatives.SwAV performs online clustering, while supervised contrastive learning was reported as more robust to corruption than cross-entropy alone.
  • Training dynamics: Contrastive objectives balance alignment of positive pairs with uniformity of representations, but NCE-based methods generally benefit from more negatives and can require larger batches and longer training.SwAV and BYOL use alternative prediction or clustering mechanisms that avoid relying on explicit negative pairs to prevent collapse.

C. AUDIO

Contrastive learning is applied to audio, graph, and relational data by defining positives through future context, transformations, structural neighborhoods, or global-local correspondence. Across these domains, encoders and heads adapt the contrastive pipeline to temporal signals, graph structure, and relational transitions.

  • Audio: CPC contrasts a contextualized representation of past audio with the true future representation and noise samples to learn predictive audio features.A strided convolutional encoder produces representations, while a GRU aggregates previous timesteps into the query context.
  • Audio: wav2vec and VQ-wav2vec extend CPC with convolutional context aggregation or discrete speech codes for future-representation prediction.VQ-wav2vec quantizes continuous speech into discrete codes before constructing the context vector.
  • Audio: Wav2vec 2.0 replaces the unidirectional context head with a bidirectional masked Transformer, while mel-spectrogram contrastive learning transfers across spoken languages for emotion classification.These methods illustrate temporal and transformation-based alternatives for audio representation learning.
  • Relational data: Relational methods represent entities and relations with embeddings or transformations, using corrupted objects or energy-based losses to distinguish valid from negative relational configurations.LRE and TransE use matrix or translation-based relation transformations, while C-SWM models state transitions as translations in embedding space.
  • Graphs: Node2vec learns neighborhood-similar node representations with biased random walks and negative sampling.The biased walks provide a flexible definition of network neighborhood used as positive keys.
  • Graphs: Graph contrastive methods create positives through subgraph sampling, graph diffusion, or local-global correspondence to learn node, patch, or graph-level representations.GCC uses subgraphs as instances, DGI maximizes local-global mutual information, and related methods target graph-level or multi-view representations.

E. MULTI-MODAL

Multimodal contrastive learning aligns representations across audio, visual, text, and other modalities, including settings with loosely aligned or separately trained data. Methods use common embedding spaces, correspondence objectives, and expanded positive sets to improve alignment and reduce collapse.

  • Cross-modal representations: Contrastive learning can learn cross-modal embeddings that enhance single-modality representations, particularly when modality-specific labels are limited.The framework supports embeddings across two or more modalities rather than only within one media type.
  • Audio-visual alignment: CMC and AVE-Net define correspondence across multiple views or audio-visual pairs, using separate encoders and contrastive alignment in a shared embedding space.AVE-Net projects audio and visual representations through nonlinear heads and measures correspondence with Euclidean-distance contrastive loss.
  • Audio-visual alignment: Cross-AVID extends audio-visual instance discrimination with Cross-modal Agreement, mining additional positive pairs from videos that agree in visual and acoustic characteristics.Within-modal contrastive learning beyond individual instances improves performance and reduces representation collapse in cross-modal settings.
  • Cross-modal objectives: Other approaches obtain negatives by temporally misaligning audio and video, or learn video representations by contrasting visual features with text captions encoded from speech.These designs exploit synchronization or caption correspondence instead of contrasting only different multimodal instances.
  • Loosely aligned data: Contrastive objectives can also map separately trained modality encoders or align audio with tags and speech with translations.Examples include probes between BERT and object-image representations, audio-tag embeddings, and speech-to-text translation networks.

F. OTHERS

Beyond standard representation pre-training, contrastive learning is used for knowledge distillation, reinforcement learning, image translation, and greedy optimization. The paper identifies open questions about learned invariances, negative sampling, architecture, and the form of representations.

  • Other applications: Contrastive Representation Distillation transfers knowledge from a large pre-trained teacher network to a smaller student network.This extends contrastive learning beyond representation learning from raw data.
  • Reinforcement learning: CPC|Action, CURL, and ACT adapt contrastive learning to reinforcement learning by encoding uncertainty, pixel observations, or temporally consistent transferable representations.CURL applies consistent augmentations across frame stacks, while ACT contrasts augmented observations with future observations using InfoNCE.
  • Image translation: Patch-wise contrastive learning improves unpaired image-to-image translation by associating corresponding patches more strongly than patches at other spatial locations.The method uses an InfoNCE objective to maximize mutual information between corresponding multilayer patches.
  • Greedy optimization: Greedy InfoMax and Local Contrastive extend mutual-information contrastive objectives to layer-wise or overlapping architectures instead of only end-to-end optimization.These methods use contrastive objectives for greedy optimization of representations.
  • Open questions: The invariances and covariances learned by instance discrimination depend on the augmentations defining positive pairs and the dataset bias on which they are applied.Aggressive cropping on centered single-object ImageNet images can induce reliance on occlusion invariance.
  • Open questions: Contrastive methods benefit from multiple negatives, creating computational costs that motivate memory mechanisms and raise questions about negative-sample quality versus quantity.The paper notes that multiple negatives can require large GPU clusters and longer training times, while careful negative selection can improve convergence and downstream performance.
  • Open questions: Architectural choices for contrastive pre-training may differ from supervised-learning designs, and learning disentangled or compositional subspaces remains under-explored.The paper highlights empirical dependence on projection heads and treats the representation form beyond a simple vector as a promising research direction.

D. AN ASYMMETRIC SCORING FUNCTION?

The paper identifies asymmetric and potentially non-transitive similarity as an open direction for contrastive scoring, alongside broader questions about loss design and task-based applications.

  • D. AN ASYMMETRIC SCORING FUNCTION?: Asymmetric scoring could distinguish relationships such as dog-cat from dog-animal, rather than treating similarity as universally symmetric.The paper asks whether contrastive losses can model non-transitive similarity relationships.
  • E. FUTURE OF THE CONTRASTIVE LOSS FUNCTION?: Future contrastive losses could improve computational and memory efficiency while incorporating multiple positive keys for one query.The paper notes that popular InfoNCE variants are efficient and simple but raises the possibility of more suitable alternatives.
  • E. FUTURE OF THE CONTRASTIVE LOSS FUNCTION?: Mutual-information maximization provides theoretical support but does not alone explain all reported successes of contrastive learning.The paper recommends examining contrastive losses from multiple perspectives to motivate new loss functions.
  • F. BEYOND LEARNING REPRESENTATION WITH CONTRASTIVE METHODS: Contrastive methods may optimize relevant tasks directly from positive and negative distributions defined over a data stream, rather than first learning a representation.This could extend task-based learning beyond a static labelled dataset.
  • VII. CONCLUSION: The paper surveys contrastive learning across image, video, text, audio, and other domains through a unified framework and taxonomy.Its synthesis draws on NLP, computer vision, audio processing, and other application areas.
  • VII. CONCLUSION: Contrastive representations suit some datasets and applications with limited labelled data, while often supporting strong downstream performance through rich general-purpose representations.The paper notes that pre-trained contrastive models may ultimately reduce computation for new tasks despite higher pre-training costs.
Loading 2010.05113v2…