PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Learning: Generative or Contrastive

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, Jie Tang

arXiv:2006.08218v5cs.LGstat.ML

TL;DR

Supervised learning is limited by manual labeling and vulnerability to attacks, motivating self-supervised representation learning from unlabeled data. The survey synthesizes methods across NLP, computer vision, and graph learning into generative, contrastive, and generative-contrastive categories, alongside theoretical analyses and open problems. It concludes that self-supervision can achieve comparable learning with fewer labels, while important limitations remain in connecting objectives such as mutual information to downstream performance.

  • Problem

    Supervised learning relies on manual labels and is vulnerable to generalization errors, spurious correlations, and adversarial attacks.

  • Method

    The survey reviews self-supervised representation-learning methods across NLP, computer vision, and graph learning, organizing them by generative, contrastive, and generative-contrastive objectives.

  • Results

    The survey concludes that self-supervised learning cannot learn more than supervised learning, but can achieve this with fewer labels.

  • Takeaways & Limitations

    Self-supervised learning provides a framework for exploiting web-scale unlabeled data to train feature extractors and context generators efficiently.

  • Takeaways & Limitations

    Maximizing mutual-information or ELBO lower bounds is not sufficient to learn representations useful for downstream tasks.

Abstract

from arXiv · show

Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

1 INTRODUCTION

Self-supervised learning is presented as an alternative to label-dependent supervised learning, using data-derived supervision for representation learning. The survey reviews the field across multiple domains, categorizes methods into three types, and discusses their advantages, limitations, theory, and future directions.

  • Supervised learning depends heavily on expensive manual labeling and faces generalization error, spurious correlations, and adversarial attacks.
  • Self-supervised learning obtains labels from data itself and predicts part of the input from other parts.
  • The survey reviews self-supervised representation learning across natural language processing, computer vision, graph learning, and related areas.
  • The survey categorizes methods as generative, contrastive, and generative-contrastive (adversarial), and examines the pros and cons of each category.
  • The survey also analyzes theoretical attempts to explain self-supervised learning and identifies open problems, limitations, and future directions.

2 MOTIVATION OF SELF-SUPERVISED LEARNING

The motivation for self-supervised learning is to exploit abundant unlabeled data while defining objectives that produce useful representations. The survey organizes these objectives into generative, contrastive, and adversarial approaches with distinct architectures and training goals.

  • Deep learning is data-hungry and can overfit or generalize poorly when supervised data are limited.
  • Self-supervised learning leverages unlabeled data and inherent co-occurrence relationships as supervision.
  • Generative: Generative methods encode x into an explicit vector z and reconstruct x from z.
  • Contrastive: Contrastive methods encode x into an explicit vector z to measure similarity, including mutual-information maximization and instance discrimination.
  • Generative-Contrastive (Adversarial): Generative-contrastive methods use an encoder-decoder to generate fake samples and a discriminator to distinguish them from real samples.
  • Conceptual Comparison: The categories differ in latent-variable treatment, discriminator use, and objectives such as reconstruction loss and contrastive learning.

3 GENERATIVE SELF-SUPERVISED LEARNING

Generative self-supervised methods learn representations by modeling or reconstructing input data, spanning autoregressive, flow-based, auto-encoding, and hybrid approaches. They provide broad modeling flexibility but face directional-context limitations and can be less competitive than contrastive methods in some classification settings.

  • Generative self-supervised learning includes autoregressive, flow-based, auto-encoding, and hybrid generative models.These methods are organized by their generative modeling objectives.
  • Autoregressive Models: Autoregressive models factorize distributions into conditionals and model context dependencies sequentially across language, images, and graphs.GraphRNN generates nodes and edges conditioned on the graph generated so far, while PixelRNN and PixelCNN generate images pixel by pixel.
  • Limitations: Autoregressive models expose tokens to context from only one direction, while masked language modeling assumes predicted tokens are conditionally independent given unmasked tokens.These are identified as inherent limitations of the respective language-modeling approaches.
  • Flow-based Models: Flow-based models learn an invertible, differentiable transformation between data x and latent variable z to estimate complex high-dimensional densities.Their dimensionality-matching and Jacobian requirements constrain model design.
  • Auto-encoding Models: Autoencoders reconstruct inputs from encoded representations, while denoising autoencoders reconstruct masked or corrupted inputs to encourage noise-robust representations.BERT masks tokens and predicts them from context, sometimes replacing tokens with original or random words instead of [MASK].
  • Hybrid Models: Permutation language modeling combines autoregressive and auto-encoding advantages by learning bidirectional contexts over factorization-order permutations.XLNet maximizes expected likelihood over all permutations of the input index sequence.
  • Limitations: Generative self-supervised learning is reported as less competitive than contrastive learning in some classification scenarios, although generative language models remain widely used for NLP classification.The survey attributes the contrastive advantage in these scenarios to its closer alignment with classification objectives.

4 CONTRASTIVE SELF-SUPERVISED LEARNING

Contrastive self-supervised learning reframes representation learning as comparing similar and dissimilar samples rather than directly modeling the data distribution. Its frameworks include context-instance and instance-instance contrast, with strong downstream performance especially for linear-protocol classification.

  • Generative models estimate p(X|Y), whereas contrastive models learn representations by modeling comparisons related to P(Y|X).Recent methods such as Deep InfoMax, MoCo, and SimCLR demonstrate the potential of discriminative objectives for representation learning.
  • Contrastive learning uses similar pairs x+ and dissimilar pairs x− with an encoder f under a noise-contrastive estimation objective.The similarity measure and encoder can vary across tasks while the comparison framework remains consistent.
  • ImageNet top-1 accuracy under the March 2021 linear-classification protocol shows self-supervised feature extraction rapidly approaching supervised ResNet50 performance.Except for BigBiGAN, the displayed models are contrastive self-supervised methods.
  • InfoNCE extends the contrastive objective by incorporating more dissimilar pairs.The survey presents InfoNCE as the formulation used when additional negative examples are involved.
  • Contrastive frameworks are divided into context-instance contrast and instance-instance contrast, both achieving strong downstream performance, particularly under linear-protocol classification.The survey distinguishes these categories by the relationship being modeled between representations.

4.1 Context-Instance Contrast

Context-instance contrast models the relationship between local features and global context, either through relative-position prediction or mutual-information maximization. These approaches extend across vision, language, and graph learning through task-specific context construction and corruption strategies.

  • Context-instance contrast models the association between a sample’s local feature and its global context representation.Examples include stripes with tigers, sentences with paragraphs, and nodes with neighborhoods.
  • Predict Relative Position: Relative-position prediction learns spatial or sequential relations among components, including patch positions, shuffled image segments, rotations, and sentence order.These pretext tasks can also create hard positive samples, as in PIRL’s jigsaw-based augmentation.
  • Maximize Mutual Information: Mutual-information methods maximize direct belonging relationships between local parts and global context while ignoring relative positions among local parts.The objective uses a sample-based estimator and commonly maximizes a lower bound with an NCE objective because mutual information is difficult to compute directly.
  • Vision: Deep InfoMax encodes an image into feature maps, applies a summary function to obtain a context vector, and contrasts local features with contexts from positive and negative images.AMDIM strengthens this paradigm by using different image views, while CMC extends multi-view contrast toward instance-instance similarity.
  • Language: In language, mutual-information objectives connect sentence-level representations with masked n-grams, whereas sentence-order prediction uses exchanged sentence positions as negatives to emphasize coherence.InfoWord samples negative contexts from the corpus; ALBERT’s SOP avoids the easier topic distinction created by NSP negatives from unrelated passages.
  • Graph Learning: In graph learning, DGI contrasts node representations with context summaries using corrupted graphs formed by permuting node features or positions while preserving subgraph structure.InfoGraph shifts the target to graph-level representations, and related work applies context prediction and attribute masking across structural and attribute objectives.
  • Limitations: Graph pre-training remains constrained by learning inductive biases across graphs because existing work is applicable only to a specific domain.This limitation is stated after methods that maximize mutual information between nodes and k-hop context subgraphs.

4.2 Instance-Instance Contrast

Instance-instance contrastive learning focuses on relationships between individual samples rather than mutual information or context-level representations. The survey traces its progression from cluster-based pseudo-labeling to instance discrimination, multi-view augmentation, and methods that eliminate negative sampling.

  • Motivation: MI-based contrastive learning’s empirical success is only loosely connected to mutual information, with encoder architecture and negative-sampling strategies receiving greater attribution.An upper-bound MI estimator produced ill-conditioned, lower-performance representations.
  • Motivation: Instance-level representations prioritize task-relevant objects or keywords over irrelevant context in image and sentence classification.The survey contrasts a dog with surrounding grass and emotional keywords with broader sentence context.
  • Instance Discrimination: CMC, MoCo, SimCLR, and BYOL supported instance-instance contrast by achieving competitive representation results under linear classification, with BYOL removing negative sampling.BYOL uses an exponential-moving-average target encoder and regression-style training to avoid collapse.
  • Cluster Discrimination: DeepCluster generates pseudo labels through clustering, whereas Local Aggregation identifies neighbors per example and directly optimizes a local soft-clustering metric.The latter changes mutual-exclusive cluster assignment and cross-entropy optimization, substantially boosting downstream representation performance.
  • Cluster Discrimination: Cluster-based methods progressed from DeepCluster and ClusterFit toward online clustering and multi-view augmentation in SwAV, which avoids a slow separate clustering stage.SwAV combines online clustering with multi-view augmentation and is more computationally efficient than instance-discrimination methods when model size is small.
  • Instance Discrimination: A batch size of 512 reduced BYOL performance by 0.3% versus 4096, compared with a 1.4% drop for SimCLR.The comparison indicates greater robustness to smaller batches for BYOL under the reported ablation.
  • Instance Discrimination: Graph instance discrimination uses two independently sampled subgraphs of each node, while graph augmentation studies show that suitable edge and node transformations improve representation performance.GCC learns transferable structural knowledge, and GraphCL evaluates four edge-perturbation and node-dropping augmentations.

4.3 Self-supervised Contrastive Pre-training for Semi-supervised Self-training

Self-supervised contrastive pre-training can be combined with semi-supervised self-training to address the gap between pre-training objectives and downstream labels. Reported studies find complementary benefits from pre-training, pseudo-labeling, and strong augmentation, including gains with limited labels.

  • Motivation: Self-supervised pre-training remains separated from supervised downstream objectives, so labels are still needed to transfer representations to tasks.The survey identifies semi-supervised learning as a way to bridge this objective gap.
  • Self-training: Semi-supervised self-training combines a small labeled set with unlabeled data by training a model to generate pseudo labels for subsequent learning.The described procedure first trains on labeled data, then labels unlabeled examples.
  • Evidence: Joint pre-training and self-training produced the best model in the cited study, while their improvements were reported as orthogonal.The study found pre-training could hurt in the examined setting, whereas self-training benefited from strong augmentation and remained helpful as labeled data increased.
  • Evidence: With only 10% of the original ImageNet labels, SimCLR v2’s ResNet-50 surpassed the supervised model using joint pre-training and self-training.The framework pre-trains, fine-tunes with 1% or 10% labels, and uses the fine-tuned network as a teacher for a smaller student.
  • Implications: Combining contrastive pre-training with semi-supervised self-training is presented as a promising direction for data-efficient deep learning, while its mechanisms remain open for study.The survey explicitly calls for further investigation of the latent mechanisms.

4.4 Pros and Cons

Contrastive learning is lightweight and effective for discriminative downstream applications, but its broader scope remains constrained by unresolved methodological and domain challenges. The survey highlights limitations in NLP scaling, negative sampling, and the theory of data augmentation.

  • Pros: Contrastive models discard the decoder because they assume classification-oriented downstream applications, making them lightweight and suited to discriminative tasks.Generative models instead retain reconstruction-oriented components.
  • Cons: Self-supervised contrastive learning remains an emerging field with several unresolved problems despite its close relationship to metric learning.The survey frames these issues as open research questions rather than settled limitations.
  • Cons: Contrastive pre-training has not yet produced convincing NLP benchmark results, and few methods apply it during the pre-training stage.The survey notes that most NLP contrastive work currently concerns supervised fine-tuning of BERT.
  • Cons: Negative sampling is often tricky, biased, and time-consuming, while the role it plays in contrastive learning remains unclear.BYOL and SimSiam reduce reliance on negative samples, but the survey identifies further improvement as necessary.
  • Cons: Data augmentation improves contrastive performance, but why and how it helps remains theoretically ambiguous, hindering transfer to discrete domains such as NLP and graph learning.The survey identifies this ambiguity as a barrier to applying augmentation beyond vision.

5 GENERATIVE-CONTRASTIVE (ADVERSARIAL) SELF-SUPERVISED LEARNING

Generative-contrastive, or adversarial, representation learning combines generative expressiveness with discriminative objectives. The survey reviews GAN-based representation methods, partial-input recovery tasks, and ELECTRA, while noting training and representation-quality limitations.

  • Overview: Adversarial representation learning reconstructs the data distribution rather than individual samples by minimizing distributional divergence.This approach arose from shortcomings of point-wise generative reconstruction.
  • Overview: Adversarial methods retain an encoder-decoder generator structure, producing reconstructive representations, whereas contrastive methods discard the decoder and learn distinguishable information.The decoder requires representations to contain information sufficient to reconstruct inputs.
  • GAN-based Representation Learning: GAN-based representation learning adds an encoder that maps real samples to latent representations while the generator acts as a decoder producing samples from prior latent codes.The discriminator distinguishes generated pairs from real sample pairs, encouraging the encoder and generator to act as inverse mappings.
  • Partial-Input Recovery: Partial-input adversarial methods recover missing image content through colorization, inpainting, and super-resolution rather than reconstructing the entire input.Colorization predicts A and B channels from L; inpainting fills arbitrary regions; SRGAN recovers high-resolution images from blurred low-resolution inputs.
  • Pre-trained Language Model: ELECTRA uses Replaced Token Detection with a small masked language-model generator and discriminator, surpassing BERT at the same computation budget.Its two-stage training avoids policy gradients for discrete tokens.
  • Limitations: ELECTRA’s binary discrimination saves computation but may harm representation quality through early embedding-space degeneration, and adversarial models are prone to training collapse.The survey notes that adversarial training requires stabilization techniques such as spectral normalization and W-GAN.

6 THEORY BEHIND SELF-SUPERVISED LEARNING

The survey examines theoretical explanations for generative and contrastive self-supervised learning, including divergence estimation, mutual-information objectives, and disentangled representations. These analyses clarify mechanisms but also show that common objective bounds do not reliably predict downstream usefulness.

  • Generative Models: GAN training can be interpreted through divergence estimation, with different divergence functions producing different GAN variants.f-GAN formulates adversarial learning as variational f-divergence estimation, while Jensen-Shannon divergence recovers the standard minmax GAN objective.
  • Disentangled Representation: InfoGAN introduces latent codes whose dimensions represent image characteristics and maximizes their mutual information with generated images to learn disentangled representations.Variational inference estimates a lower bound because mutual information is difficult to compute directly.
  • Disentangled Representation: Experiments report that InfoGAN learns good disentangled representations on MNIST, while GAN analyses examine whether internal channels correspond causally or independently to generated objects.These studies motivate the broader possibility that self-supervised methods can learn disentangled features.
  • Generative Objectives: Variational autoencoders optimize ELBO to approximate posterior and prior distributions while combining regularization and reconstruction losses.The reconstruction term models pθ(x|z), and the KL term regularizes the latent distribution.
  • Contrastive Objectives: Contrastive methods commonly maximize mutual information between inputs and representations, using estimators such as InfoNCE and related critic functions.Deep InfoMax applies a similar divergence-based objective to local and global feature distributions.
  • Contrastive Objectives: The bound I(x, y) ≥ log(N) − L becomes more accurate as the number of negative samples N grows, but larger negative sets do not necessarily improve performance.Negative sampling remains a key challenge despite the theoretical bound.
  • Theoretical Limitations: MI and ELBO are loosely connected to downstream performance: equal bound values can yield different representations, and looser bounds can produce better test accuracy.The interaction among encoder, critic, and loss function is non-trivial.

6.3 Contrastive Self-supervised Representation Learning

Contrastive pre-training can improve label efficiency, robustness, and theoretical downstream guarantees, but its benefits depend on the task, assumptions, and negative-sampling strategy. The survey also reports that alignment and uniformity jointly shape representation quality while alternative methods challenge their necessity.

  • Relationship with Supervised Learning: Self-supervised pre-training follows a pattern where it reaches higher accuracy with fewer labels but eventually converges to supervised-learning performance.Synthetic COCO experiments support this pattern rather than a permanent accuracy advantage.
  • Relationship with Supervised Learning: Self-supervised training improves robustness to adversarial examples, label corruption, and common input corruptions, and benefits out-of-distribution detection on difficult near-distribution outliers.The reported out-of-distribution performance exceeds that of fully supervised methods.
  • Alignment and Uniformity: Contrastive loss can be decomposed into alignment and uniformity terms, and experiments find that both are necessary for good representations.Directly optimizing the two terms performs consistently better than contrastive loss across computer-vision and natural-language-processing settings.
  • Alignment and Uniformity: BYOL challenges the necessity of explicit alignment and uniformity losses because it outperforms earlier contrastive methods without direct negative sampling.The survey identifies exponential moving average, batch normalization, regularization, and random initialization as possible contributors to uniformity.
  • Theoretical Guarantees: Under assumptions linking similar pairs to latent classes, theoretical analyses show that optimizing unsupervised contrastive loss can benefit downstream classification.The guarantee is restricted by sampling assumptions and encoder conditions.
  • Generalization: With sufficiently powerful encoders and sufficiently many samples, low unsupervised loss and low intraclass deviation are associated with good supervised-task performance.The bound includes generalization error and is stated probabilistically.
  • Limitations: Contrastive learning does not always select the best supervised representation because low unsupervised loss need not imply proximity to the supervised optimum.High unsupervised loss and intraclass deviation can fail to track supervised loss.
  • Limitations: Increasing negative samples raises computational cost and can hurt contrastive performance when the number exceeds a threshold.This contradicts the general belief that more negative samples always help.

7 DISCUSSIONS AND FUTURE DIRECTIONS

The survey identifies unresolved challenges in transferring self-supervised representations across tasks and datasets, while highlighting early degeneration, sampling, and theoretical foundations as directions for progress.

  • Theoretical Foundation: Theoretical analysis is needed because sampling strategies and architecture design may matter more than mutual information for the success of some MI-based methods.The survey presents theory as important for avoiding misleading empirical conclusions.
  • Transferring to downstream tasks: Pre-training objectives do not always transfer effectively to downstream tasks, and selecting suitable pretext tasks remains heuristic.BERT’s Next Sentence Prediction illustrates the risk: ALBERT found it equivalent to a naive topic model that can reduce performance.
  • Transferring to downstream tasks: Automatically designing pre-training tasks for specific downstream tasks is proposed as a direction analogous to Neural Architecture Search.
  • Transferring across datasets: Cross-dataset transfer depends on distributional assumptions that frequently fail, although broad language-model corpora can cover many language patterns within a shared embedding space.The survey notes that this advantage is less established for machine translation and graph learning.
  • Exploring potential of sampling strategies: Large negative-sample sets and augmented positive samples are promising sampling choices, but exploiting sampling strategies further remains unresolved.
  • Early Degeneration for Contrastive Learning: Contrastive methods approach supervised performance in computer vision but remain generally limited to classification, while ELETRA outperforms other generative methods on several NLP benchmarks with fewer parameters.
  • Early Degeneration for Contrastive Learning: Early degeneration occurs when models over-fit the discriminative pretext task too soon, weakening generalization.The survey calls for methods that preserve contrastive learning’s advantages while avoiding this problem.

8 CONCLUSION

The survey synthesizes self-supervised representation learning across NLP, computer vision, graph learning, and beyond. It organizes methods by training objective and closes by identifying their trade-offs, applications, and future problems.

  • The survey comprehensively reviews self-supervised representation learning approaches across NLP, computer vision, graph learning, and other domains.
  • It categorizes methods into generative, contrastive, and generative-contrastive classes according to their essential training objectives.
  • The survey presents representative methods, discusses each category’s advantages, disadvantages, and application scenarios, and lists fundamental problems and future directions.

REVISION HISTORY

The revision history records incremental corrections, literature updates, statistical updates, and additions to the survey’s theoretical and motivational coverage.

  • June 2020 revisions corrected several typos and mistakes.
  • July 2020 revisions added new papers, a theoretical analysis of contrastive objectives, and a connection between semi-supervised self-training and self-supervised contrastive learning.
  • March 2021 revisions added new papers, updated statistics in Figures 2 and 7, added motivation for using self-supervised learning, and removed the preliminary section.
Loading 2006.08218v5…