Source-linked AI summary
A Survey on Contrastive Self-supervised Learning
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, Fillia Makedon
TL;DR
The paper addresses how self-supervised learning can reduce reliance on costly annotations while learning transferable representations. It surveys contrastive methods, their pretext tasks and architectures, and their downstream evaluations, finding promising performance across multiple tasks while identifying unresolved theoretical and invariance limitations.
Problem
Supervised learning depends on expensive annotations, motivating methods that learn useful representations from unlabeled data.
Method
The paper extensively reviews contrastive self-supervised methods, covering their pretext tasks, architectures, downstream transfer, and empirical benchmarks.
Results
Contrastive methods show promising performance across image and video classification, object detection, and other downstream tasks, with some results comparable to or exceeding supervised models.
Takeaways & Limitations
Contrastive learning provides a broad framework for learning transferable representations from unlabeled data across several modalities and downstream tasks.
Takeaways & Limitations
Current approaches lack sufficient theoretical justification and can depend strongly on pretext-task choice while failing to capture some viewpoint and category-instance invariances.
Abstract
from arXiv · showhide
Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudo labels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning methods for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we have a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make substantial progress.
1 Introduction
Supervised learning is constrained by the cost of annotation, motivating self-supervised approaches that learn representations from unlabeled data. Contrastive learning brings related embeddings together and separates diverse ones, supporting transfer to downstream tasks.
- Manual annotation of millions of samples makes supervised feature learning increasingly costly and limits its scalability.
- Self-supervised methods use data-derived pseudo-labels to learn feature representations without expensive annotations.
- Contrastive learning groups similar samples in embedding space while pushing diverse samples apart using a similarity metric and contrastive loss.
- The contrastive pipeline treats an augmented version of an input as positive and other batch or dataset samples as negatives, then transfers learned representations downstream.
- SwAV, MoCo, and SimCLR produced ImageNet results comparable to state-of-the-art supervised methods, while related work highlights the effectiveness of contrastive pretext tasks.
2 Pretext Tasks
Contrastive pretext tasks create pseudo-supervision through transformations, spatial context, temporal relationships, or cross-modal prediction. These tasks define positive and negative relationships that guide representation learning across image, video, audio, and other data.
- Pretext tasks automatically generate pseudo-labels and produce representations usable for classification, segmentation, detection, and other downstream tasks.
- The main pretext-task categories are color transformation, geometric transformation, context-based tasks, and cross-modal tasks.
- Color Transformation: Color transformations alter image appearance through blurring, distortions, grayscale conversion, or noise so networks learn color-invariant similarity.
- Geometric Transformation: Geometric transformations modify spatial structure through scaling, cropping, flipping, or rotation, supporting global-to-local view prediction.
- Jigsaw puzzle: Jigsaw pretext tasks scramble image patches, using the original as an anchor and the scrambled image as a positive sample while other images are negatives.
- Temporal and predictive tasks: Temporal contrastive tasks bring clips from the same video closer and clips from different videos farther apart, while future-prediction tasks infer high-level information from past sequences.
2.4 View Prediction (Cross modal-based)
View prediction brings together multiple views or frames of the same scene by pulling related representations together and separating temporally or otherwise distinct negatives. The usefulness of such pretext tasks depends on matching the transformation to the downstream problem.
- View prediction: View prediction encourages simultaneous-view images to share nearby embeddings while separating images from different times in the same sequence.This setup learns both cross-view similarity and temporal differences between frames.
- View prediction: Video-frame pretext tasks exploit temporal coherence because nearby frames are more semantically related than frames far apart in a sequence.The approach is intended to recover temporal structure while learning visual representations.
- Choosing the task: Pretext-task selection must match the target problem because augmentation-induced invariance can help one task while harming another.Rotation may support view-independent aerial recognition but hurt tasks requiring orientation, while colorization may fail for fine-grained classification.
- Choosing the task: Rotation performs poorly on texture-based images such as those in the DTD dataset, whereas scaling and aspect-ratio changes may create easily detectable artifacts in the cited scenario.These examples show that transformations can encode unsuitable biases or shortcuts.
- Choosing the task: Images with similar shapes but different colors and textures require an appropriate pretext task rather than relying on shape similarity alone.The figure illustrates why low-level statistical differences matter when designing the task.
2.6 Pre-text tasks in NLP
NLP self-supervision uses pretext tasks over text corpora to create training signals without manual labels. The reviewed tasks predict words, sentence relationships, neighboring sentences, or the original order of shuffled text.
- NLP self-supervision: NLP models learn text representations from large corpora through pretext tasks that generate labels for otherwise unsupervised problems.The learned representations can be used in downstream language tasks.
- Word prediction: Word2Vec uses center-word prediction or neighbor-word prediction to learn word representations in vector space.The center-word variant predicts a missing word within a fixed window, whereas skip-gram predicts neighboring words from a single input word.
- Sentence prediction: Next-sentence prediction classifies whether two input sentences are consecutive, using a following sentence as positive and a random-document sentence as negative.BERT used this task for downstream problems involving sentence relations.
- Sentence prediction: Neighbor-sentence prediction asks a model to predict the previous and next sentence given a sentence.Skip-Thought Vectors applies the skip-gram idea to sentences rather than words.
- Word prediction: Autoregressive word prediction estimates the next word from preceding words or the preceding context from later words.The passage connects this task to n-gram models and GPT-family neural networks.
- Sentence permutation: Sentence permutation breaks a continuous text span into sentences, randomly reshuffles them, and trains the model to recover their original order.BART uses this pretext task on corpus text.
3 Architectures
Contrastive architectures differ mainly in how they collect negative samples: within a batch, through a memory bank, with a momentum-encoder queue, or through clustering. These choices trade off representation quality, scalability, consistency, and handling of same-class examples.
- Architecture overview: Contrastive methods treat representation learning partly as dictionary lookup and categorize architectures by how negative samples are collected.The reviewed categories include end-to-end learning, memory banks, momentum encoders, and clustering.
- End-to-End Learning: End-to-end learning uses query and key encoders trained by backpropagation, with augmented samples as positives and other batch images as negatives.Similarity loss pulls positive representations together and pushes negative representations apart.
- End-to-End Learning: SimCLR demonstrates the end-to-end trade-off by using batch size 4096 for 100 epochs, with performance favoring larger batches and more epochs.The passage describes the architecture as simple in complexity but dependent on substantial batch and training resources.
- End-to-End Learning: End-to-end methods couple negative-sample count to batch size, making GPU memory, scalability, and large-mini-batch optimization practical constraints.Effective optimization strategies may be required as batch size grows.
- Memory Bank: A memory bank stores and periodically updates feature representations for many negative samples, including moving-average embeddings for dataset instances.PIRL is cited as a visual-representation method using this mechanism.
- Memory Bank: Memory banks can be computationally expensive because stored representations become outdated quickly and require repeated updates.This creates a maintenance cost during training.
- Momentum Encoder: A momentum encoder replaces the memory bank with a queue whose current mini-batch is enqueued and oldest mini-batch dequeued.Only the query parameters are backpropagation-updated; momentum makes the key encoder evolve smoothly.
- Momentum Encoder: Momentum encoders avoid training two separate models and avoid the computational and memory inefficiency of maintaining a memory bank.The architecture preserves a dictionary of encoded keys while reducing those burdens.
4 Encoders
Encoders map inputs into latent representations that downstream models use for discrimination. The reviewed systems commonly use ResNet variants, with architecture and pooling choices adapted to images or video sequences.
- Encoder role: Encoders map input samples to a latent space, and effective feature representations support downstream classification and related tasks.Figure 15 depicts encoder training followed by knowledge transfer to downstream tasks.
- Encoder architectures: ResNet-50 is a common encoder choice among contrastive-learning methods.The survey notes that many works use some variant of ResNet.
- Image encoders: PIRL uses average-pooled ResNet-50 res5 features to produce a 2048-dimensional vector before a linear projection to 128 dimensions.The projection creates the representation used by that method.
- Image encoders: For image contrastive learning, ResNet features may be taken after average pooling and mapped to a latent space through a shallow one-hidden-layer MLP.The contrastive loss is applied in that latent space.
- Video encoders: 3D-ResNet is commonly used to extract features from sequences of image frames for action recognition.This adapts the encoder to spatiotemporal video inputs.
5 Training
Contrastive learning trains encoders by bringing similar embeddings together and separating dissimilar ones, using similarity-based losses and optimization procedures. Cosine similarity, NCE variants, temperature scaling, and optimizers such as SGD, Adam, and LARS support this training process.
- Contrastive learning brings similar instances closer in embedding space while pushing dissimilar instances farther apart.A similarity metric provides the basis for contrastive loss and backpropagation.
- NCE distinguishes an original sample from positive and negative samples using a similarity function and temperature coefficient.In the formulation, q is the original sample, k+ is positive, k_ is negative, and τ controls temperature.
- InfoNCE is used when the number of negative samples is greater, with normalization and temperature weighting helping the model learn from hard negatives.The negative samples are represented by ki.
- Training learns encoder parameters by minimizing the contrastive loss with optimization algorithms including SGD, Adam, and LARS.LARS assigns different learning rates across layers to improve stability for large-batch training, while cosine learning rates are also used.
6 Downstream Tasks
Self-supervised computer vision pipelines first learn representations through a pretext task and then transfer them to application-specific downstream tasks. Evaluation examines transfer performance across tasks and representation quality through visualizations and nearest-neighbor retrieval.
- Self-supervised computer vision pipelines combine a pretext task with a downstream task that uses the learned knowledge.Downstream applications include classification, detection, segmentation, and future prediction.
- Downstream image tasks include image classification, localization, detection, and segmentation.
- Pretrained parameters are transferred to downstream computer vision tasks through fine-tuning.Transfer performance on high-level vision tasks demonstrates the generalization ability of learned features.
- Representation quality can be evaluated with kernel visualization, feature-map visualization, attention maps, and nearest-neighbor retrieval.These analyses examine learned features and whether same-class samples are close in latent space.
- Nearest-neighbor evaluation retrieves the top-K dataset samples for an input to assess whether same-class examples cluster in latent space.
7 Benchmarks
The survey compares contrastive methods across image classification, object detection, and video classification benchmarks. Reported results show strong performance across tasks, including supervised-model-level or higher accuracy in several settings.
- ImageNet and Places are the main image-classification datasets used to compare self-supervised contrastive methods.Pascal VOC is commonly used for object-detection evaluation.
- SwAV produces ImageNet image representations with accuracy comparable to the state-of-the-art supervised model.
- SwAV and AMDIM outperform top supervised models on Places top-1 accuracy after ImageNet pretraining and linear evaluation.The result supports transfer of contrastively learned representations across datasets.
- SwAV outperforms the state-of-the-art supervised model in both linear classification and object detection on Pascal VOC7.Detection uses features fine-tuned on VOC7+12 with Faster-RCNN.
- Contrastive methods show promising video-classification results on UCF101, HMDB51, and Kinetics.
8 Contrastive Learning in NLP
Contrastive learning originated in NLP with word-level negative sampling and has expanded to sentence-level, cross-lingual, and textual representation tasks. The survey presents contrastive methods and benchmark coverage across NLP datasets.
- Mikolov et al. introduced contrastive learning to NLP in 2013 by treating co-occurring words as similar and using negative sampling for word embeddings.The approach improved learned word and phrase representations efficiently.
- The survey includes accuracy tables for Places image classification, VOC classification and detection, video classification, and NLP datasets.
- Negative sampling differentiates words from a noise distribution through logistic regression and simplifies training.
- Contrastive learning has improved NLP downstream tasks including cross-lingual pretraining, language understanding, and textual representation learning.
- CERT addresses sentence-level semantics by augmenting sentences with back-translation and predicting whether two augmentations come from the same sentence.The model fine-tunes a pretrained language representation model for this prediction task.
9 Discussions and Future Directions
The discussion identifies unresolved theoretical and practical issues in contrastive learning, including incomplete invariance, sensitivity to architecture and sampling, augmentation trade-offs, negative-sample difficulty, and data bias.
- Contrastive learning still needs more theoretical analysis despite empirical reductions in the performance gap with supervised models.
- PIRL and MoCo fail to capture viewpoint and category instance invariance that are crucial for object recognition.
- Architecture design and sampling techniques have a profound effect on contrastive-learning performance.
- Robust pretext tasks and suitable augmentations can improve representation quality, but methods using different task-selection strategies remain difficult to compare directly.
- Easy negatives contribute minimally to contrastive loss, limiting rapid convergence and motivating larger batches or other strategies for obtaining meaningful negatives.
- Self-supervised representations are influenced by underlying data biases, which are difficult to minimize even as dataset size increases.
10 Conclusion
The paper reviews contrastive self-supervised methods across vision and NLP, covering pipeline choices and downstream applications. It reports promising results while highlighting open problems requiring new techniques and paradigms.
- The paper reviews recent top-performing contrastive self-supervised methods for vision and NLP tasks.
- It explains the contrastive-learning pipeline from pretext-task and architecture selection through downstream use of learned parameters.
- Contrastive methods show promising results on image and video classification, object detection, and other NLP tasks.
- The paper concludes that current open problems require new techniques and paradigms.