A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

• CSL creates pretext-task supervision from unlabeled data, with anchor and positive features drawn together and separated from negative features.The goal is to learn representations useful for downstream tasks that may have few labeled examples.Source: §1 · p. 1, Figure 1 · p. 2
• The framework characterizes CSL through data augmentation, encoder selection, representation extraction, similarity measure, and loss function.It analyzes AMDIM [3] [19], CPC [18] [33], and SimCLR [6] as cases within this framework.Source: §1 · p. 1
• YADIM merges AMDIM and CPC data-processing pipelines and achieves comparable results on CIFAR-10, STL-10, and ImageNet while improving robustness to encoder and sampling-task choices.The paper presents this as evidence that the framework can produce variants performing as well as AMDIM, CPC, and SimCLR.Source: §1 · p. 2

2 Contrastive self-supervised learning

• CSL assumes each unlabeled instance defines its own class; two features from one instance form an anchor-positive pair, while a feature from another instance is negative.The contrastive objective combines positive and negative scores so that minimizing the loss increases positive similarity and decreases negative similarity.Source: §2 · pp. 2–3
• The framework separates CSL design into augmentation, encoder, representation extraction, similarity, and loss; augmentation can create stochastic views of the same input or features from another input.Representations may be vectors or matrices, and similarity measures produce scalar scores.Source: §2 · pp. 2–3

2.1 Special Case 1: Augmented Multiscale Deep InfoMax (AMDIM)

• AMDIM [3] maximizes mutual information between intermediate-layer features from two views of one image, using multiscale representations and dot-product similarity with NCE loss.Its encoder is a modified ResNet with five times as many feature maps as the comparable ResNet-34 architecture.Source: §2.1 · pp. 3–4

2.2 Special Case 2: Contrastive Predictive Coding (CPC)

• CPC’s data augmentation pipeline matches AMDIM’s, with a new stage appended at the end that breaks each input image into a list of overlapping q-by-q patches with d channels.For a 256×256 image, the example uses 64×64 patches with 32-pixel overlap, yielding a 7×7 list of patches.Source: §2.2 · p. 4
• CPC [18] divides images into overlapping patches and predicts features spatially below a context location, treating the prediction as anchor and the target feature as positive.Other encoder features serve as negatives; CPC uses a modified ResNet-101 and NCE loss.Source: §2.2 · pp. 4–5

2.3 Special Case 3: A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)

• SimCLR [6] uses same-image stochastic views, a generic ResNet, flattened encoder outputs, a projection head with cosine similarity, and temperature-scaled NCE loss.Its augmentation includes random crop and resize, horizontal flip, color jitter, grayscale conversion, and Gaussian blur.Source: §2.3 · p. 5

3.1 YADIM: Yet Another DIM

• CPC and AMDIM share more design elements than their different motivations suggest, so YADIM assesses their differing choices and simplifies choices with little performance impact.The framework focuses on encoder, augmentation, representation extraction, and similarity measure differences; the approaches use the same loss.Source: §3.1 · p. 6
• YADIM combines CPC’s patch transform with AMDIM’s five augmentation stages, applying all six sequentially to create anchor and positive views and a negative view from another input.The two pipelines are described as independent, enabling their union.Source: §3.1 · p. 6

3.2 Systematic investigation of CPC and AMDIM design differences

• On CIFAR-10, YADIM and CPC remain stable across standard ResNet choices, while AMDIM performance suffers.The encoder-robustness table compares the approaches across ResNet architectures.Source: Table 1 · p. 7, §3.2 · p. 7
• YADIM is less sensitive than AMDIM to the representation-extraction comparison strategy.The tested strategies include using only the last feature map, AMDIM’s comparisons, and comparisons across feature-map levels.Source: Table 2 · p. 7, §3.2 · p. 7

3.3 Final formulation of YADIM

• The final YADIM design uses the union of the CPC and AMDIM augmentation pipelines, AMDIM’s wide ResNet-34, a dot-product similarity measure, and the NCE loss.Its representation-extraction task is chosen for few assumptions while achieving near-best performance.Source: §3.1 · p. 6, §3.3 · pp. 7–8

4 Experimental setup

• CIFAR-10, STL-10, and ImageNet are evaluated, with dataset-specific image sizes, train/validation splits, and test sets.STL-10 and ImageNet models are pretrained on unlabeled training examples; CIFAR-10 results use the test set.Source: §4 · p. 8
• Evaluation pretrains an encoder on unlabeled examples, freezes it, then trains a separate network using a subset of labeled examples.The evaluation protocol follows the cited prior study; the separate network is an MLP with one 1,024-unit ReLU hidden layer.Source: §4 · p. 8

4.1 Implementation

• PyTorch Lightning implementations use shared dataset splits, fine-tuning, early-stopping, and transformation protocols to support consistent comparisons across settings.The framework also makes implementations hardware agnostic and more accessible on arbitrary hardware.Source: §4.1 · pp. 8–9
• CSL performance positively correlates with the number of negative samples; the experiments use distributed softmax to aggregate batches across GPUs.The passage also identifies memory banks as another way to increase negative samples.Source: §4.1 · p. 9
• Limited computational resources prevented thorough CPC and AMDIM hyperparameter searches and full reproduction of their reported ImageNet results.The authors attribute largely reproducing AMDIM and CPC to AMDIM’s open-source code and helpful discussion with CPC’s authors.Source: §4.1 · p. 9

5 Results and analysis

• The section presents the first comparison of AMDIM, CPC, and SimCLR using the same standardized implementation and evaluation protocol.Class separability is tested by training a classifier using all labels from CIFAR-10, STL-10, and ImageNet.Source: §5 · p. 9
• AMDIM performs close to Bachman et al.'s results on CIFAR-10 and STL-10, and on ImageNet matches the earlier version of [3] rather than the latest reported accuracy.Table 3 compares class separability across AMDIM, CPC, SimCLR, and YADIM.Source: §5 · p. 9
• YADIM achieves accuracy comparable to AMDIM and CPC on CIFAR-10, STL-10, and ImageNet.The comparison is reported alongside CPC's STL-10 and ImageNet results and SimCLR's CIFAR-10 result.Source: §5 · p. 9
• CPC sets new state-of-the-art scores for self-supervised pretrained models on STL-10, while its ImageNet result is close to the reported result.Computational constraints prevented training CPC fully to the accuracy reported by Henaff et al. [18].Source: §5 · p. 9
• SimCLR reaches 87.60 accuracy after 14 hours on one V100, compared with the reported 93.70.The passage describes the achieved result as reasonably close to the reported one.Source: §5 · p. 9

6 Related work

• Earlier image-specific pretext tasks predict patch relations [11] [21], inpaint images [35], solve jigsaws [32], colorize images [40], or detect rotations [14].Source: §6 · p. 9
• Table 3 reports test accuracy from a 1,024-unit MLP fine-tuned on frozen encoders pretrained without labels, marking the best reported and in-house results separately.The authors could not complete their SimCLR ImageNet experiment because of computational constraints.Source: Table 3 · p. 10
• AMDIM compares feature maps from two views, while CPC predicts spatially separated patches; removing CPC's context encoder yields trivial solutions for CPC and SimCLR.Source: §6 · p. 10
• CMC adds discriminator regularization to NCE and uses a memory bank for more negatives, while SimCLR adds random resize-and-crop augmentation and a projection-based similarity measure.MOCO (v2) modifies SimCLR's projection head and augmentation pipeline.Source: §6 · p. 10
• The cited work reports that CSL outperforms autoencoders and GANs, whose learned representations transfer poorly to image classification despite their performance on denoising and synthesis.Source: §6 · p. 10

7 Conclusion

• YADIM performs as well as CPC and AMDIM on CIFAR-10, STL-10, and ImageNet, while being robust to encoder architecture choice and using simpler representation extraction.Source: §7 · p. 10
• The framework characterizes CSL using five components: encoder, data augmentation, representation extraction, similarity measure, and loss function.AMDIM, CPC, and SimCLR are presented as special cases.Source: §7 · p. 10
• The comparison suggests that a wide encoder with many feature maps, a simple dot-product noise-contrastive loss, and strong augmentation can support downstream performance even with simple extraction or encoders.Source: §7 · p. 10
• The authors find that SimCLR, CMC, and MOCO differ little from one another or from CPC and AMDIM, and their design choices fit the proposed framework.Source: §7 · p. 11
• The authors release PyTorch Lightning code for the CSL approaches, aiming to enable clearer comparisons and encourage further research.Source: §7 · p. 11

A.1 Encoder

• The encoder subsection imports AMDIMEncoder from the self-supervised-learning models module.Source: §A.1 · p. 14

A.2 Transforms

• The transforms subsection constructs an STL-10 dataset with AMDIMEvalTransformsSTL10.Source: §A.2 · p. 14

A.3 Representation Extraction

• The representation-extraction example imports FeatureMapContrastiveTask and instantiates it with the identifiers '01, 02, 11'.Source: §A.3 · p. 14

A.4 Loss

• The loss example imports AmdimNCELoss and sums contrastive-loss terms for the 01, 02, and 11 feature-map pairings.Source: §A.4 · p. 14

A.5 Full AMDIM pseudocode

• The training-step pseudocode augments the batch twice, encodes both views, and passes their feature maps to the contrastive task.Source: §A.5 · p. 14

B.1 Encoder

• The encoder is imported as AMDIMEncoder.The code imports it from the self-supervised CPC module.Source: §B.1 · p. 14

B.2 Transforms

• STL-10 evaluation data use CPCEvalTransformsSTL10.The dataset is instantiated with those transforms.Source: §B.2 · p. 14

B.3 Representation extraction

• CPC forms context and target features, predicts future positions, and sums the valid losses.The code computes losses across later positions while skipping NaN values.Source: §B.3 · p. 15

B.4 Loss

• CPC flattens spatial target vectors and selected future predictions before computing their similarity scores.The target vectors are arranged by channel, while predictions select positions south of the current source.Source: §B.4 · p. 15
• Cross-entropy compares prediction logits with labels indexing the corresponding future targets.The code constructs labels from batch and spatial-position indices.Source: §B.4 · p. 15

B.5 Full CPC pseudocode

• The full CPC step encodes x into latent features Z, computes the CPC task's InfoNCE loss, and returns it.The batch provides x and y, but the shown step uses x for encoding.Source: §B.5 · pp. 15–16

C.1 Encoder

• The SimCLR code imports a ResNet-50 batch-normalized encoder.Source: §C.1 · p. 16

C.2 Transforms

• STL-10 evaluation data use SimCLREvalTransformsSTL10.The dataset is instantiated with those transforms.Source: §C.2 · p. 16

C.3 Representation extraction

• SimCLR encodes both image views into feature maps shaped (b, 2048, 2, 2).Source: §C.3 · p. 16

C.4 Loss

• NT-Xent exponentiates the full similarity matrix over temperature, sums non-self similarities, and compares them with paired-view similarities.The loss is the mean negative log of the positive-to-negative similarity ratio.Source: §C.4 · p. 16

C.5 Full SimCLR pseudocode

• The full step projects the two encoded views from feature maps to 128-dimensional vectors, then returns their NT-Xent loss.If the encoder returns feature-map lists, the implementation selects the last map.Source: §C.5 · pp. 16–17