PaperlayerView exact revision on arXiv

Source-linked AI summary

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, Rogerio Feris

arXiv:2103.13517v3cs.CV

TL;DR

The paper addresses limited evidence about how contrastive representations transfer to different domains and why they do so. It benchmarks five training objectives across classification, few-shot recognition, and detection, finding stronger transfer for contrastive approaches, especially joint objectives. Its analysis associates these representations with more low- and mid-level information and greater within-class variation.

  • Problem

    Prior work mainly emphasized in-domain evaluation, leaving the reasons for contrastive representations’ transfer advantage unclear.

    Source: §1 · p. 1

  • Method

    The study benchmarks five training objectives across downstream image classification, few-shot recognition, and object detection.

    Source: §1 · p. 1

  • Results

    Contrastive approaches transfer better across downstream tasks, and joint supervised and self-supervised contrastive training outperforms cross-entropy by 5.63% under linear evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Contrastive representations contain more low- and mid-level information and show greater within-class variation than cross-entropy representations.

    Source: §1 · p. 2

Abstract

from arXiv · show

Tremendous progress has been made in visual representation learning, notably with the recent success of self-supervised contrastive learning methods. Supervised contrastive learning has also been shown to outperform its cross-entropy counterparts by leveraging labels for choosing where to contrast. However, there has been little work to explore the transfer capability of contrastive learning to a different domain. In this paper, we conduct a comprehensive study on the transferability of learned representations of different contrastive approaches for linear evaluation, full-network transfer, and few-shot recognition on 12 downstream datasets from different domains, and object detection tasks on MSCOCO and VOC0712. The results show that the contrastive approaches learn representations that are easily transferable to a different downstream task. We further observe that the joint objective of self-supervised contrastive loss with cross-entropy/supervised-contrastive loss leads to better transferability of these models over their supervised counterparts. Our analysis reveals that the representations learned from the contrastive approaches contain more low/mid-level semantics than cross-entropy models, which enables them to quickly adapt to a new task. Our codes and models will be publicly available to facilitate future research on transferability of visual representations.

1. Introduction

The paper addresses uncertainty about why contrastive representations transfer well beyond their training domain by benchmarking five training objectives across multiple downstream tasks. It reports stronger transfer for contrastive approaches and links this pattern to richer low- and mid-level information and greater within-class variation.

1. Introduction

  • Prior work largely emphasized in-domain ImageNet evaluation, leaving the reasons for contrastive representations’ transfer advantage unclear.The paper benchmarks five objectives across downstream classification, few-shot recognition, and object detection.Source: §1 · pp. 1–2
  • Contrastive methods outperform cross-entropy models on downstream datasets, with the gap smaller after full-network fine-tuning.The same overall trend is reported for few-shot recognition and object detection; combining self-supervised contrastive loss with supervised loss also improves transfer performance.Source: §1 · pp. 1–2
  • Across downstream tasks, joint supervised and self-supervised contrastive training outperforms cross-entropy by 5.63% under linear evaluation and 3.46% in 5-shot recognition.The reported gains are 1.37% AP50 on VOC0712 and approximately 0.8% on MS COCO; the joint objective also consistently improves over its supervised-contrastive counterpart.Source: §1 · p. 2
  • Contrastive models contain more low- and mid-level information and show greater within-class variation than cross-entropy models.The authors suggest that these properties may contribute to transferability.Source: §1 · p. 2

2. Related Work

Prior work established the value of pretrained CNN features and reported transfer gains from contrastive learning. However, comparisons across transfer settings were limited, and the reasons for the gains remained unclear.

2. Related Work

  • Pretrained CNNs can provide features for SVM and logistic-regression models that outperform handcrafted features, while transfer can help train on smaller datasets without overfitting.Prior studies examined source-dataset effects, the relationship between ImageNet accuracy and downstream performance, and network depth.Source: §2 · p. 2
  • The paper investigates contrastive training’s effect on transfer performance and the underlying principle behind it.This follows prior reports of contrastive representations outperforming supervised cross-entropy models on downstream tasks.Source: §2 · pp. 2–3
  • Recent self-supervised and supervised contrastive methods improved representation quality or downstream performance, but most studies offered limited comparisons across transfer settings.The cited gaps include fixed-feature transfer, few-shot learning, and robustness.Source: §2 · p. 3

3. Analysis Setup

The study frames transfer as learning a target prediction function from source-domain knowledge and evaluates several target tasks. It compares five training objectives, including joint-loss models with separate branches over a shared backbone.

3. Analysis Setup

  • Transfer learning is framed as learning a target prediction function from source-domain knowledge, evaluated through linear evaluation, full-network fine-tuning, object detection, and few-shot classification.The setup describes source and target domains with generally different marginal distributions.Source: §3 · p. 3

3.1. Loss Functions

  • The methods compare cross-entropy, self-supervised contrastive, supervised contrastive, and two joint objectives using the stated loss formulations.MoCo supplies the contrastive setup, with an encoder and MLP projection head; the joint models use separate branches over a shared backbone.Source: §3.1 · p. 3, §3.2 · pp. 3–4, Figure 2 · p. 3

3.2. Methods

  • The study uses datasets for downstream image classification, as identified in Table 1; the cited passage does not state their number, domain categories, or training protocols.Source: Table 1 · p. 4

4. Experimental Results

Experiments use ImageNet1K pretraining for linear and full-network transfer and 12 downstream classification datasets from varied domains; Mini-ImageNet pretraining supports few-shot tests. Evaluation covers fixed-feature transfer, full-network fine-tuning, detection and segmentation, and few-shot recognition.

4.1. Datasets

  • ImageNet1K supplies 1.28M training images for linear evaluation and full-network fine-tuning, while Mini-ImageNet supplies 38k for few-shot pretraining.The 12 downstream classification datasets cover natural, satellite, symbolic, illustrative, medical, and texture domains.Source: §4.1 · p. 4

4.3. Linear Evaluation over Fixed Network

  • As fixed-feature extractors, all contrastive approaches outperform CE, and SupCon+SelfSupCon averages 5.63% better; adding SelfSupCon improves supervised variants.The linear evaluation freezes the backbone and trains only a linear layer; strong augmentation alone does not help CE much.Source: Figure 3 · p. 4, §4.3 · pp. 4–5

4.4. Full-Network Fine-Tuning

  • With full-network fine-tuning, SupCon+SelfSupCon reaches 88.56% top-1 accuracy versus CE’s 88.13%; with sufficient data, all models achieve similar performance.The table reports top-1 accuracy for downstream datasets; the authors also compare a 1,000-sample training regime separately.Source: Table 2 · p. 5, §4.4 · p. 5
  • On MS COCO, CE+SelfSupCon and SupCon+SelfSupCon improve AP over their respective counterparts by ∼0.8% and ∼0.5%.The experiments fine-tune the whole network for object detection and instance segmentation on VOC0712 and MS COCO.Source: Table 3 · p. 5, §4.4 · p. 5
  • With 1,000 training samples, CE+SelfSupCon is 1.09% and SupCon+SelfSupCon 1.48% better than CE.The authors infer contrastive-pretrained methods are slightly more effective in a limited-data regime.Source: §4.4 · p. 5

4.5. Few-shot Classification

  • Across downstream few-shot datasets, contrastive methods beat CE and joint variants score best, but SupCon+SelfSupCon trails SupCon on in-domain Mini-ImageNet.The reported few-shot averages use 600 episodes for 5-way 5-shot and 20-shot experiments.Source: §4.5 · pp. 5–6, Table 4 · p. 6

5. Discussion and Analysis

The analysis compares representation similarity, feature dispersion, calibration, and corruption robustness. It reports that contrastive models retain more low/mid-level information and generally have stronger calibration and robustness than CE, while noting metric-specific exceptions.

Representation similarity

  • Final-to-initial CKA is 0.10 for CE and 0.12 for SupCon; SelfSupCon has the most low/mid-level information, and joint models exceed their supervised counterparts.The authors interpret higher similarity between final and initial layers as evidence of more low/mid-level content in later representations.Source: §5 · p. 6
  • ResNet stages 1–3 have cross-model CKA above 0.9, but final-layer representations diverge; final-layer CKA is 0.47 for CE/SelfSupCon and 0.42 for CE/SupCon.Contrastively trained models are more similar to one another than to the cross-entropy model.Source: §5 · p. 6

Feature dispersion

  • The authors argue that greater intra-class variation may benefit transfer despite possibly harming in-domain performance; t-SNE shows broader contrastive clusters than CE.The visual compares penultimate-layer activations on ImageNet.Source: §5 · pp. 6–7, Figure 5 · p. 7

Calibration

  • Contrastive approaches produce better-calibrated predictions than CE in in-domain evaluation and transfer, although SelfSupCon has the best calibration but worst top-1 accuracy.CE+SelfSupCon improves ECE and NLL over either CE or SelfSupCon; SupCon+SelfSupCon improves calibration over SupCon or SelfSupCon.Source: Table 5 · p. 7, §5 · p. 7

Corruption robustness

  • On ImageNet-A and ImageNet-R, CE+SelfSupCon raises accuracy over CE by 5.18% and 4.38% and lowers ECE by 1.58% and 3.72%, respectively.No single contrastive model is best across every robustness metric, though contrastive loss generally improves robustness.Source: Table 6 · p. 7, §5 · p. 7

6. Ablation Studies

The ablations show that transferability responds differently from ImageNet accuracy to the SelfSupCon weight, stronger CE augmentations, and longer pretraining. Contrastive methods retain a transfer advantage, particularly with fixed features.

Ablations on weights of SelfSupCon loss

  • Transfer accuracy peaks at α = 2, while ImageNet accuracy peaks at α = 1; transfer accuracy gradually decreases when α > 2.The authors note that larger α imposes more intra-class variation, but too much may hurt transfer.Source: §6 · p. 8

Effect of augmentations on CE model

  • Contrastive approaches remain significantly better than CE and CE(strong), particularly for fixed-feature transfer, although stronger CE augmentation slightly improves CE transfer.CE(strong) adds color jitter, random grayscale, and Gaussian blur to standard CE augmentation.Source: §6 · p. 8

Does transferability improve with longer training?

  • After 80 pretraining epochs, CE transfer improves very little, whereas contrastive transfer improves with longer pretraining; joint objectives also perform better early.Figure 7 compares average linear-evaluation accuracy across downstream datasets with ImageNet validation accuracy at intermediate checkpoints.Source: §6 · p. 8, Figure 7 · p. 8

7. Conclusion

The study finds contrastive representations transfer better than standard cross-entropy representations across several downstream tasks. Joint contrastive objectives further improve transfer, while the learned representations differ in their information, class separation, and corruption robustness.

7. Conclusion

  • Contrastive models consistently outperform standard cross-entropy models in transfer learning, and combining self-supervised contrastive loss with cross-entropy or supervised contrastive loss improves performance.The study covers downstream image classification, few-shot recognition, and object detection.Source: §7 · p. 8
  • Contrastive models contain more low- and mid-level information in final layers, show larger intra-class separation, and are more robust to image corruptions than cross-entropy models.Source: §7 · p. 8

A.1. Datasets

The appendix describes evaluation across 12 datasets spanning different visual domains, followed by the ImageNet pretraining setup used for the models.

A.1. Datasets

  • The 12 downstream datasets cover natural, satellite, symbolic, illustrative, medical, and texture domains, with top-1 accuracy used unless stated otherwise.Examples include crop-disease images and flowers among natural images.Source: §A.1 · p. 12

A.2. ImageNet Pretraining

  • For ImageNet pretraining, the models use 400 epochs, a 0.01 learning rate, and cosine scheduling with five warm-up epochs.Contrastive models use color jitter, random grayscale, and Gaussian blur in addition to random crops, flips, and normalization; CE uses the standard augmentation set.Source: §A.2 · p. 12

A.3. Linear Evaluation

The appendix specifies protocols for linear evaluation, object detection, full-network fine-tuning, and few-shot recognition, including feature handling and training choices.

A.3. Linear Evaluation

  • Linear evaluation freezes the pretrained backbone and trains a linear layer on downstream data, with an intervening non-affine BatchNorm layer and validation-based hyperparameter tuning.The reported setup trains for 50 epochs; 100 epochs showed no noticeable improvement.Source: §A.3 · p. 12

A.4. Object Detection

  • Object detection fine-tunes the full network and adds BatchNorm layers to detector components such as the ROI head and feature pyramid network.Source: §A.4 · p. 13

A.5. Full-network Finetuning

  • Full-network fine-tuning trains the pretrained backbone and linear layer together, with validation-selected hyperparameters; learning rate 0.001 and batch size 32 worked best for most datasets.The protocol includes a BatchNorm layer between the backbone and linear layer.Source: §A.5 · p. 13

A.6. Few-shot Recogniton

  • Few-shot evaluation follows the CDFSL benchmark setup and reports mean accuracy over 600 randomly sampled tasks with a 95% confidence interval.A logistic regression layer is trained on extracted features during meta-testing.Source: §A.6 · p. 13

B. More Results and Analysis

This block reports supplementary transfer evaluations and analysis setups across classification, detection, few-shot recognition, and calibration. Its explicit calibration result favors contrastive models on average.

B.1. Linear Evaluation

  • Table 8 contains detailed results for fixed-feature linear evaluation.Source: §B.1 · p. 13

B.2. Object Detection

  • Object-detection results in Table 9 report standard deviations over five runs.Source: §B.2 · p. 13
  • With longer object-detection training, CE, SelfSupCon, and SupCon move closer to the joint-loss models; RetinaNet shows a similar transferability trend.The longer setting uses a 2× Detectron2 schedule; the similar RetinaNet trend supports transfer across detector types.Source: §B.2 · p. 13

B.3. Few-shot Recogniton

  • Few-shot results cover 1-, 5-, and 20-shot settings in Table 12.Source: §B.3 · p. 13

B.4. Full-network Finetune

  • Table 13 provides full-network fine-tuning results for downstream datasets.Source: §B.4 · p. 13

C.1. Experimental Setup for Analysis on ImageNet Derivatives

  • For ImageNet derivative analyses, a classifier head is trained on ImageNet1K for SelfSupCon, SupCon, and SupCon+SelfSupCon, while CE and CE+SelfSupCon use their supervised-branch head.The head-learning rate is selected from 0.01, 0.1, 1, 2, and 10 using validation accuracy; no weight decay is used.Source: §C.1 · p. 13

C.2. Model Calibration on the Downstream Datasets

  • Contrastive models have lower Expected Calibration Error than cross-entropy models on average across the 12 downstream datasets.The evaluation trains a linear layer on fixed features and reports the best validation-set ECE, where lower is better.Source: §C.2 · p. 13

C.3. Robustness of Adversarial Attack

The robustness analysis finds that all models are vulnerable to PGD attacks, with CE most robust at smaller perturbations. A separate queue-size analysis reports better transfer with larger queues.

D.1. Effect of Queue-size to Transferability

  • Higher queue sizes are associated with better fixed-feature transfer accuracy across downstream datasets.The comparison uses ImageNet1K-pretrained contrastive models with queue sizes 1024, 8192, and 65596.Source: §D.1 · p. 14

C.3. Robustness of Adversarial Attack

  • At smaller perturbations, CE is the most robust model, while the other methods are relatively vulnerable to PGD attacks.The paper attributes this tentatively to contrastive methods' sensitivity to local changes; all models remain vulnerable without adversarial training.Source: §C.3 · pp. 13–14, Figure 8 · p. 14

D.2. Results with torchvision-pretrained Model

The supplementary analyses compare transfer across pretrained models, backbones, training domains, and feature-separation measures. They also examine stronger augmentation and report that contrastive transfer advantages persist across several settings.

D.2. Results with torchvision-pretrained Model

  • The torchvision-pretrained model leads in linear evaluation, but its accuracy is similar in full-network fine-tuning; it does not exceed the reconstructed model in robustness or ImageNet accuracy.The torchvision model scores 75.7% on ImageNet versus 76.60% for the reconstructed CE model, and its mCE is 78.50% versus 60.80%.Source: §D.2 · pp. 14–15, Table 16 · p. 16, Table 17 · p. 16

D.3. ResNet50x2 as Backbone

D.4. Pretraining on Stylized ImageNet

  • On Stylized ImageNet pretraining, contrastive approaches also provide better transferable representations than the cross-entropy model in fixed-feature evaluation.Source: §D.4 · p. 16, Table 19 · p. 17

D.5. Measuring Intra-class Similarity

  • SupCon shows high intra-class variation, but CE+SelfSupCon does not show the expected higher intra-class distance; the authors caution that their distance measures may not capture the embedding landscape.The reported distances concern penultimate-layer representations on the ImageNet1K validation set.Source: §D.5 · p. 16, Table 20 · p. 17

D.6. Results for CE with Stronger Augmentation

  • CE with strong augmentation beats CE on SVHN, Sketch, and Omniglot but loses on CropDisease and ISIC; contrastive approaches generally perform better, while CE(MoCo-aug) is similar to CE.The paper notes only minor overall performance differences between CE and CE(strong), and between CE and CE(MoCo-aug).Source: §D.6 · p. 16, Table 21 · p. 18, Table 22 · p. 18

Related papers

Loading 2103.13517v3…