When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

1 Introduction

• The study asks whether unsupervised defenses can improve image-classification robustness without requiring labels, following extensions of adversarial training to semi-supervised learning.The motivation includes security concerns in high-stakes uses of image classification.Source: §1 · pp. 1–2
• Self-supervised pretraining can fail to transfer adversarial robustness to downstream tasks because pretraining and finetuning use mismatched tasks, even when their datasets share a distribution.The paper names this challenge cross-task robustness transferability.Source: §1 · p. 2
• ADVCL uses original adversarial examples and high-frequency image components to form robustness-aware and generalization-aware contrastive views.The paper frames this as a unified adversarial contrastive learning framework.Source: §1 · p. 2
• ADVCL adds pseudo-supervision from clustering unlabeled data to improve cross-task robustness transferability, rather than relying on labeled data.The paper distinguishes this approach from self-supervised defenses aided with labeled data [27].Source: §1 · p. 2
• ADVCL achieves state-of-the-art robust accuracy under PGD and Auto-Attacks using standard linear finetuning, with 3.44% and 3.45% robustness improvements on CIFAR-10 and CIFAR-100 over existing self-supervised methods.The reported Auto-Attack improvements use an 8/255 ℓ∞ perturbation and ResNet-18.Source: §1 · p. 3

2 Background & Related Work

• Contrastive learning creates positive views of the same image and learns representations by maximizing their agreement while contrasting them with negatives.The paper identifies SimCLR as a common CL framework and its focus.Source: §2 · p. 3
• In SimCLR, two randomly transformed views of each unlabeled input form a positive pair, whose projected features are optimized with NT-Xent before a linear classifier is finetuned over the fixed encoder.Transformations include random cropping and resizing, color jittering, rotation, and cutout.Source: §2 · p. 3
• Adversarial training minimizes worst-case loss over adversarially perturbed inputs, but its supervised form requires true labels.The training objective is framed as a min-max game between an attacker and defender.Source: §2 · p. 3
• Prior self-supervised adversarial defenses leave open how to preserve downstream robustness without sacrificing lightweight linear finetuning efficiency.Some prior approaches require adversarial full finetuning, while another reports a large gap between standard and adversarial linear finetuning.Source: §2 · p. 3

3 Problem Statement

• The objective is to learn robust feature representations during pretraining that yield robust downstream predictions using only lightweight finetuning.The proposed setup combines robust pretraining with linear finetuning.Source: §3 · p. 4
• The downstream classifier places a linear prediction head over a fixed feature encoder and uses supervised cross-entropy, while adversarial finetuning can instead use worst-case cross-entropy.The paper excludes standard full finetuning because it does not preserve robustness.Source: §3 · p. 4
• Cross-task robustness transferability is the challenge of transferring robustness from a self-supervised pretext task to a new downstream task.Pretraining uses unlabeled source data, whereas finetuning typically targets a new task.Source: §3 · p. 4
• The paper asks whether CL can improve adversarial robustness with standard linear finetuning and what principles preserve robustness across tasks.It also asks what insights robust self-supervised representation learning provides.Source: §3 · p. 4

4.1 View selection mechanism

• Unlike prior adversarial-view methods [27] [28], ADVCL perturbs the full image rather than transformed crops, retaining standard augmentations while adding a full-image view.The authors motivate this choice by noting that aggressive transformations can treat small image regions as positive samples, which may not suit attacks on full images.Source: §4.1 · p. 5
• The high-frequency component serves as an additional view because it is linked to stronger generalization and adversaries typically concentrate on high frequencies [53].The image is decomposed into high- and low-frequency components using Fourier transformation and its inverse.Source: §4.1 · p. 5
• ADVCL adds adversarial and high-frequency views to the two standard augmented views in a four-view contrastive objective.The objective maximizes the contrastive loss over an adversarial perturbation while including the high-frequency component of the image.Source: §4.1 · pp. 4, 6

4.2 Supervision stimulus generation: ADVCL empowered by CLUSTERFIT

• CLUSTERFIT clusters pretrained features of unlabeled data and maps cluster indices to pseudo-labels, which stimulate adversarial training with a pseudo-classification cross-entropy loss.When the cluster count is unknown, the method can average losses from classifiers paired with multiple cluster counts; the authors report that this ensemble usually improves generalization.Source: §4.2 · p. 6

5 Experiments

• The experiments assess AA, RA, and SA with ResNet-18, using PGD and Auto-Attack evaluations and comparing ADVCL with adversarial and supervised baselines.The section includes cross-task and cross-dataset evaluations, qualitative analyses, and ablations.Source: §5 · p. 6

5.1 Quantitative Results

• ADVCL substantially improves over almost all baselines and improves robustness and standard accuracy simultaneously across the reported cross-task evaluation.Table 1 covers CIFAR-10 with ResNet-18 under standard linear finetuning and adversarial full finetuning.Source: Table 1 · p. 7, §5.1 · p. 7, Table 2 · p. 7

5.2 Qualitative Results

• ADVCL representations show clearer class boundaries than baseline representations in the CIFAR-10 t-SNE visualization.The paper links clearer separation with making successful image perturbation more difficult.Source: §5.2 · p. 8, Figure 4 · p. 8
• ADVCL produces more similar texture-aligned features across seed images in the feature inversion maps of an internal neuron.Figure 5 visualizes neuron 502 under CIFAR-10 across different robust-training approaches.Source: §5.2 · p. 8, Figure 5 · p. 8
• ADVCL’s adversarial loss changes more slowly around local optima than the losses of other self-supervised pretraining approaches.The landscape is plotted with respect to model weights under SLF, using adversarial rather than standard cross-entropy loss.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 Ablation studies

• ADVCL performs best under both standard and adversarial linear finetuning, with a smaller performance gap between the two than other approaches.ADVCL with SLF also outperforms baseline approaches with ALF, indicating the learned representations are sufficient for satisfactory robustness.Source: §5.3 · p. 9, Table 3 · p. 9
• The proposed three-view contrastive loss outperforms other tested adversarial-view choices, and high-frequency components yield the best overall performance among frequency-based views.Table 4 reports robust and standard accuracy for CIFAR-10 with ResNet-18 and SLF.Source: §5.3 · p. 9, Table 4 · p. 9
• Adding supervision stimulus boosts ADVCL performance, while an ensemble over cluster counts 2, 10, 50, 100, and 500 performs better than a single cluster count.The ensemble also makes performance less sensitive to the training dataset’s actual number of labels; the best result uses an ImageNet-pretrained SimCLR model for fpre.Source: §5.3 · p. 9, Table 5 · p. 9

• Perturbations to original images, high-frequency components, and proper supervision stimulus are beneficial factors for robust contrastive learning.The paper reports state-of-the-art robust and standard accuracy using standard linear finetuning, and identifies scaling to very large datasets and models as future work for boosting cross-dataset robustness transfer.Source: §6 · p. 9

Appendices

• The supplementary material provides additional implementation details and experimental results.Source: Appendices · p. 14

A. Discussion and Broader Impact

• The discussion says standard linear finetuning gives performance comparable to computationally expensive adversarial full finetuning and notes potential mobile-device security applications.It frames these as potential broader impacts of lightweight linear finetuners that preserve adversarial robustness.Source: A. Discussion and Broader Impact · p. 14

Pretraining Details

• ADVCL pretraining uses SGD with batch size 512 for 1,000 epochs, cosine learning-rate decay, and a 10-epoch warm-up.The initial learning rate is 0.5, momentum is 0.9, weight decay is 0.0001, contrastive temperature is 0.5, and the pseudo-label cross-entropy weight is 0.2.Source: B. Implementation Details · p. 14
• Pretraining augmentations include random crops scaled from 0.2 to 1, horizontal flips, color jittering, and random grayscale.Source: B. Implementation Details · p. 14

Finetuning Details

• SLF, ALF, and AFF train for 25 epochs; ALF and AFF generate training perturbations with 10-step ℓ∞ PGD at ϵ = 8/255.SLF and ALF freeze the encoder and train a linear classifier, whereas AFF trains the full network with TRADES loss.Source: B. Implementation Details · p. 14

TRIBN: Customized batch normalization

• TRIBN uses separate batch-normalization layers for the two adversary types and for normally transformed data.The two adversary objectives maximize contrastive loss or cross-entropy loss, respectively.Source: B. Implementation Details · p. 14

C. Performance Summary under AutoAttack

• ADVCL has the best reported CIFAR-10 performance under both standard linear and adversarial full finetuning, comparing standard and robust accuracy under Auto-Attack at 8/255 ℓ∞ perturbation strength.The figure compares pretraining approaches and finetuning settings; the upper-right indicates better standard and robust accuracy.Source: Figure A1 · p. 15, C. Performance Summary under AutoAttack · p. 15

D. Comparing SLF with ALF

• Under CIFAR-10 PGD attacks, ADVCL with standard linear finetuning outperforms all baseline approaches using adversarial linear finetuning.Applying adversarial linear finetuning to ADVCL improves robustness only by a small margin, whereas it gives baselines a larger boost.Source: D. Comparing SLF with ALF · p. 15
• ADVCL is reported to outperform the supervised adversarial-training baseline under both standard and adversarial linear finetuning on CIFAR-10.The authors attribute the smaller additional gain from adversarial linear finetuning to ADVCL's more robust representations.Source: D. Comparing SLF with ALF · p. 15

E. Experiments on More Vision Datasets

• ADVCL outperforms self-supervised and supervised adversarial-training baselines in most reported additional-dataset cases, except for TinyImageNet standard accuracy, where a 0.2% drop accompanies a 3.4% robust-accuracy improvement.The experiments include SVHN and TinyImageNet in-domain evaluations and SVHN-to-STL-10 and TinyImageNet-to-STL-10 cross-domain evaluations under standard linear finetuning.Source: E. Experiments on More Vision Datasets · p. 15, Table A3 · p. 16, Table A4 · p. 16

F. Running Time Comparison

• Adversarial full finetuning takes 24× more training time per epoch than standard linear finetuning on a pretrained model.The appendix attributes the added time to min-max optimization with multiple inner maximization iterations per outer minimization step.Source: F. Running Time Comparison · pp. 15–16, Table A1 · p. 16
• ADVCL costs more to pretrain than the RoCL and ACL contrastive-learning baselines, but offline pretraining and standard linear finetuning retain a finetuning-efficiency advantage for robustness transfer.The stated cost drivers are using more than two image views and adding pseudo-supervision regularization; self-supervised pretraining methods also take more time than supervised adversarial training because they typically need more epochs to converge.Source: F. Running Time Comparison · p. 16, Table A2 · p. 16

G. Robust Transfer Learning

• For CIFAR-10-to-STL-10 transfer with standard linear finetuning, ADVCL achieves the best reported performance under all evaluated PGD attack settings.The comparison is among pretraining approaches transferred from CIFAR-10 to STL-10.Source: G. Robust Transfer Learning · p. 17, Figure A3 · p. 17

H. Ablation Studies on CIFAR-100

• ADVCL outperforms baseline approaches in most CIFAR-100 PGD-attack cases, especially as attack steps or epsilon increase.The comparison examines whether obfuscated gradients explain the observed robustness results.Source: H. Ablation Studies on CIFAR-100 · p. 17