Adversarial Self-Supervised Contrastive Learning

Minseon Kim; Jihoon Tack; Sung Ju Hwang

Source-linked AI summary

Adversarial Self-Supervised Contrastive Learning

Minseon Kim, Jihoon Tack, Sung Ju Hwang

arXiv:2006.07589v2Jun 13, 2020cs.LGcs.CVstat.ML

TL;DR

Adversarially robust training has largely relied on class labels, leaving open whether robust representations can be learned from unlabeled data alone. RoCL attacks instance-level identities and uses contrastive learning to align transformed samples with their adversarial counterparts. It achieves robustness comparable to supervised methods on targeted attacks, with stronger performance on unseen attacks, clean accuracy, and transfer learning; supervised fine-tuning improves robustness further.

Problem
Existing adversarial learning methods generally require class labels, while standard contrastive learning remains vulnerable to adversarial attacks.
Method
RoCL generates instance-wise adversarial examples from stochastic augmentations and maximizes contrastive similarity between transformed samples and their adversarial counterparts.
Results
RoCL obtains comparable robustness to supervised methods on targeted attacks, significantly higher robustness on most unseen attack types, and stronger results after supervised adversarial fine-tuning.
Takeaways & Limitations
Robust representations can be learned without class labels, and RoCL supports robust transfer learning while retaining strong clean accuracy.
Takeaways & Limitations
The broader impact discussion states that adversarial robustness against malicious attacks with deep neural networks remains a problem that has not yet been fully solved.

Abstract

from arXiv · show

Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions, which are then used to augment the training of the model for improved robustness. While some recent works propose semi-supervised adversarial learning methods that utilize unlabeled data, they still require class labels. However, do we really need class labels at all, for adversarially robust training of deep neural networks? In this paper, we propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. Further, we present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data, which aims to maximize the similarity between a random augmentation of a data sample and its instance-wise adversarial perturbation. We validate our method, Robust Contrastive Learning (RoCL), on multiple benchmark datasets, on which it obtains comparable robust accuracy over state-of-the-art supervised adversarial learning methods, and significantly improved robustness against the black box and unseen types of attacks. Moreover, with further joint fine-tuning with supervised adversarial loss, RoCL obtains even higher robust accuracy over using self-supervised learning alone. Notably, RoCL also demonstrate impressive results in robust transfer learning.

1 Introduction

The paper introduces RoCL, a self-supervised framework that trains adversarially robust representations without class labels by attacking instance identities and aligning clean and adversarial views. On benchmark datasets, RoCL matches strong supervised robustness while improving performance on unseen attacks, clean accuracy, and transfer learning.

Contribution: RoCL trains adversarially robust neural networks without class labels using a contrastive self-supervised objective.The method is designed for unlabeled data rather than relying on class labels or label-derived pseudo-labels.
Contribution: Instance-wise adversarial examples perturb stochastically augmented samples to confuse the model about each sample’s identity.The attack maximizes contrastive loss rather than targeting an incorrect class prediction.
Contribution: Contrastive training maximizes similarity between transformed samples and their instance-wise adversaries, suppressing perturbation-induced distortions in representation space.This objective treats robustness as consistency of an instance identity across adversarial views.
Evaluation: RoCL obtains comparable accuracy to strong supervised adversarial methods such as TRADES on CIFAR-10 and CIFAR-100 without labels during training.The paper evaluates the method using a robust-linear protocol against class-level adversaries.
Evaluation: Fine-tuning RoCL with class-adversarial loss yields stronger robustness without losing clean-sample accuracy, while transfer learning also shows impressive performance.The reported gains extend beyond the target attack setting to unseen attack types and downstream transfer evaluation.

2 Related Work

Prior work established adversarial training, randomized smoothing, and self-supervised representation learning as routes toward robustness, but existing adversarial learning commonly depends on class labels. The paper positions RoCL as addressing the vulnerability of standard contrastive learning while removing that label requirement.

Adversarial robustness: Adversarial robustness research includes gradient-based training methods such as FGSM, PGD, and TRADES, as well as defenses against random noise and corruptions.These approaches aim to improve robustness by training on perturbations or balancing clean accuracy with adversarial robustness.
Adversarial robustness: Existing adversarial learning techniques commonly require class labels to generate adversarial attacks.The related-work discussion identifies label availability as a common requirement for adversarial learning.
Self-supervised learning: Self-supervised learning creates supervised objectives from unlabeled data, often through manually defined pretext tasks for representation learning.Examples include predicting image-rotation angles and solving randomly generated Jigsaw puzzles.
Self-supervised learning and adversarial robustness: Prior studies report that unlabeled-data training, self-supervision, perceptual losses, and self-supervised pretraining can improve robustness or denoise adversarial perturbations.These approaches include robust representations, purifier networks, and robustness-oriented fine-tuning.

3 Adversarial Self-Supervised Learning with Instance-wise Attacks

RoCL replaces label-dependent class-wise attacks with instance-wise attacks on augmented unlabeled data, then incorporates those adversarial examples into self-supervised contrastive learning. The resulting representations are trained for robustness and evaluated through linear or robust-linear classification.

Instance-wise adversarial attacks: The attack uses projected gradient updates within an ℓ∞ norm-ball, with step size α and projection Π enforcing the perturbation constraint.The formulation generalizes gradient-based attacks, including PGD, through repeated updates and projection.
Motivation: Standard adversarial attacks require class labels, making conventional adversarial training inapplicable to unlabeled data.The paper motivates a label-free alternative to supervised attacks such as PGD and TRADES.
Instance-wise adversarial attacks: RoCL generates perturbations that maximize contrastive loss and confuse the model about an instance’s identity rather than its class.The attack operates on stochastically transformed inputs and treats other samples as negative instances.
Robust Contrastive Learning: The training objective adds instance-wise adversarial examples to the positive set and maximizes similarity between them and other augmentations of the same sample.The positive latent set contains the representation of another stochastic augmentation and the adversarial representation, while other instances remain negatives.
Evaluation: RoCL first learns robust representations without class labels, then uses a linear classifier for class-level evaluation; robust-linear evaluation additionally adversarially trains that classifier.The encoder is fixed during linear evaluation, while r-LE uses class-level adversarial examples for the linear model.

4 Experimental Results

RoCL is evaluated against white-box, black-box, unseen, smoothed, transfer-learning, and ablation settings. It achieves strong robustness without labels and gains further benefits from supervised fine-tuning and transformation smoothing.

White-box attacks: RoCL obtains significantly higher robustness than supervised adversarial-learning approaches against unseen attacks, except ℓ1 attacks with small perturbations.It also achieves much higher clean accuracy, while robust linear evaluation improves seen ℓ∞ robustness but degrades unseen ℓ1 robustness.
Fine-tuning: Adversarial fine-tuning improves robustness and clean accuracy, with RoCL + AT + SS further improving robustness over adversarial fine-tuning without the self-supervised loss.The fine-tuned RoCL models also outperform the self-supervised pretraining approach of Chen et al.
Black-box attacks: Black-box evaluations show RoCL outperforming TRADES against AT-generated attacks and matching AT against TRADES-generated attack samples.The study generates black-box examples with AT, TRADES, and RoCL models.
Transformation smoothing: Transformation smoothing improves both clean and black-box accuracy, while increasing the number of transformation samples further increases robustness.The smoothed classifier loses some robust accuracy under EoT attacks but remains reasonably robust.
Transfer learning: RoCL representations transfer effectively between CIFAR-10 and CIFAR-100, achieving better accuracy and robustness than fully supervised adversarial transfer learning in both directions.The comparison uses ResNet18 RoCL models against results from a larger WRN 32-10 architecture.
Ablation studies: Contrastive loss is the most effective tested attack-loss choice, and RoCL remains robust under 100 PGD iterations with 39.74% accuracy.The attack-loss ablation compares MSE, cosine similarity, Manhattan distance, and contrastive loss.
Ablation studies: Instance-wise adversarial examples move toward samples sharing the same instance identity after RoCL training, unlike the separated confusing samples produced on SimCLR.The result is shown through latent-space visualizations.

5 Conclusion

The paper introduces RoCL to learn robust representations without class labels by confusing instance identities and aligning transformed samples with adversarial counterparts. Across benchmarks, it achieves comparable targeted-attack robustness while improving clean accuracy and robustness to broader attacks and transfer settings.

RoCL generates instance-wise attacks that confuse a sample’s instance-level identity, then aligns transformed samples with those adversarial counterparts.
The method includes a transformation smoothed classifier that boosts performance during test inference.
RoCL achieves comparable robustness to supervised baselines on targeted attacks without using class labels.
RoCL obtains significantly better clean accuracy and robustness against black-box, unseen attacks, and transfer learning.

Broader Impact

The paper frames adversarial robustness as an unresolved deep-learning problem and argues that supervised adversarial training has limited generalization and clean accuracy. It positions adversarial self-supervised learning as a direction focused on vulnerabilities in intrinsic representations.

Adversarial robustness against malicious attacks remains a fundamental deep-learning problem that has not yet been fully solved.
Supervised adversarial training dominates current practice but suffers from limited generalization to unseen attacks and datasets and reduced clean accuracy.
The paper presents adversarial self-supervised learning as a research direction targeting vulnerability in intrinsic representation space.

A.1 Training detail and dataset

Training uses ResNet encoders with a projection head, SimCLR-style augmentations, and CIFAR-10 and CIFAR-100 data. RoCL is optimized with specified temperature, regularization, and ℓ∞ attack settings.

RoCL uses ResNet18 or ResNet50 as the encoder and a two-layer projection head with 128-dimensional embeddings.
Training minimizes the final loss with temperature τ = 0.5 and regularization parameter λ = 1/256.
The instance-wise attack uses ℓ∞ perturbations with ϵ = 0.0314 and step size α = 0.007.
Random augmentations comprise Inception crop, horizontal flip, color jitter, and grayscale.
The experiments use CIFAR-10 and CIFAR-100, each with 50,000 training and 10,000 test images, containing 10 and 100 classes respectively.

A.2 Evaluation

Evaluation freezes the encoder for linear or robust linear classification, while robustness testing uses PGD across multiple iteration counts and norm constraints. Transfer experiments move learned networks between CIFAR-10 and CIFAR-100.

Linear evaluation trains a classifier on the frozen encoder for 150 epochs using cross-entropy loss and SGD with momentum 0.9.
Robust linear evaluation uses the same frozen-encoder setup but generates class-wise adversaries with PGD under an ℓ∞ constraint.
Robustness evaluation uses white-box PGD attacks with 20, 40, and 100 steps under ℓ∞, ℓ2, and ℓ1 constraints.
The transformation smoothed classifier uses fixed-size Inception crops with scale 0.54 and varies transformation iterations across tests.
Expectation-over-transformation evaluation uses ℓ∞ perturbation ϵ = 0.0314, step size α = 0.00314, and K = 20 inner steps.
Transfer experiments evaluate networks moved from CIFAR-10 to CIFAR-100 and from CIFAR-100 to CIFAR-10.

A.5 Training efficiency of RoCL

RoCL trains by generating instance-wise adversarial examples from transformed inputs and optimizing a contrastive objective. It reaches sufficiently high clean accuracy and robustness after 500 epochs, while 1000 epochs take about 41.7 hours on two RTX 2080 GPUs.

Efficiency: 500 epochs yield models with sufficiently high clean accuracy and robustness.The reported ablation training setup uses 500 backbone-training epochs.
Efficiency: 41.7 hours are required to train RoCL for 1000 epochs using two RTX 2080 GPUs.
Training procedure: RoCL generates instance-wise adversarial examples from transformed inputs using contrastive loss.The algorithm then optimizes the model and projector parameters over the total loss.
Training procedure: The training objective includes a regularization term comparing adversarial examples with clean transformed examples.

C Results of CIFAR-10 and CIFAR-100

RoCL is evaluated on CIFAR-10 and CIFAR-100 under white-box attacks, including seen ℓ∞ and unseen ℓ2 and ℓ1 attacks, with additional evaluation against CW attacks.

RoCL achieves performance comparable to supervised adversarial learning methods on CIFAR-100.
Table 8 evaluates ResNet18 on CIFAR-10 and CIFAR-100 using robust linear evaluation and supervised adversarial training baselines.
The evaluation includes seen ℓ∞ attacks and unseen ℓ2 and ℓ1 attacks.
Table 9 reports white-box CW attack results for ResNet18 trained on CIFAR-10 with ℓ∞ training.

D Ablation

The ablations examine identity choices for instance-wise attacks and regularization, transformation functions, and the effects of regularization strength and batch size.

The ablations train the backbone for 500 epochs and the linear layer for 100 epochs.These settings produce models with sufficiently high clean accuracy and robustness.
Transformation function: Using transformed images from the same identity for instance-wise attacks is equally effective across transformation choices.
Instance-wise attack: Instance-wise attacks can use either the original transformed image or another transformation of the same instance identity.The perturbation is optimized to maximize contrastive loss between adversarial examples and same-identity images.
Regularization: Regularization varies the clean same-identity image used in the contrastive loss, choosing between t′(x) and t(x).
Experimental setup: Tables 10–12 report white-box ablations on ResNet18 trained on CIFAR-10 and CIFAR-100, including transformation, λ, and batch-size studies.
Hyperparameters: The regularization coefficient λ and contrastive-loss batch size B are important hyperparameters, and the optimal λ differs across batch sizes.