PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Pre-Training by Adversarial Contrastive Learning

Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang

arXiv:2010.13337v1cs.CV

TL;DR

Adversarially robust learning needs label-efficient representations that remain stable under small malicious perturbations, but existing representation learning does not directly enforce this invariance. The paper proposes Adversarial Contrastive Learning, which combines standard and adversarial feature consistency through several contrastive formulations. ACL consistently improves standard and robust accuracy, including 2.14% and 2.99% gains over the prior approach on CIFAR-10, while also improving semi-supervised adversarial training.

  • Problem

    Adversarially robust deep learning requires more labeled data than standard learning, motivating methods that exploit unlabeled data while addressing feature instability under adversarial perturbations.

  • Method

    ACL injects adversarial perturbations into SimCLR-style contrastive pre-training and explores standard-to-standard, adversarial-to-adversarial, adversarial-to-standard, and dual-stream consistency formulations.

  • Results

    ACL consistently outperforms existing methods, improving CIFAR-10 TA by 2.14% and RA by 2.99% over the previous unsupervised robust pre-training approach.

  • Takeaways & Limitations

    ACL pre-training produces models that are both label-efficient and robust and improves semi-supervised adversarial training when only a fraction of labels is used.

  • Takeaways & Limitations

    The evaluation notes a 1% RA drop relative to the reported result for [29] because testing uses PGD attacks with random initializations, which are stronger.

Abstract

from arXiv · show

Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits particularly well with the goal of adversarial robustness, as one cause of adversarial fragility is the lack of feature invariance, i.e., small input perturbations can result in undesirable large changes in features or even predicted labels. We explore various options to formulate the contrastive task, and demonstrate that by injecting adversarial perturbations, contrastive pre-training can lead to models that are both label-efficient and robust. We empirically evaluate the proposed Adversarial Contrastive Learning (ACL) and show it can consistently outperform existing methods. For example on the CIFAR-10 dataset, ACL outperforms the previous state-of-the-art unsupervised robust pre-training approach by 2.99% on robust accuracy and 2.14% on standard accuracy. We further demonstrate that ACL pre-training can improve semi-supervised adversarial training, even when only a few labeled examples are available. Our codes and pre-trained models have been released at: https://github.com/VITA-Group/Adversarial-Contrastive-Learning.

1 Introduction

The paper targets the joint challenge of label efficiency and adversarial robustness, where robust learning requires more labeled data and existing self-supervised approaches are limited. It proposes combining contrastive pre-training with adversarial learning to promote feature consistency under both augmentations and perturbations.

  • Adversarially robust deep learning has higher sample complexity than standard learning, intensifying the scarcity of labeled data.
  • Existing robust-learning approaches use unlabeled data through auxiliary losses, including robust regularization and pseudo-labeling.
  • The framework extends prior self-supervised robustness work beyond ad-hoc pretext tasks by using contrastive learning.
  • Contrastive learning improves unsupervised representations by maximizing consistency between differently augmented views.
  • ACL injects adversarial perturbations into contrastive pre-training to encourage feature consistency and reduce sensitivity to small input changes.

2 Our approach

The approach integrates contrastive pre-training with adversarial perturbations to learn representations consistent across standard and adversarial views. It explores several contrastive formulations, ultimately using a dual-stream design with separate batch-normalization parameters for standard and adversarial branches.

  • Adversarial Training: Adversarial Training (AT) formulates robust learning as minimax optimization over model parameters and perturbations.Standard training is the special case ε=0.
  • Contrastive Pretraining: SimCLR learns representations by maximizing agreement between differently augmented views of the same unlabeled image.In ACL, robustness components are injected into this standard-to-standard contrastive framework.
  • Adversarial Contrastive Learning: ACL evaluates Adversarial-to-Adversarial (A2A) and Adversarial-to-Standard (A2S) contrasting as alternatives to standard-to-standard contrasting.A2A applies adversarial perturbations to both augmented views, whereas A2S perturbs only one view.
  • Adversarial Contrastive Learning: A2S requires independent batch-normalization parameters for standard and adversarial branches despite shared convolutional weights.The separate statistics address differences between clean and adversarial feature distributions.
  • Dual Stream: Dual Stream (DS) combines standard-to-standard and adversarial-to-adversarial contrastive losses, using separate batch-normalization sets and equal default weighting.The four branches share convolutional weights, while standard and adversarial branches maintain distinct BN parameters.
  • Semi-Supervised Training: Semi-supervised training uses ACL pre-training, labeled-data standard training for pseudo-label generation, and adversarial training over labeled and unlabeled data.The routine initializes both the pseudo-labeling and final adversarial-training stages with the ACL-pre-trained model.

3 Experiments and analysis

Experiments evaluate ACL across supervised and semi-supervised adversarial training, robustness under unforeseen attacks, ablations, and representation quality. ACL (DS) improves standard and robust accuracy, strengthens label efficiency, generalizes across attacks, and accelerates robust fine-tuning.

  • Supervised adversarial fine-tuning: ACL (DS) improves [TA, RA] by [2.14%, 2.99%] on CIFAR-10 and [2.14%, 3.58%] on CIFAR-100 over Selfie.These gains establish new TA/RA benchmark numbers on both datasets.
  • Supervised adversarial fine-tuning: ACL (DS) contributes [0.79%, 1.26%] in [TA, RA] on CIFAR-10 with Wide-ResNet-34-10.The model reaches [85.12%, 56.72%] from a random-initialization baseline of [84.33%, 55.46%].
  • Unforeseen attacks: 2.42% averaged improvement occurs across 19 unforeseen attacks, with gains on most attacks except a 0.23% drop under gaussian noise.The comparison uses CIFAR-10 with ResNet-18.
  • Semi-supervised adversarial training: ACL (DS) exceeds Selfie by [4.50%, 0.72%] in [TA, RA] with 10% labels in semi-supervised adversarial training.Selfie improves TA by 1.37% over vanilla UAT++, while RA remains almost unchanged.
  • Semi-supervised adversarial training: With 1% labels, ACL (DS) causes only [1.00%, 0.42%] decreases in [TA, RA] versus 10% labels, while vanilla UAT++ and Selfie drop ∼13% - 30%.ACL (DS) also produces 86.73% pseudo-label accuracy, compared with 37.67% for UAT++ and 46.75% for Selfie.
  • Robustness dynamics: ACL (DS) reaches 47.38% robust accuracy after one fine-tuning epoch, whereas random initialization requires 74 additional epochs to reach the same level.Longer fine-tuning without earlier learning-rate annealing produces a 1.0% robustness drop, termed adversarial over-fitting.

4 Related work and discussions

The paper situates ACL at the intersection of self-supervised pre-training, contrastive learning, and adversarial robustness. It argues that contrastive feature consistency may target adversarial fragility more directly than handcrafted pretext tasks.

  • Pre-training and self-supervision: Unlabeled pre-training can reduce dependence on prohibitively costly labels and generally improve downstream generalization when labeled data are scarce.The paper describes pre-training as learning representations from unlabeled data before fine-tuning with few-shot labeled data.
  • Pre-training and self-supervision: Handcrafted self-supervised pretext tasks impose structural or semantic learning signals but rely on heuristics that can limit representation generality.Examples include position, order, and rotation prediction; reconstruction losses preserve input information without enforcing stronger structural priors.
  • Adversarial robustness: Adversarially robust learning requires more data than standard learning, motivating self-supervised or unlabeled objectives alongside adversarial training.Prior approaches used auxiliary losses such as robust regularizers or pseudo-label losses, while the paper follows a plug-and-play pre-training and fine-tuning scheme.
  • Contrastive learning: Contrastive learning trains representations by minimizing distances between positive pairs and maximizing distances between negative pairs, avoiding raw-input reconstruction.Its success can depend on augmentation design, architecture, projection networks, loss functions, batch size, and memory-bank choices.
  • Adversarial robustness: The paper motivates contrastive pre-training for robustness because small perturbations can cause large feature changes or label changes near samples.It therefore conjectures that feature consistency is more directly targeted to robustness than the ad-hoc pretexts used in earlier adversarial pre-training.

5 Conclusion

The paper concludes that adversarial contrastive learning enhances robustness through self-supervised pre-training. Experiments across supervised fine-tuning and semi-supervised learning show models that are both label-efficient and robust, while larger models and more diverse perturbations remain future directions.

  • ACL combines contrastive learning with adversarially robust self-supervised pre-training to improve model robustness.
  • Experiments in supervised fine-tuning and semi-supervised learning produce models that are both label-efficient and robust.
  • Future work includes defenses for larger models and datasets and the incorporation of more diverse adversarial perturbations.

Broader Impact

The paper frames adversarial defense as relevant to secure and trustworthy AI and presents ACL as a data-efficient robustness tool. It claims improvements over prior robustness-aware pre-training and higher supervised and semi-supervised adversarial-training benchmarks.

  • Adversarial defense is presented as a component of making AI systems more secure and trustworthy.
  • ACL is presented as a tool for improving model robustness in a data-efficient fashion.
  • The paper reports that ACL advances robustness-aware pre-training and raises state-of-the-art results for supervised and semi-supervised adversarial training.

Supplementary Material

The supplementary material compares ACL pre-training with Augmix under matched CIFAR-10 supervised-training settings. ACL (DS) improves both standard and robust accuracy relative to Augmix, while the comparison uses carefully aligned settings.

  • Augmix enforces feature-divergence consistency as a regularization term, whereas ACL incorporates consistency as a pre-training method.
  • The Augmix comparison keeps settings identical except for parameters unused by ACL (DS), which are grid-searched for the best configuration.
  • 0.46% TA and 0.44% RA are the improvements ACL (DS) obtains over Augmix on CIFAR-10 under supervised training.
Loading 2010.13337v1…