Source-linked AI summary
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, Zhangyang Wang
TL;DR
The paper addresses whether self-supervised pretraining can provide robustness and efficiency benefits for adversarial fine-tuning. It introduces adversarial self-supervised pretraining and finds improved robustness, faster adversarial fine-tuning, and further gains from ensembling diverse pretraining tasks.
Problem
The paper investigates whether appropriately pretrained models can improve the efficiency and adversarially robust generalization of downstream fine-tuning.
Method
The paper combines adversarial training with self-supervised pretraining, then applies the resulting robust representations to downstream supervised or adversarial fine-tuning, including task ensembling.
Results
The framework yields consistent gains over state-of-the-art adversarial training in standard and robust accuracy, including 3.83% robust-accuracy and 1.3% standard-accuracy gains on CIFAR-10, with a further 3.59% robust-accuracy improvement from ensembling.
Takeaways & Limitations
Robust pretrained models can benefit downstream fine-tuning by improving final robustness and reducing computation for adversarial fine-tuning, while diverse pretraining tasks can be combined for further gains.
Takeaways & Limitations
A prior co-optimization approach faces scalability challenges, does not provide pretrained embeddings, and leaves efficient tuning questions unanswered.
Abstract
from arXiv · showhide
Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pre-trained models have a diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.
1. Introduction
The paper asks whether self-supervised pretraining can improve the efficiency and robustness of adversarial fine-tuning, and introduces adversarially robust pretrained models to address this gap. Experiments report gains over end-to-end adversarial training and motivate ensembling diverse pretraining tasks.
- Motivation: Adversarial training is computationally expensive, motivating robust pretraining for more efficient and robust downstream fine-tuning.The paper frames this as a parallel to the benefits self-supervised pretraining provides for standard training.
- Approach: The framework introduces adversarial training into self-supervised pretraining to produce robust pretrained models for downstream tasks.It directly extends the self-supervised pretraining and fine-tuning framework into adversarial robustness.
- Results: Adversarial fine-tuning contributes the dominant portion of robustness improvement, while robust pretraining mainly speeds up adversarial fine-tuning.The study systematically evaluates combinations of pretraining and fine-tuning strategies.
- Ensembling: Different self-supervised tasks produce pretrained models with diverse adversarial vulnerabilities, motivating an ensemble of complementary tasks.The ensemble combines Selfie, Rotation, and Jigsaw pretraining.
- Ensembling: The ensemble strategy improves robust accuracy by 3.59% on CIFAR-10 while maintaining a slightly higher standard accuracy.The paper reports this as an additional improvement from combining the pretraining tasks.
2. Related Work
Prior work uses self-supervision and unlabeled data to improve representations or robust training, but this paper distinguishes its pretraining-plus-fine-tuning framework from existing auxiliary-loss approaches.
- Self-supervised learning: Self-supervised methods learn representations without manual labels for downstream tasks, using objectives such as rotation, category, filling, and spatial-composition prediction.Examples include inpainting, colorization, Rotation, Jigsaw, and Selfie.
- Adversarial robustness: Adversarial training is a strong defense, but robust learning has higher sample complexity and computational cost than standard training.These costs motivate using unlabeled data and pretrained representations for robust models.
- Ensembles: Model ensembles can improve adversarial robustness because ensemble diversity challenges adversarial-example transferability.Recent methods also encode diversity through training regularizers.
- Unlabeled data: Unlabeled data can competitively support adversarially robust training through auxiliary losses such as robust regularizers or pseudo-label objectives.Prior studies report better robust generalization with more unlabeled data.
- Research gap: The authors identify prior self-supervised robust training as a rotation auxiliary task co-optimized with supervised adversarial training, unlike their pretraining and fine-tuning framework.They characterize this prior work as the only known use of self-supervision for a target supervised robust task.
3. Our Proposal
The proposal combines self-supervised pretraining with adversarial training and downstream fine-tuning, including an ensemble of diverse pretraining tasks. Experiments compare configurations and find that adversarial fine-tuning supplies most robustness gains, while robust pretraining can reduce fine-tuning cost and transfer some robustness.
- 3.1. Setup: Self-supervised pretraining learns model parameters from an unlabeled dataset by minimizing a task-specific pretraining loss.The framework considers Selfie, Rotation, and Jigsaw as pretraining tasks.
- 3.1. Setup: Fine-tuning maps pretrained embeddings to labels using supervised loss, with either fixed pretrained parameters or jointly retrained parameters.These correspond to partial and full fine-tuning, respectively.
- 3.2. AT meets self-supervised pretraining and finetuning: Adversarial training replaces clean inputs with perturbation-optimized examples in a min-max classification objective, while standard training fixes the perturbation to zero.The same formulation can be applied to self-supervised pretraining or supervised fine-tuning.
- 3.2. AT meets self-supervised pretraining and finetuning: The study systematically compares pretraining and fine-tuning configurations, using end-to-end standard and adversarial training as baselines and measuring standard accuracy, robust accuracy, and fine-tuning epochs.Experiments use Selfie, Rotation, and Jigsaw pretraining tasks.
- 3.2. AT meets self-supervised pretraining and finetuning: Adversarial fine-tuning contributes the dominant portion of robustness improvement, whereas robust pretraining mainly accelerates adversarial fine-tuning.The paper also reports that full fine-tuning generally outperforms partial fine-tuning in robust and standard accuracy, except for the Jigsaw case noted by the authors.
- 3.3. AT by leveraging ensemble of multiple self-supervised learning tasks: The ensemble method shares a common network across multiple self-supervised tasks and adds a diversity-promoting regularizer based on their input-gradient geometry.The regularization parameter λ controls the diversity term, with λ = 0 yielding averaging ensemble training.
- 3.3. AT by leveraging ensemble of multiple self-supervised learning tasks: The diversity regularizer is maximized when input gradients become orthogonal, encouraging robustness against diversified perturbation directions.The paper uses gradient orthogonality to quantify diversity among self-supervised losses.
4. Experiments and Results
Experiments evaluate self-supervised and adversarial pretraining with multiple fine-tuning strategies, datasets, and self-supervised tasks. Adversarial pretraining improves robustness and fine-tuning efficiency, while adversarial fine-tuning and task diversity provide additional gains.
- Experimental setup: The experiments cover CIFAR-10, CIFAR-10-C, CIFAR-100, and R-ImageNet-224, with additional pretraining-size ablations using CIFAR-30K, CIFAR-50K, and CIFAR-150K.Models are evaluated on classification robustness across multiple datasets and settings.
- Experimental setup: The study systematically compares eight pretraining–fine-tuning configurations using standard, adversarial, partial, and full fine-tuning, with Selfie, Rotation, and Jigsaw tasks.Table 3 reports standard accuracy, robust accuracy, and fine-tuning iteration complexity.
- Pretraining and fine-tuning: Adversarial pretraining benefits even partial fine-tuning, adversarial partial fine-tuning yields 30% more improvement, and standard full fine-tuning can reduce robust accuracy to 0%.Adversarial full fine-tuning produces the most significant robustness improvement.
- Task diversity: Selfie and Rotation outperform Jigsaw for robustness, yielding at least 3.5% robust-accuracy improvement in adversarial-pretraining settings with partial fine-tuning.The passage reports consistency across self-supervised prediction tasks while favoring Selfie and Rotation.
- Unforeseen attacks: The approach improves standard accuracy by 1.97% and robustness against all 12 unforeseen attacks, with gains ranging from 1.03% to 6.53%.Separating pretraining from fine-tuning also provides a warm-start representation that mitigates one-shot adversarial-training computation costs.
- Task diversity: Different self-supervised tasks produce diverse transferred adversarial examples, with at least 14% non-overlap between examples attacking Rotation and Jigsaw models.This diversity motivates ensemble pretraining for further robustness improvement.
- Ablations: Increasing adversarial-pretraining data from 30K to 150K yields a 0.97% robust-accuracy gain with nearly unchanged standard accuracy.Random smoothing produces consistent findings: robust pretraining speeds adversarial fine-tuning, while full adversarial fine-tuning contributes most to robustness.
5. Conclusions
The paper combines adversarial training with self-supervision to produce robust pretrained models for downstream fine-tuning. It reports improved robustness and faster adversarial fine-tuning, with further gains from ensemble pretraining.
- Contributions: Adversarial pretraining provides robust representations that can be applied to downstream tasks through fine-tuning.The framework combines adversarial training with self-supervision.
- Findings: Adversarial pretraining boosts final model robustness and speeds subsequent adversarial fine-tuning, while adversarial fine-tuning contributes most to the robustness improvement.The paper reports consistent gains in standard and robust accuracy over state-of-the-art adversarial training on CIFAR-10.
- Ensemble strategy: An ensemble pretraining strategy further improves robustness by exploiting diversity among self-supervised tasks.The paper identifies incorporating more tasks, enlarging pretraining data, and scaling to high-resolution data as future directions.