Source-linked AI summary
Do Adversarially Robust ImageNet Models Transfer Better?
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, Aleksander Madry
TL;DR
Transfer learning seeks effective adaptation of pretrained models when downstream data or resources are limited, but ImageNet accuracy may not fully characterize transferability. The paper compares robust and standard ImageNet models across multiple transfer settings and finds that robust models consistently match or improve their counterparts despite lower ImageNet accuracy. The findings support, while not confirming, hypotheses linking robustness with improved feature representations.
Problem
Transfer learning performance is not fully explained by source-dataset accuracy, motivating investigation of other properties of pretrained models.
Method
The paper compares adversarially robust and standard ImageNet models across fixed-feature and full-network transfer for classification, object detection, and instance segmentation.
Results
Robust models consistently match or improve standard models on transfer tasks despite lower ImageNet accuracy.
Takeaways & Limitations
The findings identify adversarial robustness as a transfer-relevant factor and add support to hypotheses that robustness improves feature representations.
Takeaways & Limitations
Further work is needed to confirm or refute the feature-representation hypotheses and understand which pretrained-model properties matter for transfer learning.
Abstract
from arXiv · showhide
Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at https://github.com/Microsoft/robust-models-transfer .
1 Introduction
Transfer learning adapts ImageNet-pretrained models to downstream tasks when training from scratch is impractical. This work identifies adversarial robustness as another factor affecting transfer performance and finds that robust models can match or improve standard models despite lower ImageNet accuracy.
- Transfer learning uses information from a source task, typically an ImageNet-1K classifier, to improve performance on a target task.
- Prior work reports that higher ImageNet accuracy tends to yield better fixed-feature transfer results.
- Adversarially robust models match or improve the transfer performance of standard counterparts despite being less accurate on ImageNet.
- The study evaluates transfer using fixed features and full-network fine-tuning across downstream classification, object detection, and instance segmentation.
- The results are consistent with hypotheses that adversarial robustness produces improved feature representations, but further work is needed to test those hypotheses and identify transfer-relevant model properties.
2 Motivation: Fixed-Feature Transfer Learning
Fixed-feature transfer uses penultimate-layer representations from pretrained ImageNet classifiers, but source accuracy may not fully characterize transfer quality. This motivates testing adversarial robustness as a training prior, with experiments comparing standard and robust models across 12 downstream classification datasets.
- Fixed-Feature Transfer Learning: Fixed-feature transfer trains a simple target-task model on penultimate-layer features extracted by a pretrained ImageNet classifier.The target images are passed through the source model, and the resulting representations are used for downstream classification.
- Motivation: Higher ImageNet accuracy has been associated with better fixed-feature transfer, but learned representations are not fully described by source-dataset accuracy.The paper therefore asks whether improving ImageNet accuracy is the only route to better transfer performance.
- Motivation: Adversarial robustness imposes local stability within an ε-radius neighborhood while minimizing training risk, providing a prior on learned feature representations.The robustness level is controlled by ε in the robust optimization objective.
- Motivation: Robust representations offer better-behaved gradients, regularization-free visualization, and approximate image invertibility, suggesting potential transfer advantages despite lower source accuracy.These properties are illustrated or discussed as differences between robust and standard representations.
- Experiments: Experiments evaluate fixed-feature transfer for standard and robust ImageNet models across 12 datasets and four ResNet-based architectures with varying robustness levels.The architectures include ResNet-18, ResNet-50, and WideResNet-50 variants.
- Experiments: Robust networks consistently extract better transfer features than same-architecture standard models, especially on Aircraft, CIFAR-10, CIFAR-100, Food, SUN397, and Caltech-101.The best robustness level is selected by grid search, with separate random seeds used for final performance estimation to avoid gains from training many random robust models.
3 Adversarial Robustness and Full-Network Fine Tuning
The study evaluates robust versus standard ImageNet backbones in full-network transfer across downstream classification, object detection, and instance segmentation tasks. Robust backbones generally improve downstream performance, with experiments using standard benchmark configurations and multiple backbone initializations.
- Image classification: Full-network transfer results compare standard and robust models across 12 downstream image classification tasks.Re-weighted accuracy is used for unbalanced datasets and raw accuracy for others; error bars show one standard deviation over ten trials.
- Detection and segmentation: Robust backbones generally achieve better average precision than standard backbones for object detection and instance segmentation.The best robust backbone outperforms the standardly trained backbone for every task shown.
- Experimental setup: Object detection is evaluated on PASCAL VOC and Microsoft COCO, while instance segmentation is evaluated on Microsoft COCO.The experiments use Faster R-CNN FPN and Mask R-CNN FPN models with varying ResNet-50 backbone initializations.
- Experimental setup: The experiments vary robust backbone initializations across multiple ε values and compare them with standardly trained ResNet-50 initialization.VOC uses one standard and twelve robust networks, whereas COCO and segmentation use smaller sets because of computational constraints.
4 Analysis and Discussion
The analysis examines how robustness, accuracy, width, dataset scale, and invariance affect transfer, finding that robust models often follow different and more favorable scaling patterns than standard models.
- ImageNet accuracy and transfer performance: Robust and standard models can match or differ in transfer behavior depending on ImageNet accuracy, robustness, width, and target dataset characteristics.The section studies these factors across fixed-feature transfer and related experiments.
- ImageNet accuracy and transfer performance: The linear relationship between ImageNet accuracy and transfer accuracy is often violated when robustness varies across models.This supports the hypothesis that robustness and accuracy exert separate effects on transfer.
- ImageNet accuracy and transfer performance: When robustness is fixed, improving ImageNet accuracy improves transfer performance at about the same rate as for standard models, with higher R2 correlation.The comparison uses highly robust ε = 3 models across six architectures.
- Robust models improve with width: As width increases, transfer performance plateaus or decreases for standard models but steadily grows for robust models in the tested width regime.The tested models include ResNet-50 at x1, x2, and x4 widths, plus ResNet-18; wider robust models require more computation.
- Optimal robustness levels for downstream tasks: The optimal robustness parameter varies by dataset: ε values of 3.0 and 1.0 work best for CIFAR-10 and CIFAR-100, while smaller values tend to suit other datasets.The authors hypothesize that finer-grained datasets require smaller ε values for successful transfer.
- Comparing adversarial robustness to texture robustness: Adversarially robust models outperform texture-invariant models on all considered datasets and outperform standard ImageNet models on Stylized-ImageNet evaluation and fine-tuning.Downscaling target images to 32 × 32 before upscaling produces closely aligned accuracy curves unlike the original Figure 5 experiment.
5 Related Work
Prior work studies transfer learning across vision and other domains, while robustness research increasingly investigates whether adversarial training improves learned feature representations.
- Transfer learning: Transfer learning has been studied with CNNs across medical imaging, language modeling, object detection, and instance segmentation.Prior methods often find that fine-tuning outperforms frozen feature-based transfer.
- Adversarial robustness: Adversarial robustness research includes empirical and certified methods, alongside growing work suggesting that robust networks learn improved features relative to standard networks.The paper positions its transfer-learning analysis within this broader feature-representation literature.
6 Conclusion
The paper compares standard and robust ImageNet models across downstream tasks, finding that robust models consistently match or improve transfer performance despite lower ImageNet accuracy.
- Conclusion: The study compares robust and standard models on 12 classification tasks, object detection, and instance segmentation.It also examines the interplay among ImageNet accuracy, model width, robustness, and transfer performance.
- Conclusion: The experiments use standard and robust ImageNet models trained across multiple architectures for the transfer-learning evaluations.The models serve as the pretrained networks for the paper’s experiments.
- Conclusion: The evaluated model sets include clean standard and ℓ2-robust ImageNet classifiers, as well as ℓ∞-robust classifiers.Their clean accuracies are reported separately for the standard/ℓ2 and ℓ∞ model groups.
- Conclusion: The training procedure uses SGD with batch size 512, momentum 0.9, weight decay 1e −4, and 90 epochs with scheduled learning-rate drops.All models are trained from scratch under this fixed procedure.
- Conclusion: Standard models use cross-entropy training, while robust models use adversarial training over specified ℓ2 and ℓ∞ perturbation ranges.The robust-training setup uses three attack steps and a step size based on ε.
A.2.1 Datasets
The paper evaluates ImageNet pre-trained models on a broad suite of downstream classification datasets using fixed-feature and full-network transfer settings. Experiments use standard training procedures, publicly released models, and additional scale-controlled datasets.
- Datasets: The classification benchmark includes datasets varying in size, number of classes, and number of datapoints.The datasets are listed in Table 5 and include the standard transfer suite used in prior work.
- Fixed-feature transfer: In fixed-feature transfer, ImageNet weights are frozen and only a newly initialized final fully connected layer is trained.The new layer is trained for 150 epochs with SGD and standard augmentation.
- Full-network transfer: In full-network transfer, the same hyperparameters are used, but the pretrained ImageNet weights are not frozen.
- Unified dataset scale: The unified-scale experiment resizes datasets to 32 × 32 before applying the transfer-learning augmentations.The test pipeline then resizes images to 256 and center-crops them to 224.
- Reproducibility: The study releases the standard and robust ImageNet models and code used in its experiments.
C Object Detection and Instance Segmentation
The paper extends its transfer-learning analysis to object detection and instance segmentation with ImageNet-initialized backbones. Robust backbones generally improve average precision, although the experiments use constrained training schedules for some COCO models.
- Experimental setup: For COCO, compute limits required reduced-schedule training from every robustness initialization before selecting the top three for full-schedule training.The selected robust initializations were compared with the standard model under the full 3x schedule.
- Experimental setup: The detection and segmentation experiments use standard Detectron2 configurations rather than introducing task-specific training tricks.COCO uses R50-FPN configurations, while VOC uses R50-C4.
- Results: Robust backbones generally produce better average precision for object detection and instance segmentation.The best robust backbone outperforms the standard-trained backbone on every reported task.
- Baselines: The comparison uses torchvision ResNet-50 baselines despite known performance differences from the original ResNet-50 models targeted by Detectron2.This baseline mismatch is explicitly acknowledged in the experimental discussion.
- Related context: ImageNet-pretrained networks are established backbone models for object detection and image segmentation, and prior work links higher ImageNet accuracy with better detection accuracy.
F.1 Full-network Transfer: additional results to Figure 5
Additional results show that transfer performance is not explained by ImageNet accuracy alone: at fixed source accuracy, greater robustness generally improves transfer accuracy. When robustness is fixed, however, ImageNet accuracy becomes highly predictive of full-network transfer.
- Full-network transfer: For fixed ImageNet accuracy, generally increased robustness yields higher full-network transfer accuracy.This breaks the otherwise expected linear relationship between source accuracy and transfer performance.
- Full-network transfer: When robustness is controlled, ImageNet accuracy is highly predictive of full-network transfer performance across architectures.The comparison varies architecture while holding ε fixed.
- Model comparison: Standard, stylized, and robust ImageNet models are compared on standard transfer tasks.
F.4 Unified scale: additional results to Figure 7
Scale-controlled experiments align transfer-accuracy curves more closely than the corresponding unscaled experiment. Increasing width benefits robust models more consistently, while standard-model gains generally plateau.
- Figure layout: The plotted comparison includes ImageNet accuracy and model identity for a ResNet-50 reference, with fixed-feature transfer shown as a separate panel.
- Fixed-feature transfer: After datasets are down-scaled to 32 × 32 and up-scaled to ImageNet size, fixed-feature transfer-accuracy curves are closely aligned.Birdsnap reaches 97.0 in the reported scale-controlled figure.
- Full-network transfer: The scale-controlled experiment also reports full-network transfer accuracies across the evaluated datasets.
- Width and robustness: As model width increases, standard-model transfer accuracies generally plateau or level off, whereas robust-model accuracies steadily increase.
G.1 Fixed-feature transfer to classification tasks (Fig. 5)
The section presents fixed-feature and full-network transfer evaluations across standard and robust ImageNet models and multiple datasets, including 32x32 downsampled datasets.
- Fixed-feature transfer is evaluated for various standard and robust ImageNet models across datasets.
- Full-network transfer is evaluated for various standard and robust ImageNet models across datasets.
- Fixed-feature transfer results are separately reported for 32x32 downsampled datasets.
- Full-network transfer results are separately reported for 32x32 downsampled datasets.