PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Adversarially Robust ImageNet Models Transfer Better?

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, Aleksander Madry

arXiv:2007.08489v2cs.CVcs.LGstat.ML

TL;DR

Whether ImageNet accuracy is the only factor shaping transfer performance is unclear, since learned representations are not fully described by source accuracy. This paper compares robust and standard ImageNet models across downstream tasks and finds robust models match or improve transfer despite lower ImageNet accuracy.

  • Problem

    It is unclear whether ImageNet accuracy is the only factor affecting transfer performance, since learned representations are not fully described by source-dataset accuracy.

    Source: §2 · p. 2

  • Method

    The paper compares adversarially robust and standard ImageNet models across 12 classification tasks, object detection, and instance segmentation.

    Source: §6 · p. 9

  • Results

    Robust models consistently match or improve on standard counterparts across the evaluated transfer tasks despite lower ImageNet accuracy.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The findings add support to hypotheses suggesting that adversarial robustness is associated with improved feature representations.

    Source: §1 · p. 2

    Further work is needed to confirm or refute these hypotheses and to understand which pre-trained model properties matter for transfer learning.

    Source: §1 · p. 2

Abstract

from arXiv · show

Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at https://github.com/Microsoft/robust-models-transfer .

1 Introduction

The paper identifies adversarial robustness as another factor in transfer learning: robust ImageNet models, despite lower ImageNet accuracy, match or improve on standard counterparts in downstream transfer. It examines this trend in fixed-feature and full-network transfer across image classification, object detection, and instance segmentation.

Introduction

  • Transfer learning leverages a source task to improve a target task; the paper focuses on refining ImageNet-1K-pretrained models for target tasks.It is motivated by settings where labeled data or compute are insufficient to train models from scratch.Source: §1 · p. 1
  • Prior work links transfer performance to source-dataset choice [HAE16] [KBZ+19], higher ImageNet accuracy, and greater model depth than width [ARS+15].Source: §1 · p. 1

Our contributions

  • Despite lower ImageNet accuracy, adversarially robust networks match or improve on standard counterparts in downstream transfer.The authors first establish this trend with fixed features, then show it carries forward to full-network fine-tuning.Source: §1 · pp. 1–2
  • The study evaluates fixed-feature and full-network transfer across image classification, object detection, and instance segmentation.The image-classification suite covers 12 downstream tasks.Source: §1 · pp. 1–2, Table 1 · p. 2
  • The findings are consistent with hypotheses that adversarial robustness improves feature representations [EIS+19b] [AL20], but further work is needed to test them and identify useful pre-training properties.Source: §1 · p. 2

2 Motivation: Fixed-Feature Transfer Learning

Fixed-feature transfer depends on learned representations, not source accuracy alone, motivating the study of adversarial robustness as a training prior. The authors report that robust ImageNet models consistently yield better downstream transfer accuracy than standard models of the same architecture.

Motivation

  • Fixed-feature transfer trains a simple, often linear, classifier on target-dataset features from a pre-trained model’s penultimate layer, while prior work links higher ImageNet accuracy to better transfer.This approach has produced accurate vision classifiers and often outperformed handcrafted task-specific features [SAS+14], but the factors driving transfer performance remain incompletely understood.Source: §2 · p. 2
  • Architectures, loss functions, and data augmentation can impose distinct priors on learned features, so source-dataset accuracy alone does not fully describe representations governing transfer.Source: §2 · p. 2

The adversarial robustness prior

  • Robustness is a training prior induced by robust optimization, which minimizes training risk while promoting local stability within an ε-radius neighborhood of each training point.Adversarial robustness was first studied for security, but recent work also treats it as a prior on learned representations [EIS+19b] [ISE+19] [JBZ+19].Source: §2 · pp. 2–3

Should adversarial robustness help fixed-feature transfer?

  • Although robust models are less accurate on the source dataset, their representations have better-behaved gradients and are approximately invertible, suggesting robustness may benefit transfer.Engstrom et al. [EIS+19b] hypothesize that robust training encourages features more similar to human representations; Figure 1 illustrates regularization-free feature visualization and approximate image inversion.Source: §2 · p. 3, Figure 1 · p. 3

Experiments

  • The evaluation covers 12 transfer datasets and four ResNet-based architectures, with robust models trained at varying robustness levels ε.The datasets include those considered by and Caltech-256.Source: §2 · p. 3
  • Robust models consistently extract better transfer features than same-architecture standard models, with the largest effects on Aircraft, CIFAR-10, CIFAR-100, Food, SUN397, and Caltech-101.The WideResNet-50-4x models used a coarser robustness-level grid, so a finer search might further improve their results.Source: §2 · p. 3, Figure 2 · p. 4

3 Adversarial Robustness and Full-Network Fine Tuning

Adversarially robust ImageNet initializations consistently outperform standard networks in object detection and instance segmentation. The section also reports full-network transfer results across 12 downstream classification tasks.

Full-network transfer learning

  • Figure 3 reports full-network transfer results for standard and robust models across 12 downstream classification tasks, using re-weighted accuracy for unbalanced datasets and raw accuracy otherwise.Error bars show one standard deviation over ten random trials; the accuracy protocol follows [KSL19].Source: Figure 3 · p. 4

Object detection and instance segmentation

  • The best robust backbones outperform standard backbones on every evaluated detection and segmentation task, and robust backbones generally achieve better AP.The experiments use Faster R-CNN FPN for detection and Mask R-CNN FPN for instance segmentation; COCO results average four runs.Source: Figure 4 · p. 5, §3 · p. 5
  • Detection and segmentation are evaluated on VOC and COCO using Detectron2 defaults, without additional hyperparameter search.The benchmarks are VOC and COCO [LMB+14], with Detectron2 configurations.Source: §3 · p. 5

4 Analysis and Discussion

Previous results showed that robust models match or improve on standard models in transfer learning; this section examines how accuracy, width, downstream robustness level, and invariance relate to transfer performance.

4.1 ImageNet accuracy and transfer performance

  • Varying robustness often breaks the direct linear ImageNet-to-transfer accuracy relationship observed by.The pattern appears in fixed-feature transfer results and also in full-network transfer results reported in Appendix F.Source: Figure 5 · p. 6, §4.1 · p. 6
  • Improving robust-model ImageNet accuracy may further improve transfer, but accuracy alone does not measure feature quality, and why robust features transfer well remains open.The paper points to prior analyses of robust-network features [EIS+19b] [SSZ+19] [AL20] and mentions accuracy-improvement techniques [BGH19] [CRS+19].Source: §4.1 · p. 6
  • At fixed ε, higher ImageNet accuracy improves transfer for robust models at roughly the same rate as for standard models, with higher R2 correlation.This comparison uses highly robust (ε = 3) models spanning six architectures.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Robust models improve with width

  • Unlike standard models, whose transfer performance plateaus or declines with width, robust models keep improving across tested widths, potentially widening their transfer advantage at higher computational cost.The width comparison uses three ResNet-50 widths (x1, x2, and x4), with a ResNet-18 for reference; the standard-model trend accords with [ARS+15].Source: §4.2 · pp. 6–7, Figure 6 · p. 7

4.3 Optimal robustness levels for downstream tasks

  • The best robustness ε varies by dataset: 3.0 for CIFAR-10, 1.0 for CIFAR-100, and values smaller by an order of magnitude for other datasets.Source: §4.3 · p. 7
  • A granularity hypothesis predicts finer-grained targets need smaller ε; after resolution normalization, all 12 datasets’ ε-transfer curves nearly align with CIFAR-10 and CIFAR-100, supporting but not proving the hypothesis.The authors treat image resolution as a crude proxy for granularity and say further exploration is needed for definitive conclusions.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

4.4 Comparing adversarial robustness to texture robustness

  • Adversarially robust models outperform texture-invariant Stylized ImageNet models on all considered transfer datasets and outperform standard models on Stylized ImageNet evaluation and fine-tuning.Source: Figure 8 · p. 8, §4.4 · p. 9

5 Related Work

Prior work studies transfer methods and factors, as well as robust models’ features and accuracy tradeoffs. At their intersection, this paper complements concurrent transfer studies with broader downstream coverage and analysis of model factors.

Related Work

6 Conclusion … A.2 ImageNet transfer to classification datasets

Adversarially robust ImageNet models consistently match or improve on standard models in transfer despite lower ImageNet accuracy. The paper also examines how ImageNet accuracy, model width, robustness, and transfer performance interact.

6 Conclusion

  • Across 12 classification tasks, adversarially robust models consistently match or improve on standard counterparts despite lower ImageNet accuracy; the study also evaluates object detection and instance segmentation.The authors investigate how ImageNet accuracy, model width, robustness, and transfer performance interact.Source: §6 · p. 9
  • The analysis examines the interplay between ImageNet accuracy, model width, robustness, and transfer performance.Source: §6 · p. 9

A.1 Pretrained ImageNet models

  • The authors train standard and robust ImageNet models across several architectures and use them in the transfer-learning experiments.The architectures include standard models from Torchvision.Source: §A.1 · p. 16
  • All models are trained from scratch for 90 epochs with SGD, batch size 512, momentum 0.9, weight decay 1e −4, and a learning rate decayed tenfold every 30 epochs.Source: §A.1 · p. 16
  • Standard models use cross-entropy classification loss, while robust models use adversarial training [MMS+18].Source: §A.1 · p. 16
  • Robust training uses ℓ2 perturbation limits ε ∈ {0.01, 0.03, 0.05, 0.1, 0.25, 0.5, 1, 3, 5} and ℓ∞ perturbations, with 3 attack steps and step size ε×2/3.Source: §A.1 · p. 16

A.2.1 Datasets … B Transfer Learning with ℓ∞-robust ImageNet models

The section specifies the datasets and protocols used for transfer experiments, including full-network and fixed-feature variants, and describes additional analyses of dataset scale, reproducibility, and ℓ∞-robust models.

A.2.1 Datasets

  • Transfer starts from ImageNet-pretrained models on classification datasets used in, which vary in numbers of classes and datapoints; Table 5 lists their details.The supplied passages do not include the table's dataset entries.Source: §A.2.1 · p. 17, Table 5 · p. 17

A.2.2 Fixed-feature Transfer

  • With fixed-feature transfer, the ImageNet model is frozen and only a randomly initialized final layer is trained for 150 epochs using SGD.Training uses batch size 64, momentum 0.9, weight decay 5e−4, and an initial learning rate of 0.01 or 0.001, decayed tenfold every 50 epochs.Source: §A.2.2 · p. 17

A.2.3 Full-network transfer

  • Full-network transfer uses the fixed-feature setting's hyperparameters but does not freeze the pretrained ImageNet model's weights.Source: §A.2.3 · p. 17

A.3 Unifying dataset scale

  • To unify dataset scale, the experiment follows the A.2 setup but resizes all datasets to 32 × 32 before applying data augmentation.Source: §A.3 · p. 17

A.4 Replicate our results

  • The authors prioritize simplicity and reproducibility by using standard hyperparameters, minimizing replication tricks, and releasing the models and code.The code is available at the GitHub repository named in the passage.Source: §A.4 · p. 18

B Transfer Learning with ℓ∞-robust ImageNet models

  • Table 6 covers transfer accuracy for standard versus ℓ∞-robust ImageNet models on CIFAR-10 and CIFAR-100; the supplied passages do not state the results.The experiment investigates how other types of robust ImageNet models perform in transfer learning.Source: Table 6 · p. 19, B Transfer Learning with ℓ∞-robust… · p. 19

C Object Detection and Instance Segmentation … F Omitted Figures

The appendix reports that robust ImageNet backbones generally improve object detection and instance segmentation AP, and provides related-work context and background on adversarial robustness. It notes that robustly pretrained models can improve clean downstream accuracy despite lower ImageNet accuracy.

C Object Detection and Instance Segmentation

  • The best robust backbone outperforms the standard-trained backbone on every object detection and instance segmentation task, and robust backbones generally yield better AP.Because of compute limits, the authors searched all ε initializations with a 1x schedule, then trained the top three by Box AP and the standard model with a 3x schedule.Source: Figure 9 · p. 19, C Object Detection and Instance… · p. 19
  • Robustly initialized torchvision ResNet-50 models outperform the original baselines on all reported AP comparisons except COCO object detection, where the original baseline has 0.07 more Box AP.The robust-initialization models achieved this without additional hyperparameter searching, despite Detectron2 configurations being tuned for a slightly different ResNet-50 implementation.Source: C Object Detection and Instance… · p. 20

D Related Work

  • Prior work found that fine-tuning typically outperforms frozen-feature classification, while ImageNet top-1 accuracy strongly correlates with transfer accuracy.Earlier studies also reported that ImageNet features outperform handcrafted features on new tasks [DJV+14] [CSV+14] [SAS+14].Source: D Related Work · p. 21
  • Unlike [SSZ+19], this work examines clean downstream accuracy and finds robust, less accurate ImageNet models can transfer better; related studies associate robust features with reduced vulnerability and greater human perceptual alignment [ISE+19] [EIS+19b].Those studies also characterize robust optimization as imposing priors on learned features and report progress toward high-level input encodings [EIS+19b].Source: D Related Work · p. 21

E Background on Adversarially Robust Models

  • Adversarial examples are imperceptible perturbations that can cause model errors, and robust optimization replaces ERM with worst-case loss over a neighborhood around each training example.The radius parameter ε sets the desired robustness: ε = 0 gives standard training, while larger ε allows stability over larger radii.Source: E Background on Adversarially Robust… · p. 22
  • Adversarial training passes worst-case inputs to the network at each training iteration and remains a competitive baseline for training robust networks [GSS15; MMS+18; RWK20].The approach uses Danskin’s theorem to express the minimax gradient through the inner objective at its maximum [Dan67].Source: E Background on Adversarially Robust… · pp. 22–23

F.1 Full-network Transfer: additional results to Figure 5

For full-network transfer, accuracy alone does not explain transfer performance: at fixed accuracy, greater robustness generally yields higher transfer accuracy.

F.1 Full-network Transfer: additional results to Figure 5

  • At fixed accuracy, more robust ImageNet models generally achieve higher full-network transfer accuracy, so accuracy and transfer performance are not linearly related.The figure compares standard and robust models across image-classification datasets.Source: Figure 11 · p. 24

F.2 Varying architecture: additional results to Table 2 … G Detailed Numerical Results

Additional results show that ImageNet accuracy predicts full-network transfer when robustness is fixed, fixed-feature transfer curves align after downscaling, and increasing width benefits robust models more steadily than standard models.

F.2 Varying architecture: additional results to Table 2

  • At fixed robustness, ImageNet accuracy is highly predictive of full-network transfer performance.Table 7 varies architecture while fixing robustness.Source: Table 7 · p. 25

F.3 Stylized ImageNet Transfer: additional results to Figure 8b

  • Figure 12 compares standard, stylized, and robust ImageNet models on standard transfer tasks.Source: Figure 12 · p. 25

F.4 Unified scale: additional results to Figure 7

  • After datasets are down-scaled to 32 × 32 and up-scaled to ImageNet resolution, fixed-feature transfer accuracy curves are closely aligned.The caption contrasts these curves with the same experiment without downscaling, shown in Figure 5.Source: Figure 13 · p. 26
  • Figure 14 reports full-network transfer accuracies for datasets down-scaled to 32 × 32 and then up-scaled to ImageNet resolution.Source: Figure 14 · p. 26

F.5 Effect of width: additional results to Figure 6

  • As model width increases, robust models’ transfer accuracies generally continue to rise.Figure 15 varies width and robustness while transferring from ImageNet to various datasets.Source: Figure 15 · p. 27
  • As width increases, standard models’ transfer accuracies generally plateau or level off.This contrasts with the steady increases observed for robust models.Source: Figure 15 · p. 27

G.1 Fixed-feature transfer to classification tasks (Fig. 5) … G.3.2 Full-network (cf. Fig. 14)

The supplied appendix tables cover fixed-feature and full-network transfer using standard and robust ImageNet models, with separate evaluations on 32×32 downsampled datasets. Their captions describe evaluation scope but do not provide results or identify a better-performing model.

G.1 Fixed-feature transfer to classification tasks (Fig. 5)

  • Table 8 covers fixed-feature transfer with standard and robust ImageNet models across datasets.Source: Table 8 · p. 28

G.2 Full-network transfer to classification tasks (Fig. 3)

  • Across the standard-dataset tables, the appendix covers both fixed-feature and full-network transfer with standard and robust ImageNet models.Source: Table 8 · p. 28, Table 9 · p. 29
  • Table 9 covers full-network transfer with standard and robust ImageNet models across datasets.Source: Table 9 · p. 29

G.3.1 Fixed-feature (cf. Fig. 7 & 13)

  • Table 10 covers fixed-feature transfer on 32×32 downsampled datasets.Source: Table 10 · p. 30

G.3 Unifying dataset scale

G.3.2 Full-network (cf. Fig. 14)

  • Table 11 covers full-network transfer on 32×32 downsampled datasets.Source: Table 11 · p. 31

Related papers

Loading 2007.08489v2…