PaperlayerView exact revision on arXiv

Source-linked AI summary

Testing Robustness Against Unforeseen Adversaries

Max Kaufmann, Daniel Kang, Yi Sun, Steven Basart, Xuwang Yin, Mantas Mazeika, Akul Arora, Adam Dziedzic, Franziska Boenisch, Tom Brown, Jacob Steinhardt, Dan Hendrycks

arXiv:1908.08016v4cs.LGcs.CRcs.CVstat.ML

TL;DR

Real-world worst-case inputs may vary beyond Lp perturbations, and developers may not know which attacks they will face. ImageNet-UA evaluates robustness to unforeseen adversaries; its results show that UA2 differs from existing robustness metrics and that L2 training outperforms L∞ training on UA2.

  • Problem

    Common robustness research centers on Lp-bounded attacks, although real-world worst-case inputs may be diverse and outside the Lp ball, and developers may not know deployment attacks.

    Source: §1 · p. 1

  • Method

    ImageNet-UA uses nineteen gradient-based attacks, including eighteen novel non-Lp attacks, to benchmark unforeseen robustness on ImageNet with UA2.

    Source: §1 · pp. 1–2

  • Results

    UA2 is distinct from existing robustness metrics; L2 adversarial training yields 13.9% UA2 from 1.6%, versus 10.0% from 1.6% for L∞ training.

    Source: §1 · p. 2, §5.1 · p. 8

  • Takeaways & Limitations

    ImageNet-UA offers a standardized benchmark and progress measure for defenses targeting unforeseen adversarial robustness.

    Source: §1 · p. 1, §6 · p. 9

    The threat model assumes defenses allow gradient computation and do not use adversaries from the test-time distribution.

    Source: §3 · p. 4

Abstract

from arXiv · show

Adversarial robustness research primarily focuses on L_p perturbations, and most defenses are developed with identical training-time and test-time adversaries. However, in real-world applications developers are unlikely to have access to the full range of attacks or corruptions their system will face. Furthermore, worst-case inputs are likely to be diverse and need not be constrained to the L_p ball. To narrow in on this discrepancy between research and reality we introduce ImageNet-UA, a framework for evaluating model robustness against a range of unforeseen adversaries, including eighteen new non-L_p attacks. To perform well on ImageNet-UA, defenses must overcome a generalization gap and be robust to a diverse attacks not encountered during training. In extensive experiments, we find that existing robustness measures do not capture unforeseen robustness, that standard robustness techniques are beat by alternative training strategies, and that novel methods can improve unforeseen robustness. We present ImageNet-UA as a useful tool for the community for improving the worst-case behavior of machine learning systems.

1 INTRODUCTION

Adversarial robustness research has mainly focused on Lp-bounded attacks and defenses trained against the test-time attacks, although real-world worst-case inputs may be diverse and unknown to developers. ImageNet-UA evaluates robustness to unforeseen adversaries using a suite of gradient-based attacks.

1 INTRODUCTION

  • ImageNet-UA uses nineteen gradient-based attacks, including eighteen novel non-Lp attacks, to benchmark unforeseen robustness on ImageNet.The benchmark reports Unforeseen Adversarial Accuracy (UA).Source: §1 · pp. 1–2, Figure 1 · p. 2
  • UA2 measures generalization to diverse adversaries not seen during training, a threat model more realistic than the Lp adversaries emphasized in the literature.The benchmark addresses cases where developers lack access to the worst-case inputs their systems may encounter.Source: §1 · p. 1
  • Unforeseen robustness is distinct from existing robustness metrics; L∞-robust models generalize poorly to new attacks, while L2 is a stronger baseline.The authors also report that Lp training can be improved on by alternative training processes, and that CV benchmark progress has at least partially tracked unforeseen robustness.Source: §1 · p. 2
  • The authors propose ImageNet-UA as a standardized, expanded evaluation benchmark for unforeseen robustness and report that classical Lp-training-focused defenses can be improved on.They also measure a range of techniques and identify promising directions for generalizing adversarial robustness.Source: §1 · p. 2

2 RELATED WORK

Prior robustness evaluations and defenses largely focus on Lp-bounded attacks, while suitable non-Lp attacks and standardized tests of train-test adversary shifts have been limited. The paper situates ImageNet-UA as a unified benchmark for this gap.

2 RELATED WORK

  • Existing adversarial robustness measures almost exclusively apply to attacks optimized over an Lp ball, limiting their applicability to new deployment-time adversaries.The benchmark incorporates recommendations for sound evaluation, including measuring attack success across distortion magnitudes and using diverse differentiable attacks.Source: §2 · p. 3
  • Prior non-Lp attacks include generative-model methods that can be hard to bound or unstable, and expensive brute-force searches; the paper focuses on fast, differentiable, dataset-applicable attacks without auxiliary generative models.Earlier color-space transformations and differentiable image warps are also discussed, with Elastic adapted from the latter.Source: §2 · p. 3
  • Comparing defenses designed to generalize across train-test adversary shifts is difficult without a standardized benchmark and a sufficiently broad range of adversaries.ImageNet-UA provides a unified benchmark for testing unforeseen robustness.Source: §2 · p. 3
  • Pixel, Snow, JPEG, and Fog draw inspiration from common corruptions but are made differentiable to evaluate worst-case rather than average-case robustness.The Wood attack illustrates a separate design: PGD optimizes latent variables that parameterize sine waves overlaid on the image.Source: §2 · p. 3, Figure 3 · p. 4

3 THE UNFORESEEN ROBUSTNESS THREAT MODEL

The threat model defines adversarial examples through input-specific perturbation sets and evaluates classifier accuracy under a distribution of adversaries. It models unforeseen attacks by restricting defenders from using test-time adversaries while allowing full white-box access to attackers.

3 THE UNFORESEEN ROBUSTNESS THREAT MODEL

  • An adversarial example belongs to the input-specific perturbation set S_x and causes the classifier to predict incorrectly.The usual defense objective is high accuracy against an adversary optimizing within that set.Source: §3 · p. 4
  • Unforeseen robustness is classifier accuracy under a distribution of adversaries, rather than accuracy against a single Lp adversary.The distribution contains adversaries that define different perturbation sets.Source: §3 · p. 4
  • Attackers receive full white-box access to victim models, while defenders must allow gradient computation and cannot access adversaries in the test-time distribution.This models settings where defenders are unlikely to know all deployment-time adversaries.Source: §3 · p. 4
  • The benchmark treats unforeseen robustness as distinct from Lp robustness: models can achieve high UA2 while remaining susceptible to Lp attacks.Source: Table 1 · p. 5

4 MEASURING UNFORESEEN ROBUSTNESS

The framework evaluates unforeseen robustness with ImageNet-UA and UA2, supported by differentiable attacks whose strength can be controlled. It defines a diverse attack population and evaluates defenses at multiple distortion levels.

4.1 GENERATING ADVERSARIAL EXAMPLES

  • Attacks generate adversarial inputs through differentiable functions of an image and latent variables, with PGD used to approximately solve the resulting continuous optimization problem.The framework constrains the latent variables using Lp bounds and allows attack intensity to vary with ε.Source: §4.1 · p. 5

4.2 CORE ATTACKS

  • Core attacks span transformations including JPEG-space perturbations, pixelation, image warping, snowflakes, and color-channel shifts across horizontal bars.The attacks respond to increased optimization pressure, and increasing their distortion budget reduces model performance.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 ImageNet-UA: A NEW BENCHMARK FOR UNFORESEEN ROBUSTNESS

  • ImageNet-UA evaluates unforeseen robustness on ImageNet, while CIFAR-10-UA provides a computationally efficient counterpart for evaluating defense strategies and attack methods.Source: §4.3 · p. 6
  • UA2 measures classifier robustness across a categorical population of attacks, using each attack’s adversarial accuracy at its specified distortion level.The benchmark defines low, medium, and high distortion variants and focuses its reports on UA2med.Source: §4.3 · p. 7

5 BENCHMARKING FOR UNFORESEEN ADVERSARIAL ROBUSTNESS

The section finds that unforeseen robustness differs from existing robustness measures and that several training strategies improve UA2 beyond standard Lp training. It also relates gains to advances in computer vision, including scale, augmentation, and pretraining.

5.1 HOW DO EXISTING ROBUSTNESS MEASURES RELATE TO UNFORESEEN ROBUSTNESS?

  • UA2 is correlated with but distinct from Lp robustness, while optimized attacks resemble worst-case Lp attacks rather than ImageNet-C’s average-case profile.The authors present UA2 as a progress measure for real-world worst-case robustness.Source: §5.1 · p. 8
  • L2 adversarial training raises UA2 from 1.6% to 13.9%, exceeding L∞ training’s 10.0% from the same 1.6% standard-training baseline.The authors recommend L2 training as a strong baseline for unforeseen robustness.Source: §5.1 · p. 8

5.2 HOW CAN WE IMPROVE UNFORESEEN ROBUSTNESS?

  • 37.3 to 45.1: combining PixMix and L∞ training improves unforeseen robustness over either approach alone.The combination joins data augmentation with Lp adversarial training.Source: §5.2 · p. 8, Table 5 · p. 8
  • On CIFAR-10-UA, Meta Noise Generator training raises WRN-28-10 unforeseen robustness from 21.4% to 51.1%, exceeding the best Lp baseline.The authors leave scaling this multi-attack method to full ImageNet-UA for future work.Source: §5.2 · p. 8
  • On a 100-image ImageNet-UA subset, PAT improves over the best Lp-trained baselines from 22.6 to 26.2.PAT bounds its adversary using an estimate of human perceptual distance derived from classifier hidden states.Source: §5.2 · pp. 8–9
  • On a 100-class ImageNet-UA subset, combining Variational Regularization with PAT raises unforeseen robustness from 26.2 to 29.5 over standard PAT.Variational Regularization penalizes variance in higher-level features.Source: §5.2 · p. 9

5.3 HOW HAS PROGRESS ON CV BENCHMARKS TRACKED UNFORESEEN ROBUSTNESS?

  • UA2 rises from 1% for ResNet50 to 19.1% for ConvNeXt-V2-huge, while scaling alone shows diminishing returns.The authors also identify data augmentation and self-supervised pretraining as contributors to improved unforeseen robustness.Source: §5.3 · p. 9, Table 7 · p. 9

6 CONCLUSION

The conclusion introduces ImageNet-UA and UA2 for evaluating robustness to unforeseen adversaries, using nineteen non-Lp attacks, eighteen of them novel. The included appendices describe evaluated models, attack settings, and selected attack mechanisms.

6 CONCLUSION

  • ImageNet-UA standardizes evaluation of unforeseen adversaries with nineteen non-Lp attacks, including eighteen novel attacks, and introduces UA2 to measure unforeseen adversarial accuracy.The authors present the benchmark as groundwork for future research on real-world adversarial robustness.Source: §6 · p. 9

A.1 TRAINED MODELS

  • The trained-model appendix lists CIFAR-10 WRN-28-10 robust and TRADES models, ImageNet100 PAT-VR models, and ImageNet100 DINOv2 models.The DINOv2 models use a finetuned linear classification head, SGD at learning rate 0.001, and early stopping.Source: §A.1 · p. 13

A.2 MODEL REFERENCE

A.3 ATTACK PARAMETERS

  • Attack parameters were selected through extensive step-size sweeps to maximize attack effectiveness.The appendix also provides separate parameter tables for ImageNet-UA and CIFAR-10-UA.Source: §A.3 · p. 13, Table 8 · p. 14, Table 9 · p. 14

B DESCRIPTIONS OF THE 11 ADDITIONAL ATTACKS.

  • The additional attacks include blur, texture removal, pixel warping, edge-targeted PGD, Perlin-noise overlays, and adversarially optimized fog.Other described attacks use HSV-space PGD, block translation, image mixing, polkadots, and prison-bar patches.Source: B DESCRIPTIONS OF THE 11 ADDITIONAL… · pp. 14–15

C ATTACK COMPUTATION TIME

Most attacks take execution time not significantly greater than an equivalent PGD adversary. The accompanying evaluation materials report model results across datasets and distortion levels.

C ATTACK COMPUTATION TIME

D FULL RESULTS OF MODEL EVALUATIONS

D.2 CIFAR-10; D.3 IMAGENET100

  • Additional UA2 figures report low-, medium-, and high-distortion performance for CIFAR-10 and ImageNet100.The captions specify the datasets and distortion levels but do not state comparative outcomes.Source: Figure 9 · p. 20, Figure 10 · p. 21, Figure 11 · p. 22

D.4 EXPLORING THE ROBUSTNESS OF DINOV2

AutoAttack reduces DINOv2 accuracy to zero across the tested distortion levels, despite its strong PGD robustness. The appendices also report small seed-related UA2 variation, show attacked examples, and examine scaling.

D.4 EXPLORING THE ROBUSTNESS OF DINOV2

  • AutoAttack reduces DINOv2 accuracy to 0.0% across all distortion levels, despite its strong PGD robustness.For robust ResNet50, AutoAttack performs similarly to PGD; the authors suggest using AutoAttack as a comparison point instead of base PGD.Source: §D.4 · p. 24

D.5 PERFORMANCE VARIANCE

  • Across five seeds, UA2 standard deviations are 0.1% for ResNet50 and 0.04% for ResNet50 + L2 5, indicating minor performance variation.The evaluation used three samples of each of the two ImageNet models.Source: §D.5 · p. 24

E IMAGES OF ALL ATTACKS ACROSS DISTORTION LEVELS

F SCALING BEHAVIOUR OF OUR ATTACKS

G IMAGENET-C AND UNFORESEEN ROBUSTNESS

ImageNet-C performance is correlated with performance on optimized attacks, while optimization favors adversarially trained models. Performance on the eight core attacks also improves with ConvNeXt-V2 model scale.

G IMAGENET-C AND UNFORESEEN ROBUSTNESS

  • ImageNet-C average-case robustness correlates with performance on optimized attacks.Applying optimized versions of the attacks favors adversarially trained models.Source: Table 12 · p. 29
  • Optimized attacks favor adversarially trained models over other models.The passage reports this comparison without giving numeric values.Source: Table 12 · p. 29
  • Performance on the eight core attacks improves with ConvNeXt-V2 model scale.Source: Figure 20 · p. 30

H BENCHMARKING NON- Lp ADVERSARIAL TRAINING STRATEGIES

The section compares non-Lp robustness-training strategies with Lp baselines, reporting gains for MNG, PAT, PAT-VR, and PixMix combined with adversarial training.

H BENCHMARKING NON- Lp ADVERSARIAL TRAINING STRATEGIES

  • The evaluated alternatives include MNG, PAT, Variational Regularization, and PixMix combined with Lp adversarial training, and all beat Lp training.MNG is presented as a multi-attack robustness baseline, while PAT and Variational Regularization were designed for unforeseen robustness.Source: H BENCHMARKING NON- Lp ADVERSARIAL… · p. 31
  • 11.3% increase in UA2 on CIFAR-10-UA for MNG and 3.5% increase for PAT.MNG also improves over Lp training baselines on CIFAR-10-UA.Source: Table 13 · p. 31, H BENCHMARKING NON- Lp ADVERSARIAL… · p. 31
  • ImageNet-UA PAT and PAT-VR ResNet50s improve over Lp baselines.For computational-budget reasons, the selected Lp models were trained on a 100-image subset of ImageNet, using every 10th class.Source: Table 14 · p. 31
  • Combining PixMix with adversarial training results in large UA2 improvements on CIFAR-10.The L∞ training used the TRADES algorithm.Source: Table 15 · p. 31

I HUMAN STUDY OF SEMANTIC PRESERVATION

The human study tests whether ImageNet-UA adversarial examples retain enough semantic content for people to classify. Participants labeled images from a 100-class subset and could mark images as too corrupted.

I HUMAN STUDY OF SEMANTIC PRESERVATION

J CORRELATION OF Lp ROBUSTNESS AND ImageNet-UA

The appendix reports correlations between conventional robustness measures and ImageNet-UA, compares transfer and grid-search attacks, and discusses possible safety implications and trade-offs.

J CORRELATION OF Lp ROBUSTNESS AND ImageNet-UA

  • For adversarially trained models, Lp robustness correlates with UA2, but some models without adversarial training also improve on UA2.Source: Figure 23 · p. 34

K GRID SEARCH VS. GRADIENT-BASED SEARCH

  • Gradient-based methods outperform randomized grid search using 1,000 forward passes per datapoint.Source: Table 17 · p. 34

L TRANSFER ATTACKS

  • Transfer attacks are less effective than white-box attacks but consistently outperform randomly initialized, unoptimized attacks.The comparison uses 1,000 test samples across source and target models.Source: L TRANSFER ATTACKS · p. 35

M X-RISK SHEET

  • The authors caution that individual responses do not decisively establish whether the work is relevant to existential-risk reduction.Source: M X-RISK SHEET · p. 35

M.1 LONG-TERM IMPACT ON ADVANCED AI SYSTEMS

  • The authors describe unforeseen-attack robustness as potentially protecting advanced AI systems and screening tools, including against evasion attempts involving biothreat detection.They also identify proxy gaming and bioterrorism through screening tools as directly affected risks.Source: §M.1 · pp. 35–36

M.2 SAFETY-CAPABILITIES BALANCE

  • The benchmark quantifies UA2 progress relative to clean accuracy; the authors say adversarial training can improve unforeseen robustness while reducing clean accuracy.They also report that methods improving clean accuracy can improve unforeseen robustness, with some methods providing differential UA2 gains.Source: §M.2 · p. 36

M.3 ELABORATIONS AND OTHER CONSIDERATIONS

  • Some methods may improve unforeseen robustness without sacrificing clean accuracy, but may still incur compute overhead.Source: §M.3 · pp. 36–37
Loading 1908.08016v4…