PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoding Robustness to Image Style via Adversarial Feature Perturbations

Manli Shu, Zuxuan Wu, Micah Goldblum, Tom Goldstein

arXiv:2009.08965v3cs.CVcs.LG

TL;DR

Small-pixel adversarial training leaves natural appearance and domain shifts unaddressed. The paper introduces AdvBN, which adversarially perturbs feature statistics during training, and reports improved robustness across shifts, with feature-space perturbations complementing image-space augmentation. The authors caution that gains may trade off in-distribution performance and may not transfer from benchmarks to practitioners' settings.

  • Problem

    Pixel-level adversarial training does not address natural shifts in image style, lighting, color mapping, or domain, which can severely degrade classifier performance.

    Source: §1 · p. 2

  • Method

    AdvBN adversarially perturbs feature statistics, whose means and variances encode style, to train networks against worst-case appearance shifts.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Fine-tuning on adversarially shifted feature distributions improves robustness to domain shifts without auxiliary data, and AdvBN consistently boosts input-space data-augmentation methods.

    Source: §5.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    Feature-space perturbations can complement input-space augmentation and improve generalization, while the reported robustness includes style shifts and image corruptions.

    Source: §1 · p. 2, §6 · p. 9

    AdvBN may trade off performance on non-shifted in-distribution data, and benchmark gains may not translate to varied real-world datasets and shifts.

    Source: §6 · p. 9

Abstract

from arXiv · show

Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations. However, machine learning practitioners need models that are robust to other kinds of changes that occur naturally, such as changes in the style or illumination of input images. Such changes in input distribution have been effectively modeled as shifts in the mean and variance of deep image features. We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts. We explore the relationship between these perturbations and distributional shifts by visualizing adversarial features. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training. By fine-tuning neural networks on adversarial feature distributions, we observe improved robustness of networks to various unseen distributional shifts, including style variations and image corruptions. In addition, we show that our proposed adversarial feature perturbation can be complementary to existing image space data augmentation methods, leading to improved performance. The source code and pre-trained models are released at \url{https://github.com/azshue/AdvBN}.

1 Introduction

Pixel-level adversarial training hardens classifiers against small input perturbations but does not address natural appearance and domain shifts. The paper adapts adversarial training to feature statistics and reports robustness across multiple shifts without auxiliary-domain data.

1 Introduction

  • Small-pixel adversarial training does not address shifts in image style, lighting, color mapping, or domain, which can severely degrade classifier performance.The paper positions these natural shifts as distinct from adversarial pixel perturbations.Source: §1 · p. 2
  • The method adversarially optimizes feature statistics, whose means and variances encode style, to prepare classifiers for worst-case styles rather than styles from auxiliary images.The statistics are shifted in a min-max game to resist appearance differences including texture, color, and brightness.Source: §1 · p. 2
  • AdvBN fine-tuning improves robustness to domain shifts without auxiliary data, including appearance changes, common corruptions, and cross-domain semantic segmentation.The paper also reports improved cross-domain generalization for semantic segmentation.Source: §1 · p. 2

2 Related work

Related work spans adversarial training, feature perturbation, and robustness to distribution shifts. The paper distinguishes its direct adversarial perturbation of feature means and variances from prior feature-statistics methods and image-space augmentation.

Adversarial training

  • Prior adversarial methods address adversarial examples or generate worst-case unseen domains, while this method directly perturbs feature-map means and variances without regularization terms.The paper also distinguishes its objective and perturbation target from prior combinations of adversarial training and feature statistics.Source: §2 · p. 2

Feature Perturbation

  • Feature perturbation methods generate novel distributions through spatial noise, signal processing, or re-normalization; this method adversarially simulates worst-case appearance changes instead of using other samples' statistics.Prior re-normalization approaches use statistics from other samples or domains, or interpolate features.Source: §2 · p. 2

Robustness to distribution shifts

  • The paper focuses on major image-style shifts, including illumination, weather, and quality changes; image augmentation can be limited by available transformations, unlike feature-space approaches.This method perturbs feature statistics adversarially rather than interpolating features.Source: §2 · p. 3

3 Adversarial Batch Normalization

AdvBN perturbs deep feature distributions through PGD on batch statistics, producing adversarial features for training against domain shifts. The paper visualizes these shifts as changes in texture and color and finds that perturbation strength affects reconstructed appearance.

3 Adversarial Batch Normalization

  • AdvBN iteratively takes PGD steps on batch statistics to create feature distributions that confuse CNN classifiers, which are then used for training against domain shifts.The module is inserted into a network; the maximization uses cross-entropy loss and projected gradient descent.Source: §3 · p. 3
  • The feature re-normalization scales adversarial mean shifts by the clean feature mean, allowing larger variation when that mean has greater magnitude.This avoids making bounds and step sizes depend on the clean feature mean.Source: §3 · p. 3
  • The visualization pipeline inserts AdvBN after a VGG-19 encoder and decodes the perturbed features into images.The encoder-decoder autoencoder is trained to reconstruct its input; downstream VGG-19 classification layers provide the cross-entropy objective.Source: §3 · p. 3
  • AdvBN visualizations show inconsistent textures, including smoothed dog fur and disappearing zebra stripes, alongside suppressed colors and unnatural hues.The images also exhibit differences in color, texture, and edges.Source: §3 · pp. 3–4
  • As adversarial feature-statistic perturbations grow, reconstructed-image appearance shifts; m denotes the PGD-step count, with m = 0 representing autoencoder reconstruction without AdvBN.Source: Figure 3 · p. 4, §3 · p. 4
  • Performance on ImageNet-AdvBN is consistently degraded across evaluated methods, supporting the adversarial character of the generated features.ImageNet-AdvBN is produced by applying the visualization technique to the ImageNet validation set.Source: §3 · p. 4

4 Training with Adversarial Batch Normalization

AdvBN fine-tunes a pretrained network on clean and adversarial features, using adversarial feature perturbations to improve generalization to unseen domains without auxiliary domain data. Training modifies only later layers and requires 20 epochs.

4 Training with Adversarial Batch Normalization

  • AdvBN training fine-tunes a pretrained network on clean and adversarial features to generalize to unseen domains without auxiliary domain data.The objective is to maintain performance on the training distribution while improving generalization.Source: §4 · p. 4
  • Clean features use the original batch-normalization layers, while adversarial features use auxiliary batch-normalization layers.This separation is adopted to maintain performance on natural images.Source: §4 · p. 4
  • 20 epochs suffice for fine-tuning, and updating only later layers avoids backpropagation through the network’s first half.Starting from pretrained models yields improved robustness with little additional computation.Source: §4 · p. 5
  • The algorithm perturbs feature-map means and variances by iterative signed-loss updates, clips the perturbation, then updates the later subnetwork’s parameters.It initializes fadv from the clean feature map and repeats adversarial steps before the network-parameter update.Source: §4 · p. 5

5 Experiments

Experiments evaluate AdvBN on ImageNet variants, test design choices and feature divergence, and assess cross-domain semantic segmentation. Across these settings, the results examine generalization to distribution shifts and the effect of adversarial strength.

5.1 Evaluation on ImageNet Variants

  • On four ImageNet variants, standalone AdvBN improves a standard model on every dataset, competes with alternatives, and consistently boosts state-of-the-art augmentations.Source: Table 1 · p. 6, §5.1 · p. 7

5.2 Ablation Study

  • Among three tested placements, conv4_6 gives the worst performance across all three ImageNet variants; the authors hypothesize deeper placement may harm classification.Source: Table 2 · p. 7, §5.2 · p. 7
  • As PGD steps m increase, clean ImageNet accuracy falls, while other datasets show a turning point; both very small and overly large perturbations are less useful.Source: Figure 4 · p. 7, §5.2 · p. 7

5.3 Feature Divergence Analysis

  • AdvBN substantially reduces feature divergence in deeper fine-tuned layers across three dataset pairs, making shifted-domain and ImageNet feature distributions more similar.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4 Generalization on Semantic Segmentation

  • Across 20 source-target pairs, AdvBN achieves the best cross-domain segmentation, improves in-domain performance over standard training, while AdvProp leads in-domain.Source: Table 3 · p. 9, §5.4 · p. 9

6 Conclusion and Discussion

Fine-tuning with adversarially shifted feature distributions improves robustness to domain shifts without auxiliary data, and can complement image-space augmentation. The authors caution that improvements may trade off against in-distribution performance and may not transfer from benchmarks to practitioners’ settings.

6 Conclusion and Discussion

  • Fine-tuning on adversarially shifted feature distributions improves robustness to various domain shifts without auxiliary data.Because AdvBN operates in feature space, it can complement image-space augmentation and further improve generalization.Source: §6 · p. 9
  • Adapting feature-space perturbations to tasks beyond vision is identified as future work.The authors note that language and graph models may benefit from more structured feature-space perturbations.Source: §6 · p. 9
  • AdvBN may trade off performance on non-shifted in-distribution data, and benchmark gains may not carry over to practitioners’ settings.The authors specifically caution against assuming that performance on benchmarks such as ImageNet variants will translate to real-world datasets and shifts.Source: §6 · p. 9

A.1 Detailed results on ImageNet-C

On ImageNet-C, AdvBN alone improves the baseline model across all corruption types. The appendix details the benchmark’s severity levels and reports category mCE and subset error comparisons.

A.1 Detailed results on ImageNet-C

  • ImageNet-C comprises 75 variants across 15 corruption categories, each tested at five gradually increasing severity degrees.Degree 1 is the lowest severity.Source: §A.1 · p. 13
  • AdvBN alone improves the baseline model on all ImageNet-C corruption types.Table 4 lists category mCE; the passage does not provide the table’s individual values.Source: §A.1 · p. 13, Table 4 · p. 13
  • Table 5 reports raw error rates for ImageNet-C subsets, comparing AugMix with AugMix fine-tuned using AdvBN.The passage identifies the comparison but does not state its outcome or values.Source: §A.1 · p. 13, Table 5 · p. 13

A.2 Other architectures

The appendix describes applying AdvBN to DenseNet-121 and EfficientNet, then explains the use of separate batch-normalization statistics for clean and adversarial features and the inference-time choice between them.

A.2 Other architectures

  • For DenseNet-121 and EfficientNet, AdvBN is inserted at different network blocks and fine-tuned for 20 epochs with architecture-specific PGD settings.DenseNet-121 uses six PGD steps with τ = 0.2 and ϵ = 1.1; EfficientNet uses three steps with τ = 0.2 and ϵ = 0.5.Source: §A.2 · p. 13, Table 6 · p. 13

A.3 AdvBN at inference time

  • After fine-tuning, models with BatchNorm layers maintain separate statistics for clean and adversarial features.The auxiliary BN stores the mean and standard deviation of adversarial features, which differ substantially from the original-data statistics.Source: §A.3 · p. 14
  • At inference, the authors choose main or auxiliary BN statistics according to a heuristic based on accuracy closeness to original ImageNet.They use main BN for original ImageNet, ImageNet-Instagram, and ImageNet-C, and auxiliary BN for ImageNet-Sketch and ImageNet-Style; they identify unsupervised distance measurement as future work.Source: §A.3 · p. 14, Table 7 · p. 14

B.1 Creation of the ImageNet-AdvBN dataset

The appendix constructs two ImageNet-AdvBN datasets using VGG and ResNet autoencoders, evaluates classifiers on them against reconstructed-image controls, and provides additional VGG examples.

B.1 Creation of the ImageNet-AdvBN dataset

  • The ImageNet-AdvBN-VGG and ImageNet-AdvBN-ResNet datasets each contain 50,000 images generated from ImageNet validation data using autoencoders with AdvBN.Generation uses six AdvBN steps, stepsize 0.20, ϵ = 1.1, and batch size 32, without shuffling the validation data.Source: §B.1 · p. 14

B.2 Classification on ImageNet-AdvBN

  • Classifiers’ accuracy is significantly degraded on ImageNet-AdvBN, and the gap from reconstructed-image performance indicates that reconstruction loss alone does not account for the degradation.Table 8 compares the two AdvBN datasets with images reconstructed using their corresponding autoencoders without feature perturbation.Source: §B.2 · p. 14, Table 8 · p. 15

B.3 Additional Example Images

  • Figure 7 shows randomly selected ImageNet-AdvBN-VGG examples; the appendix omits ResNet examples because they are mostly extreme-contrast images with hard-to-observe small textures.The authors suggest either greater sensitivity of ResNet encoder features to AdvBN perturbations or the relative shallowness of the extracted ResNet-50 features as possible explanations.Source: §B.3 · p. 14, Figure 7 · p. 15

C Details concerning the training budget

This section reports the compute cost of AdvBN fine-tuning and identifies PGD-step count and layer placement as factors affecting training speed.

C Details concerning the training budget

Loading 2009.08965v3…