PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?

Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, Prateek Mittal

arXiv:2104.09425v3cs.LGcs.CRcs.CV

TL;DR

The paper addresses the cost of collecting real-world data for adversarially robust training and asks whether proxy distributions can provide useful additional samples. It formalizes robustness transfer with conditional Wasserstein distance, uses robust discrimination to assess proxies, and reports improved robustness across five datasets, with diffusion-based proxies especially effective.

  • Problem

    Collecting enough real-world training data to improve adversarial robustness is challenging and costly, while robustness transfer from synthetic data to real data remains uncertain.

  • Method

    The paper bounds robustness transfer using conditional Wasserstein distance and develops robust discrimination and synthetic scores to evaluate proxy distributions and samples.

  • Results

    PORT improves robustness across five datasets, with up to 7.5% and 6.7% gains in ℓ∞ and ℓ2 threat models and 7.6% higher certified robust accuracy on CIFAR-10.

  • Takeaways & Limitations

    Diffusion-based generative models are more helpful proxy distributions than GANs, and robust discrimination better characterizes robustness transfer than standard discrimination.

Abstract

from arXiv · show

While additional training data improves the robustness of deep neural networks against adversarial examples, it presents the challenge of curating a large number of specific real-world samples. We circumvent this challenge by using additional data from proxy distributions learned by advanced generative models. We first seek to formally understand the transfer of robustness from classifiers trained on proxy distributions to the real data distribution. We prove that the difference between the robustness of a classifier on the two distributions is upper bounded by the conditional Wasserstein distance between them. Next we use proxy distributions to significantly improve the performance of adversarial training on five different datasets. For example, we improve robust accuracy by up to 7.5% and 6.7% in $\ell_{\infty}$ and $\ell_2$ threat model over baselines that are not using proxy distributions on the CIFAR-10 dataset. We also improve certified robust accuracy by 7.6% on the CIFAR-10 dataset. We further demonstrate that different generative models bring a disparate improvement in the performance in robust training. We propose a robust discrimination approach to characterize the impact of individual generative models and further provide a deeper understanding of why current state-of-the-art in diffusion-based generative models are a better choice for proxy distribution than generative adversarial networks.

1 INTRODUCTION

The paper asks whether proxy distributions can provide cheaper additional training data for adversarial robustness and whether robustness transfers from synthetic to real data. It analyzes this transfer using distributional proximity.

  • Additional training data can improve adversarial robustness, but collecting more real-world samples is challenging and costly.
  • The paper investigates whether proxy distributions can supply cheaper additional samples while improving robust training.
  • A central question is whether robustness learned from synthetic samples transfers to real-world data.
  • The paper further examines which synthetic-data features and individual samples support synthetic-to-real robustness transfer.
  • It argues that the distribution-shift penalty is bounded by conditional Wasserstein distance between proxy and real distributions.

Q.2 How effective are proxy distributions in boosting adversarial robustness on real-world dataset?

PORT uses proxy samples to improve robust training across datasets and threat models, while robust discrimination and synthetic scores identify useful proxy distributions and samples.

  • PORT significantly improves robustness across five datasets, threat models, architectures, and empirical or certified robustness criteria.
  • 7.5% improvement in adversarial robustness is achieved over existing state-of-the-art results.
  • Robust discrimination measures proxy proximity by testing how well adversarially perturbed synthetic and real samples can be distinguished.
  • The decrease in discriminator success as perturbation size increases accurately predicts robustness transfer across generative models.
  • Synthetic scores rank individual synthetic samples by proximity to the real distribution, and lower-score selection improves performance over random selection.

2 INTEGRATING PROXY DISTRIBUTIONS IN ROBUST TRAINING

The paper formalizes robustness transfer from proxy to real distributions and develops robust discrimination as an empirical proxy for distributional proximity. These ideas motivate PORT, which combines synthetic and real samples for robust training.

  • PORT combines synthetic samples generated by a generative model with real samples in robust training.
  • Average robustness is defined as the expected distance from each sample to its closest adversarial example.
  • Transferred robustness is decomposed into distribution-shift, generalization, and empirical-training terms.
  • Theorem 1 bounds the distribution-shift penalty using conditional Wasserstein distance between labeled proxy and real distributions with identical label distributions.
  • Robust discrimination evaluates whether adversarial perturbations can make proxy samples resemble real samples, linking discriminator success to distributional separation.
  • ARC uses the decline in robust discriminator accuracy as perturbation size grows, whereas non-robust discrimination can reach near 100% accuracy and fails to measure proximity.

3 EXPERIMENTAL RESULTS

The experiments evaluate proxy-distribution training across five datasets and two threat models using clean, robust, and certified robust accuracy. They report improvements for adversarial training and randomized smoothing.

  • Proxy distributions improve adversarial training and randomized smoothing across both ℓ∞ and ℓ2 threat models.
  • The experiments cover five datasets, ℓ∞ and ℓ2 threat models, and AutoAttack-based robustness evaluation.
  • Clean accuracy and robust accuracy measure performance on unmodified test images and adversarial examples, respectively.

Synthetic data.

PORT uses synthetic proxy-distribution data, especially from diffusion models, to improve empirical and certified adversarial robustness while also improving clean accuracy. Across datasets and threat models, the approach outperforms baselines and ARC identifies which generative models best transfer robustness.

  • Adversarial robustness: Up to 7.5% and 6.7% higher robust accuracy is achieved over previous works under ℓ∞ and ℓ2 threat models, respectively.The largest gains occur on CIFAR-10 and Celeb-A.
  • Adversarial robustness: Synthetic data produces robust-accuracy gains comparable to increasing network size by an order of magnitude, including a ResNet-18 outperforming a WRN-34-20 with 16× fewer parameters.The trend holds under both ℓ∞ and ℓ2 threat models.
  • Adversarial robustness: Up to 7.5% and 7.8% higher clean accuracy is observed under ℓ∞ and ℓ2 threat models, respectively, alongside robust-accuracy improvements.This simultaneous improvement contrasts with the usual clean-accuracy versus robustness trade-off in adversarial training.
  • Proxy selection: Diffusion-based generative models outperform GANs as proxy distributions on each dataset, while ARC accurately predicts generative-model rankings for robustness transfer.ARC measures distinguishability between adversarially perturbed synthetic and real images; adaptive sampling based on ARC adds 0.5% and 0.2% average robust accuracy over random sampling.
  • Certified robustness: PORT achieves 7.6% higher certified robust accuracy than randomized smoothing at an ℓ∞ perturbation of 2/255, equivalent to an ℓ2 perturbation of 111/255.Its certified robustness is higher across ℓ2 perturbation budgets for both ResNet-18 and WRN-28-10.
  • Certified robustness: Using 10M synthetic samples raises certified robust accuracy from 63.8% for RST500K to 66.2%, despite RST500K using 500K curated real-world images.The comparison concerns CIFAR-10 certified robustness.

A.2 PROOF OF THEOREM 2

The proof establishes a tight relationship between conditional Wasserstein distance and robust distinguishing accuracy, with equality for concentric uniform ℓp spheres.

  • cwd(D, ˜D) ≥ 4ARC(D, ˜D) for distributions with equal class probabilities.Equality holds when each pair of class-conditional distributions consists of concentric uniform ℓp spheres.
  • The proof begins with a transport-based lemma that bounds discriminator distinguishability after perturbations.The construction uses a transport J and maps points toward transport midpoints when their distance is at most ϵ.
  • The argument combines the transport and discriminator bounds to complete the inequality proof and then establishes equality for the concentric-sphere case.The appendix notes that the theorem statement in the main body contains a typographical error corrected here.
  • For optimal transport between concentric spheres, every transported pair has distance ˜r − r.Consequently, the transport distance remains below ϵ whenever ϵ < ˜r − r.

A.3 EFFECT OF COMBINING PROXY AND REAL DATA ON THEOREM 1

The mixture result analyzes combining real and proxy data through a weighted distribution and shows that the distribution-shift penalty is reduced relative to using proxy data alone.

  • Theorem 6 considers a weighted mixture ¯D = p ⋅D + (1 −p) ⋅˜D of labeled real and proxy distributions with identical label distributions.The theorem applies to any classifier h ∶X →Y.
  • Because p is usually very small when proxy data greatly outnumbers original data, including original data should not substantially change the obtained shift-penalty bound.
  • The mixture satisfies cwdd(D, ¯D) ≤ (1 −p) ⋅cwdd(D, ˜D), bounding its shift penalty by the proxy-only penalty scaled by the proxy weight.The proof constructs a transport between D and ¯D using the optimal class-conditional transport between D and ˜D.

A.4 TIGHTNESS OF THEOREM 1

The authors show that the conditional-Wasserstein bound on the distribution-shift penalty is tight for a specific classifier, while leaving potentially better learning-algorithm bounds open.

  • Theorem 7 establishes that no universally better bound on the distribution-shift penalty exists for a specific classifier.It constructs a labeled distribution achieving equality with the bound from Theorem 1.
  • The proof interpolates between the original distribution and adversarially shifted distributions using a parameter α.The construction scales the conditional Wasserstein distance and the robustness difference by α.
  • Robd(h, D) − Robd(h, ˜Dα) = cwd(D, ˜Dα) = ϵ, demonstrating equality in the bound.
  • The tightness result applies only to a specific classifier and does not rule out better expected bounds for a learning algorithm.

A.5 EXPERIMENTAL VALIDATION OF MAIN THEOREM

The experiments validate the theorem’s predicted relationship between robustness transfer and conditional Wasserstein distance using class-conditional multivariate normal distributions.

  • The experiment uses a 10-class problem in which each class follows a 128-dimensional multivariate normal distribution.This setting permits exact calculation of Wasserstein distance.
  • Figure 4 compares average robustness differences with Wasserstein-2 distances across proxy distributions.
  • The difference in average robustness is upper bounded by the conditional Wasserstein distance.Wasserstein-2 is used because it has a closed-form expression for normal distributions, although the theorem’s Wasserstein-1 bound is tighter.

B ADDITIONAL DETAILS ON EXPERIMENTAL SETUP

The experiments combine standard robust-training protocols with multiple generative models, datasets, discriminator architectures, and proxy-selection metrics, while documenting substantial sampling costs.

  • Training and evaluation setup: Training uses ResNet-family architectures, stochastic gradient descent, cosine learning-rate decay, weight decay, batch size 128, and 200 epochs.The CIFAR-100 experiments use 1 × 10^-3 weight decay; other experiments use 5 × 10^-4.
  • Training and evaluation setup: Robust training evaluates both ℓ∞ and ℓ2 threat models using PGD-10 training and AutoAttack benchmarking.For CIFAR-10 and CIFAR-100, the perturbation budgets are 8/255 for ℓ∞ and 127/255 for ℓ2.
  • Generative models and data: The study compares seven generative models on CIFAR-10 and two networks on CelebA and ImageNet, using labeled or classifier-labeled synthetic images.
  • Computational cost: Sampling one million CIFAR-10 images from DDPM takes 23.8 hours on an RTX 4x2080Ti GPU cluster.The paper notes that generative-model training and sampling are one-time costs.
  • Robust discrimination and ARC: The proposed ARC metric ranks generative models by robustness transfer from their samples to real data.Its experiments compare ARC with baseline metrics such as FID and IS.
  • Robust discrimination and ARC: Non-robust discriminators are ineffective proximity measures because even shallow CNNs distinguish synthetic from real images with over 90% accuracy.CNN-4 exceeds 99% accuracy for four of seven generative models.
  • Robust discrimination and ARC: FID measured in CIFAR-10 feature space also fails to predict robustness transfer and ranks DiffBigGAN and StyleGAN relatively highly.

C.3 ADAPTIVE SAMPLING OF SYNTHETIC DATA.

Adaptive sampling selects synthetic images with the lowest robust-discriminator synthetic scores, improving robustness transfer beyond random sampling across threat models and architectures.

  • Synthetic score measures proximity to the original distribution using robust discriminators trained with 0.25/255 ℓ∞ perturbations.
  • Groups with the lowest synthetic scores achieve the highest robust test accuracy after adversarial training.The images are divided into ten equal-size groups after sorting 6M DDPM samples by score.
  • Adaptive sampling selects 10M lowest-score images from 15M candidates and outperforms random sampling across ℓ∞ and ℓ2 threat models.The improvement holds across network architectures.

D BRINGING IT ALL TOGETHER: USING SYNTHETIC SAMPLES IN ROBUST

The section analyzes synthetic proxy images and calibrates PORT before robust training. DDPM images are labeled for CIFAR-10, while γ is selected using clean and robust accuracy.

  • Analyzing synthetic images: PORT primarily uses synthetic images sampled from DDPM and StyleGAN models.These models form the main proxy-distribution sources analyzed in the section.
  • Analyzing synthetic images: DDPM synthetic images are unlabeled, so a LaNet classifier labels them for CIFAR-10; improved DDPM sampling provides labeled ImageNet images.The DDPM-generated CIFAR-10 images are nearly uniformly distributed across classes.
  • Hyperparameter selection: γ = 0.4 achieves the highest clean and robust accuracy in the PORT hyperparameter search and is used in the experiments.The search evaluates ten γ values from 0 to 1 with a ResNet-18 network.

D.1 SAMPLE COMPLEXITY OF ADVERSARIAL TRAINING

The experiments examine how synthetic-sample quantity affects adversarial training and its transfer to real data. Increasing the training set improves clean and robust performance, reduces the accuracy–robustness trade-off, and yields generalization to CIFAR-10.

  • Setup: Adversarial training on synthetic data is evaluated from 2K to 10M StyleGAN images using ResNet-18 on CIFAR-10 and CIFAR-2.StyleGAN is used because its sampling is faster, enabling generation of up to 10M synthetic images.
  • Accuracy–robustness trade-off: Increasing synthetic training samples reduces the clean-accuracy gap between natural and adversarial training on both CIFAR-2 and CIFAR-10.For CIFAR-2, the trade-off almost vanishes with sufficiently many samples.
  • Sample complexity: Both clean and robust accuracy continue to improve as the number of synthetic training samples increases.This pattern is measured on held-out synthetic test images.
  • Sample complexity: Generalization to CIFAR-10 test images also improves with the number of synthetic training samples.The experiments measure transfer from models trained only on synthetic images to the real CIFAR-10 test set.
  • Synthetic-labeling analysis: Classifier choice for labeling DDPM images affects downstream adversarial-training performance despite high CIFAR-10 clean accuracy.BiT, SplitNet, and LaNet achieve 98.5%, 98.7%, and 99.0% clean accuracy, respectively.
Loading 2104.09425v3…