PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Diffusion Models Further Improve Adversarial Training

Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, Shuicheng Yan

arXiv:2302.04638v2cs.CVcs.AIcs.CRcs.LG

TL;DR

The paper asks whether improved diffusion models can further enhance adversarial training. Replacing DDPM-generated data with EDM-generated data yields state-of-the-art robust accuracy using generated data alone, while remaining constrained by the need for millions of generated images.

  • Problem

    Whether better diffusion models can improve adversarial training beyond the gains already obtained from DDPM-generated data.

  • Method

    The authors replace DDPM-generated data with class-conditional EDM-generated data and use an almost unchanged adversarial-training pipeline without external datasets.

  • Results

    The models achieve state-of-the-art robust accuracy across CIFAR-10, CIFAR-100, SVHN, and TinyImageNet comparisons, including 70.69% on CIFAR-10 and 42.67% on CIFAR-100 under ℓ∞ threat with ε=8/255.

  • Takeaways & Limitations

    Higher-quality diffusion-generated data can substantially improve adversarial training and can outperform prior models that use external data.

  • Takeaways & Limitations

    Adversarial training remains inefficient because it requires millions of generated data points even on small datasets such as CIFAR-10.

Abstract

from arXiv · show

It has been recognized that the data generated by the denoising diffusion probabilistic model (DDPM) improves adversarial training. After two years of rapid development in diffusion models, a question naturally arises: can better diffusion models further improve adversarial training? This paper gives an affirmative answer by employing the most recent diffusion model which has higher efficiency ($\sim 20$ sampling steps) and image quality (lower FID score) compared with DDPM. Our adversarially trained models achieve state-of-the-art performance on RobustBench using only generated data (no external datasets). Under the $\ell_\infty$-norm threat model with $ε=8/255$, our models achieve $70.69\%$ and $42.67\%$ robust accuracy on CIFAR-10 and CIFAR-100, respectively, i.e. improving upon previous state-of-the-art models by $+4.58\%$ and $+8.03\%$. Under the $\ell_2$-norm threat model with $ε=128/255$, our models achieve $84.86\%$ on CIFAR-10 ($+4.44\%$). These results also beat previous works that use external data. We also provide compelling results on the SVHN and TinyImageNet datasets. Our code is available at https://github.com/wzekai99/DM-Improves-AT.

1. Introduction

Diffusion-generated data has become an important way to improve adversarial training without relying on external datasets. This paper asks whether newer, more efficient and higher-quality diffusion models can improve that approach further, and reports strong gains from EDM.

  • Motivation: Generated DDPM data significantly enhances clean and robust accuracy in adversarially trained models, making it common among RobustBench’s top models without external datasets.External datasets can improve adversarial training but are not always available.
  • Motivation: EDM achieves a lower unconditional FID of 1.97 than DDPM’s 3.17, motivating tests of whether better diffusion quality benefits adversarial training.The paper notes that DDPM and EDM images are visually indistinguishable despite the FID difference.
  • Contributions: Replacing DDPM data with EDM data, without external datasets or additional training time per epoch, yields new state-of-the-art robust accuracy on CIFAR-10 and CIFAR-100.The models also surpass previous Rank #1 models that use external data and improve clean accuracy.
  • Analysis: Ablations indicate that EDM-generated data eliminates robust overfitting and reduces the generalization gap between clean and robust accuracy.The studies examine generated-data quantity and quality, early stopping, data augmentation, and parameter sensitivity.

2. Related Work

Related work covers adversarial training enhancements, diffusion-based purification, and adversarial benchmarks. It emphasizes the computational costs and evaluation considerations surrounding diffusion-based defenses.

  • Diffusion models: Diffusion models unify denoising diffusion probabilistic modeling and score-based Langevin dynamics through stochastic differential equations.The related-work discussion presents this unified family as a foundation for later diffusion-model developments.
  • Adversarial training: Adversarial training research has incorporated external or generated data alongside metric learning, self-supervised learning, ensemble learning, fairness, and generative modeling.The paper positions its approach within these broader adversarial-training enhancements.
  • Adversarial purification: Diffusion-based adversarial purification has shown promising AutoAttack robustness, but stochastic solvers require at least tens of forward processes during inference.The discussion also notes potential vulnerability of stochastic preprocessing and test-time defenses to transfer-based and intermediate-state attacks.
  • Adversarial benchmarks: RobustBench provides a widely used benchmark built on AutoAttack for evaluating deterministic defenses such as adversarially trained models.The paper uses RobustBench to compare methods across adversarial-defense settings.

3. Experiment Setup

The experiments use WideResNet backbones, class-conditional EDM-generated data, and a TRADES-based adversarial-training pipeline. Evaluation uses AutoAttack and controls training iterations to compare generated-data settings fairly.

  • Model architectures: The study uses WRN-28-10 and WRN-70-16 backbones with Swish/SiLU activations, matching common RobustBench architectures.Both architectures are used as backbone networks.
  • Generated data: Class-conditional EDM generates CIFAR-10 and CIFAR-100 data without external datasets, selecting 1M images from 5M by pretrained-model scores.The top 20% scoring images are selected for each class; SVHN and TinyImageNet use all generated images.
  • Generated data: Generated-image pseudo-labels are determined directly by class conditioning rather than inferred from an unconditional generator.This differs from earlier approaches using unconditional DDPM data.
  • Training settings: Adversarial training uses TRADES with dataset-specific β values, weight averaging, Nesterov SGD, weight decay, and cyclic cosine-annealing learning rates.The stated β values are 5 for CIFAR-10/CIFAR-100, 6 for SVHN, and 8 for TinyImageNet.
  • Training settings: Training iterations per epoch are fixed across generated-data quantities, keeping training time constant when epochs and batch size are fixed.Each batch samples original and generated data using a fixed original-to-generated ratio.
  • Evaluation: Robustness is evaluated with AutoAttack; repeated CIFAR-10 training runs show clean accuracy of 91.12 ± 0.15% and robust accuracy of 63.35 ± 0.12%.The paper reports these repetitions to indicate low variability despite not reporting standard deviations for every experiment.

4. Comparison with State-of-the-Art

The paper compares EDM-based adversarially trained models with RobustBench methods across CIFAR-10, CIFAR-100, SVHN, and TinyImageNet. Using generated data, the models improve clean and robust accuracy and outperform prior entries under the reported threat models.

  • Evaluation scope: The comparison covers clean accuracy and AutoAttack robust accuracy across CIFAR-10, CIFAR-100, SVHN, and TinyImageNet.CIFAR-10 includes both ℓ∞ and ℓ2 threat models; the other reported comparisons use ℓ∞ with ε = 8/255.
  • Overall comparison: EDM-generated data improves both clean and robust accuracy over previous DDPM-based top-rank models and surpasses all RobustBench entries, including methods using external datasets.The method uses no external datasets, and its per-epoch training time matches the corresponding baseline when batch size is fixed.
  • CIFAR-10: 84.86% robust accuracy on CIFAR-10 under ℓ2, ε = 128/255 improves the previous SOTA result by +4.44% with WRN-70-16.Its clean accuracy is 95.54%, an improvement of +3.13%.
  • CIFAR-100: 42.67% robust accuracy on CIFAR-100 under ℓ∞, ε = 8/255 improves the previous SOTA result by +8.03% with WRN-70-16.The corresponding clean accuracy is 75.22%, improving the prior result by +11.66%.
  • SVHN and TinyImageNet: The approach also improves robustness on SVHN and TinyImageNet, reaching 64.01% robust accuracy on SVHN and 31.30% on TinyImageNet.The SVHN result improves by +2.92%, while the TinyImageNet result improves by +4.64% using class-conditional EDM-generated images.

5. How Generated Data Influence Robustness

Generated data changes adversarial-training dynamics: it narrows clean and robust generalization gaps, reduces robust overfitting, and makes longer training more useful. Larger and higher-quality generated datasets improve robustness, although gains can saturate and augmentation effects vary.

  • Early Stopping and Number of Epochs: Without generated data, robust overfitting causes test robust accuracy to degrade while training accuracy continues increasing.Early stopping is effective in this setting because longer training can worsen test performance.
  • Early Stopping and Number of Epochs: Generated data makes early stopping less important, with the best model often appearing at the end of training and adequate data mitigating robust overfitting.With 20M generated images, short training can instead cause robust underfitting, so longer training improves robust performance.
  • Amount of Generated Data: Nearly 60% robust-accuracy generalization gaps without generated data are reduced as generated data is added, while test accuracy improves.The authors attribute this pattern to generated images containing examples that are difficult to classify robustly.
  • Amount of Generated Data: After 500K generated images, additional data provides no significant improvement, suggesting model capacity limits the benefit of larger generated datasets.The authors use a larger WRN-70-16 model for state-of-the-art results and note that longer training can aid convergence with sufficient data.
  • Data Augmentation: With 1M EDM-generated images, Cutout and IDBH slightly improve robust accuracy over other augmentation choices, while common augmentation remains the default.The broader study finds rule-based and policy-based augmentations less effective for robustness when generated data is used.
  • Quality of Generated Data: Lower FID generated data yields higher clean and robust accuracy, and class-conditional generation outperforms unconditional generation.On CIFAR-10, the selected setup uses class-conditional EDM with 20 sampling steps; CIFAR-100 and SVHN use models selected after 25 steps.

6. Sensitivity Analysis

The analysis shows that large batches and moderate label smoothing improve adversarial-training performance with generated data, while excessive smoothing and large TRADES β can reduce robustness.

  • Batch size: 2048 yields the best results with 5M generated images, indicating that large batches enhance robust performance when data is sufficient.The authors use batch size 2048 for WRN-28-10 and 1024 for WRN-70-16 in their best-results settings.
  • Label smoothing: 0.1 label smoothing improves clean accuracy by 0.72% and AutoAttack accuracy by 0.52%.Label smoothing of 0.2 slightly improves clean accuracy but reduces robustness.
  • Label smoothing: Label smoothing values of 0.3 and 0.4 degrade performance, consistent with over-smoothing labels and losing information in output logits.The authors therefore set label smoothing to 0.1 throughout their experiments.
  • Effect of β: β = 5 achieves the best robustness with 1M EDM-generated data, whereas larger β values degrade robustness.The authors use β = 5 for CIFAR-10 and CIFAR-100, β = 6 for SVHN, and β = 8 for TinyImageNet.

7. Discussion

The discussion frames the work as diffusion-enhanced TRADES adversarial training, using PGD-generated adversarial examples and generated data across several image datasets. It also identifies learning efficiency as an important unresolved boundary.

  • Discussion: Learning efficiency remains unresolved because adversarial training requires millions of generated images and purification requires tens of diffusion forward processes at inference.The authors identify more efficient exploitation of diffusion models as future work.
  • Adversarial training: TRADES combines clean and robust objectives with β controlling their trade-off, and serves as the adversarial-training framework.The paper investigates β sensitivity when generated data is used.
  • PGD attack: PGD iteratively maximizes loss through gradient-ascent steps while projecting each update back into the allowed perturbation ball.The paper denotes the resulting K-step procedure as PGD-K.
  • PGD attack: The training attack uses PGD-10 with ε = 8/255 for ℓ∞ experiments and ε = 128/255 for CIFAR-10 ℓ2 experiments.Step sizes vary by threat model and dataset.
  • Generated data: Generated samples receive pseudo-labels through a pretrained WRN-28-10 for unconditional generation or directly through class conditioning.Class-conditional generation maintains balanced class counts.
  • Experimental setup: Experiments use CIFAR-10, CIFAR-100, SVHN, and TinyImageNet with WideResNet models, including WRN-28-10 and WRN-70-16.The datasets range from 10 to 200 classes and include 32×32×3 or 64×64×3 images.

B.1. Original-to-Generated Ratio

Mixing original and EDM-generated images affects performance: a 0.3 original-to-generated ratio is best with 1M generated images, while 0.2 is better when using more than 1M.

  • Original-to-Generated Ratio: A 0.3 original-to-generated ratio means three original images for every seven generated images in a training batch.The ratio measures the share of original images relative to generated images.
  • Original-to-Generated Ratio: Both clean and robust accuracy are best at a 0.3 ratio when training on 1M EDM-generated images.The experiment trains WRN-28-10 models under the ℓ∞ threat model with ε = 8/255.
  • Original-to-Generated Ratio: With 1M EDM-generated data, performance exceeds that obtained with 50K original CIFAR-10 training images.The authors conclude that generated images improve robustness when the generator produces high-quality data.

B.2. FID for CIFAR-100 and SVHN datasets

The appendix evaluates EDM sample quality for CIFAR-100 and SVHN through FID under different sampling-step counts, with 25 steps identified as optimal.

  • Related ratio comparison: Table 10 reports CIFAR-10 test accuracy under the ℓ∞ threat model with ε = 8/255 for different original-to-generated ratios.The models use 5M EDM-generated images, and the 0.2 results consistently outperform those at 0.3.
  • FID evaluation: FID is measured between 50K EDM-generated images and the CIFAR-100 or SVHN training set across diffusion sampling-step settings.The comparison is summarized in Table 11.
  • FID evaluation: 25 sampling steps achieve the best FID for the CIFAR-100 and SVHN EDM models.The paper trains its own EDM models solely on each dataset’s training set.

B.3. Ablation Studies on the Specifics of Diffusion Model Implementation

The ablations compare diffusion samplers, formulations, conditioning, and generation costs. EDM improves FID and adversarial-training accuracy under matched generation time, while unconditional generation is faster but less robust.

  • Samplers: EDM’s Heun second-order solver outperforms DDIM’s Euler solver in FID and promotes both clean and robust adversarial-training accuracy at the same generation time.DDIM uses double sampling steps for a fair runtime comparison because it requires half the time per step.
  • Samplers: Improved EDM hyperparameter selection further increases the robust performance of adversarially trained models.
  • EDM formulations: VE and VP EDM formulations achieve comparable FID, while the main paper uses the VP formulation throughout its experiments.VP uses DDPM++, whereas VE uses NCSN++ architectures.
  • Generation setting: Unconditional EDM generation is faster than class-conditional generation with a small margin, but it produces lower robust performance.The runtime comparison covers generating 5M images with different sampling steps on four A100 GPUs.
  • Training cost: Adversarial training uses four A100 GPUs for WRN-28-10 and eight for WRN-70-16, with average epoch times of 3.45 and 9.93 minutes, respectively.The reported batch sizes are 2048 for WRN-28-10 and 1024 for WRN-70-16.

B.5. Amount of Generated Data

Increasing the amount of generated data improves clean and PGD robust accuracy while substantially reducing robust overfitting. Gains plateau after 500K generated images because model capacity limits the benefit of additional data.

  • Amount of generated data: Increasing generated-data size significantly alleviates robust overfitting.
  • Amount of generated data: After 500K generated images, additional data no longer closes the clean- and robust-accuracy generalization gaps.The passage attributes this saturation to insufficient model capacity to exploit all generated data.
  • Amount of generated data: Clean and PGD robust accuracy are evaluated across different amounts of generated data.
Loading 2302.04638v2…