PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency Domain Model Augmentation for Adversarial Attack

Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, Jingkuan Song

arXiv:2207.05382v1cs.CV

TL;DR

Black-box adversarial attacks are limited by the gap between substitute and victim models, and spatial model augmentation may not create sufficiently diverse substitutes. The paper introduces frequency-domain spectrum simulation using spectrum transformations and reports improved transferability on ImageNet, including stronger performance against normally trained and defense models.

  • Problem

    Black-box attacks face low transferability because substitute and victim models can differ substantially, while spatial transformations may overlook essential model differences and yield insufficient diversity.

  • Method

    The Spectrum Simulation Attack applies DCT- and IDCT-based spectrum transformations to inputs, uses frequency-domain model augmentation, and analyzes diversity through spectrum saliency maps.

  • Results

    6.3%∼12.2% and 5.6%∼23.1% attack-success-rate improvements were reported for normally trained and defense models, respectively, over state-of-the-art transfer-based attacks.

  • Takeaways & Limitations

    Frequency-domain spectrum transformations provide a model-augmentation approach that the paper reports as more effective than traditional spatial-domain augmentation for transferable attacks.

  • Takeaways & Limitations

    The paper’s spectrum-transformation motivation relies on a lemma assuming one transformation matrix is invertible.

Abstract

from arXiv · show

For black-box attacks, the gap between the substitute model and the victim model is usually large, which manifests as a weak attack performance. Motivated by the observation that the transferability of adversarial examples can be improved by attacking diverse models simultaneously, model augmentation methods which simulate different models by using transformed images are proposed. However, existing transformations for spatial domain do not translate to significantly diverse augmented models. To tackle this issue, we propose a novel spectrum simulation attack to craft more transferable adversarial examples against both normally trained and defense models. Specifically, we apply a spectrum transformation to the input and thus perform the model augmentation in the frequency domain. We theoretically prove that the transformation derived from frequency domain leads to a diverse spectrum saliency map, an indicator we proposed to reflect the diversity of substitute models. Notably, our method can be generally combined with existing attacks. Extensive experiments on the ImageNet dataset demonstrate the effectiveness of our method, \textit{e.g.}, attacking nine state-of-the-art defense models with an average success rate of \textbf{95.4\%}. Our code is available in \url{https://github.com/yuyang-long/SSA}.

1 Introduction

Black-box attacks suffer from a large substitute–victim model gap, while spatial transformations provide insufficient model diversity. The paper addresses this with frequency-domain spectrum simulation and reports stronger transferability.

  • Black-box attacks often have low transferability because substitute and victim models differ substantially.
  • The spectrum saliency map is defined from model-loss gradients with respect to an input image’s frequency spectrum and reveals model-specific frequency interests.
  • The proposed Spectrum Simulation Attack uses DCT- and IDCT-based spectrum transformations to diversify inputs and simulate substitute models in the frequency domain.
  • Spatial-domain model augmentation may produce insufficiently diverse substitute models because it can overlook essential differences among models.
  • 6.3%∼12.2% and 5.6%∼23.1% are the reported attack-success-rate improvements for normally trained and defense models, respectively, over state-of-the-art transfer-based attacks.

2 Related Works

Prior black-box attacks improve transferability through optimization or model augmentation, while frequency-domain studies examine how image components affect model sensitivity and robustness. Existing work includes spatial transformations, frequency-based attacks, adversarial training, and input preprocessing.

  • FGSM-based black-box attacks rely on adversarial-example transferability and are a major attack family studied in this paper.
  • Optimization-based methods use momentum, Nesterov acceleration, or patch-wise strategies to improve transferability.
  • Spatial model augmentation applies random, translated, or scaled inputs to reduce over-reliance on a substitute model.
  • Frequency-domain research finds that models respond differently to high- and low-frequency components, including under adversarial training.
  • Defenses include adversarial training, input preprocessing, and combinations of transformations with ensembles or learned purification models.

3 Methodology

The method augments substitute models in the frequency domain using DCT/IDCT-based spectrum transformations, then integrates the resulting gradients into iterative attacks. Its spectrum saliency map exposes model-specific frequency interests, while the transformation theoretically produces diverse maps and empirically improves black-box attack strength.

  • 3.1 Preliminaries: The attack targets the black-box setting by crafting adversarial examples with an accessible substitute model while relying on transferability to fool the victim model.The adversarial example remains within an ℓ∞-norm constraint around the clean image.
  • 3.2 Spectrum Saliency Map: Existing spatial-domain loss-preserving transformations may generate insufficiently diverse augmented models, limiting transferability.The method is motivated by the gap between substitute and victim models and the difficulty of collecting or training many diverse models.
  • 3.2 Spectrum Saliency Map: The spectrum saliency map is the gradient of model loss with respect to an input image’s frequency spectrum, revealing model-specific frequency interests.DCT maps images to the frequency domain, while IDCT recovers the spatial image without loss.
  • 3.3 Spectrum Transformation: Spectrum Transformation applies DCT, random frequency perturbations, element-wise spectrum scaling, and IDCT to diversify input spectra.The noise is sampled from a Gaussian distribution, each scaling-mask element from U(1−ρ, 1+ρ), and DCT is applied to the whole image in the experiments.
  • 3.3 Spectrum Transformation: The authors theoretically show that spectrum transformation yields diverse spectrum saliency maps, thereby simulating diverse substitute models and narrowing the substitute–victim gap.The transformation decomposes matrix multiplication into matrix addition and Hadamard product to handle unavailable victim maps and potentially non-invertible substitute maps.
  • 3.4 Attack Algorithm: The transformation integrates with gradient-based attacks; combined with I-FGSM, it averages N augmented-model gradients before updating and clipping the adversarial example.The resulting S2I-FGSM produces more threatening examples for fooling black-box models than I-FGSM and SI-FGSM in Figure 2.

4 Experiments

Experiments evaluate the proposed attack on ImageNet-compatible images against normally trained and defense models, including single-model and ensemble settings. Ablations examine frequency-domain transformation, random variants, DCT/IDCT block size, and attention shifts.

  • Experiment Setup: Experiments use 1,000 ImageNet-compatible images and evaluate six normally trained models plus nine defense models.The normally trained models include Inc-v3, Inc-v4, IncRes-v2, Res-50, Res-101, and Res-152.
  • Attack Normally Trained Models: 58.9% success rate is achieved against IncRes-v2 when adversarial examples are crafted via Inc-v3, exceeding MI-FGSM, DI-FGSM, and PI-FGSM.The corresponding baseline success rates are 47.2%, 38.2%, and 49.6%, respectively.
  • Attack Normally Trained Models: 88.8% average success rate is achieved by S2I-MI-FGSM, outperforming SI-NI-FGSM, VT-MI-FGSM, and FI-MI-FGSM by 7.3%, 12.2%, and 6.3%.The result supports combining spectrum simulation with existing attacks to enhance transferability.
  • Attack Defense Models: S2I-TI-DIM reaches about 2× TI-DIM transferability on nine defense models, improving the average success rate from 39.0% by 35.7%.The comparison uses adversarial examples generated through a single Inc-v3 substitute model.
  • Attack Defense Models: 95.4% average success rate is obtained by S2I-SI-TI-DIM against nine defenses, outperforming SI-NI-TI-DIM, VT-TI-DIM, FI-TI-DIM, and Admix-TI-DIM.The attack uses an ensemble of Inc-v3, Inc-v4, IncRes-v2, and Res-152.
  • Ablation Study: 15.0% is the margin by which frequency-domain transformation outperforms spatial-domain transformation when attacking Inc-v3.Both random variants, ξ and M, enhance transferability; larger DCT/IDCT blocks perform better, peaking at full-image size, while Grad-CAM shows attention shifts toward mismatched regions.

5 Conclusion

The paper proposes Spectrum Simulation Attack to boost adversarial attacks from a frequency-domain perspective and narrow the substitute–victim model gap using spectrum-transformed images.

  • Spectrum Simulation Attack augments adversarial attacks from a frequency-domain perspective.
  • The method narrows the gap between substitute and victim models using a set of spectrum transformation images.
  • Extensive experiments show that the method outperforms state-of-the-art transfer-based attacks by a large margin.

A Proof

The proposition argues that the spectrum transformation produces diverse spectrum saliency maps, thereby simulating diverse substitute models. The derivation compares the original and transformed saliency maps through gradient relationships and a resulting perturbation term.

  • Proposition 1 states that the proposed spectrum transformation generates diverse spectrum saliency maps and simulates diverse substitute models.
  • The original spectrum saliency map is obtained by applying the chain rule to the gradients through the image transformations.
  • The transformed input yields a spectrum saliency map defined by differentiating the loss at T(x) with respect to the transformed image representation.
  • The derivation expresses the transformed map as (D1+K1)(D2+K2)⊙M, equivalently (Sϕ+K′)⊙M.
  • K′ combines D1K2, D2K1, and K1K2, showing that the transformation can simulate a different spectrum saliency map.

B.1 On the Standard Deviation σ of Noise ξ

The study evaluates how noise standard deviation affects S2I-FGSM transferability on normally trained and defense models. Attack success rises with σ before declining at larger values, motivating σ = 16.

  • Attack success on normally trained models increases gradually with σ and decreases when σ exceeds 16.
  • At σ = 16, defense models achieve relatively high attack success rates, so the paper sets σ = 16.
  • S2I-FGSM attack success rates are evaluated against normally trained and defense models as σ varies, using Inc-v3-generated adversarial examples.

B.3 On the Number N of Spectrum Transformations.

The number of spectrum transformations strongly affects transferability: increasing N improves performance initially, with slower gains beyond 20 transformations. Larger N also increases computational overhead.

  • N = 1 produces the lowest transferability, while increasing N initially significantly enhances adversarial-example transferability.
  • The experiment measures S2I-FGSM attack success on normally trained and defense models as the number N of spectrum transformations changes.
  • Transferability increases slowly after N exceeds 20, indicating diminishing gains from adding further spectrum transformations.
  • Larger N requires more forward and backward propagation for gradient computation, increasing computational overhead.

C Time Analysis of DCT/IDCT

The section quantifies the overhead of applying DCT/IDCT during adversarial-example generation, showing that the transformation is a minority of total runtime in the reported example.

  • 0.58s of S2I-FGSM's 3.78s average generation time is spent on DCT/IDCT for IncRes-v2.This accounts for 15.3% of the total overhead on RTX 3090 GPUs.
  • Table 4 reports average DCT/IDCT and S2I-FGSM generation times for Inc-v3, Inc-v4, IncRes-v2, and Res-152.The table lists DCT/IDCT time and S2I-FGSM time separately for each substitute model.

D.1 Spatial Domain Transformation Analysis

The analysis argues that spatial-domain changes can appear substantial while leaving frequency-based model indicators nearly unchanged. Spectrum transformation instead modifies image colors without changing semantic information.

  • Spatial Domain Transformation Analysis: The spatial saliency map is defined as the gradient of J(x, y; ϕ) with respect to x.The passage states that this definition is analogous to the proposed spectrum saliency map.
  • Spatial Domain Transformation Analysis: Spatial saliency maps change greatly after horizontal flipping, while frequency spectra and frequency saliency maps change little.The frequency saliency map is described as an indicator of model characteristics, so spatial analysis is considered unreliable for reflecting model gaps.
  • Spatial Domain Transformation Analysis: Spectrum transformation modifies image colors without changing semantic information.Visualizations compare raw input images with spectrum-transformed outputs.
  • Spatial Domain Transformation Analysis: Figure 9 compares frequency spectrum, spatial saliency map, and frequency saliency map for raw and spatially transformed images.The figure caption identifies the top row as raw images and the bottom row as spatial-domain transformed images.
Loading 2207.05382v1…