Source-linked AI summary
A Fourier Perspective on Model Robustness in Computer Vision
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer
TL;DR
The paper asks why robustness gains from data augmentation differ across corruption types. It uses Fourier-domain analyses to study Gaussian augmentation and adversarial training, finding a low-frequency bias that improves robustness to high-frequency corruptions but harms robustness to low-frequency ones. The authors suggest more diverse augmentation, supported by AutoAugment’s state-of-the-art CIFAR-10-C robustness.
Problem
Computer vision models lack robustness under distributional shift, while augmentation methods can improve some corruption types and degrade others.
Method
The paper uses Fourier-domain corruption statistics, filtering, and perturbation analyses to examine how Gaussian augmentation and adversarial training affect frequency reliance.
Results
Gaussian augmentation and adversarial training bias models toward low-frequency information, improving robustness to high-frequency corruptions while reducing robustness to low-frequency corruptions.
Takeaways & Limitations
More diverse augmentation may mitigate these trade-offs, and AutoAugment achieves state-of-the-art robustness on CIFAR-10-C.
Takeaways & Limitations
Data augmentation alone is unlikely to provide a complete solution, and AutoAugment is not strictly more robust than the baseline across frequencies.
Abstract
from arXiv · showhide
Achieving robustness to distributional shift is a longstanding and challenging goal of computer vision. Data augmentation is a commonly used approach for improving robustness, however robustness gains are typically not uniform across corruption types. Indeed increasing performance in the presence of random noise is often met with reduced performance on other corruptions such as contrast change. Understanding when and why these sorts of trade-offs occur is a crucial step towards mitigating them. Towards this end, we investigate recently observed trade-offs caused by Gaussian data augmentation and adversarial training. We find that both methods improve robustness to corruptions that are concentrated in the high frequency domain while reducing robustness to corruptions that are concentrated in the low frequency domain. This suggests that one way to mitigate these trade-offs via data augmentation is to use a more diverse set of augmentations. Towards this end we observe that AutoAugment, a recently proposed data augmentation policy optimized for clean accuracy, achieves state-of-the-art robustness on the CIFAR-10-C benchmark.
1 Introduction
The paper studies why robustness gains from Gaussian augmentation and adversarial training vary across corruption types. Fourier analysis links these trade-offs to a bias toward low-frequency information and motivates more diverse augmentation policies.
- Data augmentation can improve robustness to some corruptions while dramatically reducing performance on others.Gaussian augmentation and adversarial training improve robustness to noise and blurring but degrade performance on fog and contrast corruptions.
- Fourier analysis shows that Gaussian augmentation and adversarial training bias models toward low-frequency information.The authors connect this bias to the frequency characteristics of different corruptions.
- This low-frequency bias improves robustness to high-frequency corruptions while degrading robustness to low-frequency corruptions.
- More diverse augmentation procedures may mitigate these trade-offs.AutoAugment achieves state-of-the-art robustness on CIFAR-10-C, while a follow-up work reports state-of-the-art results on ImageNet-C.
- Fourier-domain perturbations can produce ImageNet error rates above 90% while preserving image semantics.The authors describe these as simple, single-query black-box attacks satisfying the content-preserving threat model.
2 Preliminaries
The paper defines Fourier representations and filtering operations, then uses normalized Gaussian augmentation and Fourier-basis perturbations to analyze model sensitivity across frequencies.
- The 2D discrete Fourier transform maps image matrices from R^d1×d2 to C^d1×d2, with low frequencies shifted to the spectrum center for visualization.
- High-pass filtering retains a centered square around the highest frequencies, while low-pass filtering retains a centered square around the lowest frequencies before inverse DFT.
- Gaussian data augmentation adds i.i.d. Gaussian noise to every training-batch pixel, with noise scale eσ sampled uniformly from [0, σ].Pixel values are clipped to [0, 1] in augmentation experiments.
- Fourier basis matrices have unit norm and Fourier transforms with at most two nonzero, symmetric frequency components.
- Fourier heat maps visualize test-error changes across frequency locations after adding randomly signed Fourier-basis perturbations.For multichannel images, each channel is perturbed independently.
3 The robustness problem
Models can classify images using low- and high-frequency information that is difficult for humans to recognize. These findings illustrate how non-robust statistics can support standard accuracy yet fail under distributional shift.
- Models trained on artificial shortcut signals or texture-like local statistics can perform poorly when those signals are removed or distorted.The examples connect brittleness to reliance on non-robust statistics.
- Over 30% ImageNet accuracy remains after severe low-frequency filtering, even when images appear as simple globs of color.
- 50% ImageNet accuracy is achieved under severe high-frequency filtering using features nearly invisible to humans.The high-pass images require normalized pixel statistics for visualization.
- Models can exploit correlations involving colors, local textures, shapes, and unintuitive high-frequency patterns for i.i.d. generalization.
- Imperceptibly perturbed images may contain features useful for generalization, connecting the frequency analysis to prior work.
4 Trade-off and correlation between corruptions: a Fourier perspective
Fourier analysis links augmentation-induced robustness trade-offs to corruption frequency: Gaussian augmentation and adversarial training favor high-frequency robustness while sacrificing low-frequency robustness. More varied augmentation, especially AutoAugment, mitigates these trade-offs, although frequency is not predictive in every transfer setting.
- Natural images concentrate more energy in low frequencies, whereas additive noise is relatively concentrated in high frequencies and fog and contrast in low frequencies.
- Gaussian augmentation and adversarial training improve robustness to high-frequency Fourier perturbations while reducing robustness to the lowest-frequency perturbations.This pattern appears in CIFAR-10 and ImageNet sensitivity analyses.
- For fixed-norm noise, naturally trained models are more robust to low-frequency bandwidth-3 noise, while augmented models outperform them on high-frequency bandwidths.The low-frequency error rate for naturally trained models is less than half that of the other two models.
- Low-pass filtering improves robustness to high-frequency corruptions but degrades robustness to low-frequency corruptions, while high-pass filtering causes larger accuracy drops on high-frequency corruptions.These results support a relationship between model frequency bias and corruption-specific robustness.
- Training on fog-like noise can hurt fog-corruption performance despite improving robustness to low-frequency Fourier perturbations, showing that matching marginal Fourier statistics is insufficient.The paper reports this result in Table 1 and the associated Fourier analysis.
- AutoAugment achieves 86% average corruption accuracy and mCE 64 on CIFAR-10-C, outperforming Gaussian augmentation at mCE 98 and adversarial training at mCE 108.It improves robustness over natural training on all but one corruption and also achieves the highest average corruption accuracy on ImageNet-C.
5 Conclusions and future work
The paper links robustness trade-offs to frequency biases induced by augmentation and argues that diverse augmentation can mitigate them, while data augmentation alone is unlikely to solve robustness completely.
- Gaussian data augmentation and adversarial training bias models toward low frequency information, improving robustness to high-frequency corruptions while reducing robustness to low-frequency corruptions and clean test error.
- Naively augmenting on different corruptions often transfers poorly to held-out corruptions, making robustness through data augmentation alone challenging.
- AutoAugment suggests that carefully designed augmentation can mitigate robustness trade-offs and support more useful domain-invariant features.
- Robustness benchmarks should evolve as methods induce new blind spots and trade-offs across frequencies and corruptions.
- Data augmentation alone is unlikely to provide a complete solution, motivating orthogonal methods such as architectures with better inductive biases or extrapolation-oriented loss functions.
A Comparison of model robustness on all the corruptions in CIFAR-10-C and ImageNet-C
The paper evaluates robustness using mCE and test accuracy across CIFAR-10-C and ImageNet-C, with naturally trained models providing the CIFAR-10-C baseline. The comparison includes multiple training and filtering strategies, including AutoAugment.
- mCE measures robustness improvement relative to a baseline model using test errors across K corruptions and S severities.For CIFAR-10-C, the naturally trained WideResNet is the baseline.
- The CIFAR-10-C comparison includes natural training, Gaussian augmentation, adversarial training, low-pass and high-pass front ends, and AutoAugment.
B Fourier heat maps
The paper defines Fourier heat maps by measuring how Fourier-basis perturbations change intermediate-layer outputs across validation images. It visualizes these changes and test-error patterns across WideResNet layers and ImageNet models.
- Heat-map construction: Layer outputs zh(X) are computed recursively, with the logits given by z(X) = zH(X) and predictions chosen by maximum logit.The h-th layer output is produced through the network’s weighted activations, and the final logits determine the predicted class.
- Heat-map construction: Fourier heat maps measure model stability by averaging layer-output changes across perturbed validation images.The procedure compares outputs on clean images and images modified with Fourier basis noise, then averages the output differences over n validation images.
- WideResNet visualization: Figure 8 covers five WideResNet outputs: the initial convolution, three residual blocks, and the logits, alongside a test-error heat map.The figure compares heat maps across five models, including natural training, Gaussian augmentation, adversarial training, fog-noise augmentation, and AutoAugment.
C Experiment detail
The experiment details describe normalized visualization of high-pass-filtered images and identify the model and corruption setup used in the ImageNet-C evaluation.
- Image normalization: High-pass-filtered images are visualized after normalization based on the image-wide pixel mean and standard deviation.Images are treated as X ∈ [0, 1]^(d1×d2), with RGB values optionally scaled by dividing pixel values by 255.
- Image normalization: The normalized images are displayed with matplotlib.pyplot’s imshow function.This specifies the visualization implementation rather than an additional modeling operation.
- Model comparisons: Figure 8 compares model heat maps for natural training, Gaussian augmentation, adversarial training, fog-noise augmentation, and AutoAugment.The fog-noise condition uses additive noise matching the Fourier statistics of severity-3 fog corruption.
- Model comparisons: Figure 9 reports test error in a Fourier heat map for ImageNet models across the full spectrum and a centered low-frequency square.The displayed low-frequency region is 63 × 63 within the full 224 × 224 spectrum, and a large central area has at least 95% test error.