Source-linked AI summary
Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain
Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, Yonghong Tian
TL;DR
CNN generalization remains vulnerable to common corruptions and OOD overconfidence, while prior frequency-domain accounts leave the phase spectrum’s role in robust recognition underexplored. The paper proposes APR, which recombines image phase and distracter amplitude; experiments report state-of-the-art performance across several generalization and calibration tasks.
Problem
CNNs show vulnerability to common corruptions and OOD overconfidence, while the role of phase in robust recognition remains underexplored.
Source: §1 · pp. 1–2Method
APR generates labeled training samples by combining the current image’s phase spectrum with a distracter image’s amplitude spectrum.
Source: §1 · p. 2Results
Takeaways & Limitations
The paper offers a unified explanation of common-perturbation robustness and OOD overconfidence in terms of CNN over-dependence on amplitude.
Source: §1 · p. 2The explanation is framed by assumptions that unrestricted CNNs favor amplitude and that amplitude dependence may underlie perturbation sensitivity and OOD overconfidence.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Recently, the generalization behavior of Convolutional Neural Networks (CNN) is gradually transparent through explanation techniques with the frequency components decomposition. However, the importance of the phase spectrum of the image for a robust vision system is still ignored. In this paper, we notice that the CNN tends to converge at the local optimum which is closely related to the high-frequency components of the training images, while the amplitude spectrum is easily disturbed such as noises or common corruptions. In contrast, more empirical studies found that humans rely on more phase components to achieve robust recognition. This observation leads to more explanations of the CNN's generalization behaviors in both robustness to common perturbations and out-of-distribution detection, and motivates a new perspective on data augmentation designed by re-combing the phase spectrum of the current image and the amplitude spectrum of the distracter image. That is, the generated samples force the CNN to pay more attention to the structured information from phase components and keep robust to the variation of the amplitude. Experiments on several image datasets indicate that the proposed method achieves state-of-the-art performances on multiple generalizations and calibration tasks, including adaptability for common corruptions and surface variations, out-of-distribution detection, and adversarial attack.
1. Introduction
The introduction links CNN robustness failures to amplitude-sensitive frequency reliance, contrasting this with human use of phase, and presents APR as a phase-focused augmentation.
1. Introduction
- CNNs benefit from high-frequency image components but are vulnerable when amplitude is disturbed by noise or common corruptions.The introduction contrasts this reliance with human recognition based more on phase-related components.Source: §1 · pp. 1–2
- After amplitude is exchanged, CNN predictions can follow the new amplitude while humans still recognize the object from its structure.The introduction reports this pattern in both in-distribution and OOD examples.Source: §1 · p. 2, Figure 2 · p. 2
- Amplitude-Phase Recombination (APR) trains on samples combining the current image’s phase with a distracter image’s amplitude, retaining the current image’s label.Distracters are either other images (APR-P) or augmentations of the current image (APR-S).Source: §1 · p. 2
- The paper frames CNNs’ amplitude dependence as a common explanation for perturbation sensitivity and OOD overconfidence.It notes that OOD images can have different structures yet share similarities in high-frequency amplitude components.Source: §1 · p. 2
- APR achieves state-of-the-art performance across common-corruption and surface-variation adaptability, OOD detection, and adversarial attack tasks.The paper also presents a unified explanation of perturbation robustness and OOD overconfidence through CNNs’ over-dependence on amplitude.Source: §1 · p. 2
2. Related Work
Related work examines CNN behavior through frequency components and surveys established data-augmentation strategies, while noting limits in prior analyses and methods.
2. Related Work
- Frequency-domain studies report CNN use of high-frequency features, low-frequency features, and phase features in face-forgery detection [32].Other work finds current defenses remain vulnerable to low-frequency perturbations [45]; the cited discussion says qualitative study of amplitude and phase roles in generalization was lacking.Source: §2 · pp. 2–3
- Figure 3 contrasts in-distribution CIFAR-10 and OOD SVHN examples, where swapping amplitude can exchange CNN predictions while humans retain category judgments from contours.The OOD examples are also described as receiving overconfident predictions.Source: Figure 3 · p. 3
- Conventional augmentation includes mixup [55], CutMix [53], AutoAugment, and AugMix [21], addressing overfitting or corruption robustness through different strategies.The passage notes that many methods can degrade non-adversarial accuracy or require adaptive, complex task-specific parameters [36].Source: §2 · p. 3
3. The Secret of CNN in the Frequency Domain
The qualitative study finds that CNN behavior is more sensitive to amplitude than phase, including under high-frequency perturbations and OOD comparisons. It interprets phase as encoding templates that preserve image structure.
3.1. Qualitative Study on the Frequency Domain
- The ResNet-18 evaluation reconstructs images from combinations of amplitude and phase spectra across frequency bands, then compares classification accuracy.The bands are low-, intermediate-, and high-frequency, separated using low-pass, band-pass, and high-pass filters.Source: §3.1 · p. 3
- Combining amplitude and phase performs better across tested combinations, while accuracy changes more when amplitude is varied than when phase is varied.The authors interpret this pattern as CNN convergence relying more on amplitude and neglecting phase.Source: §3.1 · p. 4, Figure 4 · p. 4
- Gaussian noise changes high-frequency amplitude despite being visually imperceptible, and the authors infer that CNNs may misclassify when that spectrum changes.Assumption 1 says CNNs without effective training restrictions tend to perceive amplitude more than phase; Corollary 1 describes a corresponding failure under bounded amplitude perturbation.Source: §3.1 · p. 4
- High-frequency amplitude does not distinguish CIFAR-10 in-distribution samples from CIFAR-100 OOD samples, which the authors connect to overconfidence.Assumption 2 attributes sensitivity to common perturbations and OOD overconfidence to amplitude over-dependence; Corollary 2 gives an example, and the corollaries are offered as alternative explanations of CNN generalization behavior.Source: §3.1 · p. 4, Figure 5 · p. 4
3.2. The Role of the Phase Spectrum
- The paper interprets Fourier frequencies as four sign-defined template contrasts encoded in phase, which can preserve image structure and help locate targets.The authors connect this structured information to the human visual system's recognition of images without the original amplitude spectrum.Source: §3.2 · p. 5, Figure 6 · p. 5
4. Amplitude-Phase Recombination
Amplitude-Phase Recombination (APR) is a parameter-free augmentation that recombines image spectra to reduce dependence on amplitude and emphasize phase. It has pair-sample and single-sample variants that can be combined.
4. Amplitude-Phase Recombination
- APR recombines image spectra in single or paired samples to encourage phase-based structure and reduce dependence on amplitude.The method constructs new training examples from the spectra of current and distracter images.Source: §4 · p. 5
- APR-P exchanges amplitude spectra between paired images while keeping phase spectra and their corresponding labels unchanged.It generates both exchanged pairings and uses the phase labels as targets.Source: §4 · p. 5
- APR-S treats a training image and a transformed version as different samples with the same label, using transformation sets based on different seeds or sequences.The transformations may be random or deterministic.Source: §4 · p. 5
- APR-P and APR-S can be combined, and the figure compares their generated samples; APR-S inversions are less different from the original image than APR-P samples.Table 1 is described as reporting average classification error for common corruptions and surface variations.Source: §4 · p. 5, Figure 7 · p. 6, Table 1 · p. 6
5. Experiments
APR-SP improves corruption robustness on ImageNet while maintaining test accuracy, and APR variants improve results across CIFAR corruption, OOD-detection, and adversarial evaluations. Increasing phase-prediction weight is also associated with stronger performance on corruption, surface-variation, and OOD tasks.
5. Experiments
- Experiments use CIFAR-10/100 and ImageNet, evaluate 15 corruption types at five severity levels, and test OOD detection with CIFAR-10 as in-distribution data.Corruptions are withheld from training; the reported OOD evaluation includes SVHN, resized LSUN and ImageNet, and CIFAR-100.Source: §5 · p. 6, §5.1 · pp. 6–7
5.1. CIFAR-10 and CIFAR-100
- APR-P achieves 6% lower absolute corruption error than pair-image Mixup or CutMix on CIFAR-100, while APR-SP improves performance by 5% over AugMix.APR-S with cross-entropy also performs better than AugMix on common corruptions and surface variations.Source: §5.1 · p. 7
- APR-P improves AUROC by 2% over Cutout on CIFAR-10 while maintaining test accuracy, and APR-SP exceeds CSI in almost all OOD tasks.The evaluation uses maximum softmax probability to detect OOD samples.Source: §5.1 · p. 7, Table 2 · p. 7
- APR-SP improves performance on clean images and against AutoAttack; APR-S improves more than APR-P, and combining both strategies performs better.Cutout is reported as ineffective against the attack relative to the baseline with revisiting adversarial training.Source: §5.1 · p. 7, Table 3 · p. 7
5.2. ImageNet Classsification
- 15% improvement over the baseline 80.6% mCEc: APR-SP improves ImageNet corruption performance while maintaining test accuracy.APR-SP is reported to outperform APR-S and APR-P by about 5%, and APR-SP with DeepAugment improves 6% over reproduced DeepAugment.Source: §5.2 · pp. 7–8, Table 4 · p. 8
5.3. Labeled by Amplitude or Phase?
- As phase-prediction weight increases, accuracy improves especially on common corruptions, surface variations, and OOD detection, while OOD detection ability strengthens.The final prediction combines phase and amplitude predictions using λ.Source: §5.3 · p. 8, Figure 9 · p. 8
6. Conclusion & Outlook
The conclusion presents APR as a phase-focused augmentation method and identifies further research directions; the appendix summarizes supplemental analyses and visualizes amplitude-spectrum distributions.
6. Conclusion & Outlook
- The authors propose APR to emphasize phase information and report state-of-the-art performance across multiple generalization and calibration tasks.They identify future work on representing part-whole hierarchies and developing CNN models or convolution operations that capture more phase information.Source: §6 · p. 8
Appendix
- The appendix describes supplemental analyses of frequency-domain distributions, phase-spectrum templates, APR-S implementation, Fourier analysis, and ImageNet evaluation.It says the ImageNet results clarify the method’s scalability.Source: Appendix · p. 11
- Figure 10 visualizes amplitude-spectrum types, with red for CIFAR-10 original or in-distribution samples and gray for corrupted, adversarial, or CIFAR-100 out-of-distribution samples.Source: Figure 10 · p. 11
A. More Studies on the Frequency Domain.
The appendix relates amplitude-spectrum differences to CNN behavior, describes how phase templates represent image structure, and specifies augmentation choices used in APR-S.
A. More Studies on the Frequency Domain.
- Corrupted and adversarial samples differ from originals in intermediate- and high-frequency amplitude, while their low-frequency amplitude remains indistinguishable.The authors relate these changes to CNN misclassification of corrupted or adversarial images.Source: A. More Studies on the Frequency Domain · p. 11
- CIFAR-100 out-of-distribution samples can be indistinguishable from in-distribution samples in any amplitude-frequency band, creating a risk of overconfident in-distribution predictions.Source: A. More Studies on the Frequency Domain · p. 11
- The authors relate perturbation sensitivity and OOD overconfidence to CNN over-dependence on amplitude, noting that similarity in any amplitude band may matter.Source: A. More Studies on the Frequency Domain · p. 11
B. Templates from the Phase Spectrum
- Phase-spectrum templates encode contrasts across scales and directions, from coarse low-frequency partitions to fine high-frequency partitions and moderate intermediate partitions.The authors say phase templates can recover image structure without the original amplitude spectrum.Source: B. Templates from the Phase Spectrum · pp. 11–12
C. Augmentation Operations
- APR-S uses augmentation operations from [21] but omits contrast, color, brightness, sharpness, and Cutout to avoid overlap with CIFAR-C and ImageNet-C corruptions.Source: C. Augmentation Operations · p. 12
D.1. Fourier Analysis
Fourier-basis perturbations reveal frequency-specific robustness differences across models. APR-SP improves robustness to most tested frequency perturbations.
D.1. Fourier Analysis
- High-frequency Fourier-basis perturbations push the standard model’s error rate above 80%, despite its robustness to low-frequency perturbations.Source: §D.1 · p. 12
- APR-S improves high-frequency robustness while retaining low-frequency robustness, but remains sensitive to intermediate-frequency additive noise.Source: §D.1 · p. 12
- APR-SP, which combines APR-S and APR-P, substantially improves robustness to most frequency perturbations.Source: §D.1 · p. 12
D.2. Clean Error
On CIFAR-10, APR-SP is reported to improve clean-image classification and adaptability across several settings; the accompanying frequency analysis reports increased robustness to most perturbations.
D.2. Clean Error
- APR-SP achieves the best CIFAR-10 clean-error performance across the reported backbone networks.Source: §D.2 · p. 12, Table 5 · p. 13
- APR-SP substantially improves robustness to most frequency perturbations, according to the accompanying model-sensitivity figure.Source: Figure 13 · p. 13
- APR-SP improves adaptability to common corruptions, surface variations, and OOD detection, as well as classification accuracy on clean images.Source: §D.2 · p. 13
D.3. OOD Detection on ImageNet.
On ImageNet-1K and ImageNet-O, APR-SP improves OOD detection over standard augmentations, with about 22% improvement on AUROC. Additional CAM examples show APR-SP focusing on target objects under common corruptions and surface variations.
D.3. OOD Detection on ImageNet.
- ImageNet-O provides 2K examples from ImageNet-22K that exclude ImageNet-1K; ResNet-50 is trained on ImageNet-1K and tested on both datasets.The experiments use ImageNet-O as the out-of-distribution dataset for ImageNet-1K.Source: §D.3 · p. 13
- OSCR jointly evaluates in-distribution accuracy and OOD detection; its thresholded CCR counts correct in-distribution predictions, while FPR counts OOD samples labeled in-distribution.Source: §D.3 · p. 13
- On a large and difficult dataset, APR-SP performs better than standard augmentations, achieves about 22% improvement on AUROC, and improves OOD detection performance according to OSCR while maintaining test accuracy.Source: §D.3 · p. 13
D.4. More CAM Visualization Examples
- APR-SP focuses on target objects under common corruptions and surface variations in the additional CAM examples.Figures 14 and 15 compare baseline and APR-SP visualizations alongside original and corrupted or varied images.Source: §D.4 · p. 13, Figure 14 · p. 14, Figure 15 · p. 15