Source-linked AI summary
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, Ekin D. Cubuk
TL;DR
The paper addresses the challenge of achieving both clean-data accuracy and robustness to naturally occurring corruptions despite an observed trade-off between them. It introduces Patch Gaussian, which adds Gaussian noise to random image patches and interpolates between Gaussian and Cutout. Patch Gaussian improves robustness and clean accuracy, reaches state-of-the-art Common Corruptions results, and transfers to other regularization, augmentation, and detection settings.
Problem
Models that perform accurately on clean images can remain vulnerable to real-world corruptions, while existing augmentations often improve robustness at the expense of clean accuracy.
Method
Patch Gaussian adds Gaussian noise to randomly selected image patches, interpolating between whole-image Gaussian noise and Cutout.
Results
Patch Gaussian improves robustness while maintaining or improving clean accuracy, achieves state-of-the-art CIFAR-C and ImageNet-C performance, and improves results with AutoAugment and COCO object detection.
Takeaways & Limitations
Patch Gaussian is reported as generally useful across common-corruption classification, combined regularization and augmentation strategies, object detection, and frequency-based robustness analysis.
Takeaways & Limitations
The observed robustness–generalization pattern initially appears consistent with prior claims that robustness comes at the cost of generalization, motivating further investigation.
Abstract
from arXiv · showhide
Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions. While architectural advances have led to improved accuracy, building robust models remains challenging. Prior work has argued that there is an inherent trade-off between robustness and accuracy, which is exemplified by standard data augment techniques such as Cutout, which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy. To overcome this trade-off, we introduce Patch Gaussian, a simple augmentation scheme that adds noise to randomly selected patches in an input image. Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNetCommon Corruptions benchmarks while also improving accuracy on clean data. We find that this augmentation leads to reduced sensitivity to high frequency noise(similar to Gaussian) while retaining the ability to take advantage of relevant high frequency information in the image (similar to Cutout). Finally, we show that Patch Gaussian can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment, and improves performance on the COCO object detection benchmark.
1 Introduction
Real-world deployment requires models that combine clean-data accuracy with robustness to naturally occurring corruptions. Patch Gaussian is introduced to overcome the observed trade-off between these goals while extending performance across benchmarks and settings.
- Motivation: Modern neural networks perform well on curated image datasets but remain vulnerable to common real-world corruptions.Examples include translations, scale changes, blur, additive noise, inserted objects, and distribution shifts.
- Contributions: Figure 1 indicates that increasing Patch Gaussian σ improves both mean corruption error and clean accuracy, unlike the competing trends for Gaussian and Cutout.More robust and accurate models appear down and to the right.
- Prior work: Robustness research has examined benchmarks, failure modes, architectural changes, pre-training, and data augmentation.Data augmentation enlarges training-set diversity and can learn invariances that are difficult to encode architecturally.
- Motivation: Prior augmentation methods often improve either robustness or clean accuracy, motivating claims of a fundamental trade-off.The paper frames this trade-off as a limitation of existing individual augmentation methods.
- Contributions: Patch Gaussian interpolates between Cutout and Gaussian and achieves state-of-the-art Common Corruptions performance on CIFAR-C and ImageNet-C.The paper also reports compatibility with regularization and augmentation policies, improved COCO detection, and frequency-based differences in model sensitivity.
2 Preliminaries
Cutout and Gaussian provide complementary benefits: Cutout improves clean accuracy, while Gaussian noise can improve robustness to Gaussian corruption. Their behavior reveals an apparent accuracy–robustness trade-off.
- Standard augmentations: Cutout replaces a randomly located square patch with the dataset’s mean pixel, while Gaussian adds independent noise to every pixel.Both augmentations use randomly sampled strength or patch parameters during training.
- Observed trade-off: Cutout improves clean-test accuracy but does not increase robustness to Gaussian noise.The comparison evaluates models on clean data and images corrupted by several Gaussian-noise standard deviations.
- Observed trade-off: Higher Gaussian-noise strength can improve robustness to Gaussian noise but decreases clean accuracy.This produces the complementary behavior to Cutout.
- Observed trade-off: Figure 2 plots each augmentation and hyper-parameter setting as a dot, with the y-axis showing corrupted-minus-clean accuracy change.The diamond marks the hyper-parameters selected by the method in Section 3.2.
- Observed trade-off: These results initially appear consistent with a robustness cost to generalization, which the later method investigates as a possible limitation of augmentation strategies.The passage presents this as an initial interpretation rather than a settled conclusion.
3 Method
Patch Gaussian adds Gaussian noise within a randomly located square patch, allowing a continuum between whole-image Gaussian noise and Cutout-like information removal. Hyper-parameters are selected using clean-accuracy and Gaussian-robustness criteria.
- Patch Gaussian: Patch Gaussian adds Gaussian noise to a randomly positioned W × W image patch.The patch center is sampled within the image, as in Cutout.
- Patch Gaussian: Varying patch size and maximum noise standard deviation interpolates between whole-image Gaussian augmentation and an approximation of Cutout.Increasing patch size approaches Gaussian, while increasing σmax approaches Cutout.
- Implementation: Image transformations operate on unnormalized pixels in [0, 1], with clipping after noise augmentation and standard flips and crops applied according to dataset-specific order.The implementation specifies different flip-and-crop ordering for CIFAR-10 and ImageNet.
- Hyper-parameter selection: Hyper-parameter selection prioritizes robustness while requiring a minimum clean accuracy Z; if none qualifies, the highest-clean-accuracy model is reported.Z varies by dataset and model.
- Hyper-parameter selection: Robustness is average accuracy relative to clean accuracy across Gaussian corruptions with σ values 0.1, 0.2, 0.3, 0.5, 0.8, and 1.0.The metric is reported as correlated with mCE and is used to reduce overfitting to the Common Corruptions benchmark.
- Experimental setting: Experiments use CIFAR-10 and ImageNet with Wide-ResNet, Shake-shake, ResNet-50, and ResNet-200 models under specified training schedules.Standard flips, crops, and model-specific optimization settings are used as baselines.
4 Results
Patch Gaussian overcomes the observed accuracy–robustness trade-off, improving clean accuracy and corruption robustness across classification and detection settings. It also remains compatible with other regularization and augmentation strategies.
- Patch Gaussian ablations: The selected Patch Gaussian hyper-parameters improve Gaussian-noise robustness while also improving clean accuracy, combining properties associated with Gaussian and Cutout.The ablation varies either patch size with σ fixed at 1.0 or σ with patch size fixed at 350.
- Common Corruptions: Patch Gaussian establishes state-of-the-art mean Corruption Error on both CIFAR-C and ImageNet-C.ImageNet-C results are reported for both the as-released JPEG-compressed benchmark and a version without extra compression.
- Common Corruptions: Patch Gaussian improves robustness beyond noise-based corruptions, including corruptions such as fog where Gaussian augmentation hurts performance.The comparison uses Original mCE (-noise) and mCE (-noise) to exclude Gaussian Noise, Shot Noise, and Impulse Noise.
- Model capacity: Capacity affects the magnitude of gains: lower-capacity models show diminished clean-accuracy and robustness improvements, while mCE still increases substantially relative to other strategies.The comparison contrasts ResNet-50 and Wide ResNet with higher-capacity ResNet-200 and Shake 112 models.
- Regularization: Patch Gaussian can be combined with existing regularization methods, with Dropblock plus Patch Gaussian producing the best robustness despite reducing clean accuracy.Label smoothing has mixed effects, while Dropblock is a strong regularizer under the 90-epoch training setting.
- Augmentation policies and detection: Patch Gaussian improves accuracy and robustness over AutoAugment alone and improves clean COCO detection, including a 1% mAP gain over baseline.On corrupted COCO data with σ = 0.25, Gaussian and Patch Gaussian reach 26.1% mAP, compared with 11.6% for baseline.
5 Discussion
The frequency analysis explains how Patch Gaussian combines Gaussian’s reduced high-frequency sensitivity with Cutout’s retention of useful high-frequency information. This combination addresses the accuracy–robustness weaknesses observed for the two standard augmentations.
- Analysis method: The analysis perturbs images with Fourier-space noise across orientations and frequencies, then measures activation changes and test error.The resulting heatmaps represent sensitivity to frequency and orientation perturbations.
- Analysis method: For CIFAR-10, the analysis covers the full Fourier domain; for ImageNet, it focuses on lower, more visually salient frequencies.The perturbation norms are 4 for CIFAR-10 and 15.7 for ImageNet.
- Frequency-based analysis: Gaussian encourages low-pass filtering, reducing high-frequency sensitivity but discarding valuable information at low layers.The authors suggest this may explain Gaussian’s lower test accuracy.
- Frequency-based analysis: Cutout encourages high-frequency information but does not reduce test-error sensitivity, limiting its robustness improvement.Its high-frequency use may help explain improved generalization performance.
- Frequency-based analysis: Patch Gaussian permits high-frequency information through lower layers while reducing test-error sensitivity at high frequencies.This combines properties associated with Cutout and Gaussian in the frequency-based analysis.
- Frequency-based analysis: High-pass-filtered-image accuracy remains similar to the baseline and Cutout for Patch Gaussian models, whereas Gaussian fails to maintain it.The comparison tests whether models retain useful high-frequency information.
6 Conclusion
The paper introduces Patch Gaussian as an augmentation that improves robustness to common corruptions without reducing clean accuracy. It also reports concurrent clean-accuracy and robustness improvements in several settings and compatibility with other training strategies.
- Conclusion: Patch Gaussian improves robustness to common corruptions without incurring a drop in clean accuracy.For models large relative to dataset size, including ResNet-200 on ImageNet and all CIFAR-10 models, both improve concurrently.
- Conclusion: Patch Gaussian interpolates between Cutout and Gaussian while combining their strengths without inheriting their weaknesses.The paper connects this behavior to its frequency-sensitivity analysis.
- Conclusion: Patch Gaussian works with other regularization and data augmentation strategies and improves object detection model performance.The conclusion presents this as evidence of broader usefulness.
Appendix
The appendix provides supplementary figures, tables, implementation fragments, and visualizations covering augmentation behavior, hyperparameters, corruption results, and Fourier analyses.
- Supplementary figures: Figure 6 presents the accuracy/robustness trade-off for Cutout and Gaussian on ResNet-50 models.The figure is referenced as supplementary detail for the trade-off analysis.
- Supplementary figures: Figure 7 shows a Patch Gaussian hyperparameter sweep on CIFAR-10 and ImageNet.Patch Gaussian approaches Gaussian with larger patch sizes and Cutout with larger σ; colors encode σ.
- Supplementary figures: Figure 8 illustrates Patch Gaussian’s ability to overcome the accuracy/robustness trade-off on CIFAR-10 and ResNet-50.The top row covers CIFAR-10 and the bottom row covers ResNet-50.
- Supplementary figures: Figure 9 displays images modified with centered Patch Gaussian patches across different W and σ values.The visualization varies patch width and noise scale.
- Implementation: The patch mask is generated from a randomly sampled location, boundary clipping, padding dimensions, and a tiled three-channel mask.These fragments implement patches that may intersect image boundaries.
- Implementation: The TensorFlow implementation samples a patch size and noise scale, applies clipped Gaussian noise to the selected patch, and leaves the rest unchanged.The implementation normalizes pixels, constructs a mask, and rescales the output.
- Supplementary tables: Tables 6–8 provide selected augmentation hyperparameters and full original and corruption errors for ImageNet models.Table 6 marks manually chosen stronger hyperparameters with an asterisk and denotes maximum-uniform sampling with “≤”.