Source-linked AI summary
SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization
A. F. M. Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae
TL;DR
Deep CNNs risk overfitting, while random patch selection can pair labels with source regions that do not represent the object. SaliencyMix selects patches around salient regions before mixing them with target images, and reports strong classification results alongside improved robustness and object detection performance.
Problem
Deep CNNs can overfit, and random patch selection may choose uninformative regions whose mixed labels encourage unexpected feature representations.
Method
SaliencyMix extracts a saliency map, selects a patch around the peak salient region, and mixes that representative source patch with the target image.
Results
SaliencyMix improves classification, robustness, and object detection performance, including best known top-1 errors of 2.76% and 16.56% for WideResNet on CIFAR-10 and CIFAR-100.
Takeaways & Limitations
Careful source-patch selection keeps information about the source object and supports more appropriate feature representation and class-probability assignment.
Takeaways & Limitations
SaliencyMix requires slightly longer training time because of saliency-map generation, and top-down saliency methods may generalize poorly beyond their training data.
Abstract
from arXiv · showhide
Advanced data augmentation strategies have widely been studied to improve the generalization ability of deep learning models. Regional dropout is one of the popular solutions that guides the model to focus on less discriminative parts by randomly removing image regions, resulting in improved regularization. However, such information removal is undesirable. On the other hand, recent strategies suggest to randomly cut and mix patches and their labels among training images, to enjoy the advantages of regional dropout without having any pointless pixel in the augmented images. We argue that such random selection strategies of the patches may not necessarily represent sufficient information about the corresponding object and thereby mixing the labels according to that uninformative patch enables the model to learn unexpected feature representation. Therefore, we propose SaliencyMix that carefully selects a representative image patch with the help of a saliency map and mixes this indicative patch with the target image, thus leading the model to learn more appropriate feature representation. SaliencyMix achieves the best known top-1 error of 21.26% and 20.09% for ResNet-50 and ResNet-101 architectures on ImageNet classification, respectively, and also improves the model robustness against adversarial perturbations. Furthermore, models that are trained with SaliencyMix help to improve the object detection performance. Source code is available at https://github.com/SaliencyMix/SaliencyMix.
1 INTRODUCTION
Deep CNNs can overfit when training examples are insufficient, motivating augmentation methods that improve generalization without discarding informative pixels or selecting irrelevant source patches. SaliencyMix addresses this by selecting source patches around salient regions before mixing them with target images.
- Motivation: Large CNN parameter counts increase overfitting risk when training examples are insufficient, reducing model generalization.CNNs typically have 10 to 100 millions of learnable parameters.
- Existing augmentation: Regional dropout improves regularization by removing image regions, but discards a notable portion of informative pixels.Dropout and regional dropout guide models away from focusing on small image regions or internal activations.
- Existing augmentation: CutMix avoids blank regions by replacing an image region with another image patch, but random selection can choose background patches irrelevant to the source object.Such patches may leave augmented images without information about the corresponding object.
- SaliencyMix: SaliencyMix extracts a saliency map and selects a patch surrounding the source image’s peak salient region before mixing it with the target image.The selected patch is intended to contain relevant source-object information and support more appropriate feature representation learning.
- Reported results: SaliencyMix reports best known top-1 errors of 2.76% on CIFAR-10 and 16.56% on CIFAR-100 for WideResNet, plus ImageNet results for ResNet-50 and ResNet-101.The ImageNet top-1 errors are 21.26% for ResNet-50 and 20.09% for ResNet-101.
2 RELATED WORKS
The paper situates SaliencyMix among augmentation and saliency-detection methods, emphasizing representative source patches and saliency models that generalize beyond task-specific training data. It connects careful patch selection with improved performance and robustness while noting a small computational burden from saliency detection.
- Data augmentation: Data augmentation increases the diversity of existing data through transformations and can improve model performance and robustness.The paper describes augmentation as useful when collecting labeled data is cumbersome and time-consuming.
- Data augmentation: Prior augmentation methods include Cutout, which removes image regions, and MixUp, which blends images and linearly interpolates their labels.These methods are presented as alternatives to manually designed transformations.
- SaliencyMix: SaliencyMix uses careful source-patch selection to retain source-object information, addressing the class probability assignment problem and improving performance and robustness.The paper links representative patch content to more appropriate label assignment during mixing.
- Saliency detection: Saliency detection methods include bottom-up approaches using low-level visual features and top-down approaches using supervised, task-driven learning.The paper reviews contrast, frequency-domain, contextual-attention, and deep-learning-based saliency methods.
- Saliency detection: SaliencyMix uses bottom-up saliency because it is unsupervised, scale-invariant, and more robust for unseen data than task-specific top-down models.The effectiveness of training-based saliency methods may depend on the quality and quantity of their training data.
3 PROPOSED METHOD
SaliencyMix selects a source patch around the most salient region, then combines it with a target image and mixes labels according to patch size. The method examines saliency detectors and patch-selection schemes to determine effective configurations.
- Source-patch selection: SaliencyMix extracts a saliency map from a source image and selects a patch surrounding its peak salient region to retain source-object information.The saliency map highlights objects of interest, while the peak identifies the most salient pixel used to position the patch.
- Source-patch selection: The source patch is selected around the most salient pixel so that it comes from the object region rather than the background.The patch size is controlled by a combination ratio λ sampled from the uniform distribution (0, 1).
- Patch and label mixing: SaliencyMix forms an augmented image by combining a masked source patch with the complementary masked target image.The binary mask retains the selected source region, while its complement retains the other target-image regions.
- Patch and label mixing: The augmented label combines source and target labels according to the size of their mixed patches using the combination ratio λ.The method defines the mixed label as ya = λyt + (1 − λ)ys.
- Experimental examination: The method evaluates saliency detectors and selection schemes on CIFAR-10 and Tiny-ImageNet using ResNet-18 and ResNet-50 experiments.The saliency-detector experiments run for 200 epochs on CIFAR-10 and 100 epochs on Tiny-ImageNet; results are averaged across five and three runs for the two datasets, respectively.
- Selection and mixing schemes: Among the investigated schemes, Salient to Corresponding balances occluding important target regions with preserving regularization, so it is used by default.Schemes selecting non-salient source regions produce larger classification errors, while Salient to Salient and Salient to Non-Salient have different target-occlusion drawbacks.
4 EXPERIMENTS
Experiments evaluate SaliencyMix across image classification, object detection, CAM analysis, adversarial robustness, and training time. The method generally outperforms comparison augmentations, while requiring slightly longer training because of saliency-map generation.
- Image Classification: SaliencyMix outperforms other methods on CIFAR datasets except PuzzleMix, achieving 2.76% and 16.56% top-1 error with WideResNet-28-10 on CIFAR-10 and CIFAR-100.Without traditional augmentation, it reduces CIFAR-10 error over CutMix by 1.85%, 2.35%, and 1.14% across ResNet-18, ResNet-50, and WideResNet-28-10; CIFAR-100 reductions are 5.69%, 6.76%, and 3.76%.
- Image Classification: SaliencyMix drops ResNet-50 ImageNet top-1 error by 1.66%, 1.31%, and 0.14% versus Cutout, Mixup, and CutMix, respectively.For ResNet-101, it achieves 20.09% top-1 error and 5.15% top-5 error.
- Object Detection: SaliencyMix pre-training improves Faster RCNN object detection by +1.77 mAP after fine-tuning on Pascal VOC.The authors attribute this to augmented images containing more foreground or object parts than background.
- CAM Analysis: CAM results show SaliencyMix guides models to focus more precisely on target objects and corresponding features than the compared augmentation techniques.Mixup can confuse object recognition, while Cutout can suffer from uninformative image regions.
- Adversarial Robustness: SaliencyMix improves adversarial top-1 accuracy by 1.96% over the nearly comparable CutMix method on perturbed ImageNet validation images.The experiment uses FGSM-generated adversarial examples and ResNet-50 ImageNet models.
- Computational Complexity: SaliencyMix requires slightly longer training than other methods because it generates saliency maps.The authors consider this additional cost negligible relative to the performance improvement.
5 CONCLUSION
SaliencyMix is designed to improve CNN classification and generalization by selecting representative source patches and directing models toward overall object regions. Across classification, detection, and robustness evaluations, it reports improved performance with a small saliency-detection cost.
- Conclusion: SaliencyMix selects representative source patches to prevent inappropriate feature representations and guide models toward overall object regions.The strategy is designed for training CNNs to improve classification performance and generalization ability.
- Conclusion: SaliencyMix achieves 2.76% and 16.56% top-1 error with WideResNet on CIFAR-10 and CIFAR-100, respectively.These are reported as new best known results in the conclusion.
- Conclusion: SaliencyMix achieves 21.26% and 20.09% top-1 error with ResNet-50 and ResNet-101 on ImageNet, respectively, while Faster RCNN transfer improves by +1.77 mAP.The conclusion also reports strengthened robustness against adversarial attacks.