Source-linked AI summary
ColorFool: Semantic Adversarial Colorization
Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, Andrea Cavallaro
TL;DR
Restricted adversarial attacks have limited black-box transferability and defense robustness, while unrestricted attacks can be visually noticeable. ColorFool uses semantic, natural-color modifications in a black-box unrestricted attack and achieves state-of-the-art success rate, defense robustness, and transferability across the evaluated classification settings.
Problem
Restricted attacks have limited success against unseen classifiers and defenses, whereas unrestricted attacks can produce perceptually unnatural colors.
Method
ColorFool selectively changes colors within natural ranges for semantic regions using the Lab color space without changing lightness.
Results
ColorFool achieves state-of-the-art success rate, robustness to defenses, and transferability across scene and object classification evaluations against five attacks.
Takeaways & Limitations
Semantic colorization can produce same-size adversarial images that remain natural-looking while misleading seen and unseen classifiers.
Takeaways & Limitations
Future work will evaluate the attacks under more defenses and in object detection and semantic segmentation.
Abstract
from arXiv · showhide
Adversarial attacks that generate small L_p-norm perturbations to mislead classifiers have limited success in black-box settings and with unseen classifiers. These attacks are also not robust to defenses that use denoising filters and to adversarial training procedures. Instead, adversarial attacks that generate unrestricted perturbations are more robust to defenses, are generally more successful in black-box settings and are more transferable to unseen classifiers. However, unrestricted perturbations may be noticeable to humans. In this paper, we propose a content-based black-box adversarial attack that generates unrestricted perturbations by exploiting image semantics to selectively modify colors within chosen ranges that are perceived as natural by humans. We show that the proposed approach, ColorFool, outperforms in terms of success rate, robustness to defense frameworks and transferability, five state-of-the-art adversarial attacks on two different tasks, scene and object classification, when attacking three state-of-the-art deep neural networks using three standard datasets. The source code is available at https://github.com/smartcameras/ColorFool.
1. Introduction
ColorFool addresses limitations of restricted adversarial perturbations in black-box transferability, defense robustness, and visual naturalness by selectively changing colors in semantic image regions. It is evaluated against five attacks across scene and object classification settings.
- 1. Introduction: Restricted attacks control pixel changes through L_p norms, whereas unrestricted attacks span a wider range of intensity changes.The restricted forms include L∞, L0, and L2 constraints on pixel changes, perturbed-pixel counts, or energy.
- 1. Introduction: Restricted perturbations can be detected by defenses, while unrestricted attacks are more robust but may create unnatural, perceptible colors.Defenses include re-quantization, median filtering, JPEG compression, adversarial training, and modified loss functions.
- 1. Introduction: ColorFool proposes a black-box, unrestricted attack that selectively alters colors in semantic regions while preserving human-sensitive regions and natural appearance.It operates on Lab color channels and chooses natural-color ranges for specific semantic categories.
- 1. Introduction: ColorFool is validated on three DNNs and three datasets for scene and object classification, including seen and unseen classifiers and defense frameworks.The evaluation compares ColorFool with five state-of-the-art attacks.
2. Adversarial attacks
Adversarial attacks are organized by whether perturbations are norm-restricted or unrestricted, with different mechanisms for generating and transferring adversarial images. The section compares BIM, TI-BIM, DeepFool, SparseFool, SemanticAdv, BigAdv, and ColorFool across their perturbation types, attack settings, and constraints.
- Restricted attacks: BIM iteratively follows the input gradient under an L∞ constraint, while clipping keeps adversarial images within both the ε-neighborhood and valid pixel range.TI-BIM extends BIM by convolving gradients with a predefined kernel to improve transferability to unseen classifiers.
- Restricted attacks: DeepFool seeks a minimal L2 perturbation toward the closest decision boundary, whereas SparseFool projects onto an approximated boundary using an L1 objective to produce sparse changes.SparseFool uses a local hyperplane approximation and clipping to maintain valid pixel values.
- Attack comparison: Table 1 organizes attacks by perturbation norm or color manipulation, white- versus black-box access, and object- versus scene-classification tasks.It also records datasets and class counts, including ImageNet, CIFAR-10, P-Places365, and MNIST for JSMA.
- Unrestricted attacks: SemanticAdv shifts hue and saturation while preserving HSV value, and BigAdv fine-tunes a colorization model for each image to generate unrestricted color perturbations.BigAdv may severely distort colors when its color hints are not carefully chosen.
- Unrestricted attacks: ColorFool is an unrestricted black-box attack that perturbs colors only in selected semantic regions and chosen natural ranges, unlike SemanticAdv and BigAdv.SemanticAdv changes HSV colors without considering image content, while BigAdv uses a trained colorization model and per-image fine-tuning.
3. ColorFool
ColorFool separates semantic regions by their sensitivity to color and selectively perturbs them in Lab space using natural-color ranges. It generates adversarial images by progressively applying randomized color changes while preserving valid pixel values.
- 3.2 Color perturbation: Non-sensitive regions can undergo arbitrary-range color changes, after which modified sensitive and non-sensitive regions are recombined into the adversarial image.Quantization keeps pixel values within the valid dynamic range, and inverse Lab-to-RGB conversion produces the final image.
- 3.1 Semantic image regions: ColorFool decomposes each image into semantic regions and distinguishes color-sensitive categories from non-sensitive regions using segmentation masks.Sensitive categories include person, sky, vegetation, and water; masks are produced by a segmentation architecture trained on 150 semantic region types.
- 3.2 Color perturbation: The attack modifies colors in the perceptually uniform Lab space, which separates chromatic information in a and b from brightness in L.The a channel spans green to red, b spans blue to yellow, and L spans black to white.
- 3.2 Color perturbation: Sensitive regions receive randomized perturbations from natural-color ranges defined by semantic class, observed colors, and prior color-perception knowledge.The perturbation is applied to the a and b channels while preserving the lightness channel; Table 2 specifies the semantic color ranges and excludes person regions from color changes.
- 3.2 Color perturbation: ColorFool progressively scales randomly selected perturbations across trials until one misleads the classifier or the maximum trial count is reached.The scaling factor is α = n/N, reducing large color changes in early trials.
4. Validation
ColorFool is evaluated against five adversarial attacks across scene and object classification, multiple classifiers, datasets, transfer settings, defenses, and image-quality measures. It achieves strong transferability and defense robustness while preserving natural-looking image quality.
- Experimental setup: The validation compares BIM, TI-BIM, DeepFool, SparseFool, SemanticAdv, ColorFool-r, and ColorFool across ResNet50, ResNet18, AlexNet, and three datasets.The evaluation measures success rate, transferability, robustness to defenses, and adversarial-image quality using standardized attack implementations and data-processing software.
- Success rate and transferability: ColorFool achieves high success rates on seen and unseen classifiers, reaching 0.97 on R18 and 0.69 and 0.41 when evaluated with AlexNet and R50, respectively, on CIFAR-10.Restricted attacks reach at most 0.41 on unseen classifiers, whereas unrestricted attacks reach up to 0.77; SemanticAdv can transfer better but severely distorts colors.
- Robustness to defenses: Unrestricted attacks consistently exceed 60% success after re-quantization, median filtering, or JPEG compression, while restricted attacks are the least robust to filtering.Figure 3 reports post-filter success rates across datasets and classifiers; restricted perturbations can be removed before classification.
- Robustness to defenses: ColorFool is least detectable across applied filters on P-Places365, with 25% detectability versus 27% for SemanticAdv and ColorFool-r.The paper attributes this pattern partly to low-frequency perturbations in unrestricted attacks; JPEG compression is reported as the most effective detection framework.
- Robustness to defenses: ColorFool remains robust under PCL combined with adversarial training, maintaining a success rate above 99% against ResNet110 while restricted attacks decline considerably.Table 4 evaluates CIFAR-10 attacks against ResNet110 under softmax, PCL, and combined PCL plus adversarial training defenses.
- Quality: ColorFool preserves perceived image quality despite large L_p-distance changes, matching or exceeding clean-image NIMA scores across datasets and classifiers.ColorFool-r and ColorFool score over 5.19 on P-Places365 and ImageNet; on CIFAR-10, SemanticAdv, ColorFool-r, and ColorFool score over 4.96.
5. Conclusion
ColorFool is a black-box adversarial attack that changes colors in semantic regions using human color-perception priors. The authors report state-of-the-art success, robustness, transferability, and reduced detectability, while identifying broader defenses and tasks for future evaluation.
- ColorFool achieves state-of-the-art success rate, transferability, and robustness to filtering, adversarial training, and improved loss-function defenses.The comparison covers seen and unseen classifiers and emphasizes reduced detectability relative to restricted attacks, especially under JPEG compression.
- Future work will evaluate attacks against more defenses and extend colorization-based analysis to object detection and semantic segmentation.