Source-linked AI summary
IGAff: Benchmarking Adversarial Iterative and Genetic Affine Algorithms on Deep Neural Networks
Sebastian-Vasile Echim, Andrei-Alexandru Preda, Dumitru-Clementin Cercel, Florin Pop
TL;DR
Black-box adversarial attacks can reveal weaknesses in deep networks, but assessing robustness is challenging when model details are inaccessible. The paper benchmarks two affine-based black-box algorithms across image datasets, architectures, and attack and training configurations. It reports that AGA is the strongest evaluated attack, while adversarial augmentation particularly benefits vision transformers.
Problem
Assessing the robustness of newer deep neural network architectures against adversarial attacks is important, especially when models are difficult to interpret or their details are inaccessible.
Source: §1 · p. 1Method
The paper benchmarks ATA, an iterative affine-transformation attack, and AGA, a genetic algorithm combining affine transformations and random noise, across datasets, architectures, and attack and augmentation setups.
Source: §1 · pp. 1–2Results
AGA is the strongest evaluated attack, and vision transformers are more vulnerable to undefended AGA attacks but can become more robust than CNNs after adversarial augmentation.
Source: §5 · p. 9Takeaways & Limitations
ATA is more stable and less sensitive to tuning, making it more suitable for data augmentation or as part of a more complex adversarial algorithm.
Source: §5 · p. 9Hardware memory constrained AGA population size, and adversarial training required more time as iteration counts increased.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
Deep neural networks currently dominate many fields of the artificial intelligence landscape, achieving state-of-the-art results on numerous tasks while remaining hard to understand and exhibiting surprising weaknesses. An active area of research focuses on adversarial attacks, which aim to generate inputs that uncover these weaknesses. However, this proves challenging, especially in the black-box scenario where model details are inaccessible. This paper explores in detail the impact of such adversarial algorithms on ResNet-18, DenseNet-121, Swin Transformer V2, and Vision Transformer network architectures. Leveraging the Tiny ImageNet, Caltech-256, and Food-101 datasets, we benchmark two novel black-box iterative adversarial algorithms based on affine transformations and genetic algorithms: 1) Affine Transformation Attack (ATA), an iterative algorithm maximizing our attack score function using random affine transformations, and 2) Affine Genetic Attack (AGA), a genetic algorithm that involves random noise and affine transformations. We evaluate the performance of the models in the algorithm parameter variation, data augmentation, and global and targeted attack configurations. We also compare our algorithms with two black-box adversarial algorithms, Pixle and Square Attack. Our experiments yield better results on the image classification task than similar methods in the literature, achieving an accuracy improvement of up to 8.82%. We provide noteworthy insights into successful adversarial defenses and attacks at both global and targeted levels, and demonstrate adversarial robustness through algorithm parameter variation.
1 Introduction
Adversarial attacks can reveal weaknesses in difficult-to-interpret deep networks, but assessing newer architectures under black-box attacks remains an important robustness question. This paper introduces and benchmarks two affine-based black-box attacks across several image datasets and model architectures.
1 Introduction
- Adversarial attacks can expose weaknesses in black-box models and can also help train more robust versions.The paper motivates this work by noting robustness questions for newer deep-learning architectures, including vision transformers.Source: §1 · p. 1
- Genetic algorithms have been applied to adversarial attacks by combining and mutating promising candidate inputs, including without access to model architecture or weights.Prior applications also extend beyond image tasks to language, cybersecurity, and tabular settings.Source: §1 · p. 1
- The study benchmarks ATA, an iterative affine-transformation attack, and AGA, a genetic attack combining affine transformations and random noise, across three datasets and four architectures.The evaluations cover attack, defense, and data-augmentation setups.Source: §1 · p. 1
- The paper evaluates global and targeted attacks, data augmentation, parameter variation, and qualitative comparisons with Pixle and Square Attack.Its targeted experiments cover ten classes on a dataset and examine potential training uses for particular class confusion.Source: §1 · p. 2
2 Related work
Prior work studies robustness across attack settings, model families, and affine transformations, while targeted attacks use varied mechanisms and tasks. The paper situates its algorithms among this work and depicts ATA and AGA as iterative methods with additional genetic stages for AGA.
2.1 Adversarial Robustness
- Vision-transformer robustness has been studied more recently than CNN robustness; prior work reports settings where transformers are robust, as well as lower attack transferability.Reported explanations include reduced reliance on high-frequency features and stronger generalization against spurious correlations after large-scale pretraining.Source: §2.1 · p. 2
2.2 Adversarial Affine Transformations
- Affine transformations preserve parallel lines and include translations, rotations, scalings, and shears; they are also used to create synthetic augmentation data.The cited literature includes affine-based defense, attack, and intermediate-layer applications.Source: §2.2 · p. 2
- Prior affine-transform defense experiments reported accuracy recovery against several attacks, but did not assess granular transformation effects or include gray-box and black-box attacks.Other work applies affine transformations to face-recognition attacks or CNN activation maps.Source: §2.2 · p. 2
2.3 Targeted Attacks
- Prior targeted attacks span audio noise with genetic search, multi-target perturbations, cross-model attacks, and white-box fashion recommendations.Reported outcomes include up to 35% target-matching audio inputs [46], 100% MNIST targeted success, an 18.7% average improvement over the state of the art [6], and up to 100% success in specific source-to-target fashion experiments.Source: §2.3 · p. 2
- ATA and AGA share iterative selection that maximizes the attack score; AGA additionally uses population initialization, mutation, and crossover.The algorithms produce mutated images for either training-time augmentation or testing-time attack.Source: Figure 2 · p. 3
3 Methodology
The methodology benchmarks affine and genetic black-box attacks across datasets and model architectures, using targeted and untargeted configurations alongside data augmentation and parameter variation.
3.1 Dataset
- Caltech-256, Tiny-ImageNet-200, and Food-101 span class-support diversity factors of 0.46, 2.75, and 9.90, respectively.Images are resized to 224 × 224 pixels and split into train, validation, and test sets at 0.8/0.1/0.1.Source: §3.1 · p. 3
3.2 Model Architectures
- The tested architectures are ResNet-18, DenseNet-121, Vision Transformer, and SwinV2-T; DenseNet-121 uses dropout 0.4.ViT processes 16×16 image patches as tokens, and the SwinV2-T configuration uses C = 96 with blocks {2, 2, 6, 2}.Source: §3.2 · p. 3
3.3 Adversarial Algorithms
- ATA applies randomly sampled affine transformations, while AGA adds genetic mutation, bounded noise, and crossover; both select adversarial images using attack scores.AGA repeats the highest-scoring candidate to reinitialize its population; the experiments use 7 iterations, population size 3, mutation and crossover probabilities of 0.3, and intensity 0.1.Source: §3.3 · p. 4
3.4 Experimental Setup
- Experiments vary attack iterations and AGA parameters, assess data augmentation, and test untargeted attacks against targeted attacks on ten Caltech-256 classes.Accuracy and macro and weighted F1 are reported, with attack success rate used for adversarial experiments; training uses 12 epochs and batch size 32.Source: §3.4 · pp. 4–5
- Randomized experiments are repeated five times and reported as mean and standard deviation, except targeted attacks, which report average success rate only.Hardware memory limits AGA population size to 3 and batch size to 32 for ViT; adversarial training time increases with attack iterations.Source: §3.4 · p. 5
4 Results
Across the three datasets, AGA achieved the highest reported undefended attack success rates, while classification accuracy peaked at 90.16% on Caltech-256 and exceeded Sandru et al. by up to 8.82%.
4.1 Model Performance
- 85.42% average accuracy on Tiny-ImageNet-200 was 8.82% above Sandru et al.; Swin Transformer V2 led Caltech-256 (90.16%) and Food-101 (84.88%).For Caltech-256 and Food-101, the reported Swin Transformer V2 results were 90.16% without augmentation and 84.88% using ATA augmentations, respectively.Source: Table 1 · p. 5, §4.1 · p. 5
4.2 Adversarial Attacks
- AGA achieved the highest undefended attack success rates: 43.1% for Swin Transformer V2 on Caltech-256, 56.09% for Swin Transformer V2 on Food-101, and 63.97% for ViT on Tiny-ImageNet-200.AGA also exceeded ATA's SR by 29.14% on Caltech-256, 38.4% on Food-101, and 38.17% on Tiny-ImageNet-200 in undefended attacks.Source: Table 2 · p. 6, §4.2 · p. 6
- Adversarially trained vision transformers defended better than CNNs, but were significantly affected when undefended; ViT on Tiny-ImageNet-200 had the largest AGA SR drop, −49.6%.The reported best ATA defense reductions were 5.89% for ResNet-18 on Food-101 and 12.17% for DenseNet-121 on Tiny-ImageNet-200.Source: Table 2 · p. 6, §4.2 · p. 6
- Untargeted attacks confused every model more than targeted attacks; targeted SR peaked at 7.77% for Swin Transformer V2 under AGA.ViT's maximum targeted SR was 1.41% under AGA.Source: Table 3 · p. 6, §4.2 · pp. 6–7
- DenseNet-121 showed ATA targeted-class accuracy improvements for 7/10 classes, while Swin Transformer V2's AGA improved 4/10; Swin exceeded ViT by eight improved classes.DenseNet-121 had five more improved classes than ResNet-18.Source: Figure 4 · p. 7, §4.2 · p. 7
4.3 Algorithm Parameter Variation
- AGA SR rose with iterations, peaked at 100% mutation, and increased with noise intensity; crossover probability was least sensitive, with 7–8% standard deviation.Swin Transformer V2 was most affected by increasing parameters on Caltech-256 and Food-101, while ViT was most vulnerable on Tiny-ImageNet-200.Source: Figure 5 · p. 7, §4.3 · p. 8
4.4 Qualitative Results
- AGA fooled the classifier on 8/12 core images, versus ATA 6/12, Square Attack 5/12, and Pixle 4/12.ATA selects the best candidate after one affine-transform pass, whereas AGA iteratively updates fit candidates and accumulates noise; Square Attack uses patches and stripes, while Pixle swaps pixels.Source: Figure 6 · p. 8, §4.4 · pp. 8–9
4.3 Algorithm Parameter Variation
- ATA attack improvements were 0.5–4% with higher iteration counts; SR maxima were around 27% on Tiny-ImageNet-200, 19% on Food-101, and 15% on Caltech-256.Swin Transformer V2 had up to 6–8% more SR than ViT, though attack SR was visibly smaller for both transformer architectures than other models.Source: Figure 7 · p. 8, §4.3 · p. 8
5 Conclusions
The conclusions report that AGA is the strongest evaluated attack, while ATA is more stable and less sensitive to tuning. They also find that transformers’ relative vulnerability to undefended attacks reverses with adversarial training, while noting limits in task and targeted-attack coverage.
5 Conclusions
- The benchmark reports data-augmentation gains over a similar method, defense against impactful attacks, and targeted-attack potential for improving target-class classification.It also examines algorithm-parameter variation and qualitatively compares ATA, AGA, Pixle, and Square Attack.Source: §5 · p. 9
- Transformers achieve the best regular-training accuracy but are more vulnerable to undefended AGA attacks; adversarial augmentation can make them more robust than CNNs.The authors conclude that future robustness research should focus more on transformers.Source: §5 · p. 9
- AGA is the strongest evaluated attack, with performance mainly driven by iteration count, mutation rate, and noise intensity; crossover rate has limited impact.ATA is more stable and less sensitive to heavy tuning, making it more suitable for data augmentation or as part of a more complex attack.Source: §5 · p. 9
- Evaluation is limited to image classification; potential use in natural language processing, scene text detection, and face recognition remains future work.Hyperparameters were adjusted to available hardware, and testing with increased parameter values remains future work.Source: §5 · p. 9
- Targeted attacks covered only ten classes for one dataset, so their benefits across multiple datasets remain to be tested.The authors also identify further work on qualitative results, explainability, multi-objective genetic algorithms, affine-transform ablations, comparisons with other attack types, and progression toward white-box attacks.Source: §5 · p. 9