Attacking the Madry Defense Model with $L_1$-based Adversarial Examples

1 INTRODUCTION

• Adversarial examples can mislead a well-trained network and transfer from the model they target to other models trained for the same task.The introduction also notes that some examples can be virtually indistinguishable to human perception.Source: §1 · p. 1
• The MNIST challenge limits each pixel’s perturbation to ε = 0.3 in L∞, restricting attacks not optimized for that metric.The authors characterize this constraint as unrealistic for an attacker.Source: §1 · p. 1
• EAD’s L1-based adversarial examples transfer in targeted and non-targeted cases and have minimal visual distortion despite high L∞ distortion.The authors argue these findings call into question using L∞ alone to quantify visual distortion.Source: §1 · p. 1

2 EXPERIMENT SETUP

• Table 1 compares tuned PGD, I-FGM, C&W, and EAD across confidence levels, reporting ASR and distortion metrics averaged over successful examples.ASR means attack success rate (%).Source: Table 1 · p. 2
• The Madry Defense Model is evaluated on MNIST, where it was trained against perturbations no larger than ε = 0.3.The study notes that visually imperceptible examples are difficult to generate on simple grayscale MNIST.Source: §2 · pp. 1–2
• Attacks are evaluated for targeted and non-targeted transfer from either one undefended model or an ensemble of three naturally trained models.The authors expect examples that remain adversarial across multiple models to be more likely to transfer to unseen models.Source: §2 · p. 2
• EAD combines L1 and L2 penalties, with β controlling their trade-off, and κ setting the target-class confidence margin.The β = 0 case is equivalent to the C&W L2 attack.Source: §2 · p. 2
• PGD and I-FGM are compared as L∞ attacks, with ε tuned by grid search; I-FGM denotes PGD without random starts.FGM uses the sign of the input gradient of the training loss, and I-FGM iterates it with clipping to an ε-ball.Source: §2 · p. 2

3 EXPERIMENT RESULTS

• The highest EAD ASR in both targeted and non-targeted cases occurs at β = 0.01, the largest β tested.The authors interpret the trend as indicating the importance of minimizing L1 distortion for transferability.Source: §3 · p. 2
• Increasing β improves ASR more significantly at lower κ, which the authors link to minimizing L1 distortion for transferable examples with minimal visual distortion.PGD and I-FGM were tuned over ε values from 0.1 to 1.0 in increments of 0.1.Source: §3 · p. 3
• EAD outperforms C&W at every κ in both targeted and non-targeted settings; at targeted κ = 50, EAD also has lower average L2 distortion.The comparison uses EAD and C&W at β = 0.01, alongside PGD and I-FGM settings selected for their highest ASR.Source: §3 · p. 3
• At targeted κ = 30, EAD outperforms PGD and I-FGM with much lower L1 and L2 distortion.For the non-targeted case, PGD and I-FGM achieve higher ASR than EAD, but their larger distortions indicate perceptible perturbations.Source: §3 · p. 3

4 VISUAL COMPARISON

• PGD produces visually apparent noise even at ε = 0.3, where attack performance is weak.The figure compares non-targeted EAD and PGD examples with similar average L∞ distortion.Source: §4 · p. 3, Figure 1 · p. 3
• At average L∞ distortion 0.8, EAD’s elastic-net minimization yields less visual distortion than PGD, challenging L∞ as a sole measure of visual distortion.The comparison uses EAD with β = 0.01 and κ = 10 and PGD with ε = 0.8.Source: §4 · p. 3

5 CONCLUSION

• EAD outperforms PGD in targeted transfer and generates less visually distorted transferable examples at comparable L∞ distortion.The authors attribute the visual improvement to reduced L1 and L2 distortion.Source: §5 · p. 3
• The results demonstrate EAD's transferability capabilities.Source: §5 · p. 3
• The authors suggest incorporating L1 or L2 alongside L∞ for possible adversarial-training and adversary-detection defenses.Source: §5 · p. 3

6.1 RETRAINING THE MADRY DEFENSE MODEL (TABLE 2)

• The Madry Defense Model could not be successfully adversarially trained with PGD when ε exceeded 0.3.The authors note that distortion at such large ε is clearly perceptible.Source: §6.1 · p. 5

6.2 ATTACK DETAILS

• Targeted attacks use x0 and x for the original and adversarial examples, respectively, and t for the target class.Source: §6.2 · p. 5

6.2.1 EAD

• EAD combines L1 and L2 penalties, with β controlling their trade-off and κ setting the target-class margin; β = 0 recovers the C&W L2 attack.The implementation uses 9 binary-search steps for c, 1,000 iterations per step, and projected FISTA for EAD.Source: §6.2.1 · p. 5

6.2.2 PGD

• PGD is treated as projected gradient descent on negative loss; the experiments use 40 steps, with step size ε/40 without random starts and 2ε/40 with them.The EAD and C&W optimizers differ: EAD uses projected FISTA, while C&W uses ADAM.Source: §6.2.2 · p. 5, §6.2.1 · p. 5

6.3 PGD ADVERSARIAL EXAMPLES (FIGURE 2)

• As ε increases, PGD adds more noise to all pixels in the image.Source: §6.3 · p. 6
• At ε = 0.3, the PGD example is visibly noisy but does not successfully transfer.Source: §6.3 · p. 6
• At higher ε, the example appears to change a handwritten ‘1’ into a ‘3’, though the authors say the result is unclear under such large visual distortion.Source: §6.3 · p. 6

6.4 VISUAL COMPARISON IN THE TARGETED CASE (FIGURE 3)

• EAD at β = 0.01 and κ = 10 has average L∞ distortion 0.876, and is compared with PGD at ε = 0.8 and ε = 0.9.Source: §6.4 · p. 6
• In the targeted case, further distortion is needed to generate transferable adversarial examples.Source: §6.4 · p. 6
• Despite the similar average L∞ distortion, EAD's elastic-net minimization visibly improves example quality over PGD.Source: §6.4 · p. 6

6.5 SINGLE MODEL RESULTS (TABLE 3)

• EAD outperforms C&W at every κ; at targeted κ=50, both outperform PGD and I-FGM with much lower L1 and L2 distortion.At κ≥50, EAD also has lower average L2 distortion than C&W.Source: §6.5 · p. 6
• In non-targeted attacks, PGD and I-FGM exceed EAD in ASR, but their large L1 and L2 distortions are visually perceptible.Source: §6.5 · p. 6

6.6 BETA SEARCH FOR EAD (TABLES 4 AND 5)

• At optimal κ, β tuning has little effect on ASR, but at lower κ—where visual quality matters—increasing β significantly improves ASR.Source: §6.6 · p. 6

• PGD and I-FGM can induce misclassification with large visual distortion, but versus EAD they are less transferable and more perceptible, with L1 and L2 rising sharply at high ε.Source: §6.7 · p. 6, Figure 4 · p. 9, Figure 5 · p. 9
• PGD adversarial retraining does not successfully train the Madry Defense Model when ε>0.3.Source: Table 2 · p. 7
• In the Figure 2 PGD sweep, examples with ε≤0.3 fail to transfer to the Madry Defense Model.Source: Figure 2 · p. 7
• Figure 3 compares targeted EAD and PGD examples at similar average L∞ distortion; EAD's average L∞ is 0.876, alongside PGD examples at ε=0.8 and ε=0.9.Source: Figure 3 · p. 7