Source-linked AI summary
Understanding Measures of Uncertainty for Adversarial Example Detection
Lewis Smith, Yarin Gal
TL;DR
The paper asks which uncertainty measures best detect adversarial examples and whether common dropout approximations capture the relevant uncertainty. It compares these measures, visualizes their behavior, and proposes probabilistic ensemble extensions. Mutual information is the only measure reported to obtain a useful AUC on the cats-versus-dogs dataset, while the discussion concludes that better uncertainty can improve robustness without explicitly modeling adversarial examples.
Problem
The paper investigates which uncertainty measures are useful for detecting adversarial examples and whether dropout adequately captures model uncertainty.
Method
The paper compares uncertainty measures, visualizes them in a two-dimensional MNIST latent space, analyzes softmax variance and mutual information, and examines dropout failures.
Results
Only mutual information obtains a useful AUC on the cats-versus-dogs dataset; most other measures perform worse than random guessing.
Takeaways & Limitations
The results provide evidence that improving uncertainty estimates can improve robustness to adversarial examples without explicitly referencing the adversarial setting.
Takeaways & Limitations
Dropout alone is not a convincing defense: white-box attacks can still succeed while remaining imperceptible to humans.
Abstract
from arXiv · showhide
Measuring uncertainty is a promising technique for detecting adversarial examples, crafted inputs on which the model predicts an incorrect class with high confidence. But many measures of uncertainty exist, including predictive en- tropy and mutual information, each capturing different types of uncertainty. We study these measures, and shed light on why mutual information seems to be effective at the task of adversarial example detection. We highlight failure modes for MC dropout, a widely used approach for estimating uncertainty in deep models. This leads to an improved understanding of the drawbacks of current methods, and a proposal to improve the quality of uncertainty estimates using probabilistic model ensembles. We give illustrative experiments using MNIST to demonstrate the intuition underlying the different measures of uncertainty, as well as experiments on a real world Kaggle dogs vs cats classification dataset.
1 Introduction
Adversarial examples exploit neural networks’ vulnerability to small, confidence-preserving perturbations, motivating uncertainty-based detection. The paper compares uncertainty measures, examines dropout’s failures, and proposes improved uncertainty estimation.
- Small input perturbations can induce erroneous, over-confident classifications, threatening safety-critical and security-sensitive applications.
- Distance-based detection is difficult because pixel-wise distance poorly reflects perceptual similarity and density modeling does not scale easily to high-dimensional images.
- Bayesian model uncertainty is proposed as a proxy for distance from the image manifold, where uncertainty should increase away from training data.
- Existing work uses multiple uncertainty measures, but has mostly overlooked how the chosen measure affects adversarial-example detection.
- The paper relates softmax variance to mutual information, exposes dropout gaps, and studies adversarial and non-adversarial off-manifold images.
2 Background
The background distinguishes Bayesian, variational, and uncertainty concepts relevant to adversarial detection. It motivates mutual information because predictive entropy does not separate model uncertainty from data ambiguity.
- Bayesian neural networks treat weights as random variables with a prior and marginalize parameter uncertainty during inference.
- Dropout provides a computationally tractable variational approximation, commonly implemented by randomly zeroing network units.
- Variational inference replaces an intractable posterior with a simpler distribution and turns integration into an optimization problem.
- Epistemic uncertainty reflects limited knowledge, whereas aleatoric uncertainty reflects irreducible stochasticity or noisy data.
- Predictive entropy is unsatisfactory because it combines epistemic and aleatoric uncertainty, while mutual information can distinguish them.
- The paper analyzes uncertainty measures for adversarial detection, connects softmax variance with mutual information, and evaluates the resulting ideas empirically.
3 Understanding Measures of Uncertainty for Adversarial Example Detection
The paper distinguishes uncertainty measures for adversarial detection and explains why mutual information and softmax variance can be effective. It argues that predictive entropy may also respond to non-adversarial ambiguity, then empirically studies these claims.
- 3.1 Adversarial Examples and Uncertainty: Mutual information and predictive entropy should increase for inputs far from the natural-image manifold, but predictive entropy can also rise for inherently ambiguous inputs.Ambiguous inputs may have high predictive entropy but low mutual information, causing false detections despite being non-adversarial.
- 3.1 Adversarial Examples and Uncertainty: Adversarial crafting tends to minimize prediction entropy because attack objectives concentrate probability on a selected class.This follows from probability normalization and affects the uncertainty patterns of crafted examples.
- 3.2 Mutual Information and Softmax Variance: The softmax variance is the leading term in a Taylor-series expansion of mutual information, making it a principled approximation rather than merely an ad-hoc score.The first expansion term is identical to mean sample variance up to a multiplicative constant.
- 3.2 Mutual Information and Softmax Variance: This relationship could explain why softmax variance detects adversarial examples effectively, with mutual information increasing off-manifold but not on image interpolations.The variance follows similar trends, although predictive variance can also increase on interpolated images.
- Empirical Study: The paper empirically studies the uncertainty measures to demonstrate the proposed distinctions and relationships.The experiments are intended to test the preceding claims about uncertainty behavior and detection.
4 Empirical Evaluation
The experiments compare uncertainty measures as proxies for distance from the image manifold and evaluate their adversarial-detection behavior. Mutual information distinguishes on-manifold from off-manifold interpolations, but MC dropout can remain spuriously confident and ensemble methods only partially address this.
- Measures of Uncertainty on Image Interpolations: Mutual information distinguishes on-manifold latent interpolations from off-manifold image-space interpolations, whereas predictive entropy fails to do so.This supports using mutual information as a proxy for whether an image belongs to the learned manifold.
- Visualization in Latent Space and Dropout Failures: A two-dimensional variational-autoencoder latent map visualizes uncertainty measures across decoded MNIST inputs.The method projects uncertainty into a low-dimensional space to compare predictive entropy, mutual information, and dropout behavior.
- Visualization in Latent Space and Dropout Failures: Predictive entropy is high between ambiguous classes, while mutual information remains low near familiar data and can peak on out-of-sample interpolations.Entropy therefore does not reliably distinguish inherent ambiguity from unfamiliar inputs in these visualizations.
- Visualization in Latent Space and Dropout Failures: MC dropout produces confident predictions in latent-space holes for decoder-generated nonsense, indicating that it captures local posterior behavior rather than the full posterior.The approximation can underestimate posterior uncertainty by fitting a local mode.
- Visualization in Latent Space and Dropout Failures: An ensemble of dropout models mitigates some spuriously confident latent-space regions, but this depends on local modes being represented with roughly equal likelihood and probability.The authors do not guarantee that such ensembles capture the true posterior.
- Evaluation on Cats and Dogs Dataset: On the cats-versus-dogs dataset, only mutual information achieves a useful adversarial-example AUC, while most other measures perform worse than random guessing.The authors attribute entropy’s high false-positive rate to many naturally ambiguous examples; its low AUC suggests greater confidence on adversarial than natural examples.
- Evaluation on Cats and Dogs Dataset: Monte Carlo estimation lowers this model’s accuracy relative to point estimates despite producing sensible uncertainty estimates, possibly because dropout is excessive in transfer-learned layers.The authors describe this as a quirk of the particular model and training setup.
5 Discussion & Conclusion
The paper argues that improving uncertainty estimation can improve robustness to adversarial examples without explicitly modeling the adversarial setting. However, dropout is only a limited defense because white-box attacks can still succeed imperceptibly.
- The results support viewing adversarial examples as potentially arising from training and inference procedures rather than being intrinsic to neural-network function classes.
- Better uncertainty estimates improve robustness to adversarial examples without requiring assumptions about their distribution.
- Dropout networks are more difficult to attack than deterministic networks, but attacks can still succeed while remaining imperceptible in the investigated white-box setting.
- The authors attribute dropout’s limited protection to its crude approximation and significant underestimation of uncertainty.
- Improving scalable uncertainty estimation beyond dropout is identified as an important direction for future research.