Source-linked AI summary
Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
Long Zhao, Ting Liu, Xi Peng, Dimitris Metaxas
TL;DR
The paper addresses the difficulty of generating hard adversarial perturbations for robustness under data shifts and corruptions. It derives a maximum-entropy regularizer from the Information Bottleneck principle, achieving state-of-the-art performance across three benchmarks, including 5% and 4% accuracy gains on CIFAR-C datasets.
Problem
Adversarial data augmentation needs effective fictitious target distributions containing hard perturbations substantially different from source data, because real-world shifts and corruptions degrade model performance.
Method
The method derives a maximum-entropy regularizer from the Information Bottleneck principle to generate perturbations that enlarge the current model’s predictive uncertainty.
Results
The method achieves state-of-the-art performance on MNIST, PACS, and CIFAR-10/100-C, improving accuracy by 5% on CIFAR-10-C and 4% on CIFAR-100-C.
Takeaways & Limitations
Maximum-entropy regularization produces hard adversarial perturbations that improve robustness to large domain shifts and corruptions encountered during deployment.
Takeaways & Limitations
Exact computation of the Information Bottleneck compression term is intractable for high-dimensional data and therefore requires approximation.
Abstract
from arXiv · showhide
Adversarial data augmentation has shown promise for training robust deep neural networks against unforeseen data shifts or corruptions. However, it is difficult to define heuristics to generate effective fictitious target distributions containing "hard" adversarial perturbations that are largely different from the source distribution. In this paper, we propose a novel and effective regularization term for adversarial data augmentation. We theoretically derive it from the information bottleneck principle, which results in a maximum-entropy formulation. Intuitively, this regularization term encourages perturbing the underlying source distribution to enlarge predictive uncertainty of the current model, so that the generated "hard" adversarial perturbations can improve the model robustness during training. Experimental results on three standard benchmarks demonstrate that our method consistently outperforms the existing state of the art by a statistically significant margin.
1 Introduction
The paper addresses the difficulty of generating hard adversarial perturbations for unseen distribution shifts. It proposes an information-theoretic regularizer that uses maximum entropy to improve adversarial data augmentation.
- Motivation: Distribution mismatches, corruptions, and adversarial attacks can substantially degrade deep neural network performance.Adversarial data augmentation generates fictitious target distributions to resemble unforeseen shifts and improve robustness during training.
- Motivation: Existing adversarial data augmentation methods make it difficult to define effective fictitious target distributions containing hard perturbations far from the source distribution.
- Motivation: The method regularizes adversarial data augmentation using the Information Bottleneck principle, whose effectiveness in this setting remains unclear.
- Approach: Maximizing mutual information between inputs and latent representations is used to produce hard fictitious target domains shifted from the source domain.Direct mutual-information optimization is challenging because mutual information is intractable.
- Contribution: The paper derives a maximum-entropy regularizer that bounds the Information Bottleneck objective and reports consistent, statistically significant improvements on three standard benchmarks.The regularizer is described as requiring only one line of code with minor computational cost.
2 Background and Related Work
The paper situates its method in domain generalization and adversarial data augmentation, formalizing robustness to distributions near a single source domain. It uses latent-space transport costs and a Lagrangian formulation to make the worst-case objective tractable.
- Information Bottleneck: The Information Bottleneck Lagrangian minimizes I(X; Z) − λI(Y; Z), trading off input compression against predictive information.The compression term is associated with robustness, generalization error, and out-of-distribution detection.
- Domain Generalization: Domain generalization seeks features that transfer to unseen domains, while single domain generalization learns from only one source domain.
- Adversarial Data Augmentation: Adversarial data augmentation trains on fictitious target distributions generated around a source distribution to address deployment under unforeseen shifts.
- Adversarial Data Augmentation: The worst-case objective considers distributions within a distance ρ of the source domain, with network loss evaluated using classifier parameters and latent representations.
- Robust Optimization: The latent-space Wasserstein metric assigns transport cost based on representation distance while making label changes infinitely costly.A Lagrangian relaxation with penalty parameter γ is used because the arbitrary-radius formulation is intractable for deep networks.
3 Methodology
The method incorporates the Information Bottleneck principle into adversarial data augmentation by replacing intractable mutual-information maximization with a tractable maximum-entropy objective. Alternating maximization and minimization phases generate adversarial perturbations that enlarge predictive uncertainty and are then used to update the model.
- Information Bottleneck formulation: The supervised Information Bottleneck loss combines cross-entropy with a compression penalty βI(X; Z).This adapts the IB Lagrangian for classification by replacing the prediction-information constraint with supervised risk.
- Adversarial training procedure: Training alternates K maximization and minimization phases, generating adversarial samples under a distribution-shift constraint and updating parameters with the IB loss.ME-ADA solves the maximization phase, while stochastic gradient descent updates the robust surrogate loss in the minimization phase.
- Maximum-entropy regularization: Because I(X; Z) is intractable for high-dimensional data, the method uses prediction entropy H(Ŷ) as a tractable lower bound during adversarial augmentation.For deterministic networks with fixed parameters, I(X; Z) ≥ I(X; Ŷ) = H(Ŷ).
- Maximum-entropy regularization: Maximum-entropy regularization produces perturbations that cannot be effectively compressed by the current model and may represent large domain shifts.The formulation enlarges the current model’s predictive uncertainty during the maximization phase.
- Empirical implementation: The entropy term is computed from model softmax outputs, making the maximization objective efficient to implement.The empirical formulation estimates expected prediction entropy from source-domain samples.
- Non-deterministic conditions: The maximum-entropy formulation extends approximately to stochastic models when their predictions are ε-close to deterministic, yielding H(Ŷ|X) ≈ 0.The bound’s complexity is mainly controlled by the finite label-set size, and it does not depend on the input-space size.
4 Experiments
Experiments evaluate ME-ADA across large domain shifts, PACS domain generalization, and common image corruptions. The method consistently achieves strong performance, including statistically significant gains over baselines and large corruption-robustness improvements.
- Experimental Settings: Experiments cover MNIST-to-domain-shift transfer, PACS single-domain generalization, and CIFAR-10-C/CIFAR-100-C corruption robustness.MNIST models are evaluated on SVHN, MNIST-M, SYN, and USPS; PACS uses four object-style domains; CIFAR-C includes fifteen corruption types at five severities.
- MNIST with Domain Shifts: On MNIST domain shifts, ME-ADA achieves the best classification performance, with BNNs further improving results through predictive-uncertainty estimation.USPS gains are less significant because USPS is highly similar to MNIST.
- MNIST with Domain Shifts: ME-ADA improves SVHN, MNIST-M, and SYN performance over ERM and [68] with statistically significant gains across different iteration counts K.Figure 1 averages results over ten runs while varying K and fixing γ and β.
- PACS: On PACS, ME-ADA performs best among methods without domain identification and approaches MetaReg, which uses domain identification.Its largest margin appears when sketch is the test domain, the most shifted and only colorless PACS domain.
- CIFAR-10 and CIFAR-100 with Corruptions: ME-ADA improves previous corruption-robustness state of the art by 5% accuracy on CIFAR-10-C and 4% on CIFAR-100-C across architectures and datasets.Gains are distributed across many corruptions, with approximately 25% accuracy improvements for glass blur, Gaussian, shot, and impulse noises.
- CIFAR-10 and CIFAR-100 with Corruptions: Figure 2 compares ERM with ME-ADA across fifteen CIFAR-10-C corruption types using WideResNet (40-2), averaging over five corruption strengths.The method shows small performance drops only for fog, brightness, and contrast.
5 Conclusion
The paper introduces maximum-entropy regularization for adversarial data augmentation to generate hard perturbations that enlarge predictive uncertainty. It reports state-of-the-art robustness across several domain-shift and corruption benchmarks, while noting that the method does not directly extend to regression.
- Conclusion: Maximum-entropy regularization produces hard adversarial perturbations that enlarge the current model’s predictive uncertainty during augmentation.The resulting fictitious target distributions support training for large domain shifts and corruptions.
- Conclusion: The method achieves state-of-the-art performance on MNIST, PACS, and CIFAR-10/100-C and is extremely simple to implement.
- Conclusion: The method cannot be directly applied to regression because its maximum-entropy lower bound remains difficult to compute.The paper suggests exploring alternative information measurements in future work.
Broader Impact
The method produces fictitious target domains substantially shifted from the source data to reduce bias in object-classification models, while introducing additional task-specific hyperparameters.
- The method generates efficient fictitious target domains that are largely shifted from the source training dataset.
- Training on these adversarial domains is intended to make object-classification models less biased.
- The approach introduces new hyperparameters that must be tuned for different tasks.
- Compared with methods requiring larger datasets for similar robustness, the proposed approach can reduce cross-domain data collection, development time, and related costs.
A.1.1 Proof of Proposition 3
The proof bounds entropy-estimation errors by combining deviation control from McDiarmid’s inequality with an expected-bias bound, yielding the stated proposition.
- The proof decomposes entropy-estimation error into deviation from the estimator’s expectation and expected estimation bias.
- McDiarmid’s inequality provides a high-probability bound on deviations of the entropy estimator from its expected value.
- Changing one sample alters two joint empirical probabilities by 1/m and changes the relevant marginal estimates by at most 1/m.
- The entropy estimator changes by at most |T| log(m)/m under a single-sample replacement.
- A plug-in entropy bias bound upper-bounds the expected bias by (|T| −1)/m.
- Combining the deviation and bias bounds gives the proposition’s entropy-estimation bound.
A.1.2 Proof of Corollary 1
The corollary follows by applying an entropy-continuity bound to distributions sharing the same input marginal, then using determinism to set conditional entropy to zero.
- A continuity lemma bounds the difference between conditional entropies of two joint distributions with identical Z marginals and bounded L1 distance.
- The bound controls |H(p(Y|X)) − H(˜p(Y|X))| using the distributions’ L1 distance and the finite outcome set size.
- For deterministic Y = f(X), the conditional entropy H(p(Y|X)) equals zero, which completes the corollary’s bound.
A.2 Implementation Details
The implementation includes a Bayesian neural-network variant using variational inference, while experiments specify training settings across PACS and corruption benchmarks.
- A.2.1 BNN Variant: The BNN variant models network outputs as p(y|x, θ), with θ representing latent network parameters and D the training examples.
- A.2.1 BNN Variant: Variational inference approximates p(θ|D) with q(θ|ψ) by minimizing their KL divergence through variational free energy.
- A.2.1 BNN Variant: The variational objective is approximated using T Monte Carlo samples from the variational posterior.
- A.2.1 BNN Variant: The posterior uses a diagonal-covariance Gaussian parameterized by ψ = (µ, σ), with weights sampled by θ = µ + σ ◦ ϵ.
- A.2.1 BNN Variant: The prior is a zero-centered mixture of two Gaussians with −log σ1 = 0, −log σ2 = 6, and π = 0.25.
- A.2.2 PACS / A.2.3 CIFAR-10 and CIFAR-100: PACS experiments report target-domain settings, while CIFAR corruption experiments specify settings for multiple architectures and optimization procedures.