Source-linked AI summary
Generalizing to Unseen Domains via Adversarial Data Augmentation
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, Silvio Savarese
TL;DR
The paper addresses generalization to unseen domains when only one source distribution is available and target data are inaccessible. It formulates robustness around the source, generates adversarial target-like examples through iterative augmentation, and reports improved performance across unknown domains in digit recognition and semantic segmentation. For classification, it uses an ensemble because the target shift magnitude is unknown.
Problem
The paper addresses learning models that perform well on populations different from a single training distribution without access to unknown target-distribution data.
Method
The method iteratively augments the source dataset with adversarial examples from fictitious hard target distributions and trains models across different shift distances.
Results
The method improves performance across a priori unknown target distributions on digit recognition and cross-season or cross-weather semantic segmentation tasks.
Takeaways & Limitations
The approach provides broad generalization behavior from a single source domain, without requiring samples from multiple training domains.
Takeaways & Limitations
The test-time heuristics for selecting among ensemble models cannot be applied straightforwardly to semantic segmentation, and extending them remains future work.
Abstract
from arXiv · showhide
We are concerned with learning models that generalize well to different \emph{unseen} domains. We consider a worst-case formulation over data distributions that are near the source domain in the feature space. Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model. We show that our iterative scheme is an adaptive data augmentation method where we append adversarial examples at each iteration. For softmax losses, we show that our method is a data-dependent regularization scheme that behaves differently from classical regularizers that regularize towards zero (e.g., ridge or lasso). On digit recognition and semantic segmentation tasks, our method learns models improve performance across a range of a priori unknown target domains.
1 Introduction
The paper targets generalization from one source distribution to unseen populations without target-domain data. It formulates robust learning around the source and uses adversarial augmentation, with experiments showing broader performance across unknown domains.
- Models that validate well on source-like data can perform poorly on populations different from the training distribution.
- The setting differs from domain adaptation because training uses a single source domain and no samples from fixed target distributions.
- The method solves a worst-case problem over distributions near the source, using Wasserstein distance in semantic space to represent realistic covariate shifts.
- The iterative procedure adds adversarially perturbed samples and, for softmax losses, regularizes toward parameters associated with the true label rather than zero.
- The approach learns models that improve performance across a priori unknown target distributions in digit recognition and cross-season or cross-weather semantic segmentation.
2 Method
The method measures domain proximity through learned feature representations and alternates between generating worst-case examples and updating the model on the augmented data. Because the shift magnitude is unknown, classification uses an ensemble spanning multiple distance levels.
- The semantic-space distance is defined through transportation costs computed from the network’s last hidden-layer representation.
- Training alternates a maximization phase that generates adversarial examples with a minimization phase that updates model parameters on the augmented dataset.
- Because the covariate-shift magnitude is unknown, the method trains models for different γ values and selects a classification model using the greatest softmax score.
3 Theoretical Motivation
The paper interprets its adversarial augmentation procedure as adaptive data augmentation and, for softmax classification, as data-dependent regularization. The regularizer pulls the model toward the classifier associated with the true label rather than toward zero.
- The maximization phase augments the dataset with adversarially perturbed samples generated under the current model.These samples are drawn from fictitious hard target distributions and can be interpreted through a semantic-space perturbation.
- The augmented semantic point corresponds to a Tikhonov-regularized Newton step on the loss at the current model.The gradient-ascent updates approximate this step without directly computing backsolves on a large dense matrix.
- Under sufficient smoothness and γ > L1, Theorem 1 provides a bound for the robust surrogate when the feature map image is Rp.The stated assumptions require Lipschitz control of the loss, gradient, and Hessian in semantic space.
- 3.2 Data-Dependent Regularization: For softmax classification, the robust surrogate is an approximate regularizer on the classification-layer weights.The analysis applies to the m-class softmax loss and assumes γ exceeds a norm-dependent quantity L(θ).
- 3.2 Data-Dependent Regularization: The regularization term minimizes the distance between the average estimated linear classifier and the classifier corresponding to the true label.Unlike ridge or lasso, it penalizes deviations from the true-label parameter vector rather than deviations toward zero.
4 Experiments
The experiments evaluate the method on digit classification and semantic segmentation with target domains unknown during training. Across these settings, the method generally improves out-of-domain performance, with limitations for USPS and test-time model selection in segmentation.
- Experimental setup: The evaluation covers digit classification and semantic segmentation, with models trained on one source domain and tested on other domains.Digit models use MNIST and target datasets SVHN, MNIST-M, SYN, and USPS; segmentation models use SYNTHIA scene and condition splits.
- Digit classification: Semantic-space constraints improve unseen-domain performance compared with pixel-space constraints, and the method outperforms ERM across tested γ values on SVHN, MNIST-M, and SYN.The comparison uses K = 1 for the space constraint and K = 2 for varying γ values.
- Digit classification: With γ = 1.0, the method improves performance on SVHN, MNIST-M, and SYN, statistically outperforming both ERM and Dropout.The experiments vary the number of iterations K and compare against ERM and Dropout.
- Digit classification: Combining the method with ridge regularization produces additive benefits in the reported digit experiments.The comparison is between ridge alone and ridge combined with the method using K = 1 and γ = 1.0.
- Digit classification: An ensemble using models trained with different γ values improves accuracy across testing scenarios, with out-of-sample performance increasing as K becomes larger.Softmax confidences select the model at test time; USPS shows no improvement in the ensemble setting.
- Semantic scene segmentation: In semantic segmentation, the method mostly generalizes better to unknown distributions and always significantly outperforms the baseline for models trained on Night scenarios.The comparison reports average mIoU across test conditions for Highway, New York-like City, and Old European Town splits.
5 Conclusions and Future Work
The paper presents adversarial data augmentation and an ensemble method for broad generalization across unseen domains. It reports broad generalization on digit recognition and cross-season and cross-weather semantic segmentation, while identifying extensions to ensemble decision rules and theoretical analysis as future work.
- The paper studies adversarial data augmentation and defines an ensemble method for classification to generalize across unseen data distributions.
- Experimental results show broad generalization on digit recognition and cross-season and cross-weather semantic segmentation tasks.
- Future Work: The proposed ensemble heuristics apply only to classification, motivating future decision rules for broader task classes including semantic segmentation.
- Future Work: Quantifying the behavior of the paper’s data-dependent regularization schemes remains an open theoretical question.
A.1 Proof of Theorem 1
The proof establishes the stated result through assumptions on the feature map and loss curvature, approximation bounds, and a sequence of inequalities leading to the final conclusion.
- The proof fixes θ, x0, y0, and z0 = g(θf; x0), then uses the assumption Im(g(θf; ·)) = R^p.
- The first- and second-order approximations of the loss around z = z0 are introduced to support the analysis.
- Because the loss Hessian is bounded by L1 < γ, the Newton-based point attains the maximum in the relevant problem.
- The eigenvalue bound λmin(−∇zzh(z)) ≥ γ − L1 supplies the curvature condition used in the proof.
- The proof combines approximation bounds, Lemma 2, and bounds on the distance between z0 and the Newton point before reaching the final result.
- The argument concludes after substituting the preceding inequalities into the bound.
A.2 Proof of Lemma 3
The lemma proves stability of an approximate minimizer by applying a perturbation result to strongly convex and Lipschitz-related functions.
- Lemma 4 states that under a second-order growth condition, an approximate minimizer remains controlled when the objective is perturbed by a locally Lipschitz difference.
- The proof applies Lemma 4 by setting f0 to a γ-strongly convex function and f1 to the negative loss plus a quadratic penalty.
- The difference f0 − f1 is shown to be 2L0-Lipschitz, completing the application of Lemma 4.
A.3 Proof of Theorem 2
The proof bounds the gap between the adversarial objective and the loss using Taylor expansions and Lipschitz gradients, then establishes the required gradient regularity.
- The proof introduces notation for the feature representation and loss, with z = g(θ, ·) in R^p.
- A key claim bounds φγ(θ; (z, y)) − ℓ(θ; (z, y)) using the gradient norm and the condition γ > L.
- Taylor’s theorem provides matching upper and lower bounds involving 1/2(γ − L) and 1/2(γ + L) times the squared gradient norm.
- Combining the two bounds proves the claim, after which the proof reduces to showing that the loss gradient is L-Lipschitz.
- Figure 3 compares ADDA results across MNIST-M, SYN, and USPS while plotting the number of target samples on the x-axis.
B Additional Experimental Results
The experiments evaluate Algorithm 1 on digit recognition and semantic segmentation under cross-domain testing. Comparisons include ERM baselines and, for digit recognition, the unsupervised domain adaptation method ADDA.
- Semantic segmentation: Semantic segmentation models are trained on Highway or New York-like City and tested across New York-like City, Old European Town, and Highway.The comparison uses ERM rows and Ours rows with K = 1 and γ = 1.0.
- Comparison with ADDA: On MNIST →USPS, ADDA clearly outperforms the proposed method, while on MNIST →MNIST-M the proposed method is slightly lower than ADDA.The comparison is not fair because ADDA uses target data during training whereas the proposed method does not.
- Comparison with ADDA: On MNIST →SYN, the proposed method outperforms ADDA despite ADDA having access to many target-domain samples.MNIST →SVHN results are not provided for ADDA because it would not converge on that split.
MNIST-M
Table 2 reports semantic-segmentation mIoUs on the SYNTHIA dataset, comparing ERM with the proposed method across training and test sets.
- MNIST-M: Table 2 organizes SYNTHIA results by training set, method, and test set, reporting mIoUs for ERM and Ours.The Ours configuration uses K = 1 and γ = 1.0.