Source-linked AI summary
Generalizing Across Domains via Cross-Gradient Training
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, Sunita Sarawagi
TL;DR
The paper addresses classification across unseen domains without target-domain adaptation, where relying only on input features can overfit observed domains. CROSSGRAD uses Bayesianly motivated, domain-guided cross-gradient augmentation, and experiments report consistent improvements over baseline and LABELGRAD, while DAN provides little improvement.
Problem
The problem is learning label classifiers that generalize from labeled multi-domain training data to unseen domains without labeled or unlabeled target-domain data.
Method
CROSSGRAD jointly trains label and domain classifiers, augmenting inputs with perturbations guided by each other’s loss gradients.
Results
Across four classification datasets, CROSSGRAD improves accuracy over the baseline, with a consistent advantage over LABELGRAD while DAN provides little improvement.
Takeaways & Limitations
Domain-guided augmentation can use partially correlated domain signals for domain generalization without explicit distributional assumptions about their effects on inputs.
Takeaways & Limitations
CROSSGRAD’s behavior for multidimensional, nonlinear domain-feature effects is difficult to diagnose, and its extension to more dimensions or truly categorical domains remains open.
Abstract
from arXiv · showhide
We present CROSSGRAD, a method to use multi-domain training data to learn a classifier that generalizes to new domains. CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain. Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training. In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains. We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances. CROSSGRAD parallelly trains a label and a domain classifier on examples perturbed by loss gradients of each other's objectives. This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions. Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and that (2) data augmentation is a more stable and accurate method than domain adversarial training.
1 INTRODUCTION
The paper studies classification that must generalize from labeled examples in several training domains to unseen domains without target-domain adaptation. CROSSGRAD addresses overfitting by using domain-guided input perturbations rather than removing domain signals.
- Existing methods commonly require labeled or unlabeled target-domain data and a separate adaptation step before prediction.
- The goal is accurate label prediction both within observed domains and in domains absent from training.
- Predicting labels without domain information is typically harder than predicting them conditioned on the domain.
- Prior work assumed an explicit domain geometry, whereas this paper relies on deep networks to discover implicit domain features.
- Conventional training can overfit observed domains, while domain-adversarial training may fail to safeguard the network as a whole.
- CROSSGRAD uses a Bayesianly motivated augmentation scheme in which label and domain objectives generate perturbations through each other’s input gradients.
2 RELATED WORK
Related approaches often learn domain-invariant representations or use generic adversarial perturbations, but these strategies can discard useful domain information or remain vulnerable to overfitting. CROSSGRAD instead models domain variation continuously and perturbs instances along domain-loss directions.
- Domain adaptation spans settings with two or multiple domains, labeled or unlabeled target data, paired examples, or attached domain features.
- Domain-adversarial networks learn hidden representations that make domain discrimination difficult, assuming domains are visible during training.
- Representation-based domain-generalization methods reduce domain dissimilarity or learn shared features across training domains.
- Such representations may ignore additional information that domain features provide about labels.
- DANs can be fooled by a representation that fits training domains while hiding domain information from the adversarial classifier.
- Generic adversarial training augments examples along classifier-loss gradients, whereas CROSSGRAD perturbs them along domain-loss gradients to model domain variation.
- CROSSGRAD’s Bayesian model differs from related work by representing domain variation continuously and projecting perturbations from that space onto instances.
3 OUR APPROACH
CROSSGRAD models labels, domains, and inputs with latent continuous domain features, then trains label and domain classifiers using cross-guided perturbations. Its alternating updates augment each classifier with examples perturbed by the other classifier’s loss gradient.
- Bayesian formulation: The model assumes label and domain are separate, weakly correlated tags whose values can be changed independently in principle.
- Bayesian formulation: A Bayesian network represents discrete label and domain variables, latent continuous domain features, and the observed input.
- Bayesian formulation: At inference, the method estimates the label posterior from the input while accounting for uncertainty about the unobserved domain.
- Bayesian formulation: Generalization assumes training domains provide adequate support for the latent continuous domain-feature distribution.
- Domain-guided augmentation: The method uses small input perturbations because imperfectly inferred domain features could otherwise alter the label.
- Domain-guided augmentation: A domain-feature extractor is trained to predict domain labels using a softmax classifier and cross-entropy domain loss.
- Domain-guided augmentation: CROSSGRAD generates domain perturbations along the input gradient of domain loss while training the domain extractor to resist label-induced shifts.
- Cross-gradient training: Alternating updates train both classifiers on original and cross-perturbed minibatch examples, weighting the augmented losses by α_l and α_d.
4 EXPERIMENTS
Experiments across four classification tasks and multiple architectures show that CROSSGRAD improves domain generalization, with domain-guided perturbations consistently outperforming alternatives. Analyses indicate that continuous domain embeddings support interpolation and hallucinated samples, while benefits are largest when training domains are scarce and do not cover test variation.
- Datasets and setup: The experiments covered character recognition across fonts, handwriting recognition across authors, rotated-MNIST recognition, and spoken-word recognition across users.Domains were fonts, writers, rotation angles, and speakers, respectively, with disjoint training, validation, and test domains.
- Overall comparison: CROSSGRAD improved accuracy on all four datasets, while DAN was worse than LABELGRAD and the gap between CROSSGRAD and LABELGRAD was consistent.The comparison included DAN, LABELGRAD, and a no-special-training baseline; the MNIST baselines were CCSA and D-MTAE.
- Changing model architecture: CROSSGRAD surpassed baseline accuracy and LABELGRAD even with a higher-capacity 2-block ResNet on Fonts and Handwriting.The ResNet model was significantly better than LeNet, but CROSSGRAD retained the advantage across both datasets.
- Why does CROSSGRAD work?: Domain embeddings for intermediate rotation angles lay between embeddings for neighboring angles, indicating that the domain classifier extracted a continuous representation from categorical domain labels.This pattern appeared for domains 30, 45, and 60, and again for 0, 15, and 30 degrees.
- When is domain generalization effective?: CROSSGRAD’s gains were largest with 40 training domains and disappeared beyond the baseline’s 88.3% accuracy when training used more than 1000 domains.As training data covered more domain variation, marginal gains decreased; the 1000-domain gap was not statistically significant.
- When is domain generalization effective?: The multidimensional, nonlinear role of domain features in determining inputs remained difficult to diagnose, so the mechanism analysis focused on a restricted one-dimensional rotation setting.The restricted setting treated MNIST rotation angle as a continuous domain variable.
- When is domain generalization effective?: On rotated MNIST, CROSSGRAD was beaten only for the extreme test angles M0 and M75, while intermediate angles benefited from interpolation through hallucinated domains.The domain-loss gradient did not point in the direction needed for the extreme endpoints.
5 CONCLUSION
CROSSGRAD treats domain and label signals symmetrically, using domain-guided perturbations to improve generalization without explicit distributional assumptions or target-domain adaptation data.
- CROSSGRAD trains label and domain predictors on examples perturbed by each other’s loss gradients.This creates domain-guided input augmentation while treating domain and label signals symmetrically.
- CROSSGRAD can use partially correlated domain and label signals without requiring explicit assumptions about how they affect inputs.
- CROSSGRAD performs best when training domains are scarce and do not directly cover test domains well.
- The method does not require labeled or unlabeled target-domain data, target-domain features, or a separate adaptation phase.
APPENDIX
The appendix derives how perturbations in learned domain features can be translated into input perturbations using the Jacobian and its transpose-based inverse-kinematics method.
- The derivation connects intuitive input perturbations with corresponding perturbations in the learned domain representation.
- Perturbing learned domain features produces an augmented input instance through a corresponding input-space update.
- The Jacobian relates changes in the input x to changes in the domain features ĝ.
- For non-square or low-rank Jacobians, the derivation uses a Jacobian transpose method to invert the feature-change relationship.The method recasts the problem as minimizing squared L2 error with gradient descent.
- An initial gradient-descent step changes x by ϵJ^T∆ĝ to affect a desired change in domain features.The Jacobian can be computed by back-propagation.