Source-linked AI summary
Unsupervised Domain Adaptation by Backpropagation
Yaroslav Ganin, Victor Lempitsky
TL;DR
Domain adaptation is needed when labeled training data come from a distribution shifted from the unlabeled target data. This paper uses feature alignment through a gradient reversal layer trained by standard backpropagation, achieving successful adaptation across several challenging image-domain shifts while also showing a failure case.
Problem
Deep models require abundant labeled data, but available labeled source data may differ in distribution from the real-world target data encountered at test time.
Method
The approach jointly learns source-label-discriminative and domain-invariant features by combining label and domain classifiers with a gradient reversal layer in a standard feed-forward network.
Results
The method successfully adapts across challenging synthetic-to-real and digit-domain shifts, covering two thirds of the source-only-to-target-supervised gap in one experiment.
Takeaways & Limitations
Unsupervised domain adaptation can be incorporated into deep feed-forward models using standard backpropagation and implemented with common deep-learning packages.
Takeaways & Limitations
The formulation assumes similar but different source and target distributions and requires large labeled source and unlabeled target samples during training.
Abstract
from arXiv · showhide
Top-performing deep architectures are trained on massive amounts of labeled data. In the absence of labeled data for a certain task, domain adaptation often provides an attractive option given that labeled data of similar nature but from a different domain (e.g. synthetic images) are available. Here, we propose a new approach to domain adaptation in deep architectures that can be trained on large amount of labeled data from the source domain and large amount of unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of "deep" features that are (i) discriminative for the main learning task on the source domain and (ii) invariant with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a simple new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation. Overall, the approach can be implemented with little effort using any of the deep-learning packages. The method performs very well in a series of image classification experiments, achieving adaptation effect in the presence of big domain shifts and outperforming previous state-of-the-art on Office datasets.
1. Introduction
The paper addresses deep learning when labeled data are scarce but abundant training data come from a shifted distribution, such as synthetic data versus real test-time data. It proposes learning features that support label prediction while becoming invariant to the source–target domain shift.
- Deep architectures achieve major advances across machine-learning tasks, but their strongest performance depends on large amounts of labeled training data.
- Synthetic or semi-synthetic data can provide abundant, fully labeled training sets, yet their distribution differs from real data encountered at test time.
- The approach jointly uses a label predictor and a domain classifier, optimizing the feature mapping to reduce label loss while increasing domain-classification loss.This optimization encourages domain-invariant features to emerge during training.
2. Related work
Related work includes unsupervised domain adaptation methods that match source and target feature distributions, as well as methods that gradually transition the training distribution from source to target. A deep variant performs this transition through layerwise training of sequential deep autoencoders while progressively replacing source samples with target samples.
- Feature-distribution matching: Unsupervised domain adaptation methods commonly match feature distributions between source and target domains.Some methods reweight or select source-domain samples, while others seek an explicit feature-space transformation.
- Gradual domain transition: Other approaches gradually transition from source to target domains by changing the training distribution.This family includes methods that progressively alter the data used for training.
- Gradual domain transition: A deep approach performs this transition by layerwise training a sequence of deep autoencoders while gradually replacing source-domain samples with target-domain samples.The passage states that this improves over a similar method training a single deep autoencoder for the transition.
3. Deep Domain Adaptation
The method jointly learns source-discriminative and domain-invariant features by combining label prediction with adversarial domain classification. A gradient reversal layer makes this saddle-point training implementable through standard stochastic gradient descent and supports target-domain prediction.
- Problem setup: The model assumes similar but shifted, complex, unknown source and target distributions over inputs and labels, with classification labels drawn from a finite set.The approach is described as generic beyond finite-label classification, although the section analyzes classification problems with Y = {1, 2, . . . L}.
- Learning objective: Source label-loss minimization makes the learned features discriminative and preserves prediction performance for the feature extractor and label predictor.Only the source portion of the training set is annotated, while target-domain samples are unlabeled.
- Learning objective: At the saddle point, the domain classifier minimizes domain loss, while feature parameters minimize label loss and maximize domain loss; λ controls their trade-off.This adversarial optimization drives features toward domain invariance without sacrificing source-task discriminativeness.
- Gradient reversal: The gradient reversal layer multiplies the backpropagated feature gradient by −λ, reducing the proposed updates to standard SGD in a feed-forward model.Without the negative factor, SGD would instead make features dissimilar across domains to reduce domain-classification loss.
- Outcome: After training, the label predictor can classify samples from both the target and source domains using features that are simultaneously domain-invariant and discriminative.The resulting architecture consists of a feature extractor, label predictor, domain classifier, and gradient reversal connection.
- Theoretical interpretation: Reversed-gradient backpropagation decreases α(Gd), effectively reducing dHp∆Hp(S, T ) and improving the approximation of target error εT(Gy) by source error εS(Gy).This analysis assumes the domain-classifier family is rich enough to contain the symmetric-difference hypothesis set of the label-predictor family.
4. Experiments
The experiments evaluate the proposed adaptation method across digit, synthetic-to-real, and OFFICE domain shifts using source-labeled and target-unlabeled data. Results show successful feature alignment, improved target performance, and state-of-the-art accuracy on OFFICE.
- Training procedure: The model uses batches split evenly between labeled source samples and unlabeled target samples, while the adaptation factor λ increases from 0 to 1 during training.The schedule uses γ = 10 in all experiments and was not optimized or tweaked.
- Visualizations: Feature-distribution overlap in t-SNE visualizations corresponds strongly to successful target-domain classification adaptation.The visualizations color-code domains and compare feature distributions at different points in the network.
- MNIST → MNIST-M: The proposed method aligns features and successfully adapts from MNIST to MNIST-M despite backgrounds and strokes making the target domain distinct for a CNN trained on MNIST.The source-only model performs poorly, whereas the proposed approach aligns feature distributions and improves adaptation.
- OFFICE: On the standard OFFICE dataset, the proposed method outperforms competing approaches and establishes a new state-of-the-art result.OFFICE contains 2817 labeled images across 31 categories in its largest domain, motivating fine-tuning from ImageNet-pretrained CNNs.
- Synthetic-to-real adaptation: The method covers two thirds of the gap between source-only training and target-labeled training on the synthetic-numbers-to-SVHN setting, while SA yields no significant accuracy improvement.The domains remain distinct, particularly because SVHN contains structured background clutter.
5. Discussion
The paper proposes unsupervised domain adaptation for deep feed-forward architectures using annotated source-domain data and unannotated target-domain data, aligning feature distributions across domains.
- The approach enables large-scale adaptation using annotated source-domain data alongside unannotated target-domain data.
- Adaptation is achieved by aligning feature distributions across the two domains, following a principle used by prior shallow and deep domain-adaptation methods.
- The proposed approach differs from previous methods in how this cross-domain alignment is accomplished.
Appendix A. An alternative optimization approach
Appendix A presents an alternative two-loss optimization construction that produces the same updates as gradient reversal while allowing other domain-loss pairs. A binomial cross-entropy variant may yield stronger early gradients for dissimilar domains, but showed no significant experimental improvement.
- Alternative optimization construction: An alternative construction replaces the gradient reversal layer with two domain-classifier losses, L_d+ and L_d−, producing the same updates (4)–(6).Minimizing L_d+ improves domain discrimination, whereas minimizing L_d− favors distinct domains.
- Alternative optimization construction: The gradient reversal layer is the special loss pair (L_d, −λL_d), while other domain-loss pairs are also possible.Different parameters participate in optimizing different losses in this framework.
- Binomial cross-entropy variant: Swapping domain labels yields the adversarial binomial cross-entropy loss pair, which can produce stronger early gradients when domains are quite dissimilar.In the experiments, this loss choice produced no significant improvement.
Appendix B. CNN architectures
The experiments used four CNN architectures selected for MNIST, SVHN, SYN SINGS →GTSRB, and OFFICE domain settings. OFFICE adaptation used pre-trained AlexNet with a domain-classifier branch attached to fc7, while the branch design remained tunable.
- Architecture selection: Four CNN architectures were used across the experiments, with the first three shown in Figure 5.
- MNIST: The MNIST source domain used a smaller architecture inspired by classical LeNet-5.
- SVHN: SVHN experiments used an architecture adopted from Srivastava et al. (2014).
- SYN SINGS →GTSRB: The SYN SINGS →GTSRB setting started from the single-CNN baseline of Cires¸an et al. (2012).
- OFFICE: OFFICE experiments used pre-trained AlexNet from the Caffe-package, with an adaptation architecture identical to Tzeng et al. (2014).A 2-layer domain classifier, x →1024 →1024 →2, was attached to the 256-dimensional bottleneck of fc7.
- Architecture limitation: The domain-classifier branch was somewhat arbitrary, and tuning it might improve adaptation performance.
Appendix C. Training procedure
The training procedure uses stochastic gradient descent with momentum and a progress-dependent learning-rate annealing schedule. The SVHN architecture additionally uses dropout and ℓ2-norm restriction.
- Training uses stochastic gradient descent with 0.9 momentum and learning-rate annealing.
- The schedule uses training progress p from 0 to 1, with µ0 = 0.01, α = 10 and β = 0.75.The schedule was optimized to promote convergence and low error on the source domain.
- The SVHN architecture uses dropout and ℓ2-norm restriction during training.