Source-linked AI summary
Multi-Source Domain Adaptation with Mixture of Experts
Jiang Guo, Darsh J Shah, Regina Barzilay
TL;DR
Multi-source unsupervised domain adaptation must use heterogeneous sources without losing source-specific information or inducing negative transfer. The paper combines source experts using an example-to-domain point-to-set metric learned through meta-training, and reports consistent gains across sentiment analysis and POS tagging, including robust handling of unrelated sources.
Problem
Existing adaptation methods commonly focus on a single source, while combining multiple heterogeneous sources can ignore distinct source–target relations and cause negative transfer.
Method
A mixture-of-experts model uses a Mahalanobis point-to-set metric to weight source predictors for each target example and learns the metric through unsupervised meta-training.
Results
The approach consistently outperforms the best single-source and unified multi-source baselines, achieving 7% relative error reduction on Amazon reviews and 13% on SANCL on average.
Takeaways & Limitations
The method can robustly handle negative transfer from unrelated sources while improving adaptation results on sentiment analysis and POS tagging.
Takeaways & Limitations
The method allows examples from one domain to attend to different sources because domains may contain multiple latent sub-domains.
Abstract
from arXiv · showhide
We propose a mixture-of-experts approach for unsupervised domain adaptation from multiple sources. The key idea is to explicitly capture the relationship between a target example and different source domains. This relationship, expressed by a point-to-set metric, determines how to combine predictors trained on various domains. The metric is learned in an unsupervised fashion using meta-training. Experimental results on sentiment analysis and part-of-speech tagging demonstrate that our approach consistently outperforms multiple baselines and can robustly handle negative transfer.
1 Introduction
The paper addresses multi-source unsupervised domain adaptation by modeling relationships between individual source domains and target examples. Its mixture-of-experts model learns source-specific weighting through an unsupervised point-to-set metric and improves results across sentiment analysis and POS tagging.
- Motivation: Multiple complementary sources can better fit target distributions than a single source, especially when no individual source matches the target well.Product-review domains may combine heterogeneous subdomains that cannot be perfectly aligned to one source.
- Motivation: Combining sources into one domain ignores distinct source–target relationships and may wash out informative domain characteristics, causing negative transfer.
- Approach: The proposed mixture-of-experts model weights source predictors for each target example according to its proximity to each source domain.The weights represent the modeled relationship between the example and individual source domains.
- Approach: The point-to-set metric uses Mahalanobis distance in hidden representation space and is learned jointly with the model through meta-training on source-derived adaptation tasks.Adversarial learning additionally aligns source and target encoding spaces.
- Results: 7% relative error reduction on Amazon reviews and 13% on SANCL were achieved on average over the evaluated adaptation settings.The approach consistently improved over the best single-source and unified multi-source models.
2 Related Work
Prior multi-source adaptation methods differ in how they represent source–target relations, including equal weighting, supervised or predefined similarity, and example-to-domain selection. The paper distinguishes its approach by learning example-specific source relations without additional target supervision while retaining individual source characteristics.
- Unsupervised Domain Adaptation: Unlike prior representation-learning approaches, this method both learns a shared target-generalizing space and captures informative characteristics of individual source domains.
- Multi-Source Domain Adaptation: Earlier methods may assume all source domains are equally important to the target domain.
- Multi-Source Domain Adaptation: Other approaches learn global domain similarity from labeled data or rely on predefined similarity measures.
- Multi-Source Domain Adaptation: Example-to-domain similarity has also been used to select source data for training a single classifier, while related attention-based work learns relations with limited labeled target data.
3 Methodology
The method combines source-specific experts through example-to-domain confidence weights learned from a point-to-set metric, while jointly learning transferable and source-specific representations. Unsupervised meta-training treats each source in turn as a meta-target, and optional MMD aligns target and source representations.
- Architecture: The model uses a shared encoder, source-specific classifiers, a metric component, and an optional adversary to produce weighted multi-source predictions.Each input is encoded once, passed to every source classifier, and combined using metric-derived weights.
- Mixture of Experts: A point-to-set Mahalanobis metric measures an example’s relation to each source domain in hidden representation space and determines expert confidence weights.The metric uses a learned positive semidefinite matrix, parameterized as M = UU⊤, and confidence scores are normalized into α.
- Mixture of Experts: The metric accounts for task structure: binary classification uses category-distance differences, whereas sequence tagging computes token-level distances directly.For binary classification, distance to a domain mean can be counter-intuitive near the decision boundary; sequence tagging has more complex boundaries and imbalanced labels.
- Meta-Training: Meta-training learns the metric without target labels by treating each source domain as a meta-target and the remaining domains as meta-sources.The learned metric is applied over all source domains for each target example at testing time.
- Joint Learning: Training jointly optimizes the MoE and supervised multi-task objectives, with MMD optionally minimizing divergence between target and source marginal representations.The encoder is shared across supervised source objectives, and the adversarial term is omitted when unlabeled target data is unavailable.
- Joint Learning: The model regularizes α over all sources and benefits from allowing examples within one domain to attend to different sources, consistent with latent sub-domains.The authors report that a one-hot source-attention constraint is less beneficial in practice.
4 Experimental Setup
Experiments evaluate multi-source adaptation on sentiment classification and POS tagging, using established datasets, round-robin target selection, and comparisons against single-source, unified multi-source, and adversarial baselines.
- Datasets: Sentiment experiments use Amazon reviews datasets containing four domains: Books, DVDs, Electronics, and Kitchen appliances.Each domain provides 2,000 labeled training examples, split evenly between positive and negative reviews.
- Protocol: Each dataset selects target domains in round-robin fashion and uses cross-validation over source domains for hyper-parameter selection.In adversarial settings, the target training set supplies unlabeled target data; ZISER17 therefore uses a transductive setting.
- Datasets: POS tagging experiments use SANCL annotations from five web domains: Emails, Weblogs, Answers, Newsgroups, and Reviews.Newsgroups, Reviews, and Answers serve as target domains when both validation and test sets are available.
- Baselines: The comparisons include MoE, the best single-source model, and a unified multi-source model trained on combined source data.The unified multi-source model is a common strong baseline, while adversarial variants are denoted with ∗-A.
- Baselines: Adversarial CHEN12 experiments additionally compare against mSDA and MDAN, which use unlabeled target-domain data.MDAN extends domain-adversarial adaptation to multiple sources by selectively backpropagating domain-discrimination loss according to classification error.
- Implementation: Encoders vary by task and dataset: CHEN12 uses a 5,000-to-500 multilayer perceptron, ZISER17 uses a convolutional encoder, and POS tagging uses hierarchical BiLSTMs.Training uses Adam with weight decay, mini-batches of 32 samples per domain, tuned λ and η, and γ=1; MMD uses 19 RBF kernels.
5 Results
Across sentiment and POS tagging experiments, MoE consistently performs strongly and uses learned source relationships to improve robustness against negative transfer. Ablations and analyses support both selective source weighting and the value of entropy regularization.
- Sentiment analysis: MoE consistently achieves the best performance across Amazon review settings and tasks.
- Sentiment analysis: In the CHEN12 “B,D,K-E” negative-transfer scenario, MoE significantly outperforms the unified multi-source model.
- Adversarial adaptation: Adversarial loss generally improves baseline systems and often further improves MoE, but not in every task.The paper attributes some exceptions to more compact representations that make source-specific metric learning harder.
- Metric analysis: The α distributions are often skewed, indicating that MoE draws on a few informative source domains for each example.
- Metric analysis: A Kitchen review mentioning “charger” and “battery” is closer to Electronics, a relation captured by its α distribution.
- Ablation: Entropy regularization benefits MoE under both non-adversarial and adversarial settings on CHEN12 and ZISER17.
- Part-of-speech tagging: Adding Twitter causes negative transfer for the unified model, while MoE handles it robustly and can benefit from the additional source.
- Part-of-speech tagging: With increasing Twitter data, unified-model performance worsens for Answers and Reviews, whereas MoE remains stable and improves consistently.
6 Conclusion
The proposed approach is evaluated on POS tagging with multiple source domains and is reported to outperform baselines while handling negative transfer. Its learned metric reduces the influence of unrelated sources such as Twitter.
- Table 4 evaluates POS tagging on SANCL using Web, Emails, and Twitter as source domains.
- Table 5 reports the average metric-value distribution α across all tokens in the SANCL test set.
- Table 6 examines POS tagging accuracy as the amount of Twitter training data varies.
- The method consistently outperforms various baselines on sentiment classification and part-of-speech tagging.