Source-linked AI summary
Maximum Density Divergence for Domain Adaptation
Li Jingjing, Chen Erpeng, Ding Zhengming, Zhu Lei, Lu Ke, Shen Heng Tao
TL;DR
Domain adaptation must transfer knowledge from labeled source data to unlabeled target data despite distinctive distributions and unresolved alignment limitations in existing metrics and adversarial methods. The paper proposes MDD and integrates it into adversarial training as ATM, reporting improved performance across four benchmarks, including SVHN→MNIST.
Problem
Domain adaptation seeks to mitigate distribution shifts between labeled source and unlabeled target domains, while adversarial confusion alone may not guarantee domain alignment.
Method
ATM integrates Maximum Density Divergence, which minimizes inter-domain divergence and maximizes intra-class density, into adversarial domain adaptation.
Results
96.1% on SVHN→MNIST improved from 89.2%, and experiments on four benchmarks report significant advantages over previous state-of-the-art methods.
Takeaways & Limitations
MDD provides an additional distribution-divergence loss that can be incorporated into adversarial domain adaptation to alleviate its equilibrium challenge.
Takeaways & Limitations
ATM’s advantage is not quite clear on MNIST→USPS and USPS→MNIST because their distribution gap is relatively marginal.
Abstract
from arXiv · showhide
Unsupervised domain adaptation addresses the problem of transferring knowledge from a well-labeled source domain to an unlabeled target domain where the two domains have distinctive data distributions. Thus, the essence of domain adaptation is to mitigate the distribution divergence between the two domains. The state-of-the-art methods practice this very idea by either conducting adversarial training or minimizing a metric which defines the distribution gaps. In this paper, we propose a new domain adaptation method named Adversarial Tight Match (ATM) which enjoys the benefits of both adversarial training and metric learning. Specifically, at first, we propose a novel distance loss, named Maximum Density Divergence (MDD), to quantify the distribution divergence. MDD minimizes the inter-domain divergence ("match" in ATM) and maximizes the intra-class density ("tight" in ATM). Then, to address the equilibrium challenge issue in adversarial domain adaptation, we consider leveraging the proposed MDD into adversarial domain adaptation framework. At last, we tailor the proposed MDD as a practical learning loss and report our ATM. Both empirical evaluation and theoretical analysis are reported to verify the effectiveness of the proposed method. The experimental results on four benchmarks, both classical and large-scale, show that our method is able to achieve new state-of-the-art performance on most evaluations. Codes and datasets used in this paper are available at {\it github.com/lijin118/ATM}.
1 INTRODUCTION
Unsupervised domain adaptation transfers knowledge from labeled source data to unlabeled target data despite distribution shifts. The paper introduces MDD and ATM to align domains more explicitly while addressing limitations of adversarial training and existing metrics.
- Domain adaptation transfers knowledge between interconnected domains with distinctive data distributions, making distribution-shift mitigation its central challenge.
- Existing methods either minimize divergence metrics or confuse a domain discriminator to learn domain-invariant features.
- MDD jointly minimizes inter-domain divergence and maximizes intra-domain density, while incorporating class information to optimize conditional as well as marginal distributions.
- Adversarial training can fully confuse the discriminator without guaranteeing that the two domains are well aligned because adversarial optimization faces an equilibrium challenge.
- ATM incorporates MDD into adversarial domain adaptation as an additional distribution-gap loss or regularizer to alleviate the equilibrium challenge.
- 96.1% on SVHN→MNIST improved from 89.2% with the proposed ATM across four benchmark evaluations.
2 RELATED WORK
Prior domain adaptation work formulates adaptation through distance metrics or adversarial training. The paper positions its approach as combining explicit metric optimization with adversarial learning to address alignment concerns.
- Distance-based domain adaptation formulates knowledge transfer as minimizing a metric that measures distribution discrepancy.
- Existing work explores MMD, covariance, H-divergence, and Kullback–Leibler divergence for measuring distribution differences.
- The paper presents ATM as an adversarial domain adaptation method that simultaneously optimizes MDD and adversarial loss.
- Metric-based approaches explicitly optimize similarity functions, whereas adversarial methods train a discriminator with a feature network to learn domain-invariant features.
- Vanilla adversarial domain adaptation may not guarantee well-aligned domains even when the domain discriminator is fully confused, due to the equilibrium challenge.
3 THE PROPOSED METHOD
The proposed method introduces Maximum Density Divergence (MDD) to align domain distributions while increasing within-domain compactness, then combines this loss with adversarial training in ATM. The paper provides practical batch-wise formulations, theoretical divergence bounds, and an adversarial objective that explicitly aligns distributions.
- 3.1 Maximum Density Divergence: MDD combines an inter-domain distance term with two intra-domain terms that increase the density of the source and target distributions.The first term aligns P and Q, while the remaining terms make each domain more compact.
- 3.1 Maximum Density Divergence: The practical loss uses same-label samples for intra-domain terms and relative-position source-target pairs to make batch-wise computation feasible.The implementation addresses all-pairs computation and sampling challenges in deep-network training.
- 3.1 Maximum Density Divergence: On a finite probability space, MDD is a lower bound of symmetric KL-divergence and is bounded by 4δ2(P, Q).The paper states these properties in Lemmas 1 and 2 and relates the upper bound to total variation distance.
- 3.1 Maximum Density Divergence: MDD is implemented on learned features f_i = F(x_i), rather than directly on the original input space.F is the feature representation network used to learn domain-invariant features.
- 3.1 Maximum Density Divergence: The MDD formulation considers both inter-domain divergence and intra-domain density, thereby addressing marginal and conditional distributions.The intra-domain terms use pseudo labels that are dynamically updated during training.
- 3.2 Cross-Domain Adversarial Tight Match: ATM simultaneously minimizes MDD and confuses the domain discriminator to address the equilibrium challenge in adversarial domain adaptation.The adversarial framework combines the MDD loss with supervised source classification and adversarial training components.
4 EXPERIMENTS
Across four benchmarks, ATM generally improves domain-adaptation accuracy over strong baselines, including on challenging and large-scale evaluations. Additional analyses indicate that MDD contributes to alignment, stability, and performance gains.
- Digits Recognition: ATM improves digits-recognition accuracy over previous state-of-the-art methods, with gains of 0.5%, 0.9%, and 6.9% on three transfers.The largest improvement occurs on SVHN→MNIST, where ATM reaches 96.1% versus CDAN's 89.2%.
- Office-31: On Office-31, ATM exceeds CDAN by 3.1% and 4.2% on the hardest transfers and by 2.1% and 3.1% on Avg 1 and Avg 2.Avg 2 excludes W→D and D→W to better reflect performance on harder tasks.
- ImageCLEF-DA: On ImageCLEF-DA, ATM generally improves accuracy over CDAN by around 2% and achieves a 2.3% average gain across more diverse scenarios.The Pascal-target setting is more challenging, while other evaluations exceed 90% accuracy.
- Model Analysis: ATM converges within 20 epochs on SVHN→MNIST, produces smoother loss curves than CDAN, and achieves smaller A-distance than CDAN.The analyses also report that all three MDD terms contribute and that ATM can correctly classify confusing handwritten categories.
5 CONCLUSION
The paper concludes that Adversarial Tight Match (ATM) addresses unsupervised domain adaptation by combining adversarial learning with Maximum Density Divergence (MDD). ATM aligns domains while tightening intra-class density and outperforms prior state-of-the-art methods in extensive experiments.
- ATM is proposed as a method for unsupervised domain adaptation.
- MDD aligns domains by minimizing inter-domain divergence and maximizing intra-class density.
- MDD is tailored for smooth incorporation into adversarial domain adaptation frameworks.
- Jointly optimizing an additional distribution-divergence loss can alleviate adversarial learning’s equilibrium challenge in domain adaptation.
- Extensive experiments on four benchmarks show ATM outperforming previous state-of-the-art methods with significant advantages.