Joint Distribution Optimal Transportation for Domain Adaptation

Nicolas Courty; Rémi Flamary; Amaury Habrard; Alain Rakotomamonjy

Source-linked AI summary

Joint Distribution Optimal Transportation for Domain Adaptation

Nicolas Courty, Rémi Flamary, Amaury Habrard, Alain Rakotomamonjy

arXiv:1705.08848v2May 24, 2017stat.MLcs.LG

TL;DR

Unsupervised domain adaptation must predict target labels without target supervision despite shifts in feature and conditional distributions. The paper proposes JDOT, which jointly optimizes a source-target joint-distribution coupling and a target prediction proxy. JDOT is theoretically linked to minimizing a target-error bound and reaches or surpasses state-of-the-art performance across real-world classification and regression tasks.

Problem
Unsupervised domain adaptation lacks target labels and must handle cases where both marginal feature and conditional distributions differ.
Method
JDOT minimizes optimal transport loss between the source joint distribution and the estimated target joint distribution (X, f(X)) while jointly optimizing the coupling and prediction function.
Results
JDOT reaches or surpasses state-of-the-art results on real-world classification and regression problems.
Takeaways & Limitations
JDOT provides a general domain-adaptation framework that aligns feature and conditional distributions without explicitly learning a transformation.
Takeaways & Limitations
The method assumes probabilistic Lipschitzness linking labels across close source and target instances under a coupling, and its naive learning formulation has NsNt terms.

Abstract

from arXiv · show

This paper deals with the unsupervised domain adaptation problem, where one wants to estimate a prediction function $f$ in a given target domain without any labeled sample by exploiting the knowledge available from a source domain where labels are known. Our work makes the following assumption: there exists a non-linear transformation between the joint feature/label space distributions of the two domain $\mathcal{P}_s$ and $\mathcal{P}_t$. We propose a solution of this problem with optimal transport, that allows to recover an estimated target $\mathcal{P}^f_t=(X,f(X))$ by optimizing simultaneously the optimal coupling and $f$. We show that our method corresponds to the minimization of a bound on the target error, and provide an efficient algorithmic solution, for which convergence is proved. The versatility of our approach, both in terms of class of hypothesis or loss functions is demonstrated with real world classification and regression problems, for which we reach or surpass state-of-the-art results.

1 Introduction

Unsupervised domain adaptation addresses target prediction when source and target data-generating processes differ and target labels are unavailable. JDOT aligns joint feature-label distributions by jointly optimizing a coupling and target prediction function, with theoretical and empirical support.

Domain adaptation transfers knowledge from an annotated source domain to an unlabeled target domain whose acquisition process may differ.Differences can involve acquisition conditions, devices, preprocessing, and compression.
Covariate-shift methods assume feature distributions change while conditional label distributions remain unchanged, an assumption not generally justified.More general adaptation seeks to align both marginal features and conditional distributions, but target labels are unavailable.
JDOT finds a prediction function by minimizing optimal transport loss between the source joint distribution and an estimated target joint distribution (X, f(X)).The method is designed for unsupervised adaptation between joint distributions.
JDOT minimizes a bound on target error and provides an efficient optimization procedure for regression and classification with broad hypothesis and loss-function classes.The paper reports applicability to kernel machines and neural networks.
Real-world experiments report JDOT performance reaching or surpassing state-of-the-art results.The paper evaluates both regression and classification problems.

2 Joint distribution Optimal Transport

JDOT extends optimal transport from feature marginals to joint feature-label distributions, using a prediction proxy for unavailable target labels. Its separable joint cost balances feature and label alignment, while alternating optimization estimates the coupling and predictor.

2 Joint distribution Optimal Transport: The adaptation setting contains distinct source and target joint distributions, with corresponding feature marginals μs and μt and unknown target labels.The source has labeled data, whereas the target testing set has unknown labels.
2.1 Optimal transport in domain adaptation: Monge optimal transport seeks a map pushing the source marginal toward the target marginal, whereas Kantorovich optimization seeks a coupling with prescribed marginals.The Kantorovich relaxation generalizes the Monge problem with fewer existence and uniqueness constraints.
2.2 Joint distribution optimal transport loss: JDOT aligns source and target joint distributions through a coupling rather than explicitly learning a transformation.This targets simultaneous feature and conditional-distribution alignment.
2.2 Joint distribution optimal transport loss: The joint cost D(x1, y1; x2, y2) = αd(x1, x2) + L(y1, y2) combines feature distance with label discrepancy.The parameter α balances feature-space alignment against label loss; as α → +∞, feature distance dominates.
2.2 Joint distribution optimal transport loss: Because target labels are unavailable, JDOT replaces them with proxy predictions f(x2) and jointly seeks a coupling and predictor.The empirical coupling is a transportation matrix, and the predictor is chosen to match source labels with aligned target instances.
Learning with Joint Distribution OT: The method uses alternating block coordinate descent over the coupling and prediction function, with regularization added in practice to reduce overfitting.The paper describes the optimization problem as smooth with separable constraints.

3 A Bound on the Target Error

The paper bounds target error under probabilistic transfer Lipschitzness, connecting the JDOT objective to source–estimated-target alignment and joint prediction error.

Error and assumptions: The target error is defined as the expected loss under the target distribution, with an analogous source-domain error.The loss is assumed bounded, symmetric, k-Lipschitz, and to satisfy the triangle inequality.
Transfer assumption: Probabilistic transfer Lipschitzness bounds the probability that nearby source–target instances under a coupling receive different labels.It relaxes classic Lipschitzness to accommodate marginal-label relationships.
Theoretical guarantee: The main theorem bounds target error using the JDOT alignment objective, sampling terms, a probabilistic-Lipschitzness term, and joint error of an ideal labeling function.The theorem assumes bounded inputs, a suitable coupling, a PTL-compatible Lipschitz labeling function, and an admissible loss.
Interpretation: Adaptation is possible when the source and estimated target joint distributions align well and the remaining terms, including joint error, are sufficiently small.The joint error term requires good prediction in both domains, while φ(λ) measures failure of probabilistic Lipschitzness.

4 Learning with Joint Distribution OT

JDOT alternates optimization of the transport coupling and prediction function, supporting broad hypothesis and loss classes while providing convergence to critical points.

Function classes: The framework accommodates RKHS and parameterized function spaces, including linear models, neural networks, and kernel methods.A regularization term Ω(f) can be defined according to the chosen hypothesis space.
Optimization procedure: The optimization uses two-block coordinate descent, alternately updating the coupling γ and the function f.With fixed f, the problem is classical optimal transport; with fixed γ, it becomes a supervised-style learning problem.
Label transfer and complexity: The fixed-coupling learning objective transfers source labels through estimated target labels weighted by the optimal transport matrix.This produces NsNt terms in general, although RKHS formulations can use Nt parameters.
Convergence: Every limit point of the alternating-optimization sequence is a critical point of the objective, provided the sequence has limit points.This follows from the differentiable two-block Gauss–Seidel structure and closed, non-empty, convex constraints.
Regression: For squared-loss regression, fixing γ reduces the learning problem to weighted averages of source target values.This simplification relies on the quadratic loss and need not hold for more complex regression losses.
Classification: In one-against-all hinge-loss classification, each target sample uses a convex combination of positive- and negative-label hinge losses weighted by γ.The transported class proportions are represented by ˆP = 1/Nt γᵀPs.

5 Numerical experiments

The experiments evaluate JDOT on classification and regression transfer tasks, showing strong performance across visual adaptation, Amazon reviews, and WiFi localization.

Caltech-Office classification: JDOT consistently outperforms the baseline on Caltech-Office, improving average accuracy by 5 points.It also achieves the best mean accuracy and average ranking, and is statistically better than most compared methods.
Amazon review classification: JDOT surpasses DANN in 11 of 12 Amazon review transfer tasks, except books→dvd.The experiment uses a neural-network classifier and demonstrates adaptation with different loss functions.
Experimental design: The evaluation compares JDOT with kernel, projection, transfer-component, adversarial, and optimal-transport baselines across classification and regression settings.The compared methods include KRR, SurK, DIP, DIP-CC, GeTarS, CTC, CTC-TIP, SA, ARTL, OT-IT, OT-MM, and DANN.
WiFi localization regression: JDOT performs best in one of three transfers across periods and exceeds 98% average accuracy across devices.For every across-device task, it is at least ten points ahead of the best competing method.
WiFi localization regression: The WiFi experiments measure correctness within three meters across periods and six meters across devices.Reported values are means and standard deviations over ten repetitions using randomly sampled source and target data.

6 Discussion and conclusion

The conclusion presents JDOT as an optimal-transport approach that jointly aligns feature and conditional distributions while learning a target predictor. Experiments support its effectiveness and flexibility across tasks, losses, and hypothesis classes.

Method and theory: JDOT jointly optimizes a coupling between source and target joint distributions and a prediction function.The approach assumes a transfer map between source and target joint distributions and aligns feature and conditional distributions.
Method and theory: The paper proves that JDOT learning is equivalent to minimizing a bound on the target distribution.
Empirical findings: Experiments show superiority over several state-of-the-art methods on classification and regression benchmarks.The comparisons include optimal-transport methods, domain-adversarial neural networks, and transfer components.
Flexibility: JDOT accommodates multiple loss functions, including mean squared error and hinge loss, and multiple hypothesis classes.

A Illustration on a simple example

A three-class rotated-Gaussian toy experiment illustrates how JDOT iteratively adapts decision boundaries and how the α parameter affects classification accuracy. Combining input-space and label losses gives the central comparison.

Toy setup: The toy classification problem uses three 2D Gaussian source classes and rotates the source distribution by π/4 to form the target domain.The experiment uses hinge loss with linear and RBF kernels in a reproducing kernel Hilbert space.
Iterative adaptation: Directly applying a source-trained classifier performs poorly after the target rotation, while block coordinate descent recovers the target labels.
Effect of α: Using mostly label cost with α = 0.1 deteriorates final accuracy, whereas using only input-space distance with α = 50 or 100 improves performance.The input-only setting is equivalent to the cited prior method [14].
Figure 2: Figure 2 tracks accuracy over 15 iterations for six α values and shows decision boundaries at selected iterations.The figure distinguishes source samples as crosses and target samples as class-colored circles.
Optimization: The method is solved with block coordinate descent by alternating optimal-transport and learning updates until convergence.Each iteration first solves the transport problem with fixed f, then solves the learning problem with fixed γ.

C BCD iterations on real data

On real data, the BCD optimization generally stabilizes after about 10 iterations, although empirical performance can fluctuate rather than improve monotonically.

The empirical result generally stabilizes at around 10 iterations.
Performance increases are not monotonic on the real dataset, unlike in the toy example.
Accuracy of the estimated model is tracked across BCD iterations on the Caltech-office dataset.

D Proof of Theorem 3.1

The theorem analyzes target-error guarantees for joint-distribution domain adaptation under boundedness, loss regularity, and probabilistic transfer Lipschitzness assumptions. It applies to samples consisting of labeled source instances and unlabeled target instances.

The target error is defined as the expected loss between target labels and predictions, with an analogous source error and inter-function loss.
The estimated target distribution Pf_t is defined from the target marginal and a labeling function f.
The transport loss combines feature distance and label discrepancy, while the loss is assumed symmetric, bounded, k-Lipschitz, and triangle-inequality satisfying.
Theorem D.1 considers a hypothesis class over a compact finite-dimensional input space and an output space, with source, target, and estimated target joint distributions.
The sampling setup uses Ns labeled source instances and Nt unlabeled target instances, with the bound holding for sufficiently large sample sizes and probability at least 1 − δ.
The guarantee assumes a Lipschitz labeling function satisfying probabilistic transfer Lipschitzness relative to an optimal coupling and minimizing joint source-target error among compatible functions.

Proof

The proof bounds target error by combining transport-based discrepancies, sampling concentration, probabilistic transfer Lipschitzness, and the joint error of an ideal labeling function. It establishes the theorem via Wasserstein inequalities and a union bound.

The proof uses symmetry and triangle inequality of the loss to relate target prediction error to source and target errors.
Kantorovich-Rubinstein duality bounds the relevant discrepancy for any coupling, allowing the argument to use the optimal coupling Π*.
Probabilistic transfer Lipschitzness controls most coupled instances, while the bounded deviation M contributes the additional kMφ(λ) term where it fails.
The transport cost is a metric under bounded support and bounded loss, enabling empirical Wasserstein concentration for source and estimated-target distributions.
Combining the inequalities and applying the union bound yields the theorem with probability at least 1 − δ for every f ∈ H.
When Ps = Pt and f* is strongly transfer-Lipschitz, the bound connects to a classical generalization bound because terms involving f* can vanish.

E Empirical concentration result for Wasserstein distance

The empirical concentration result provides Wasserstein-distance control for empirical measures under transport-inequality and moment assumptions. Its sample requirement depends on dimension and the desired accuracy.

The result is attributed to Bolley and co-authors and is used to control empirical Wasserstein distances.
For d′ greater than the dimension of Z, sufficiently large samples provide concentration at accuracy ε with a sample-size condition involving ε^−(d′+2).
The concentration constants depend on d′ and square exponential moments of the underlying measure.
The underlying measure is assumed to satisfy Talagrand’s transport inequality T1(c), expressed through relative entropy.