PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Transport for Domain Adaptation

Nicolas Courty, Rémi Flamary, Devis Tuia, Alain Rakotomamonjy

arXiv:1507.00504v2cs.LG

TL;DR

Domain adaptation must handle differing source and target data distributions, often with labels only in the source domain. The paper proposes regularized optimal transport that aligns the distributions while constraining source class structure, and reports frequent superiority over competing state-of-the-art methods. Its approach also addresses computational scaling through a generalized conditional-gradient method.

  • Problem

    Unsupervised domain adaptation requires transferring knowledge when labels are available only in the source domain and source and target data distributions differ.

  • Method

    The paper learns a regularized optimal transportation plan aligning source and target distributions while encoding source class structure, and uses generalized conditional-gradient splitting for computation.

  • Results

    Optimal transportation domain adaptation schemes frequently outperform competing state-of-the-art methods across synthetic and real-world datasets.

  • Takeaways & Limitations

    The framework supports unsupervised adaptation and extends to semi-supervised adaptation when a few target labels are available.

  • Takeaways & Limitations

    Discrete optimal transport has computational complexity O((n_s + n_t)n_sn_tlog(n_s + n_t)), which limits utility on large datasets without faster regularization.

Abstract

from arXiv · show

Domain adaptation from one data space (or domain) to another is one of the most challenging tasks of modern data analytics. If the adaptation is done correctly, models built on a specific data space become more robust when confronted to data depicting the same semantic concepts (the classes), but observed by another observation system with its own specificities. Among the many strategies proposed to adapt a domain to another, finding a common representation has shown excellent properties: by finding a common representation for both domains, a single classifier can be effective in both and use labelled samples from the source domain to predict the unlabelled samples of the target domain. In this paper, we propose a regularized unsupervised optimal transportation model to perform the alignment of the representations in the source and target domains. We learn a transportation plan matching both PDFs, which constrains labelled samples in the source domain to remain close during transport. This way, we exploit at the same time the few labeled information in the source and the unlabelled distributions observed in both domains. Experiments in toy and challenging real visual adaptation examples show the interest of the method, that consistently outperforms state of the art approaches.

1 INTRODUCTION

Unsupervised domain adaptation addresses distribution drift when labels exist only in a source domain. The paper proposes regularized optimal transport to align domains while preserving source class structure and enabling target classification.

  • Domain adaptation transfers knowledge from a labeled source domain to an unlabeled target domain whose data follow a different probability distribution.
  • The proposed approach transports source samples toward the target distribution, then trains a classifier using the transported labeled samples.The transportation map is data-dependent and usually nonlinear.
  • Optimal transport provides distribution distances that can be computed from empirical observations and remain meaningful when distribution supports do not overlap.
  • Regularization terms encode source class information or preserve neighborhood structures to favor transformations suited to adaptation.
  • The framework includes an efficient optimization algorithm and can be extended to semi-supervised adaptation when a few target labels are available.
  • Related work: Unlike reviewed global representation methods, the proposed approach defines a local transformation for each source sample under marginal-distribution preservation.

2 OPTIMAL TRANSPORT AND APPLICATION

The paper formulates domain adaptation as transporting the source marginal distribution to the target while preserving label information. It uses optimal transport plans and cost-based mappings, with squared Euclidean distance chosen empirically in the described implementation.

  • Domain adaptation assumes distinct source and target joint distributions, with μ_s and μ_t denoting their respective marginal data distributions.
  • The framework assumes domain drift arises from an unknown, possibly nonlinear transformation that preserves the conditional label distribution.Under this assumption, label information is preserved through the transformation.
  • A transport map T from μ_s to μ_t provides the adaptation pipeline: estimate both marginals, find T, transport labeled source samples, and train a classifier.
  • The Kantorovich formulation represents transport with a coupling γ whose marginals are μ_s and μ_t; γ₀ is the resulting transportation plan.
  • The implementation uses squared ℓ2 Euclidean cost and therefore evaluates distances through the corresponding squared Wasserstein distance.The paper reports that this choice provided the best average experimental result, while other cost functions remain possible.

3 REGULARIZED DISCRETE OPTIMAL TRANSPORT

The section formulates discrete optimal transport, adds entropy regularization for smoother and faster coupling computation, and maps source samples toward the target distribution. Under specific affine conditions, the resulting transport recovers the exact transformation and preserves labels.

  • 3.1 Discrete optimal transport: Discrete optimal transport learns a probabilistic coupling between empirical source and target distributions by minimizing transport cost under marginal constraints.The cost matrix records the cost of moving probability mass between samples, using squared Euclidean distance in this framework.
  • 3.1 Discrete optimal transport: The unregularized coupling is sparse, with at most ns + nt −1 nonzero entries, but solving its linear program has complexity O((ns + nt)nsntlog(ns + nt)).This computational cost limits utility on large datasets.
  • 3.2 Regularized optimal transport: Entropy regularization produces a denser, smoother coupling by increasing the entropy of the transport plan and can be interpreted through a KL divergence to a uniform joint distribution.As λ increases, sparsity decreases and source probability mass spreads across more target points.
  • 3.2 Regularized optimal transport: The regularized formulation enables efficient computation through the Sinkhorn-Knopp matrix-scaling approach.The paper identifies this algorithm as a key component of its methodology.
  • 3.3 OT-based mapping of the samples: A computed coupling transports source samples through Wasserstein interpolation; at t = 1, the barycentric mapping places transported samples on the target distribution’s support.With squared ℓ2 cost, each mapped source sample is a weighted average of target samples and lies in their convex hull.
  • 3.4 Discussing optimal transport for domain adaptation: For discrete distributions related by a strictly positive definite affine transformation with equal weights and squared ℓ2 cost, optimal transport recovers the exact affine transformation and fully preserves label information.The paper states that a classifier trained on the mapped samples has no generalization loss under these conditions.

4 CLASS-REGULARIZATION FOR DOMAIN ADAPTATION

The paper introduces class-aware regularization for optimal transport, using source labels and neighborhood structure to preserve class information during domain adaptation. It also extends the formulation to semi-supervised adaptation by constraining matches involving labeled target samples.

  • Regularizing the transport with class labels: Class-based regularization incorporates source labels into the transport plan rather than using them only for classifier training.The objective adds a class-based penalty to the regularized optimal transport problem.
  • Regularizing the transport with class labels: Two class regularizers promote same-label couplings or locally smooth, class-regular transported source samples.The alternatives are group sparsity and graph Laplacian regularization.
  • Regularization with group-sparsity: The convex ℓ1−ℓ2 group-lasso regularizer induces sparse target representations by grouping transport coefficients according to source classes.Its convexity permits an efficient generic optimization scheme.
  • Regularization with group-sparsity: The group-sparsity formulation assumes equal source and target label distributions, although small deviations in label proportions do not prevent the method from working.The required condition is Ps(y) = Pt(y).
  • Regularization with graph structure: Graph Laplacian regularization preserves source-sample relationships during transport and can sparsify similarities between samples of different classes.When target similarities are available, a symmetric source-target Laplacian trade-off can also be used, but target similarities cannot generally exploit class labels.
  • Regularizing for semi-supervised domain adaptation: For semi-supervised adaptation, an infinite matching cost excludes source-target pairs with different known labels while allowing matches to unlabeled target samples.The parameter-free term modifies the original cost matrix.

5 GENERALIZED CONDITIONAL GRADIENT FOR SOLVING REGULARIZED OT PROBLEMS

The paper solves its regularized optimal transport problem with a generalized conditional gradient algorithm. The method separates differentiable and convex nonsmooth components, while entropy-regularized subproblems are efficiently solved by Sinkhorn–Knopp scaling.

  • Existence and uniqueness: The objective has a solution because its regularized objective is continuous over a convex, closed, bounded transport constraint set.If the regularizer is strictly convex, the minimum is unique; this holds for the Laplacian regularizer.
  • Generalized conditional gradient: Generalized conditional gradient is selected for scalability while maintaining feasible transportation plans at every iterate.The method extends conditional gradient to composite objectives.
  • Generalized conditional gradient: GCG linearizes only the differentiable component of a composite objective, leaving the convex possibly nonsmooth component unlinearized.For this problem, f(γ) = ⟨γ, C⟩F + ηΩc(γ) and g(γ) = λΩs(γ).
  • Algorithmic implementation: Each GCG search-direction subproblem is an entropy-regularized optimal transport problem that can be solved with Sinkhorn–Knopp scaling.This provides the efficient inner solver used by the algorithm.
  • Algorithmic implementation: The algorithm applies to both Laplacian and group-lasso regularizers, despite group-lasso nondifferentiability at zero under a suitable nonzero initialization.Sinkhorn–Knopp exponentiation preserves nonzero grouped coefficients across iterates.

6 NUMERICAL EXPERIMENTS

The experiments evaluate regularized optimal transport for domain adaptation on a controllable two-moons problem and challenging visual recognition tasks. Across the visual experiments, class-based OT regularization performs strongly, while severe geometric deformation and some high-class-count settings remain difficult.

  • 6.1 Two moons: simulated problem with controllable complexity: The two-moons experiment varies target-domain rotation to control adaptation difficulty in a nonlinear, two-dimensional setting.The source and target each contain two classes represented by rotated moons, with performance evaluated over repeated trials.
  • 6.1 Two moons: simulated problem with controllable complexity: Optimal-transport methods outperform state-of-the-art methods particularly at low rotation angles, while class-label regularization helps at mid-range angles below 40°.At large angles such as 90°, the error rate reaches 0.5 because an opposite rotation can produce similar empirical distributions with inverted labels.
  • 6.1 Two moons: simulated problem with controllable complexity: For rotations above 70°, no OT method is a clear winner because sparse regions of the source distribution are not satisfactorily transported.The resulting classification depends heavily on training samples concentrated in denser portions of the source distribution.
  • 6.2.3 Results on unsupervised domain adaptation: With DeCAF features, OT methods improve classification significantly, by more than 20 points in some Office-Caltech transfers such as D→A and A→W.Using seventh-layer rather than sixth-layer DeCAF features does not strongly improve accuracy, suggesting overlap between higher-layer processing and optimal transport.

7 CONCLUSION

The paper presents a regularized optimal-transport framework for unsupervised domain adaptation, extending it to semi-supervised settings and scalable real-world computation. Experiments on synthetic and real datasets show frequent outperformance of competing state-of-the-art methods, while the authors identify regularization design and multidomain adaptation as directions for further work.

  • 7 CONCLUSION: The framework uses optimal transport to solve unsupervised domain adaptation and incorporates class structure through two regularization schemes.These regularizers enforce that samples from the same class undergo similar transformations during transport.
  • 7 CONCLUSION: The approach extends to semi-supervised domain adaptation and uses generalized conditional gradient splitting to scale to real-world datasets.The semi-supervised extension applies when a few target-domain labels are available.
  • 7 CONCLUSION: The proposed optimal-transport domain adaptation schemes frequently outperform competing state-of-the-art methods on synthetic and real-world datasets.The evaluation covers both toy and real-world datasets.
  • 7 CONCLUSION: The authors identify proper regularization as an open issue and suggest dedicated physical constraints and multimarginal optimal transport for future extensions.They also plan to extend the framework to multidomain adaptation, where several distributions are matched.
Loading 1507.00504v2…