Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation

1 Introduction

• Optimal Transport measures the minimum cost of transporting a source distribution to a target and has applications in generative modeling and domain adaptation.In Wasserstein GANs, the approximate distance between real and generated distributions is minimized; domain adaptation uses distance minimization between source and target feature distributions.Source: §1 · pp. 1–2
• Even 5% outliers can produce poor couplings and substantially change Wasserstein distance estimates because OT’s marginal constraints weigh every sample equally.The figure contrasts clean couplings, couplings with 5% outliers, and robust Wasserstein couplings.Source: §1 · p. 2, Figure 1 · p. 2
• Existing unbalanced OT relaxes marginal constraints, but its exact dual optimization can converge poorly and behave unstably in large-scale deep-learning applications such as GANs.This motivates a more practical dual formulation.Source: §1 · p. 2
• The proposed dual form reduces to weighted OT, assigning low weights to outlier samples; discrete and continuous stochastic-relaxation solvers demonstrate stability in large-scale deep learning.The resulting instance weights can indicate how difficult input samples are to generate for a given task.Source: §1 · p. 2
• Robust OT improves adversarial domain adaptation by ∼5% and enables GANs to ignore outliers, while its distance is bounded by a constant factor of true OT under mild assumptions.The bound applies to any outlier distribution.Source: §1 · p. 2

2 Background on Optimal Transport

• OT minimizes the cost of transporting one probability distribution to another over joint distributions whose marginals match the input distributions.The exact marginal constraints force outliers to receive nonzero coupling weights, potentially increasing transportation cost.Source: §2 · p. 3
• The Kantorovich-Rubinstein dual optimizes over 1-Lipschitz functions, which can be represented by neural networks with constraints enforced by weight clipping or gradient penalties.The section describes this dual as simpler than the primal optimization.Source: §2 · p. 3
• Entropy regularization and marginal relaxation are two popular OT relaxations; this work uses marginal relaxations to handle outlier noise in machine-learning applications.The paper distinguishes its unbalanced Kantorovich formulation from prior work on Monge unbalanced OT.Source: §2 · p. 3

3 Robust Optimal Transport

• Outliers are modeled as samples with large noise, and the section uses relaxed OT formulations to address them.Source: §3 · p. 3

3.1 Unbalanced Optimal Transport

• Unbalanced OT [6] relaxes marginal constraints, but its dual is difficult to train in deep learning because it uses two constrained discriminators and standard SGD failed to converge in the authors’ χ²-divergence GAN test.The authors motivate a new dual reformulation as a scalable alternative.Source: §3.1 · pp. 3–4

3.2 Our Duality

• Robust OT bounds the f-divergence of relaxed marginals to control relaxation and yields a dual with one function rather than the two used by unbalanced OT.The single-function form is presented as suited to deep-learning applications.Source: §3.2 · p. 4
• The discrete formulation optimizes empirical sample weights, while the continuous stochastic relaxation represents those weights as functions and can be trained with stochastic gradient descent.The discrete weight optimization is a second-order cone program; the continuous formulation uses weight functions that can be implemented with neural networks.Source: §3.2 · pp. 4–5

3.3 Can robust OT handle outliers?

• Robust OT is upper bounded by a constant factor of the true Wasserstein distance under the stated outlier model, with the bound depending on ρ.The result is derived for one-sided robust OT, the form used in the GAN and domain-adaptation experiments.Source: §3.3 · p. 5
• When the outlier fraction γ is known, the section gives ρ = γ²/[2(1−γ)] as an appropriate choice; otherwise, validation performance can guide its selection.Source: §3.3 · p. 5

4.1 Generative modeling

• On CIFAR-10 with MNIST or uniform-noise outliers, robust WGAN ignores outliers and maintains good FID as their fraction rises, unlike WGAN.Source: Figure 2 · p. 6, §4.1 · p. 7

4 Experiments

• GAN training minimizes the one-directional robust Wasserstein measure in dual form, and the robust formulation can extend to non-saturating and hinge losses.Source: §4 · p. 5, §4.1 · p. 6

4.1 Generative modeling

• On DomainNet sketches, robust SNGAN generates sketch-like samples while ignoring outliers, whereas SNGAN also generates non-sketch images.Source: Figure 4 · p. 7, §4.1 · p. 7

4.2 Domain adaptation

• On VISDA-17, the robust adversarial objective improves performance by ∼5% over the standard adversarial objective and reaches parity with UDA-specific competitors when entropy-regularized.Source: Table 2 · p. 7, Table 3 · p. 8, Table 4 · p. 8

4.1 Generative modeling

• On clean CIFAR-10 and CIFAR-100, robust WGAN shows no FID drop, which the authors take as evidence that generated distributions do not drop modes.Source: §4.1 · p. 7, Table 1 · p. 7
• In CIFAR-MNIST and DomainNet-sketch examples, outliers receive lower weights than in-distribution samples, and the authors interpret weights as indicators of generation difficulty.Source: §4.1 · pp. 7–8

• The method is designed to make robust optimal transport insensitive to outliers, defined as samples with large noise.Source: §5 · p. 9
• A computationally efficient dual form makes the robust OT objective suited to deep learning applications.Source: §5 · p. 9
• In GANs and domain adaptation, the proposed method is reported to handle outliers effectively and achieve good performance improvements.Source: §5 · p. 9

• OT's sensitivity to input noise can make it problematic for large-scale machine learning, where noise is inevitable.Source: §6 · p. 9
• The developed learning method is described as efficient, provably robust against outliers, and amenable to deep generative modeling and domain adaptation.Source: §6 · p. 9
• The authors state that the work may bridge theoretical and applied machine learning, and they identify no negative ethical or societal outcomes.Source: §6 · p. 9

A Proofs

• In metric spaces, the robust OT dual can be expressed using a single discriminator, making it easier to train in large-scale deep learning problems such as GANs.Source: Duality · pp. 12–13
• When β satisfies the stated constraint, mixtures of outlier and clean transport plans are feasible for robust OT, but their cost is only an upper bound because the mixture need not be optimal.Source: Provable robustness · pp. 13–14

B.1 Dual Objective

• The unbalanced OT dual objective is derived from a primal penalized by f-divergences between perturbed and original marginals, using f(x) = (x − 1)^2/2.Source: §B.1 · pp. 14–15
• For the quadratic divergence, the derivation evaluates the conjugate across cases and substitutes its maximizing value into the dual.Source: §B.1 · pp. 14–15

B.2 Practical Implementation and Training Issues

• The unbalanced OT implementation constrains two discriminator networks, and training failed to converge to a proper solution despite hyperparameter sweeps.Source: §B.2 · p. 15
• The practical dual produces CIFAR-like samples, while the unbalanced OT dual has a flat loss curve and produces noisy images rather than a proper solution.Source: §B.2 · p. 15, Figure 5 · p. 16, Figure 6 · p. 16

C Choosing ρ and the tightness of the bound

• When the outlier fraction γ is known, the section recommends ρ = γ/2(1 −γ); otherwise, it proposes estimating ρ from the inflection point of an elbow-shaped robust OT curve.Source: C Choosing ρ and the tightness of the… · pp. 16–17
• The reported robust OT upper bound closely approximates the true robust OT measure in the plotted example.Source: C Choosing ρ and the tightness of the… · p. 17, Figure 7 · p. 17
• The table caption reports that the robust GAN generates male images even when males make up only 2% of the imbalanced CelebA input.Source: Table 6 · p. 17

• The assumed outliers are samples with large noise, far from the true data distribution in the Wasserstein sense.Source: D Biases And Mode Drop · p. 17
• Even with males comprising 2% of CelebA inputs, the robust GAN generated male images; the authors also report no clean-CIFAR-10 FID drop.They interpret these observations as evidence that the model drops high-noise samples rather than rare modes.Source: D Biases And Mode Drop · p. 18
• Rare modes can still be dropped; the authors propose using learned weights to visualize dropped modes and, in future work, boosting to train a balanced mixture model.Source: D Biases And Mode Drop · p. 18

E Properties of Robust OT

• Robust Wasserstein is non-negative because both the cost function and transportation map are non-negative.Source: E Properties of Robust OT · p. 18
• Robust Wasserstein satisfies identity: transporting a distribution to itself has a feasible zero-cost solution, so its value is zero.Source: E Properties of Robust OT · p. 18
• Robust Wasserstein is generally asymmetric when ρ1 and ρ2 differ, because the two marginals then have different f-divergence constraints.Source: E Properties of Robust OT · p. 18

F.1 Generative modeling

• The robust Wasserstein GAN formulation can be extended to spectral-normalization GAN and non-saturating GAN variants.Source: §F.1 · p. 18

• CIFAR-10 datasets were corrupted with MNIST or uniform-noise outliers at varying fractions γ, with each dataset kept at 50,000 samples and models trained using robust Wasserstein loss.Experiments used DCGAN and ResNet architectures.Source: §F.1.1 · p. 19
• FID scores were reported against clean CIFAR-10 without outliers, using Inception-feature means and covariances from clean data and generated samples; the mean and covariance matrices were computed over 50,000 samples.Source: §F.1.1 · p. 19
• The appendix identifies a sketch-domain DomainNet experiment using robust spectral-normalization GAN with ResNet generator and discriminator architectures.Source: §F.1.1 · p. 19
• Figures 8 and 9 visualize samples generated from CIFAR-10 corrupted with MNIST and uniform-noise outliers, respectively.Source: Figure 8 · p. 21, Figure 9 · p. 21

F.1.2 Clean datasets

• Clean-data experiments train an unconditional robust Wasserstein GAN on CIFAR-10 and a conditional GAN on CIFAR-100.The CIFAR-100 model uses conditional batch normalization, similar to.Source: §F.1.2 · p. 21

F.1.3 Architectures and Hyper-parameters

• Architecture and hyperparameter details are referred to in separate tables for the ResNet model, DCGAN, and weight network.Source: §F.1.3 · p. 21

F.2 Domain Adaptation

• Domain adaptation is evaluated on VISDA-17 with ResNet-18, ResNet-50, and ResNet-101; target weights use the discrete optimization version, and target logits receive entropy regularization.Source: §F.2 · p. 22
• The source-only baseline trains the feature network and classifier on labeled source data, while adversarial alignment adds domain-discrepancy minimization and is described as an unweighted version of the proposed objective.Source: §F.2 · p. 22
• The domain-adaptation feature network uses ImageNet-pretrained ResNets with the final linear layer removed; a linear classifier is trained from scratch, and the discriminator is a spectrally normalized three-layer MLP.Source: §F.2 · pp. 22–23
• The appendix visualizes learned target weights as histograms and shows target samples sorted by weight.Source: §F.2 · p. 23, Figure 10 · p. 23, Figure 11 · p. 23