PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal transport for Gaussian mixture models

Yongxin Chen, Tryphon T. Georgiou, Allen Tannenbaum

arXiv:1710.07876v2math.PReess.SY

TL;DR

The paper addresses computationally expensive optimal transport on structured probability densities by developing an explicit framework for Gaussian mixture models. It defines a metric supporting comparison, interpolation, and averaging while preserving Gaussian-mixture structure, with lower computational cost and good approximation under small component covariances.

  • Problem

    Optimal mass transport on the full manifold of probability densities is computationally expensive, motivating a framework for structured Gaussian mixture densities.

  • Method

    The paper restricts optimal mass transport to the submanifold of Gaussian mixture distributions and develops a metric, geodesic interpolation, and barycenter construction within that space.

  • Results

    The resulting distance is computed by a linear program with variables growing linearly in the number of Gaussian components, and it closely approximates standard optimal transport when component covariances are small.

  • Takeaways & Limitations

    The framework provides a structured way to compare, interpolate, and average Gaussian mixture models while retaining their mixture structure.

  • Takeaways & Limitations

    The relation between the proposed distance and the restriction of the standard Wasserstein distance to Gaussian mixtures remains unclear.

Abstract

from arXiv · show

We present an optimal mass transport framework on the space of Gaussian mixture models, which are widely used in statistical inference. Our method leads to a natural way to compare, interpolate and average Gaussian mixture models. Basically, we study such models on a certain submanifold of probability densities with certain structure. Different aspects of this framework are discussed and several examples are presented to illustrate the results. This method represents our first attempt to study optimal transport problems for more general probability densities with structures.

I. INTRODUCTION

The paper motivates optimal transport on Gaussian mixture models because full probability-density transport is expensive, while mixtures have structure worth preserving. It frames OMT as a geometry for comparing, interpolating, and averaging distributions, then seeks a structured submanifold framework.

  • Gaussian mixture models represent subpopulations with weighted Gaussian components and are widely used because inference is mathematically simple and computationally efficient.
  • OMT provides a mass-preserving, minimum-cost framework whose induced geometry supports comparing, interpolating, and averaging probability densities.
  • Transport on the full probability-density manifold is computationally expensive, motivating OMT formulations on structured, parameterized submanifolds.
  • The paper focuses on Gaussian mixture models as a structured density family while leaving extensions to more general structured densities for future work.

A. Gaussian marginal distributions

For Gaussian marginals, the optimal transport problem has a closed-form solution derived by optimizing over Gaussian joint distributions. The resulting displacement interpolation remains Gaussian, and the Gaussian Wasserstein space formally extends Euclidean space.

  • A. Gaussian marginal distributions: Gaussian marginal transport simplifies to a closed-form solution obtained by minimizing over Gaussian joint distributions under a semidefinite constraint.
  • A. Gaussian marginal distributions: The displacement interpolation between Gaussian distributions remains Gaussian, with linearly interpolated means and an explicitly determined covariance.
  • A. Gaussian marginal distributions: Singular Gaussian distributions are handled by replacing matrix inverses with pseudoinverses in the Wasserstein distance formula.
  • A. Gaussian marginal distributions: The Wasserstein space of Gaussian distributions formally extends Euclidean space R^n.

III. OMT FOR GAUSSIAN MIXTURE MODELS

The paper constructs Gaussian mixture models as discrete measures on the Wasserstein space of Gaussian distributions. Applying discrete OMT to these measures yields a metric on the mixture-model space while retaining Gaussian mixture structure.

  • III. OMT FOR GAUSSIAN MIXTURE MODELS: The framework addresses the loss of Gaussian mixture structure under ordinary displacement interpolation by seeking a geometry that preserves it.
  • III. OMT FOR GAUSSIAN MIXTURE MODELS: Gaussian mixture models are viewed as discrete distributions on the Wasserstein space of Gaussian distributions, enabling OMT to be applied to their components.
  • III. OMT FOR GAUSSIAN MIXTURE MODELS: The component-level formulation uses a discrete OMT problem whose transport cost is the squared Wasserstein metric between Gaussian components.
  • III. OMT FOR GAUSSIAN MIXTURE MODELS: The resulting distance d defines a metric on the space M(R^n) of Gaussian mixture distributions.

0. Therefore,

The construction yields a triangle-inequality relation for the Gaussian-mixture distance through metric and Minkowski inequalities.

  • The Gaussian-mixture distance satisfies d(µ0, µ2) ≤ d(µ0, µ1) + d(µ1, µ2), using the Wasserstein metric and Minkowski inequality.

A. Geodesic

The proposed geodesic between Gaussian mixtures remains a Gaussian mixture and has constant-speed distance under d.

  • A. Geodesic: d(µs, µt) = (t −s)d(µ0, µ1) for 0 ≤s < t ≤1, so the path has constant speed.The equality follows by combining the upper and lower length bounds.
  • A. Geodesic: The geodesic connecting two Gaussian mixtures is itself a Gaussian mixture formed from weighted Gaussian distributions νij.The construction preserves the mixture-model structure along the interpolation.
  • A. Geodesic: The solution is unique for generic µ0, µ1 ∈ M(Rn), although some instances admit multiple solutions.Thus, nonuniqueness is not expected in most applications.

B. Relation between d and W2

The distance d induced on the Gaussian-mixture submanifold generally exceeds W2, but can efficiently approximate it when components are narrow and well separated.

  • B. Relation between d and W2: In general, d > W2 because restricting transport to the Gaussian-mixture submanifold makes the transport plan suboptimal.Equality holds when both distributions contain only one Gaussian component.
  • B. Relation between d and W2: The relation between d and W2 is not fully established: it is unclear whether d is the restriction of W2 to M(Rn).This is an explicit limitation of the proposed distance relation.
  • B. Relation between d and W2: When Gaussian-component variances are small relative to mean differences, d is generally a very good approximation of W2.This regime supports using d as an approximation for distributions with separated, concentrated components.
  • B. Relation between d and W2: For mixtures with few components, computing d requires solving (9), avoiding potentially costly density discretization and discrete OMT.The resulting computation is described as extremely efficient when component counts are small.

IV. BARYCENTER OF GAUSSIAN MIXTURES

The paper formulates Gaussian-mixture barycenters within the mixture family, reducing the problem to linear programming and obtaining dimension-independent complexity apart from Gaussian cost evaluation.

  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The Gaussian barycenter remains Gaussian, and a fixed-point iteration is available, although convergence of the practical iteration lacks a proof.This limitation is stated for the iteration used in practice.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The unrestricted barycenter problem is difficult in high dimensions and can lose Gaussian-mixture structure, motivating the modified formulation.For general distributions, the problem is affected by the curse of dimensionality.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The modified barycenter problem restricts the optimizer to Gaussian mixtures and replaces W2 with the relaxed distance.This preserves the Gaussian-mixture structure while changing the optimization problem.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The resulting barycenter is a weighted mixture of Gaussian minimizers ν1,...,νN, with N bounded by N1N2 · · · NL.The candidate components arise from optimal Gaussian solutions associated with component combinations.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The barycenter weights are obtained by solving a linear programming problem.After optimization, µ = p1ν1 + p2ν2 + · · · + pNνN.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: The formulation is independent of the dimension of Rn, while complexity depends on the numbers of mixture components.The dimension affects only the cost computation, for which a closed form is available.
  • IV. BARYCENTER OF GAUSSIAN MIXTURES: A specified-component variant allows the number N of barycenter components to be chosen small and is solved by the same linear-programming approach.The prescribed Gaussian components need not have the form generated by component combinations.

V. NUMERICAL EXAMPLES

Numerical examples illustrate the framework for computing distances, geodesics, and barycenters.

  • V. NUMERICAL EXAMPLES: Several examples illustrate the framework for computing distance, geodesic, and barycenter.The examples cover the paper’s three main Gaussian-mixture transport operations.

A. d vs W2

The proposed distance differs from W2, while the proposed interpolation preserves Gaussian-mixture structure that standard OMT interpolation loses.

  • A. d vs W2: The two distances are not equivalent, and d is always bounded below by W2.
  • B. Geodesic: The proposed interpolation preserves the Gaussian-mixture structure, whereas standard OMT interpolation loses it in intermediate densities.The two Gaussian components of the proposed interpolation are shown in Figure 4.
  • B. Geodesic: In a two-dimensional example, standard OMT again undermines the Gaussian-mixture structure during interpolation.Figures 5–7 compare the marginal distributions and the two interpolation methods.

C. Barycenter

The barycenter example compares the proposed method with traditional OMT for Gaussian mixtures and reports better averaging with the proposed method.

  • C. Barycenter: The barycenter experiment uses three Gaussian mixture distributions with equally distributed component masses.
  • C. Barycenter: The proposed method gives a better Gaussian-mixture average, while traditional OMT damages the Gaussian-mixture structure.Both methods are applied with two sets of weights.
Loading 1710.07876v2…