PaperlayerView exact revision on arXiv

Source-linked AI summary

COT-GAN: Generating Sequential Data via Causal Optimal Transport

Tianlin Xu, Li K. Wenliang, Michael Munn, Beatrice Acciaio

arXiv:2006.08571v2stat.MLcs.LG

TL;DR

Generating realistic sequential data is challenging because spatial structure and temporal dependencies interact. COT-GAN trains implicit generative models using causal optimal transport and a mixed Sinkhorn divergence. The paper reports performance gains across time-series and video datasets and reduced mini-batch bias.

  • Problem

    Training implicit generative models on dynamical data is difficult because temporal dependencies are strongly correlated with spatial features.

    Source: §1 · p. 1

  • Method

    COT-GAN trains implicit generative models with causal transport plans and a mixed Sinkhorn divergence for mini-batches.

    Source: §1 · pp. 1–2, §3 · p. 3, §3.1 · p. 3

  • Results

    COT-GAN outperforms existing methods across datasets from traditional time series to high-dimensional videos, and the mixed Sinkhorn divergence mitigates biased parameter estimation.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Experiments suggest that constraining transport plans to be causal is a promising direction for generating sequential data.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce COT-GAN, an adversarial algorithm to train implicit generative models optimized for producing sequential data. The loss function of this algorithm is formulated using ideas from Causal Optimal Transport (COT), which combines classic optimal transport methods with an additional temporal causality constraint. Remarkably, we find that this causality condition provides a natural framework to parameterize the cost function that is learned by the discriminator as a robust (worst-case) distance, and an ideal mechanism for learning time dependent data distributions. Following Genevay et al.\ (2018), we also include an entropic penalization term which allows for the use of the Sinkhorn algorithm when computing the optimal transport cost. Our experiments show effectiveness and stability of COT-GAN when generating both low- and high-dimensional time series data. The success of the algorithm also relies on a new, improved version of the Sinkhorn divergence which demonstrates less bias in learning.

1 Introduction

Generating realistic dynamical data is challenging because temporal and spatial features are strongly correlated. COT-GAN uses causal optimal transport and a mixed Sinkhorn divergence, and the paper reports better performance across traditional time series and high-dimensional videos.

1 Introduction

  • Dynamical data synthesis is a challenging unsupervised learning problem with scientific and practical applications.Examples include video, audio, physiological recordings, and financial traces.Source: §1 · p. 1
  • Temporal dependencies are strongly correlated with spatial features, while prior work often separates models or losses into static and dynamic components.The paper introduces an adversarial objective based on optimal transport with transport plans constrained to respect causality.Source: §1 · p. 1
  • COT-GAN outperforms existing methods across datasets ranging from traditional time series to high-dimensional videos.The paper also proposes a mixed Sinkhorn divergence to mitigate biased parameter estimation.Source: §1 · pp. 1–2

2 Background

The background introduces adversarial implicit generative models and optimal transport as tools for distribution matching. Entropic regularization enables Sinkhorn computation, while the Sinkhorn divergence corrects a nonzero self-distance.

2.1 Adversarial learning for implicit generative models

  • An implicit generative model maps latent samples to generated data, with a discriminator judging generated and real samples in a zero-sum training game.The generator induces νθ, which is trained to be close to the real distribution μ.Source: §2.1 · p. 2

2.2 Optimal transport and Sinkhorn divergences

  • For finite supports, classical optimal transport has super-cubic complexity, while entropic regularization enables Sinkhorn computation and the Sinkhorn divergence corrects nonzero self-distance.The regularized distance is defined using the regularized transport optimizer; the Sinkhorn divergence subtracts the two self-costs.Source: §2.2 · p. 2

3 Training generative models with Causal Optimal Transport

COT-GAN formulates sequential-data generation using causal transport plans and a regularized transport objective. Its mixed Sinkhorn divergence addresses finite-mini-batch bias, and the resulting adversarial training learns a robust distance between real and generated distributions.

3 Training generative models with Causal Optimal Transport

  • The method formulates generation for d-dimensional, T-long sequences as distributions on path space and builds its objective from a special class of transport plans.The coordinates are organized by time, with associated filtrations describing information available through each time step.Source: §3 · p. 3

3.1 Causal Optimal Transport

  • Causal plans transport mass non-anticipatively: target mass at time t depends on the source only through time t.Causality is also expressed through martingales and conditional independence between y_t and future source coordinates given the source through time t.Source: §3.1 · p. 3

3.2 Regularized Causal Optimal Transport

  • Entropic regularization yields a regularized COT distance between the COT distance and independent-coupling loss, closer to COT for small ε; reformulation makes it a worst-case family of regularized OT costs.The regularized problem is optimized over causal plans and can be computed with Sinkhorn; Figure 1 compares distances and divergences across mini-batch sizes.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 Reducing the bias with mixed Sinkhorn divergence

  • In Example 3.4, regularized distance and ordinary Sinkhorn divergence miss θ = 0.8, while mixed Sinkhorn divergence finds the correct minimizer across batch sizes.It also outperforms ordinary Sinkhorn divergence when that alternative is allowed a larger batch, though adding more batch terms increases computational cost and has no absolute empirical winner.Source: §3.3 · p. 5

3.4 COT-GAN: Adversarial learning for sequential data

  • COT-GAN parameterizes adapted processes with time-dependent architectures, penalizes nonzero average martingale increments, and trains the generator and discriminator in opposing directions.It approximates the causal function class by truncating sums and computes the mixed divergence with Sinkhorn iterations.Source: §3.4 · p. 5

end

  • The discriminator learns a robust worst-case distance between real and generated distributions, with per-iteration complexity O((J + d)LTm^2).The cost-function family originates from causality.Source: end · p. 6

4 Related work

Prior video GANs separate spatial and temporal modeling through architecture, while sequential GANs often use recurrent networks but remain domain-specific. COT-GAN instead puts transport causality in the objective and proposes a mixed Sinkhorn divergence usable beyond time series.

Video generation

  • Video GANs address spatio-temporal dependencies through architectures including two-stream generation, discriminator weight clipping, or separate frame-quality and motion discriminators.VGAN combines static-background and moving-foreground networks; TGAN adds weight clipping; MoCoGAN uses two discriminators.Source: §4 · p. 6

Sequential generation

  • Sequential GANs often deploy recurrent networks, with TimeGAN adding teacher forcing and WaveGAN adopting WaveNet's causal structure, but they are generally domain-specific.COT-GAN aims to offer a more general sequential framework by incorporating transport causality into its objective.Source: §4 · p. 6

COT-GAN's distinction

  • Unlike [21], COT-GAN adds a causal component to the cost, and its mixed Sinkhorn divergence is proposed to reduce parameter-estimation bias and extend to IGMs beyond time series.The passage describes the divergence as a generic training divergence, not limited to time-series settings.Source: §4 · p. 6

5 Experiments

Across time-series and video experiments, COT-GAN generally matches or outperforms the compared methods, with results varying by dataset and metric. The experiments also find the mixed Sinkhorn divergence especially important in the AR-1 ablation.

Experiments

  • For low-dimensional time series with complex temporal structure, COT-GAN is compared with TimeGAN, WaveGAN, and SinkhornGAN; AR-1 statistics are means and standard deviations over 16 runs with different random seeds.The two statistics are the time-averaged sum of absolute differences between channel correlation coefficients and the absolute difference between real-sample and generated-sample correlation coefficients.Source: §5.1 · p. 7

5.1 Time series

  • COT-GAN best matches multivariate AR-1 correlations and is most stable across runs; it also best matches noisy-oscillation samples and EEG correlations across the compared baselines.AR-1 stability is inferred from the small standard deviation across runs; EEG comprises 64 channels and the first 100 time steps.Source: §5.1 · pp. 7–8
  • In the AR-1 ablation, each component contributes to the best result, and the mixed Sinkhorn divergence is the most important factor for performance improvement.The variants include the original unmixed divergence, removal of the martingale penalty, and direct minimization of mixed or original Sinkhorn divergences.Source: §5.1 · p. 8

5.2 Videos

  • On Sprites, COT-GAN outperforms both alternatives on FVD and KVD, while mixed Sinkhorn minimization scores better on FID and KID; on human actions, COT-GAN leads except on KVD.Table 1 reports scores estimated using 10,000 generated samples; lower values indicate better sample quality.Source: §5.2 · p. 9, Table 1 · p. 9

6 Discussion

The discussion presents causal transport as a promising approach to sequential-data generation. The paper’s approximations suffice for its experiments, while several extensions and theoretical analyses remain future directions.

6 Discussion

  • The paper introduces causal transport theory to machine learning and presents causal transport plans as a promising direction for generating sequential data.The authors point to COT-GAN’s performance as evidence for this direction.Source: §6 · p. 9
  • The mixed Sinkhorn distance and truncated sum suffice to produce good experimental results, while motivating further theoretical analysis.Source: §6 · p. 9
  • Future work includes flexible-length data, conditional COT-GAN, improved enforcement of the martingale property for M, and better parameterization of the causality constraint.Source: §6 · p. 9

7 Broader impact

COT-GAN is suited to datasets of i.i.d. sequences or stationary time series, and the paper describes possible applications of sequential-data synthesis. It also notes that financial time-series generation requires a modified algorithm.

7 Broader impact

  • COT-GAN is suitable for generating sequential data when the real dataset consists of i.i.d. sequences or stationary time series.Source: §7 · p. 10
  • Synthetic time series may be useful when privacy concerns, high costs, or data scarcity limit access to abundant training data for predictive models.Source: §7 · p. 10
  • Financial time-series generation remains under development because financial data’s high non-stationarity requires different features and architectures.The paper also describes possible uses in pricing, hedging, portfolio selection, risk management, and stress testing.Source: §7 · p. 10

Supplementary material

The supplement states limiting behaviors for regularized causal optimal transport and outlines a reformulation of its causality constraint. It also describes how the causal constraint can be represented in the optimization.

A.1 Limits of regularized Causal Optimal Transport

  • As ε approaches zero, the left- and right-hand sides of (A.2) both tend to Kc(µ, ν).Source: §A.1 · p. 13
  • For sufficiently large ε, the optimizer is the independent coupling µ ⊗ ν, and Kc,ε(µ, ν) equals Eµ⊗ν[c(x, y)].Source: §A.1 · p. 13

A.2 Reformulation of the COT problem

  • The causal constraint (3.2) is expressed using a characteristic function to rewrite (3.3), with the min-max equality justified by convexity and compactness.Source: §A.2 · p. 13

A.3 Sinkhorn divergence at the level of mini-batches

The supplement compares mini-batch Sinkhorn-divergence variants and reports that COT-GAN performs strongly across low- and high-dimensional experiments. It also details the low-dimensional time-series and sprite-animation data used in those experiments.

A.3 Sinkhorn divergence at the level of mini-batches

  • For sinusoid distributions with varying maximum amplitude, the sample Sinkhorn divergence does not recover the true optimizer θ = 0.8, while the mixed divergence does.Source: §A.3 · p. 14
  • In low-dimensional experiments, COT-GAN and COT-GAN6 significantly outperform the other tested mini-batch Sinkhorn-divergence variants, though their relative performance differs by task.COT-GAN outperforms COT-GAN6 on the 1-D noisy oscillation but underperforms it on multivariate AR-1.Source: §A.3 · pp. 14–15, Figure 6 · p. 14, Figure 7 · p. 15
  • COT-GAN performs best on all video evaluation metrics except KVD for Sprites animation; COT-GAN and COT-GAN6 both outperform MoCoGAN on the two tasks.COT-GAN6 requires more mini-batches and is about 1.5 times slower than COT-GAN.Source: §A.3 · p. 15, Table 2 · p. 15
  • As ε approaches infinity, the ordinary mini-batch Sinkhorn divergence converges to a biased MMD estimator, whereas the mixed divergence converges to an unbiased estimator.Source: §A.3 · p. 15
  • The mixed divergence may still be biased relative to the true Sinkhorn divergence, although in the reported example its minimum occurs near the true parameter.Source: §A.3 · p. 16

B.1 Low dimensional time series

  • For noisy oscillations, the experiments compare pixel-value marginals and the joint distribution of bump locations at adjacent time steps; the EEG experiment uses 2,841 training and 969 test sequences from 43 control-group subjects.The noisy-oscillation data depict a bump moving rightward in a zig-zag pattern.Source: §B.1 · pp. 16–17

B.2.1 Sprite animations

  • The sprite dataset contains 6,352 character sequences, padded to 13 frames of 64 × 64 × 3 pixels after cropping and dropping the alpha channel.Source: §B.2.1 · p. 18

B.2.2 The Weizmann Action database

The Weizmann Action experiments use short, downscaled video sequences and LSTM-based generators, with fixed training settings and substantial training time.

B.2.2 The Weizmann Action database

  • The training set contains 89 sequences, each with 16 frames at 64×64×3 resolution, sampled from videos after skipping the first five frames and then every other frame.The original clips are 2–7 seconds long, with 25 frames per second at 144×180×3 resolution.Source: §B.2.2 · p. 19
  • A 2-layer LSTM maps per-step latent variables into time-dependent features, which a 4-layer deconvolutional network maps to frames.The latent variable has shape 5 × 5 per time step; dense layers connect the LSTM and deconvolutional network.Source: §B.2.2 · p. 19
  • Training uses batch size and sequence length 16, λ = 0.01, ϵ = 6.0, and Sinkhorn L = 100, and runs for 3–4 days on one NVIDIA Tesla P100 GPU.Each iteration takes roughly 1.5 seconds.Source: §B.2.2 · p. 19

C Sprites and human action results without cherry-picking

Random, uncherry-picked samples show good-quality Sprite sequences from mixed Sinkhorn minimization and COT-GAN, while human-action samples reveal visual artifacts in both methods and MoCoGAN.

C Sprites and human action results without cherry-picking

  • For Sprites, mixed Sinkhorn minimization and COT-GAN produce good-quality samples, while MoCoGAN captures only rough frame patterns and lacks smooth motion evolution.The samples are random and shown without cherry-picking.Source: C Sprites and human action results… · p. 19, Figure 10 · p. 20
  • Human-action samples from COT-GAN and mixed Sinkhorn minimization show blurriness or person disappearance when clothing resembles the background; MoCoGAN also has this issue and appears to show some mode collapse.The comparison uses generators of similar capacity, trained for 65,000 iterations.Source: C Sprites and human action results… · p. 20, Figure 11 · p. 21
Loading 2006.08571v2…