PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Gradient Barriers in Pushforward Deep Generative Models

Edric Tam, Qingyang Liu, Barbara Engelhardt, Ray Bai

arXiv:2610.34484v1stat.MLcs.LGmath.STstat.TH

TL;DR

DGMs can struggle to represent heavy-tailed and multimodal targets, raising the question of what regularity the target itself requires of a transport map. The paper derives gradient and Jacobian lower bounds and shows that bijective reparametrizations can reduce the learned map’s burden when approximating a known density.

  • Problem

    DGMs can underrepresent heavy-tailed and multimodal targets, motivating the question of what regularity those targets impose on any map transporting a simple source distribution.

    Source: §1 · pp. 1–2

  • Method

    The paper uses Poincaré-type inequalities for heavy-tailed targets and isoperimetric and coarea inequalities for separated modes, and proposes bijective reparametrizations for density approximation.

    Source: §1 · p. 3

  • Results

    Large centered target moments and low-probability valleys between well-separated modes require large transport derivatives; Gaussian-source flows also underrepresent Student-t tails and omit a mixture mode.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Known bijective reparametrizations can absorb part of the required derivative growth, reducing the burden on the learned map while the composite map remains subject to the lower bounds.

    Source: §1 · p. 3

    Global Lipschitz constraints can improve numerical stability but limit faithful representation of heavy-tailed targets and well-separated modes.

    Source: §6 · p. 13

Abstract

from arXiv · show

Broad classes of deep generative models (DGMs) operate by learning to map a simple source distribution, such as a standard Gaussian, to a complex high-dimensional target distribution. Despite their success, DGMs often empirically struggle to accurately represent distributions with heavy tails or well-separated modes. In this work, we show that these difficulties reflect intrinsic properties of the underlying transport problem rather than particular neural network architectures or training procedures. We establish quantitative lower bounds on the gradients and Jacobians of pushforward mappings that transport log-concave source distributions to such targets. For heavy-tailed target distributions, the required gradient magnitude grows as the target develops increasingly large moments. For multimodal target distributions, the low-probability regions between well-separated modes similarly force large gradients in the transport map. We discuss how such large derivatives relate to trade-offs between expressivity and numerical stability. Our results hold under substantially weaker regularity conditions than previous analyses and remain stable under perturbations of the target distribution. Finally, we show that these obstructions can be mitigated when the goal is approximation of a given density rather than generative modeling. Specifically, we propose simple classes of bijective reparametrizations that can alleviate the difficult geometric structure and reduce the gradient burden placed on the learned generative map. Numerical experiments support our theoretical analysis.

1. Introduction.

This section examines why pushforward generative models struggle with heavy-tailed and multimodal targets, developing derivative lower bounds and reparametrizations that address these barriers. It also considers approximate pushforwards, showing that small W2 error relative to the square root of the target’s centered second moment does not eliminate the gradient requirement.

1. Introduction.

  • For well-separated modes, low-probability valleys force large transport gradients even when target moments remain bounded.The introduction illustrates mode omission for a bimodal Gaussian mixture under a Gaussian source.Source: §1 · p. 3
  • Known bijective reparametrizations can absorb part of the derivative growth for density approximation, improving representation of heavy tails and both mixture modes.The composite map remains subject to the lower bounds, but the learned component carries less derivative burden.Source: Figure 1 · p. 2, §1 · p. 3

4. Gradient Bounds for Deep Generative Models via Functional Inequalities.

  • Large centered target moments force large transport gradients and Jacobians, with heavy-tailed lower bounds applying independently of architecture or training procedure.The results require Sobolev regularity for moment bounds rather than a global Lipschitz bound.Source: §1 · p. 3, §2 · p. 4

2. Related Work.

  • The pushforward framework covers explicit and continuous-flow maps and discretized diffusion samplers; the analysis uses weaker regularity than prior global-Lipschitz approaches.Moment bounds require Sobolev regularity, while multimodal bounds require only local Lipschitz continuity; models without a natural pushforward mechanism are outside scope.Source: §2 · p. 4

4.1. Gaussian Source Distributions.

  • For Gaussian sources, Poincaré inequalities relate the output covariance to the transport Jacobian, so large centered second moments require large Jacobians.The scalar Gaussian inequality is dimension-independent and holds under weak Sobolev regularity.Source: §4.1 · pp. 6–7

4.2. Lp–Lq Poincaré Inequalities.

  • Lp Poincaré inequalities extend moment-to-gradient bounds to higher moments, and large centered component moments likewise require large Jacobian norms.The Gaussian vector-valued result bounds expected pth powers of Jacobian norms.Source: §4.2 · p. 7

4.3. More General Log-Concave Distributions.

  • The moment-to-gradient lower bounds extend from Gaussian sources to general log-concave sources, with constants depending on the source distribution.For isotropic log-concave sources, the Poincaré constant is at most C log(ed); a strong curvature lower bound yields Gaussian-like inequalities.Source: §4.3 · p. 8

4.4. Approximate Pushforward Maps.

  • The second-moment bound is stable for approximate pushforwards when the generated distribution is within ε of the target in W2.The proposition assumes a finite-second-moment target and a map with components in W 1,2(γ).Source: §4.4 · p. 8
  • Proposition 4.7 establishes that small-error W2 approximation cannot eliminate the gradient requirement for approximate pushforward maps.The proof appears in Section S2 of the Supplement.Source: §4.4 · p. 9

5. Gradient Bounds for Multimodal Targets.

For multimodal targets, substantial probability on both sides of a low-probability valley forces large Jacobians, even when target moments remain bounded. Isoperimetric bounds locate this derivative burden and extend to approximate transports and log-concave sources.

5. Gradient Bounds for Multimodal Targets.

  • Two substantial-probability regions separated by a low-probability valley force large Jacobians, with quantitative bounds determined by their separation and probability masses.The argument uses isoperimetric inequalities and complements analyses based on global Lipschitz assumptions.Source: §5 · p. 9

5.1. A Localized Jacobian Bound.

  • The localized bound requires a Jacobian contribution of at least Δ·I(b) from source points mapped into the valley, increasing linearly with separation for fixed b.It requires no integrability assumption on the Jacobian and does not depend on source or target dimension.Source: §5.1 · pp. 9–10

5.2. Approximation of Multimodal Targets.

  • The bounds apply to locally Lipschitz maps and describe the geometry of separated regions without requiring a density or mixture representation.The approximation results use total variation error and the target’s probability assigned to the separating region.Source: §5.2 · pp. 10–11
  • For approximate transports with total variation error ε, the expected Jacobian lower bound grows linearly with separation, while each fixed higher Jacobian moment diverges as valley mass δ and ε shrink, provided Δ and w stay bounded away from zero.For fixed log-concave sources, the corresponding lower bound also diverges under these conditions.Source: §5.2 · pp. 10–11
  • In the two-component Gaussian mixture example, exact transports require diverging Jacobians as the components narrow, although every fixed absolute moment of the target remains bounded.The derivative at the center of the explicit monotone transport also diverges as σ ↓ 0, while the target density there tends to zero.Source: §5.2 · p. 11

6. Practical Implications for Deep Generative Modeling.

Large transport derivatives are intrinsic to heavy-tailed and well-separated multimodal targets, creating a stability–expressivity trade-off. When the target density is evaluable up to a normalizing constant, bijective coordinate changes can reduce the derivative burden on the learned map.

6. Practical Implications for Deep Generative Modeling.

  • Large transport derivatives remain necessary without a global Lipschitz constraint: heavy-tail bounds track target moments, while multimodal bounds identify contributions over low-probability valleys.The bounds apply to Gaussian and log-concave sources under weak Sobolev regularity for moment bounds, and require local Lipschitz continuity for multimodal bounds.Source: §6 · p. 12
  • For neural ODEs, large transport derivatives require a large time integral of the vector-field Jacobian norm, and the required derivatives can grow rapidly with mode separation.Large vector-field Jacobian norms are associated with numerical instability in fixed-step solvers and more function evaluations in adaptive solvers.Source: §6 · pp. 12–13
  • Global Lipschitz constraints can improve numerical stability but limit expressivity, ruling out exact heavy-tailed targets or preventing accurate representation of well-separated modes.Large Jacobians can amplify small input errors into larger output errors, including errors from rounding and finite-precision computation.Source: §6 · pp. 12–13

7. Approximating Densities via Bijective Reparametrizations.

  • When a target density is evaluable up to a normalizing constant, bijective reparametrizations can alter its geometry while retaining the simple source and reducing the Jacobian norms required of the learned map.The learned transport operates in transformed coordinates, and applying the known inverse transformation recovers samples in the original coordinates.Source: §7 · p. 13, §6 · p. 13

7.1. Reparametrizations for Heavy Tails.

  • For regularly varying tails, the one-dimensional reparametrization converts polynomial tails into exponential tails without requiring the tail index, giving the transformed variable finite moments of every order.The transformation can also be applied coordinate-wise to multivariate targets without requiring coordinate independence.Source: §7.1 · p. 14
  • For the transformed heavy-tailed target, changing coordinates yields an exact transport with finite expected squared derivative.The stated result applies for 0 < κ ≤ 2.Source: §7.1 · p. 14

7.2. Reparametrizations for Multimodal Target Densities.

  • For multimodal targets, coordinate changes compress gaps between reference component locations while preserving their neighborhoods, reducing the Jacobian norms required of the learned map.With reference-location error η ≤ r, the transformed transport can have uniformly bounded gradients even as original component separation increases; the original transport's expected squared derivative grows exponentially with squared separation.Source: §7.2 · pp. 15–16

8. Empirical Illustrations.

Numerical illustrations using explicit transport maps examine heavy-tailed and multimodal targets, then show how known coordinate reparametrizations change transport derivatives.

8.1. Heavy-Tailed Targets.

  • Poincaré lower bounds closely track evaluated Jacobian norms across three heavy-tailed families, with both increasing sharply as tail parameters approach 2 from above.The variance diverges at this boundary.Source: §8.1 · p. 16, Figure 2 · p. 17

8.2. Multimodal Targets.

  • As the low-probability valley widens, both the lower bound and the mean transport derivative increase, and the bound captures the increase rate well.The comparison uses two-component Gaussian mixtures.Source: §8.2 · p. 18, Figure 4 · p. 18

8.3. Reparametrizations of Known Transport Maps.

  • The asinh reparametrization substantially reduces expected squared Jacobian norms across Student-t, Pareto, and Fréchet targets.Original-map values rise sharply near tail parameter 2, while transformed-map values remain moderate across the displayed range.Source: §8.3 · pp. 18–19, Figure 5 · p. 19
  • For five-component mixtures, the original expected squared derivative rises rapidly with component separation, while the reparametrized quantity levels off.Component widths, transformation parameters, and reference-location errors are held fixed.Source: §8.3 · p. 19, Figure 6 · p. 19

Declaration of AI Use.

This material records the authors’ use of AI tools and introduces weak-gradient, Sobolev, and tail-moment concepts used in the analysis.

Declaration of AI Use.

  • The authors used AI tools to refine writing, check mathematical derivations, and assist with code writing, then reviewed and edited the content.They state that they take full responsibility for the final manuscript.Source: Declaration of AI Use · p. 20

S1.1. Weak Gradients.

  • Weak gradients coincide with classical gradients for differentiable functions and allow piecewise-smooth functions with finitely many kinks to be analyzed.The examples include absolute value and ReLU functions.Source: S1.1. Weak Gradients · p. 23

S1.2. Sobolev Spaces.

  • The space W 1,2(ν) requires only weak differentiability and a square-integrable gradient, making it less restrictive than a global Lipschitz condition.It is the natural domain for the Poincaré and log-Sobolev inequalities used in the analysis.Source: S1.2. Sobolev Spaces · p. 23

S1.3. Background on Heavy-Tailed Distributions.

  • The paper defines a real-valued variable as heavy-tailed when E[et|X|] is infinite for every t > 0; such distributions typically have only finitely many moments.Light-tailed distributions admit moments of all orders.Source: S1.3. Background on Heavy-Tailed… · p. 24

Moments and Polynomial Tail Bounds.

  • Polynomial tail decay corresponds to moments existing only up to a critical order: a tail bounded below by t^-α has divergent moments of order p ≥ α.A tail bounded above by t^-α implies finite moments for p < α.Source: Moments and Polynomial Tail Bounds · p. 24

Slow and Regular Variation.

This background section defines tail variation and canonical distribution families, then introduces log-concavity and source-measure boundary and Poincaré constants.

Slow and Regular Variation.

  • Regular variation expresses a function as x^ρL(x), where L varies slowly; regularly varying tails have a power-law form with a slowly varying factor.The paper identifies the Student-t tail index with its degrees of freedom.Source: Slow and Regular Variation · pp. 24–25

Some Canonical Heavy-Tailed Distributions.

  • For Student-t, Pareto, and Fréchet families, smaller tail or shape parameters yield heavier tails; Pareto and Fréchet variances are finite only when their shape parameters exceed 2.For those two families, the kth moment exists exactly when k is below the corresponding shape parameter.Source: Some Canonical Heavy-Tailed… · p. 25

S1.4. Log-Concave and Strongly Log-Concave Distributions.

  • A log-concave density obeys a geometric-interpolation inequality, and strong log-concavity adds a positive curvature lower bound.The standard Gaussian is 1-strongly log-concave.Source: S1.4. Log-Concave and Strongly… · p. 25

S1.5. Isoperimetric Profiles.

  • An isoperimetric profile lower-bounds the boundary measure of every region with a specified probability, describing how probability grows as a region is enlarged.For the Gaussian profile, half-spaces attain the bound.Source: S1.5. Isoperimetric Profiles · p. 26

S1.6. Cheeger and Poincaré Constants.

  • The Cheeger constant quantifies the boundary needed to separate regions, while the variance Poincaré constant controls variance through expected squared gradients.The standard Gaussian has variance Poincaré constant 1.Source: S1.6. Cheeger and Poincaré Constants · p. 26

S2. Proofs for Section 4.

The proofs use Poincaré inequalities componentwise and directionally, and extend smooth-function arguments to the weakly differentiable Sobolev settings used in the paper.

S2. Proofs for Section 4.

  • The proof of Theorem 4.2 applies the scalar Gaussian Poincaré inequality to each transport-map component, then takes expectations to obtain the stated bound.The proof also applies the inequality to scalar projections of the map and takes traces of a positive-semidefinite bound.Source: S2. Proofs for Section 4 · pp. 27, 29
  • The proof of Theorem 4.5 invokes separate results for one-dimensional isotropic measures and for dimensions d ≥ 2, then applies the resulting inequalities to smooth W 1,2(ν) functions.The proof notes that the referenced spectral gap is the reciprocal of the paper’s variance Poincaré constant.Source: S2. Proofs for Section 4 · p. 28
  • The proof extends Poincaré inequalities from smooth functions to Sobolev classes by approximating with smooth functions and taking limits in the Sobolev norm.The cited density argument applies with Lp norms for every 1 ≤ p < ∞.Source: S2. Proofs for Section 4 · p. 28

S3. Proofs for Section 5.

The proofs localize transport-gradient control to the preimage of a separating region, then combine it with isoperimetric and Cheeger inequalities. They extend the resulting bounds to approximate transports using total-variation and moment arguments.

S3. Proofs for Section 5.

  • Gaussian isoperimetry and coarea yield a Cheeger inequality, which combines with the localized gradient estimate to prove the localized bound.The proof also establishes a first inequality and a second inequality by integrating the isoperimetric profile.Source: S3. Proofs for Section 5 · pp. 29–31
  • The truncated-distance function has gradient bounded by the transport Jacobian only on the preimage of the separating region.Its gradient vanishes where the function reaches either endpoint of its range.Source: S3. Proofs for Section 5 · p. 30
  • For approximation, total-variation bounds give a positive separation parameter, and Hölder’s inequality and nonnegativity extend the resulting estimate to global moments.The proof notes that the inequalities also hold when the moment is infinite.Source: S3. Proofs for Section 5 · pp. 30–31

S3.1. Calculations for Example 5.4.

Example 5.4 shows that a Gaussian transport can require a diverging gradient lower bound even when every fixed target moment stays bounded. The example also verifies the derivative formula for its monotone transport.

S3.1. Calculations for Example 5.4.

  • Every fixed absolute moment of the target remains bounded uniformly as σ ↓0.The bound is sup 0<σ≤1 E|Xσ|^r ≤ E(a + |G|)^r < ∞ for every fixed r > 0.Source: S3.1. Calculations for Example 5.4 · p. 32
  • The gradient lower bound diverges as the separating-region probability tends to zero, although every fixed target moment remains bounded.For each fixed q > 1, the numerator tends to aI(1/2) > 0 while the denominator tends to zero.Source: S3.1. Calculations for Example 5.4 · p. 32
  • The example verifies the derivative formula for the smooth, locally Lipschitz monotone exact transport.The transport is defined using the target CDF and its inverse.Source: S3.1. Calculations for Example 5.4 · pp. 32–33

S4. Proofs for Section 7.

The proofs establish moment properties for bijective reparametrizations and analyze a multimodal example. In that example, reparametrization keeps transport derivatives bounded independently of component separation, unlike the original transport’s expected squared derivative.

S4.1. Proofs for Bijective Reparametrizations.

  • Under the stated tail-rate condition, the reparametrized variable has finite exponential moments for 0 < t < α and finite polynomial moments of every order.The proof first establishes the tail-rate formula (7.3).Source: S4.1. Proofs for Bijective… · pp. 33–34

S4.2. Calculations for Example 7.2.

  • The bijective reparametrization Ψ is a C1 bijection with a C1 inverse and 0 < Ψ′ ≤ 1.Its inverse is C1 because its derivative is continuous and strictly positive.Source: S4.2. Calculations for Example 7.2 · p. 37
  • The reparametrized transport derivative is bounded independently of component separation, while the original map’s expected squared derivative grows rapidly with separation.The pointwise bound for the transformed map is 0 < S′D(z) ≤ C.Source: S4.2. Calculations for Example 7.2 · pp. 38, 40

S5.1. Details of the Heavy-Tail Illustrations.

The heavy-tail illustrations compare the covariance-trace lower bound with the transport Jacobian’s expected squared norm. They use product targets in two dimensions, which the passage says represent higher-dimensional behavior.

S5.1. Details of the Heavy-Tail Illustrations.

  • These families are used to assess bound tightness and practical relevance, with interpretable tail parameters allowing systematic variation in tail heaviness.The passage notes that explicit Gaussian transports allow direct inspection of gradients and Jacobians.Source: S5.1. Details of the Heavy-Tail… · p. 41
  • The targets are product measures with independent coordinates from Student-t, Pareto, and Fréchet families, shown in two dimensions without loss of generality.The passage says the behavior in two dimensions is representative of higher-dimensional cases.Source: S5.1. Details of the Heavy-Tail… · p. 41
  • The simulations compare the target covariance trace with the Gaussian-source expectation of the squared norm of the transport Jacobian’s diagonal entries.The latter expectation is computed numerically using Gauss–Hermite quadrature.Source: S5.1. Details of the Heavy-Tail… · p. 41

S5.2. Empirical Illustrations with a Log-Concave Source Distribution.

The experiments replace the standard Gaussian source with an isotropic product Laplace source to test whether heavy-tail-driven Jacobian growth persists for a non-Gaussian log-concave source. Across three heavy-tailed target families, the numerical Jacobian estimates track the predicted lower bounds and show rapid growth in heavy-tail regions.

S5.2. Empirical Illustrations with a Log-Concave Source Distribution

  • The experiments use an isotropic product Laplace source, which is log-concave but not strongly log-concave, instead of the standard Gaussian.Its scale is chosen so its variance is 1; the target families and overall procedure otherwise match Section 8.Source: S5.2. Empirical Illustrations with a… · p. 41
  • The study compares numerically evaluated expected squared Frobenius Jacobian norms with Poincaré lower bounds and plots log absolute Jacobian determinants across the input space.For the log-concave-source experiments, the Poincaré constant is not generally known, so the experiments use a working value of 2.Source: S5.2. Empirical Illustrations with a… · pp. 41–42
  • Across Student-t, Pareto, and Fréchet targets, the expected squared Jacobian norm tracks the log-concave Poincaré lower bound as tail heaviness varies.The heatmaps also show rapid Jacobian growth in regions associated with heavy-tailed behavior.Source: S5.2. Empirical Illustrations with a… · p. 42, Figure S1 · p. 42

S5.3. Empirical Illustrations with Correlated Heavy-Tailed Targets.

The supplementary experiments test whether heavy-tail-related Jacobian growth persists for correlated targets and document numerical setups for the correlated-target and density-fitting experiments. The results extend the observed bound-tracking behavior beyond independent product targets, while the density-fitting setup uses supplied reference component locations rather than evaluating a procedure to locate them.

S5.3. Empirical Illustrations with Correlated Heavy-Tailed Targets

  • The correlated-target experiments apply a Cholesky-factor linear map to independent heavy-tailed coordinates, then compare Jacobian norms with Gaussian Poincaré lower bounds.The setup varies tail-heaviness parameters and correlation ρ ∈ {0.5, 0.9}.Source: S5.3. Empirical Illustrations with… · pp. 42–43
  • For correlated Student-t, Pareto, and Fréchet targets, expected squared Jacobian norms closely track Gaussian Poincaré lower bounds across the tested constructions.The targets are formed by linearly transforming independent heavy-tailed coordinates, with correlations ρ = 0.5 and ρ = 0.9.Source: S5.3. Empirical Illustrations with… · p. 43, Figure S3 · p. 44

S5.4. Details of Numerical Experiments

  • The Figure 5 heavy-tail experiment compares the exact maps T and S = Ψ ◦ T for two-dimensional product Student, Pareto, and Fréchet targets.It applies arsinh coordinate-wise, with s = √κ for Student-t and s = 1 otherwise, and evaluates expectations using Gauss–Legendre quadrature.Source: S5.4. Details of Numerical Experiments · pp. 43–44
  • The Figure 6 multimodal experiment uses five equally weighted, unit-standard-deviation Gaussian components whose means are spaced by D.The supplied reference component locations are synthetic, so the experiment does not evaluate a procedure for locating components.Source: S5.4. Details of Numerical Experiments · pp. 44–45

S5.5. Density-Based Fitting for Figure 1

  • The Figure 1 density-fitting experiments use a two-dimensional Student-t target and a two-component Gaussian mixture with a standard Gaussian source.Normalizing flows are trained by minimizing reverse KL against the original target density using a four-layer affine-coupling RealNVP flow.Source: S5.5. Density-Based Fitting for Figure 1 · p. 45

Related papers

Loading 2610.34484v1…