PaperlayerView exact revision on arXiv

Source-linked AI summary

Density-Level Backward Error for Low-NFE Generative Flows

Ewsey Obzherin, Valentin Khrulkov

arXiv:2610.29475v1cs.LGcs.NAmath.NAstat.ML

TL;DR

Low-NFE generative ODE sampling depends on how evaluations are placed, while existing schedule approaches do not isolate which numerical defect changes the evolving marginal. The paper projects modified-equation error onto the density-active component and uses it to guide allocation and velocity-representative choice. Its experiments and dense-grid comparisons support faster approximation of the same high-NFE Euler limit.

  • Problem

    Low-NFE sampling quality depends on evaluation placement, but schedule methods do not isolate the local numerical defect that changes the evolving marginal.

    Source: §1 · pp. 1–2

  • Method

    The framework projects the solver’s modified-equation defect onto the density-changing component and uses the residual for solver-order-aware allocation and representative selection.

    Source: §1 · p. 2

  • Results

    Controlled and pretrained experiments validate the mechanism, and matched-seed dense-grid convergence shows faster approximation of the same high-NFE Euler limit.

    Source: §9 · p. 9

  • Takeaways & Limitations

    The approach allocates evaluations for a fixed low-order solver rather than changing the integration formula, distinguishing it from higher-order solver comparisons.

    Source: §1 · p. 2, §9 · p. 9

    The quantile grid exactly minimizes the finite-N local surrogate, not necessarily the full nonlinear terminal error.

    Source: §6 · p. 6

Abstract

from arXiv · show

Low-NFE generative ODE sampling is constrained not only by solver order, but also by how a fixed budget of network evaluations is distributed along the flow. We develop a density-level backward-error framework that identifies the component of a solver's modified-equation defect that changes the evolving marginal, while excluding solenoidal trajectory distortion that is invisible in Wasserstein tangent geometry. The resulting residual induces a solver-order-aware compute-allocation rule and, through gauge freedom, an objective for selecting numerically favorable velocity representatives of the same density path. This separates our approach from schedule search driven by trajectory or endpoint surrogates and from higher-order solver design. Exact Gaussian and mixture models establish the mechanism, including cases where curvature-based allocation fails and continuum-optimal transport is not discretization-optimal. On a pretrained CIFAR-10 EDM probability-flow ODE, density-aware allocation consistently improves fixed-update Euler sampling in the low-NFE regime. Matched-seed convergence to a dense Euler reference confirms that the method accelerates approximation of the same high-NFE flow rather than changing the asymptotic sampler.

1 INTRODUCTION

With few network evaluations, generation quality depends on where a fixed solver places them, but existing schedule methods do not isolate which part of numerical error changes the marginal. This paper connects density-level error analysis to timestep allocation and velocity-representative selection.

1 INTRODUCTION

  • Earlier schedule-search methods optimize placement but do not isolate marginal-changing defect components or exploit equivalent velocity fields.The introduction contrasts these approaches with trajectory- or reference-solution-driven schedule objectives.Source: §1 · p. 1
  • Density-aware design separates solver choice, evaluation placement, and selection among velocity fields that generate the same continuum density path.The paper connects schedule and representative design through density-level backward error.Source: §1 · p. 1
  • The projected modified-equation defect determines leading marginal error, excluding solenoidal trajectory motion invisible to the density.The framework connects its norm to the leading local marginal W2 defect.Source: §1 · p. 2
  • For order-r solvers, allocation depends on propagated residual K(1,t)Rr(t), with exponent 1/(r+1), and its quantile grid minimizes the stated finite-N local surrogate.The allocation and representative-selection problems respectively distribute evaluations and choose a continuum-equivalent field.Source: §1 · p. 2
  • The fixed-update comparisons hold network, update rule, NFE, and sampling cost constant; higher-order samplers are shown only as context.The offline allocation profile is reusable across generated samples and NFE choices.Source: §1 · pp. 2–3

2 RELATED WORK AND POSITIONING

The paper positions its contribution at the intersection of solver-error analysis, schedule optimization, and gauge freedom. Rather than optimize the density curve or solver formula, it optimizes numerical realization of a fixed marginal path.

2 RELATED WORK AND POSITIONING

  • Unlike continuum transport methods that optimize a path or endpoint map, this work fixes the marginal curve and optimizes its gauge-equivalent velocity lifts.The paper also distinguishes Wasserstein spline and second-order transport models, which optimize the curve itself.Source: §2 · p. 3
  • Schedule-optimization methods vary discretization, while this work derives an allocation quantity by projecting modified-equation error through the continuity equation.The resulting formulation also treats the marginal-preserving velocity representative as an optimization variable.Source: §2 · p. 3
  • The weighted Hodge projection isolates the modified-equation defect’s gradient component as the contributor to leading marginal W2 error, leaving gauge freedom for numerical-integrability optimization.Other gauge-related approaches address conservativity, particle diversity, or field design; curvature and strain/vorticity methods diagnose trajectory-level difficulty.Source: §2 · p. 3
  • The schedule layer is a continuous resource-allocation problem, whereas representative selection is PDE-constrained optimization with weighted Poisson projections.A local theorem bridges solver-level error and the variational limit connects discrete and continuum optimization.Source: §2 · p. 3

3 MARGINAL-PRESERVING GAUGES

For a fixed density path, marginal-preserving gauges are velocity additions whose weighted divergence vanishes. These fields preserve the continuum density evolution, but nontrivial gauge optimization requires multiple dimensions under standard decay assumptions.

3 MARGINAL-PRESERVING GAUGES

  • The analytic setting assumes positive densities and sufficient smoothness, with integrations by parts requiring no boundary contribution.The stated domains are a flat torus, a smooth compact boundaryless domain, or Euclidean space with sufficient decay.Source: §3 · p. 4
  • A solenoidal gauge is a velocity addition u satisfying ∇·(p_tu_t)=0, so adding it to a field leaves the density curve unchanged.The constraint may equivalently be read in distributions for smooth fields.Source: §3 · p. 4
  • Under standard decay assumptions, the feasible gauge set is trivial in one dimension, making nontrivial gauge optimization intrinsically multidimensional.In one dimension the constraint implies p_tu_t is constant in space, and decay forces that constant to zero.Source: §3 · p. 4

4 SOLVER-AWARE DENSITY DEFECT

The solver-aware density defect measures the marginal effect of a numerical step through the first nonzero modified-equation differential. Its solver-specific form includes Euler acceleration and extends to higher-order methods.

4 SOLVER-AWARE DENSITY DEFECT

  • Theorem 1 relates the numerical step’s leading density evolution defect to the weighted divergence of the first modified-equation differential.The identity holds in distributions and in negative Sobolev norms controlled by the stated remainder.Source: §4 · p. 4
  • Euler’s leading modified-equation differential is acceleration a_w=∂_tw+∇_ww, while midpoint and Heun first contribute at order h^2 in the modified field.The same density-level construction applies by passing each method’s first nonzero differential through weighted divergence.Source: §4 · pp. 4–5
  • The relevant solver defect is its density-active component, which determines leading marginal transport error rather than the full trajectory distortion.The framework projects the modified-equation differential onto the weighted Wasserstein tangent space.Source: §4 · pp. 4–5

5 DENSITY-ACTIVE WASSERSTEIN GEOMETRY

The density-active projection isolates the part of a solver defect that changes the marginal, excluding solenoidal motion invisible to Wasserstein tangent geometry.

5 DENSITY-ACTIVE WASSERSTEIN GEOMETRY

  • The weighted-gradient projection is characterized by orthogonality to the solenoidal component under the density-weighted inner product.The weak projection problem is the normal equation for projection onto the closed gradient subspace.Source: §5 · p. 5
  • Only the gradient projection contributes to the leading marginal defect; solenoidal modified differentials can bend trajectories without producing leading density error.The weighted divergence and weighted negative norm of a field are determined by this projection.Source: §5 · pp. 5–6
  • Applying the projection to the modified-equation differential yields a solver-aware local residual tied to the local Wasserstein defect.The resulting theorem identifies the leading local marginal W2 defect with a Wasserstein tangent norm.Source: §5 · p. 5

6 SOLVER-ORDER TIMESTEP ALLOCATION

The section allocates steps according to solver-order-dependent residual impact, including its amplification to the terminal time. It also characterizes schedule sensitivity and shows why propagation weighting can matter even when the local residual is constant.

6 SOLVER-ORDER TIMESTEP ALLOCATION

  • The propagated density defect weights the geometric residual by its amplification from the step time to the terminal time.The canonical source norm identifies the geometric residual with Rr(t), giving Wr(t) = K(1, t)Rr(t).Source: §6 · p. 6
  • The unique continuous allocation minimizes the stated finite-N surrogate by placing steps according to the terminally weighted residual raised to the solver-order-dependent exponent.For order r, the exponent is 1/(r+1); Euler, second-order, and third-order methods use powers 1/2, 1/3, and 1/4.Source: §6 · pp. 6–7
  • The quantile grid exactly minimizes the finite-N surrogate, but this is not a claim of exact minimization of the full nonlinear terminal error.Its local step asymptotics and O(N^-2) endpoint approximation are stated in the appendix.Source: §6 · pp. 6–7

6.1 ALLOCATION ROBUSTNESS AND PROPAGATION

  • If the estimated allocation profile differs in log by at most ε, the surrogate ratio is bounded by exp(2qε/(q + 1)).Schedule quality depends on normalized allocation density rather than absolute residual scale, while regularization controls sensitivity near zeros.Source: §6.1 · p. 7
  • Propagation weighting is consequential even for a constant local residual, for which the residual-only objective ratio can grow in constructed smooth scalar flows.The passage establishes the construction without specifying the growth expression here.Source: §6.1 · p. 7

7 GAUGE OPTIMIZATION AND ITS DISCRETE LIMIT

The section selects velocity representatives within a fixed density path to reduce discrete solver error. Under stated regularity conditions, the continuum Euler objective is the variational limit of the discrete local-W2 objective.

7 GAUGE OPTIMIZATION AND ITS DISCRETE LIMIT

  • The feasible optimization direction is the solenoidal projection of the ambient gradient, keeping the search within fields that realize the same density path.The discrete local-W2 objective is defined for a partition, and its continuum counterpart minimizes the solver-induced local transport defect while holding the marginal curve fixed.Source: §7 · pp. 7–8
  • Under uniform expansion, bounded-remainder, and Lipschitz assumptions, discrete Euler local-W2 objectives Γ-converge to the continuum representative-selection functional.Convergent sequences of discrete minimizers have continuum-minimizing limits; the result also holds on noncompact sets with an equicoercive regularizer.Source: §7 · p. 8
  • For Gaussian paths, zero leading Euler bias is characterized by matching the covariance derivative with twice the velocity-covariance product.The isotropic feasible lifts add a skew component to the radial gradient lift.Source: §7 · p. 8
  • The radial Gaussian gradient lift is minimum-kinetic-energy, yet its Euler covariance defect is (d/2)|s¨(t)|h² + O(h³), while a distinct skew lift achieves O(h³).When s¨/s > 0, exact skew gauges exist in even dimensions; this lift cancels the leading covariance defect.Source: §7 · p. 8

8 CONTROLLED AND PRETRAINED VALIDATION

Controlled models test the allocation and representative-selection mechanisms, while the pretrained CIFAR-10 EDM study isolates timestep placement by comparing Euler grids at matched cost. The proposed grid improves low-NFE Euler FID and approaches the same dense Euler limit faster.

8 CONTROLLED AND PRETRAINED VALIDATION

  • Exact Gaussian and analytic mixture checks support the predicted local-W2 orders, gauge obstruction, quantile surrogate, propagation weighting, and square-root allocation.Hard-GMM comparisons support competitiveness rather than uniform dominance.Source: §8 · p. 8
  • The proposed Euler grid reduces FID relative to log-σ at 8, 16, and 20 NFE and improves over EDM-ρ at every displayed budget.At 35 NFE the gain nearly vanishes; network, update rule, NFE, seeds, and runtime are matched so timestep placement is the only design variable.Source: §8 · p. 9, Table 1 · p. 9
  • Across tested budgets from 8 to 512 NFE, the proposed grid is closest to a 1000-step Euler reference, while all three Euler schedules are indistinguishable at 1000 NFE.This supports faster convergence to the same Euler limit, complementary to higher-order solver engineering.Source: §8 · p. 9

9 CONCLUSION

The paper identifies density-active modified-equation error as the leading source of marginal transport error, then uses it to allocate evaluations and select velocity representatives. Controlled and pretrained experiments validate the mechanism, and matched-seed tests show faster approximation of the same high-NFE Euler flow.

9 CONCLUSION

  • The density-active component of a solver’s modified equation determines its leading marginal transport error.The framework projects the defect onto the Wasserstein tangent space, excluding solenoidal motion invisible to the density.Source: §9 · p. 9
  • The framework yields a solver-order allocation law and a representative-selection objective over marginal-preserving gauges, with a discrete-to-continuum justification for Euler.Source: §9 · p. 9
  • Controlled and pretrained experiments validate the mechanism, while matched-seed dense-grid convergence shows faster approximation of the same high-NFE Euler limit.DPM++ and EDM Heun change the integration formula, whereas this method allocates a fixed low-order solver’s evaluation budget.Source: §9 · p. 9

A PRETRAINED EDM CIFAR-10 DETAILS

The appendix checks the density-aware allocation and gauge mechanisms in exact Gaussian and mixture settings, then evaluates fixed-budget Euler scheduling on pretrained CIFAR-10. The tests distinguish timestep placement from stronger solver formulas and compare low-NFE endpoints with a dense numerical Euler reference.

A PRETRAINED EDM CIFAR-10 DETAILS

  • The Euler comparison holds the network, update rule, and NFE fixed, changing only timestep placement; its reusable profile allocates more evaluations to the low-noise region.The offline schedule uses quantiles of the square root of the propagated density-active profile.Source: A PRETRAINED EDM CIFAR-10 DETAILS · pp. 12–13, Figure A.1 · p. 13
  • The density-aware grid has the smallest endpoint RMSE at every tested budget against a 1000-step Euler reference.The comparison uses 1024 matched samples and budgets from 8 to 512 NFE; the reference is numerical, not an exact ODE solution.Source: A PRETRAINED EDM CIFAR-10 DETAILS · p. 13, Figure A.2 · p. 13
  • At 1000 NFE, the three independent FID-5k values differ by less than 0.002, consistent with faster convergence rather than a different asymptotic sampler.Generation times stay within a few percent.Source: A PRETRAINED EDM CIFAR-10 DETAILS · p. 14

B.1 GAUSSIAN CANCELLATION AND PARITY

  • Exact Gaussian checks find zero minimized linear residual in dimensions 2 and 4, but a positive residual in dimensions 3 and 5.The parity obstruction reflects cancellation by an orthogonal complex structure in even dimensions, with one scalar direction remaining in odd dimensions.Source: §B.1 · p. 15

B.2 WHY FULL MATERIAL ACCELERATION IS NOT THE ALLOCATION SIGNAL

  • In a four-dimensional skew-gauge example, the density-active residual is about 5 × 10^-15 while the full material-acceleration proxy reaches about 136.This is an exact counterexample to treating greater trajectory curvature as necessarily greater marginal error.Source: §B.2 · pp. 15–16

D UNIFIED ADDITIONAL BASELINES AND ESTIMATOR VALIDATION

  • The unified hard-GMM protocol also evaluates particle-only dual residuals, directional propagation weighting, a trajectory-coupling oracle, and fair-NFE Heun.It includes Jo–Choi velocity-variation resampling and a hybrid experiment that allocates Heun corrections to proxy-selected intervals.Source: §C.3 · p. 16, §3 · p. 18

C.3 VALIDATION-SELECTED AND MIXED-ORDER BASELINES

  • For hybrid Euler/Heun allocation, the three nonuniform selectors choose nearly the same late-time intervals and all improve over uniform selection.The caption attributes their close proxy values to shared late-time concentration on this benchmark, not equivalence of the underlying error signals.Source: Table C.5 · p. 17, Figure C.6 · p. 17

C HARD-GMM ALLOCATION STUDY

  • On the hard-GMM path, the density-active schedule improves over uniform Euler at the predicted exponent and remains competitive with simpler proxies.The proxies are strongly correlated on this benchmark, so the study tests whether the theory-linked schedule remains useful under that condition.Source: C HARD-GMM ALLOCATION STUDY · p. 16, §C.2 · p. 16, §C.3 · p. 18

D.2 UNIFIED RESULTS

The unified results examine density-aware low-NFE Euler schedules, particle-only residual estimation, and checks that distinguish density-level from trajectory error while exploring propagation weighting and gauge choice. They also characterize projected gauge optimization, including when feasible velocity changes can cancel leading Euler bias and when Gaussian covariance paths cannot be matched at second order.

D.2 UNIFIED RESULTS

  • At NFE 8 and 16, the validation-selected density schedule reduces MMD from 0.02013±0.00232 to 0.00939±0.00138 and from 0.00646 ± 0.00184 to 0.00462 ± 0.00161, respectively.Its NFE 8 result is comparable to the Euler–Heun AMR and Jo–Choi schedules, whose MMD values are 0.01052 ± 0.00179 and 0.00960 ± 0.00140.Source: §D.2 · pp. 18, 20, Table D.6 · p. 19, Figure D.7 · p. 19

D.3 PARTICLE-ONLY RESIDUAL

  • Across 31 profile times, the finite RBF particle-dual estimator correlates 0.9760 (Pearson) and 0.9907 (Spearman) with the exact score–divergence profile, and its schedule achieves MMD 0.00995 ± 0.00166 at NFE 8 and 0.00398 ± 0.00142 at NFE 16.The schedule recovers difficult time-region ordering without access to ∇log pt; reported profile costs imply amortization after about 182 batches for the particle-dual profile versus about 80 for the exact profile, under the stated implementation.Source: §D.3 · p. 20, Figure D.8 · p. 20

E.1 ANALYTIC MIXTURE PATH AND OBJECTIVE SPECIFICITY

  • The analytic mixture checks show that local marginal error, paired-trajectory error, and terminal feature error can favor different timestep geometries.The density grid is closer to the trajectory oracle than to the terminal-feature oracle, while the terminal marginal oracle optimizes a separate fixed random-feature MMD objective.Source: §E.1 · p. 21, Table E.7 · p. 21

E.2 ORDER, PROPAGATION, AND PROFILE ROBUSTNESS

  • On the anisotropic propagation example, residual-only scheduling can underperform because it ignores amplification before the terminal time, while K-weighting nearly matches the direct oracle and square-root allocation is less sensitive to multiplicative profile noise than linear-power allocation.The K-weighted result motivates the empirical terminal factor used in the CIFAR-10 experiment.Source: §E.2 · p. 21, Table E.8 · p. 21

F APPLICATIONS AND OPTIMIZATION RELEVANCE

  • Adaptive scheduling uses the marginal backward-error residual and terminal propagation profile K(1, t)Rloc(t), while the density-active residual also diagnoses where finite-step bias arises and where steps or higher-order integration may help.A high full material-acceleration interval need not matter when its acceleration is not density-active; a Gaussian curvature-distractor experiment provides a counterexample.Source: F APPLICATIONS AND OPTIMIZATION… · p. 22
  • Gauge-aware design can select numerically favorable density-equivalent velocities or add diversity-promoting gauges subject to a bias constraint, but the controlled experiments favor conservative corrections with line search or validation.Projected descent follows the negative projected bias gradient over pt-solenoidal directions and decreases bias for sufficiently small positive steps when that gradient is nonzero.Source: F APPLICATIONS AND OPTIMIZATION… · p. 22, I PROJECTED GAUGE OPTIMIZATION · p. 27

G.2 ISOTROPIC GAUSSIAN SEPARATION

  • In the isotropic Gaussian example, a nonzero rotational, non-gradient velocity representative is Euler-optimal to the displayed order despite not being the Otto-horizontal minimum-kinetic-energy representative.The Euler covariance agrees with the gradient lift through order h^2.Source: §G.2 · p. 24

H SOLVER-AWARE EXTENSIONS AND VARIATIONAL LIMITS

  • The order-r result links the leading modified-equation defect to a Wasserstein density perturbation, while the variational-limit argument supports convergence of discrete minimizers under uniform expansion and compactness, or equicoercivity on noncompact sets.Pointwise local asymptotics alone do not guarantee convergence of minimizing sequences.Source: §H.1 · p. 24, §H.5 · p. 26

I PROJECTED GAUGE OPTIMIZATION

  • Weighted Poisson projection decomposes a velocity defect into a gradient component and a remainder orthogonal to all gradients, separating density-visible change from solenoidal distortion.The potential solves ∇·(pt∇ϕt) = ∇·(ptzt), with normalization modulo constants.Source: I PROJECTED GAUGE OPTIMIZATION · pp. 27–28
  • For regularized representative selection, the residual-derived term plus λu lies in the weighted gradient subspace at a local minimizer, supporting projected-gradient descent over feasible gauges.The descent result assumes a finite-dimensional parameterization or discretization and a Lipschitz gradient.Source: I PROJECTED GAUGE OPTIMIZATION · pp. 28–29
  • Leading Euler density bias is exactly cancellable by a feasible linearized gauge iff its negative lies in the defect map’s range; otherwise the minimal residual is the distance to the attainable set.If the range is not closed, arbitrarily small residual requires membership in its closure, even when exact cancellation is unavailable.Source: I PROJECTED GAUGE OPTIMIZATION · p. 28

J EXACT AND OBSTRUCTED GAUSSIAN GAUGES

  • For Gaussian paths, a linear lift has zero leading Euler covariance bias iff it satisfies the path’s Lyapunov constraint and a Riccati-type condition, equivalently making its acceleration density-invisible.The isotropic case reduces to a parity obstruction for skew-symmetric controls.Source: J EXACT AND OBSTRUCTED GAUSSIAN GAUGES · p. 29
  • Some Gaussian covariance paths cannot achieve exact one-step second-order Euler covariance matching with any linear marginal-preserving gauge because the required covariance-cone condition fails.The condition requires a positive-semidefinite right-hand side.Source: J EXACT AND OBSTRUCTED GAUSSIAN GAUGES · p. 30

J.1 ISOTROPIC CLASSIFICATION AND PARITY OBSTRUCTION

The appendix characterizes when marginal-preserving gauges cancel Euler bias, extends the analysis to additional density paths, and contrasts solver-aware representatives with optimal-transport lifts.

J.1 ISOTROPIC CLASSIFICATION AND PARITY OBSTRUCTION

  • Even-dimensional isotropic paths with positive ¨s/s admit a manifold of exact cancellation gauges, while odd dimensions retain residual and nonpositive ¨s/s favors A = 0.The exact gauges are parametrized by orthogonal complex structures, isomorphic to O(2m)/U(m).Source: §J.1 · pp. 30–31

L.1 PROOF OF THEOREM 3

  • The residual-only schedule is exact when the transport norm is nonexpansive along the reference flow; directional amplification gives a propagation-aware extension.Source: §L.1 · p. 32

L.2 OFFLINE PROFILE COST AND AMORTIZATION

  • Profile cost depends on acceleration and divergence estimation choices, while reusing a profile across samples amortizes its cost; wall-clock breakeven remains implementation-specific.Source: §L.2 · p. 32

L.3 A PARTICLE-ONLY DUAL RESIDUAL ESTIMATOR

  • A particle-only variational estimator uses samples and acceleration evaluations to recover the geometric residual without separately estimating score or divergence.At the population optimum, its objective equals the squared norm of the projected acceleration; a finite RBF version is validated on the hard-GMM path.Source: §L.3 · p. 33

M MIXTURES AND PRACTICAL PARAMETERIZATIONS

N ADDITIONAL EXPLICIT EXAMPLES

  • Gauge availability and cancellation depend on the path: decaying one-dimensional paths have no nontrivial gauges, while anisotropic and radial paths allow cancellation under stated conditions.The conditions are ¨s/s > 0 for the conformal anisotropic Gaussian path and nonnegativity of the radial expression in equation 123.Source: §N.1 · p. 33, §N.2 · p. 34, §N.3 · p. 35
  • Continuum-invisible gauges can create Euler marginal bias, while Gaussian lifts can have zero leading bias despite arbitrarily large full material acceleration.Source: §N.5 · p. 35, §N.6 · p. 36

O CONNECTION VIEWPOINT

  • The solver selects a horizontal representative by minimizing modified-equation density bias, distinct from the kinetic-energy criterion of optimal transport.The density curve is fixed; the optimization acts over its Lagrangian lifts and is induced by the specified solver's leading defect.Source: O CONNECTION VIEWPOINT · pp. 36–37

Related papers

Loading 2610.29475v1…