PaperlayerView exact revision on arXiv

Source-linked AI summary

Drifting Fields are not Conservative

Leonard T. Franz, Sebastian Hoffmann, Tim Weiland, Bernhard Schölkopf, Georg Martius

arXiv:2604.06333v3cs.LGcs.CV

TL;DR

Drifting trains generators by moving samples toward data with a vector field, raising whether this procedure is equivalent to scalar-loss optimization. The paper introduces sharp normalization to obtain a conservative score-difference field for radial kernels, and reports performance comparable to the original drift field.

  • Problem

    Drifting uses a vector-valued transport field, which can represent non-conservative motions, leaving open whether its training implicitly optimizes a scalar loss.

    Source: §1 · p. 2

  • Method

    The paper replaces the original normalization with sharp-kernel normalization, yielding a conservative field that is a difference of sharp-KDE log-scores.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Sharp normalization performs comparably to the original drift field, while the original field is generally non-conservative except for the Gaussian kernel.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The sharp field has a scalar log-KDE potential and exact equilibrium identifiability, extending score-difference transport to non-Gaussian radial kernels.

    Source: §1 · p. 2, §7 · p. 9

    Because the potential depends on the current generated distribution through stop-gradient, training follows a changing sequence of landscapes rather than one fixed objective.

    Source: §4 · p. 6

Abstract

from arXiv · show

Drifting models have recently gained attention for generating high-quality samples in a single forward pass. During training, they learn a push-forward map by following a vector-valued field, the drift field. We ask whether this procedure is equivalent to optimizing a scalar loss and find that, in general, it is not: drift fields are not conservative and cannot be written as the gradient of any scalar potential. We identify the position-dependent normalization as the source of non-conservatism, with the Gaussian kernel as the unique radial exception. Guided by this, we introduce the sharp kernel $k^\#$ and a sharp-normalized drift field that is conservative for general radial kernels. The resulting vector field is the gradient of a scalar potential that can be optimized directly using stochastic gradient descent. Moreover, the field has the form of a score difference of kernel density estimates, and gives exact equilibrium identifiability. Thus, sharp normalization closes the gap to related literature, such as Wasserstein gradient-flows and denoising score matching, also for non-Gaussian kernels. Empirically, sharp normalization preserves the performance of the original drifting objective, suggesting that the non-conservative flexibility is not required for high-quality generation.

1 Introduction

The introduction asks whether drifting’s vector-field training implicitly optimizes a scalar loss, since its stop-gradient objective can implement non-conservative transports. It argues that the original normalization generally prevents a scalar-potential interpretation, while sharp normalization supplies a conservative alternative with comparable empirical performance.

1 Introduction

  • The paper asks whether drifting implicitly optimizes a scalar loss, because its stop-gradient objective can implement transport fields that are not gradients of scalar potentials.The question is whether drifting’s additional flexibility is needed or whether a conservative field suffices.Source: §1 · p. 2
  • For most kernel choices, the original position-dependent normalization yields a non-conservative drift field that no scalar potential can reproduce.The paper describes the earlier claim as correct for that specific normalization, but overstated as a claim that loss-based training with normalization is impossible.Source: §1 · p. 2
  • Replacing the original kernel with the sharp kernel yields a conservative field whose log-KDE potential is a KL divergence between sharp-kernel KDEs of generations and data.The paper presents this potential as an explicit loss for the sharp-normalized field.Source: §1 · p. 2
  • The sharp field has the form of a difference between data and generated-distribution KDE log-scores, extending score-difference transport to non-Gaussian radial kernels.The field uses sharp-kernel KDEs.Source: §1 · p. 2
  • Sharp normalization performs comparably to the original drift field, and the scalar-potential formulation removes the need to explicitly construct a sample transport field.The experiments suggest the non-conservative component does not contribute significantly to performance.Source: §1 · p. 2

2 Background

The background defines drifting as generator training through a field that transports generated samples toward the data distribution. It introduces kernels, KDE normalization, and conservatism as the condition for representing a field through a scalar potential.

2 Background

  • A drifting generator maps noise to samples and updates its outputs toward the data distribution by following a drift field in a fixed-point training procedure.The field is antisymmetric and vanishes when the generated distribution equals the data distribution; under stated regularity conditions, a vanishing field implies equality.Source: §2 · pp. 2–3
  • A radial kernel depends only on squared distance, and the normalization factors used in drifting are KDEs of the data and generated distributions.For positive-definite kernels, these KDEs can also be viewed as pointwise kernel mean embeddings.Source: §2 · p. 3
  • A field is conservative when it is the gradient of a scalar potential; equivalently, its Jacobian is symmetric, with off-diagonal asymmetry measuring deviation from conservatism.The paper also characterizes conservatism through path-independent line integrals.Source: §2 · p. 3

3 Drift Fields Are Not Generally Conservative

The section shows that drifting fields are generally non-conservative because of their position-dependent normalization, while the Gaussian kernel is an exception. It traces the obstruction to a normalization mismatch and relates the unnormalized field to a conservative potential.

3 Drift Fields Are Not Generally Conservative

  • The Laplacian and rational quadratic drift fields have nonzero curl, whereas the Gaussian field has zero curl and admits a scalar potential.A nonzero curl means no scalar potential has the corresponding field as its gradient.Source: §3 · p. 3

3.1 Nonconservatism is Caused by the Normalization

  • Sharp and flat profiles differ for Laplacian, Matérn-7/2, and rational quadratic kernels, while Gaussian profiles agree up to scale.The figure compares normalized radial profiles.Source: Figure 2 · p. 5
  • For radial kernels, the unnormalized drifting field is conservative and can be produced by a squared-MMD potential; adding position-dependent normalization introduces non-conservatism.The unnormalized field’s conservative property identifies normalization as the source of the obstruction.Source: §3 · p. 3, §3.1 · pp. 4–5

3.2 The Gaussian Kernel: An Exception where Normalization Retains Conservatism

  • The Gaussian kernel is the exception for which normalization retains conservatism, with the drift field expressible through a scalar potential.For Gaussian kernels, the drift sub-fields satisfy the relation used to identify that potential.Source: §3.2 · p. 5

4 Sharp Normalization

Sharp normalization makes drifting conservative for general radial kernels and supports training through a scalar potential. It also relates the field to KDE score differences and changes how drift magnitudes behave relative to the original field and the MMD gradient.

4 Sharp Normalization

  • Sharp normalization yields a conservative field for any radial kernel.It replaces the original position-dependent normalization.Source: §4 · p. 5
  • The sharp-normalized field is a difference of log-scores for KDE-smoothed target and generated distributions, extending score-difference transport to non-Gaussian radial kernels.Source: §4 · p. 6
  • Conservatism lets training use an expected scalar-potential loss with standard automatic differentiation, without explicitly constructing the transport field.Source: §4 · p. 6
  • With a connected domain and a strictly positive, continuously differentiable, characteristic sharp kernel of constant finite mass, V#=0 everywhere if and only if p=q.Source: §4 · p. 6
  • Among radial kernels, only the Gaussian makes the original and sharp-normalized fields coincide up to a global constant.Source: §4 · p. 6

4.1 Comparing the Original and Sharp-Normalized Fields

  • Independent rescaling generally changes field magnitude and direction; Figure 3 shows the sharp field between the original field’s unbounded tails and the MMD gradient’s fast decay.The sharp field also has improved tail behavior over the MMD gradient, including a bounded signal for the Laplacian kernel.Source: §4.1 · p. 7, Figure 3 · p. 7

5 Experiments

Experiments compare the log-KDE objective with drifting variants on MNIST, Fashion-MNIST, and ImageNet. The scalar loss matches or beats drifting in FID across all three datasets, with bandwidth and classifier-free guidance affecting the ImageNet setting.

5 Experiments

  • Classifier-free guidance is essential for competitive ImageNet results with the log-KDE objective.The objective incorporates guidance using a weighted logsumexp over a mixture of generated and unconditional data distributions.Source: §5 · p. 8
  • Pixel-space experiments compare the original drifting objective, its corrected implementation, and log-KDE, while ImageNet experiments replace only the loss in Deng et al.'s codebase.MNIST and Fashion-MNIST omit several enhancements applicable to either objective; ImageNet uses the ablation-default configuration.Source: §5 · pp. 7–8

5.1 Results

  • Across all three datasets, log-KDE matches or beats drifting in FID; on MNIST, Gauss# scores 7.21 versus 37.62 and Laplace# scores 29.12 versus 45.02.The corrected Naive implementation also generalizes to smaller kernel widths and consistently outperforms Deng et al.'s original implementation.Source: §5.1 · p. 8, Table 2 · p. 9
  • On ImageNet, both log-KDE variants perform similarly and need larger bandwidths (σ ∈[0.2, 0.4]) than drifting (σ ∈[0.05, 0.2]); the reason remains unresolved.Laplace# tolerates smaller widths than the other log-KDE variant.Source: §5.1 · p. 9
  • The results support log-KDE as an effective scalar-loss alternative, suggesting drifting's non-conservative expressiveness is not required for high-quality generation.Figure 4 plots FID-50k against kernel width for MNIST, Fashion-MNIST, and ImageNet; its lines show medians and its shaded regions show the range across seeds.Source: Figure 4 · p. 8, §5.1 · p. 9

6 Related Work

The paper connects sharp-normalized drifting to Wasserstein gradient flows, score matching, and kernelized particle flows. It also distinguishes the Gaussian case from other radial kernels and relates sharp kernels to companion kernels and equilibrium identifiability.

6 Related Work

  • Sharp-normalized drifting connects to Wasserstein gradient flows and denoising score matching through their shared score-difference transport form.The Wasserstein KL-flow particle velocity is ∇log p −∇log qt, and the sharp field has the same form using sharp-kernel KDEs.Source: §6 · p. 9
  • The Gaussian is the unique radial kernel with k# ∝k, so only in that case does the original drifting field already coincide with the KDE score difference.For other radial kernels, sharp normalization restores this correspondence.Source: §6 · p. 9
  • For companion-elliptic kernel families, including Matérn kernels, Lee's companion kernel coincides with the sharp kernel up to scale; sharp normalization also yields a simple equilibrium-identifiability argument.The argument uses the same kernel in both numerator and denominator of the KDE log-ratio.Source: §6 · p. 9

7 Conclusion

The conclusion attributes non-conservatism to a mismatch between the original normalization and the sharp-kernel KDE gradient. Sharp normalization restores a conservative score-difference field, with comparable experimental performance, while practical convergence and kernel assumptions remain limitations.

7 Conclusion

  • For radial kernels other than the Gaussian, the original position-dependent normalization creates transversal magnitude variations and a denominator mismatch that prevents general conservatism.The numerator is the gradient of a sharp-kernel KDE, but the original denominator uses the unsharp kernel's KDE.Source: §7 · p. 9
  • Dividing by the corresponding sharp KDE restores the score difference ∇log pKDE[k#] −∇log qKDE[k#], yielding a conservative scalar log-KDE objective.The formulation also gives exact equilibrium identifiability and extends the Gaussian score-flow interpretation to non-Gaussian radial kernels.Source: §7 · p. 9
  • Sharp normalization performs comparably to the original drifting objective, suggesting its original denominator's non-conservative component is not required for high-quality generation.Source: §7 · p. 9
  • Population-level theory does not directly imply convergence with minibatch KDE estimates, finite-batch optimizers, neural generator parameterizations, feature-space losses, or classifier-free guidance.The cleanest theory assumes radial kernels with suitable sharp companions; the denoising interpretation additionally requires positive, integrable k#, and performance is bandwidth-sensitive.Source: §7 · p. 10

Impact Statement

The paper frames its contribution as theoretical analysis of drifting objectives and their relation to conservative fields and kernel density estimation. It discusses potential benefits and risks, then gives appendix results on conservatism, dimensionality, and kernel requirements.

Impact Statement

  • The work is a theoretical analysis of generative-model training objectives, introducing no datasets, large-scale pretrained models, or deployment-ready systems.Source: Impact Statement · p. 12
  • Potential benefits include simpler implementation of one-step generative models and closer ties to established frameworks; potential misuse includes synthetic media for misinformation, fraud, or non-consensual imagery.The risk is described as indirect and shared with generative modeling more broadly; the experimental models are small, benchmark-focused, weaker than public systems, and unreleased.Source: Impact Statement · p. 12

A.1 Curl and Path Independence

  • A vector field is conservative if and only if its Jacobian is symmetric.Source: §A.1 · p. 12

A.2 Relation of conservatism of Vunnorm.

  • For all p and q, conservatism of the composite unnormalized field implies that each subcomponent must be individually conservative.Source: §A.2 · p. 13

A.3 Nonconservatism in Arbitrary Dimensions

  • A two-dimensional counterexample extends to every dimension n ≥2, using restriction and embedding arguments.Restriction preserves conservatism, while embedding a counterexample produces one in the next dimension.Source: §A.3 · p. 14

A.4 Radiality is a Requirement for Conservatism

  • For the unnormalized field to be generally conservative, the kernel must be radial: it depends only on the distance between its inputs.Source: §A.4 · p. 14

B Detailed Kernel Derivations

The appendix derives sharp and flat kernels for common radial kernels. Gaussian kernels are unchanged up to scale, while other examples yield distinct forms and Matérn order shifts.

B.1 Gaussian Kernel

  • For Gaussian kernels, both the sharp and flat are proportional to the original kernel, with k#(x, y) = σ2k(x, y).The Gaussian is the fixed point of these operations up to constants.Source: §B.1 · p. 15

B.2 Laplacian Kernel

  • For the Laplacian kernel, the sharp is σ(∥x −y∥+ σ)k(x, y), while the flat is k(x, y)/(σ∥x −y∥) and is singular at x = y.The singular flat corresponds to the p = 0 boundary case.Source: §B.2 · p. 15, §B.4 · p. 17

B.3 Rational Quadratic Kernel

  • For the rational quadratic kernel k(x, y) = (1 + ∥x − y∥²/σ²)⁻², the sharp is k#(x, y) = (σ²/2)k(x, y)¹/².Source: §B.3 · p. 16

B.4 Half-Integer Matérn Kernels (ν = p + 1

  • For half-integer Matérn kernels, the sharp shifts the order from ν to ν + 1 and the flat shifts it to ν − 1 for p ≥ 1, with corresponding length-scale rescaling.The rescaling tends to 1 as ν → ∞, recovering the Gaussian fixed point; the Laplacian case has no regular flat.Source: §B.4 · p. 18

C Conservative Training

The appendices derive a scalar training objective when the drift field is conservative, connect a stop-gradient form to squared MMD, and establish sharp-normalized equilibrium identifiability. They also analyze a normalization mismatch in the referenced algorithm and summarize experimental configurations.

C Conservative Training

  • When the drift field is conservative, its potential yields a simplified scalar training objective through the multivariate chain rule and reparameterization.Source: C Conservative Training · pp. 18–19

D Squared MMD as a Stop-Gradient Objective

E Identifiability of Sharp-Normalized Equilibria

F Wrong Drifting Normalization

  • Concatenating positive and negative samples before softmax produces a shared normalization factor NposZp(xi) + NnegZq(xi), rather than separate factors.The passage states that softmax over y+ and y− independently would yield the separate normalization factors.Source: F Wrong Drifting Normalization · p. 21

F.1 Naive Drifting Algorithm

  • The naive algorithm computes separate normalized positive and negative kernels, forms their products, and uses the resulting weights to subtract the two sample averages.Source: §F.1 · p. 22

G Additional Training Details

  • The experimental details include training and architecture configurations, hardware requirements, and ADM-suite evaluation metrics for MNIST and Fashion-MNIST.Source: G Additional Training Details · p. 23

G.1 Class-Conditional Data Loading

The appendix replaces per-class sample queues with direct class-conditional sampling for MNIST and Fashion-MNIST, describes feature normalization using nearest-neighbor statistics, and adapts classifier-free guidance to the log-KDE objective.

G.1 Class-Conditional Data Loading

  • For MNIST and Fashion-MNIST, direct class-conditional batch sampling avoids the queue’s memory overhead and warmup while sampling uniformly from each class distribution.The queue approach can require at least 201GB per host for 1,024 pixel-space ImageNet samples per class and may not reflect the true class-conditional distribution.Source: §G.1 · pp. 23–24

G.2 Feature Normalization

  • For MNIST and Fashion-MNIST, per-class EMAs of mean k-nearest-neighbor distances normalize real and generated features, making kernel widths interpretable in mean-neighbor-distance units.The method skips the first 300 optimization steps while the EMA stabilizes.Source: §G.2 · p. 24

G.3 Classifier-Free Guidance

  • The classifier-free-guidance log-KDE objective replaces the conditional pushforward expectation with one over a mixture of conditional generated samples and unconditional data samples.The mixture weights are stated to match the target mixture in expectation for any sampled α.Source: §G.3 · p. 24

H Additional Results

The additional results show uncurated ImageNet samples from a log-KDE-Laplace# run and samples from MNIST and Fashion-MNIST across drift variants and log-KDE. The ImageNet run is an ablation and is not directly comparable to the cited state-of-the-art runs.

ImageNet

  • The ImageNet results are from ablation runs after 30,000 steps and are not directly comparable to Deng et al. [5] runs using larger models and 200,000 training steps.Source: H Additional Results · p. 26
  • FID-50k = 9.35 for uncurated ImageNet samples from log-KDE-Laplace# after 30,000 steps with kernel width 0.3.Source: Figure 5 · p. 26

MNIST and Fashion-MNIST

  • MNIST and Fashion-MNIST figures show real and generated samples across drift variants and log-KDE, using each variant’s kernel width with its lowest FID score.Source: Figure 6 · p. 27

Related papers

Loading 2604.06333v3…