Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

1 Introduction

• For data x = Az with orthonormal columns of A and d < D, the score splits into on-support and orthogonal components that encode latent distribution information and drive subspace recovery, respectively.Here d is the intrinsic dimension and D the ambient dimension.Source: §1 · p. 2
• The proposed encoder-decoder network with skip-layer connections has score-approximation guarantees, with size exponential in 1/ϵ and an exponent depending on intrinsic dimension d.The network also yields an L2 score-estimation guarantee whose rate depends on d and early stopping time t0, not ambient dimension D.Source: §1 · p. 2
• On-support distribution recovery converges in total variation at e O(n^−1/(2(d+5))); the orthogonal component vanishes and the data support is approximately recovered.The paper obtains these guarantees by simulating a discretized backward process with the learned neural score estimator.Source: §1 · p. 2

1.1 Related work

• Earlier sampling analyses assume an accurate score estimator; De Bortoli (2022) studies low-dimensional manifold data but obtains a rate with exponential dependence on manifold diameter.Other cited sampling guarantees include assumptions such as bounded moments or a log-Sobolev inequality.Source: §1.1 · p. 3
• Block et al. (2020) provide L2 score-estimation guarantees, but their error bound depends on unknown Rademacher complexity, whereas this work specifies a neural network class and an explicit error bound.Song et al. (2020a) and Liu et al. (2022) study score and distribution estimation from an asymptotic-statistics perspective.Source: §1.1 · p. 3

Forward and backward SDEs

• The forward Ornstein–Uhlenbeck process adds Gaussian noise, has conditional law N(α(t)X0, h(t)ID), and approaches a standard Gaussian marginal at a sufficiently large horizon.The paper uses this forward process as the basis for generating data by reversing time.Source: §2 · pp. 3–4
• The reverse-time SDE uses the score ∇log pt, but practical generation substitutes a standard Gaussian initialization and a neural-network score estimator before discretized simulation.The true score and the forward process's terminal distribution are unknown.Source: §2 · p. 4

Score matching

• Because the score-based quadratic loss is intractable, denoising score matching instead minimizes an equivalent objective using the Gaussian forward transition kernel and its analytical score.The transition-kernel score corresponds to the noise added to X0 at time t.Source: §2 · pp. 4–5
• Empirical training samples t uniformly from [t0, T] and avoids times below t0 to prevent score blow-up and stabilize training; the paper sets g(t)=1 for simplicity.The empirical objective uses w(t)=1/(T−t0), with sufficient sampling of the forward transition and time assumed inexpensive.Source: §2 · pp. 4–5

3 Score decomposition

• For x=Az on an unknown d-dimensional subspace, the score separates into orthogonal on-support and off-support components; s∥ lies in span(A) and depends on A⊤x.Here A has orthonormal columns and the latent variable z has a density.Source: §3 · pp. 5–6
• As t→0, s⊥ blows up because h(t)→0, consistent with score blowup for manifold data (Song and Ermon, 2020; Kim et al., 2021; Pidstrigach, 2022; De Bortoli, 2022).Source: §3 · pp. 5–6
• The backward process splits into latent-informed on-support dynamics and simpler linear off-support dynamics, whose drift coefficient eventually vanishes as the data support is perfectly recovered.Source: §3 · p. 6
• In the Gaussian latent example, s∥ is linear and carries the latent covariance information that guides distribution recovery, while s⊥ blows up as t approaches 0.Source: §3 · p. 6
• The on-support score s∥ is Lipschitz continuous over z and t1,t2∈[0,T], a property used to develop score approximation and estimation results.Source: §3 · p. 7

4.1 Score approximation

• SNN encodes x through V⊤x, applies a latent network, and decodes through V, while a shortcut contributes −x/h(t); score matching later recovers the unknown data subspace.The network retains an encoder–decoder structure, and the estimated score's Lipschitz property is essential to bounding distribution recovery error.Source: §4.1 · pp. 7, 9
• Under Assumptions 1–3, SNN approximates score functions in L2 over unbounded inputs for t in [t0, T], with network size depending on intrinsic dimension d.The latent density is twice continuously differentiable with sub-Gaussian tails, and the on-support score is β-Lipschitz; the construction's Lipschitz constraints do not undermine approximation power.Source: §4.1 · pp. 8–9

4.2 Score estimation theory

• The squared L2 score-estimation error converges at O~(1/(t0 n^(2/(d+5)))) for sufficiently large n, with the rate depending on intrinsic dimension d.Theorem 2 is presented as the first explicit sample-complexity bound for score matching; δ(n) is negligible when n is sufficiently large.Source: §4.2 · p. 10
• As t0 approaches zero, the estimation bound becomes vacuous for fixed n because the score blows up; larger t0 improves the bound but worsens distribution recovery.The paper addresses this tension with a tradeoff on t0 in the following section.Source: §4.2 · p. 10

Theorem 3

• With high probability, Theorem 3 guarantees recovery of the unknown subspace and bounds on-support distribution error in total variation and Wasserstein-2 under a finite-KL condition and a step-size constraint.The theorem uses T = Θ(log n) and t0 = O(min{c0, 1/β}), where c0 is the minimum eigenvalue of the latent covariance.Source: §5 · pp. 10–11

Vanishing in the orthogonal space

• The continuous-time generated distribution’s orthogonal pushforward is Gaussian with covariance bounded by ct0 I, and the orthogonal component approximately vanishes as t0 approaches zero.The theorem states the covariance bound for the projection orthogonal to the learned subspace; the interpretation also relates vanishing to the true data subspace.Source: §5 · p. 11

Subspace recovery error

• The learned column span approaches the data subspace with error proportional to score-estimation error and dependent on the latent covariance’s minimum eigenvalue.This does not imply that the matrices themselves are close in Frobenius norm, because the column span is invariant under orthogonal transformations; Theorem 3 therefore allows an orthogonal transformation.Source: §5 · p. 11

Tradeoff on t0

• As t0 decreases, total-variation error from score estimation increases while Wasserstein-2 bias decreases, creating a tradeoff in recovering the latent distribution.The section proposes making the two errors the same order when choosing t0.Source: §5 · p. 11

6.1 Proof sketch of Theorem 1

• Theorem 1 handles the unbounded input domain by truncating to a hypercube, approximating the score piecewise linearly inside it, and setting the network to zero outside.The tail behavior of the noisy distribution bounds the error outside the hypercube; Figure 3 illustrates the local piecewise construction.Source: §6.1 · pp. 11–12, Figure 3 · p. 12

6.2 Proof of Theorem 2

• Theorem 2 bounds the oracle loss by separating finite-sample, truncation, and network-approximation errors, using Bernstein-type concentration for the statistical terms.The proof selects the network from Theorem 1, controls truncation separately, and combines the bounds to obtain the desired result.Source: §6.2 · p. 13

6.3 Proof of Theorem 3

• For Theorem 3, item 1 uses amplified orthogonal-score blowup to force the estimated subspace to match the data subspace, while item 3 analytically solves the orthogonal backward process.For item 2, the proof studies continuous-time generation and obtains the discrete result by adding discretization error.Source: §6.3 · p. 14
• For Theorem 3, item 2 converts the score error to latent score error, then uses Girsanov’s Theorem to bound process KL divergence through drift error.The proof first compares the ground-truth and learned latent backward processes.Source: §6.3 · p. 14

• The score can be accurately approximated and estimated, and the resulting diffusion model learns the data distribution at a rate free of the curse of ambient dimensionality.The estimation and distribution-learning convergence rates depend on the data’s intrinsic dimension.Source: §7 · p. 14

Linear subspace assumption

• Linear-subspace analysis remains challenging, and the paper calls for follow-up work under more general assumptions such as manifold data.Even under the linear-subspace assumption, analyzing diffusion behavior in the on-support and orthogonal subspaces requires non-trivial analysis.Source: §7 · p. 14

End-to-end distribution learning

• Rather than first estimating a subspace and then fitting a distribution, the end-to-end scheme captures unknown linear structure and the data distribution with fast estimation guarantees.The paper argues that this scheme better reflects practical diffusion learning than the two-step PCA-like approach.Source: §7 · p. 15

A.1 Proof of Lemma 1

• The Lemma 1 proof derives the score using the latent variable and the forward process.It then substitutes the conditional density into the score expression.Source: §A.1 · p. 19
• Because the conditional density is continuously differentiable in x, the proof uses this property in the score expression.Source: §A.1 · p. 19
• The score component s∥ depends only on A⊤x, so it can be written with that projected input of intrinsic dimension d.Source: §A.1 · p. 20

A.2 Computation in Example 1

• In Example 1, the marginal distribution of the forward process remains Gaussian.The passage also introduces its density function.Source: §A.2 · p. 20
• The Example 1 calculation checks spatial Lipschitz continuity by considering two cases, including λd > 1, and then combines them.It also addresses Lipschitz continuity with respect to time by taking a time derivative.Source: §A.2 · p. 20

B.1 Proof of Theorem 1

• Taking V = A reduces construction of the score network to approximating g(z′, t) in the d-dimensional projected coordinates z′ = A⊤x.Source: §B.1 · p. 21
• Because the input domain is unbounded, the proof approximates on a compact set S and sets the network output to zero outside S.The tail behavior of Pz allows the L2 approximation error to remain controlled.Source: §B.1 · p. 21
• Choosing the truncation radius R using the tail bound controls the L2 approximation error outside the compact region.Source: §B.1 · pp. 21, 23
• On the rescaled domain, coordinate maps are approximated using a partition of unity, with ReLU implementations obtained by adapting Chen et al. (2020, Lemma 10).The construction partitions the spatial and time coordinates and uses trapezoid functions.Source: §B.1 · p. 22
• The constructed score approximator is 1/h(t) A f̄θ(A⊤x, t) − 1/h(t)x, with an L2 approximation error bounded for t ∈ [t0, T].Source: §B.1 · p. 23

B.2 Proof of Theorem 2

• The oracle inequality decomposes the loss into empirical concentration, truncation, and statistical and approximation error terms.Source: §B.2 · pp. 24, 26
• The concentration step bounds an induced class of truncated losses, while the proof separately controls truncation using the sub-Gaussian tail assumption.Source: §B.2 · pp. 24–26
• The network-class covering number accounts for both the orthonormal matrix V and network function fθ in deriving the bound.Source: §B.2 · pp. 27–28
• Theorem 2’s bound follows after combining the error terms, with factors in d, β, τ, log D, and log t0 omitted.Source: §B.2 · pp. 27–28

B.3 Conditional covariance bound

• The conditional covariance proof uses the on-support score expression and differentiates it with respect to z′.Source: §B.3 · p. 29

B.4 Truncation error

• Under Assumption 2, Lemma 2 bounds the truncation error by ε in L2(Pt) for every t in [t0, T], for its specified choice of R.Source: §B.4 · p. 29
• The truncation-error proof treats two terms separately, using independence of the on-support and orthogonal Gaussian components and a condition on R.Source: §B.4 · p. 30

C.1 Subspace Error and Latent Score Matching Error

• Lemma 3 states an assumption that includes the existence of an orthonormal matrix U ∈ R^{d×d}; the condition following “such that” is not shown in these passages.Source: §C.1 · p. 31

C.2 Backward processes

• The backward-process analysis gives latent distribution estimation error bounds for both undiscretized and discretized learned SDEs.The argument works in latent space and decomposes the ground-truth backward process there.Source: §C.2 · pp. 31–32

C.3 Orthogonal process

• For T > 1, the orthogonal process has Y_{T−t0} ∼ N(0, σ^2I) with σ^2 ≤ e^t0 when t0 ≤ 1; its discretized version has σ^2 ≤ e^(t0 + η) when t0 + η ≤ 1.The discretized bound assumes step size η and T − t0 = K_Tη.Source: §C.3 · p. 32

C.4 Proof of Theorem 3

• Theorem 3’s proof establishes its first item and total-variation bound, bounds W2 using finite second moment, and derives the third item from the orthogonal process.The Wasserstein-2 argument adapts a technique of Chen et al. (2022b, Lemma 16), whose proof is said to require only finite second moment.Source: §C.4 · p. 33

D.1 Proof of Lemma 3

• The proof of Lemma 3 decomposes the error and uses the best L2 approximation in Im(V), with a lower bound from the component of AY orthogonal to that subspace.The accompanying lemmas relate this analysis to the smallest eigenvalue of an expected outer product.Source: §D.1 · pp. 34–35

D.1.1 Evolution of score function

• The score-evolution section derives a result for the latent distribution’s score and notes that the resulting bound is tight for Gaussian distributions.The supplied fragments do not show the score formula or bound.Source: §D.1.1 · pp. 38–39

D.1.2 Other lemmas.

• Lemma 10 assumes a β-Lipschitz score ∇log pz(z) and states a bound involving its expected squared norm.The bound’s right-hand side is not visible in the supplied passage.Source: §D.1.2 · p. 40

D.2 Proof of Lemma 4, Undiscretized Setting

• Under finite score-function Lipschitz constants and sub-Gaussian Z0, the proof establishes Novikov’s condition for the undiscretized process.Source: §D.2 · pp. 40–41
• Girsanov’s theorem, Pinsker’s inequality, and data processing handle the learned process’s N(0, Id) initialization, with exponential O-U convergence used in the resulting bound.Source: §D.2 · p. 42

D.3 Proof of Lemma 4, Discretized Setting

• For T − t0 = KTη, the discretized setting also satisfies Novikov’s condition and yields a KL-divergence bound between backward-process path laws.Source: §D.3 · pp. 42–43
• The discretized proof of Lemma 4, part 2, adds an error term ϵdis(T − t0).Source: §D.3 · p. 45

D.4 proof of Lemma 5

• The proof of Lemma 5 concludes that YT−t0 is zero-mean Gaussian with covariance bounded by et0I.Source: §D.4 · p. 45

E Helper lemmas

• The helper lemmas include a Bernstein-type concentration bound for bounded function classes, using covering numbers and i.i.d. samples.Source: E Helper lemmas · p. 47
• The appendix provides tail bounds for Gaussian integrals, used in score approximation and estimation analyses.Source: E Helper lemmas · p. 49
• A matrix-norm lemma treats orthonormal-column matrices whose spans are approximately equal, supporting score-estimation error bounds in Theorem 3.Source: E Helper lemmas · p. 50