PaperlayerView exact revision on arXiv

Source-linked AI summary

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

Yang Song, Sahaj Garg, Jiaxin Shi, Stefano Ermon

arXiv:1905.07088v2cs.LGstat.ML

TL;DR

Score matching is limited by the cost of computing Hessian traces in deep or high-dimensional models. The paper proposes sliced score matching, which uses random score projections and Hessian-vector products, and reports scalable density estimation plus accurate score estimation for implicit-distribution applications.

  • Problem

    Score matching avoids intractable partition functions but is limited by expensive Hessian-trace computation for deep models and high-dimensional data.

  • Method

    Sliced score matching compares scores after projection onto random directions, requiring Hessian-vector products and adapting to deep score networks for implicit distributions.

  • Results

    Experiments show sliced score matching learns deep energy-based models effectively and produces more accurate score estimates than kernel estimators for implicit VAEs and WAEs.

  • Takeaways & Limitations

    The estimator scales to deep models and high-dimensional data while remaining implementable in modern automatic-differentiation frameworks.

  • Takeaways & Limitations

    Score-network outputs need not be gradients of scalar functions, although the paper states the integration-by-parts argument remains valid.

Abstract

from arXiv · show

Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random vectors before comparing them. This objective, called sliced score matching, only involves Hessian-vector products, which can be easily implemented using reverse-mode automatic differentiation. Therefore, sliced score matching is amenable to more complex models and higher dimensional data compared to score matching. Theoretically, we prove the consistency and asymptotic normality of sliced score matching estimators. Moreover, we demonstrate that sliced score matching can be used to learn deep score estimators for implicit distributions. In our experiments, we show sliced score matching can learn deep energy-based models effectively, and can produce accurate score estimates for applications such as variational inference with implicit distributions and training Wasserstein Auto-Encoders.

1 INTRODUCTION

Score matching avoids intractable partition functions but is computationally limited by Hessian-trace evaluation. Sliced score matching projects scores onto random directions, enabling scalable deep-model training and score estimation.

  • Score matching learns unnormalized models by comparing data and model scores, which do not depend on the intractable partition function.
  • Hessian-trace computation requires work proportional to data dimension, limiting score matching for deep neural networks and high-dimensional data.
  • Sliced score matching compares score projections along random directions and requires only Hessian-vector products, simplifying implementation in modern automatic-differentiation frameworks.
  • The method has consistent and asymptotically normal parameter estimates under regularity conditions and can scale to deep unnormalized models.
  • Experiments find sliced score matching more scalable or accurate than existing variants for density estimation and better than kernel estimators for implicit VAE and WAE applications.

2 BACKGROUND

Score matching removes dependence on an intractable partition function, but its usable objective requires estimating a Hessian trace. Score estimation extends these ideas to implicit distributions, where samples are available but densities are not.

  • An unnormalized model has an intractable partition function, yet its score remains independent of that function.
  • Score matching: Integration by parts converts Fisher divergence into a trainable objective containing the Hessian trace of the model log-density.
  • Score matching: Computing the Hessian trace naively requires D additional backward passes, making score matching expensive when data dimension D is large.
  • Implicit distributions: Implicit distributions have tractable sampling processes but no tractable density, making objectives involving their densities difficult to optimize.
  • Implicit distributions: Estimating an implicit distribution’s score from samples can support entropy optimization and related variational inference objectives.

3 DENSITY AND SCORE ESTIMATION WITH SLICED SCORE MATCHING

Sliced score matching replaces full score comparison with random projections and uses integration by parts to obtain a sample-based objective. The same objective trains deep score networks for implicit distributions.

  • Sliced score matching: Sliced score matching compares data and model scores after projection onto random directions, replacing the high-dimensional Fisher divergence objective.
  • Sliced score matching: Valid projection distributions have positive-definite E[vv^T] and finite E[||v||^2], including Gaussian, multivariate Rademacher, and hypersphere distributions.
  • Sliced score matching: Integration by parts removes the unknown data score and yields an equivalent objective up to a constant under regularity conditions.
  • Sliced score matching: Variance reduction analytically integrates part of the objective; the resulting SSM-VR estimator empirically performs better than the original estimator.
  • Sliced score matching: For M projections, sliced score matching uses M + 1 gradient operations, compared with D + 1 typically needed for the full Hessian trace when D ≫ M.
  • Score estimation: A deep vector-valued network h(x; θ) is optimized with the sliced objective, and h(x; θ̂) approximates the score of an implicit distribution.
  • Score estimation: The score network need not be the gradient of a scalar function, because the integration-by-parts argument does not require that structure.

4 THEORETICAL ANALYSIS

The paper establishes consistency and asymptotic normality for sliced score matching under regularity conditions, and connects the method to existing estimation principles. These results characterize its statistical behavior and show how projection count affects variance.

  • Consistency: Sliced score matching is consistent for any M as N →∞ under suitable assumptions.The assumptions include normalizability, compactness, well-specification, identifiability, positivity, and Lipschitz conditions.
  • Consistency: The consistency proof combines identification of the true parameter with uniform convergence of the empirical objective, regardless of M.The paper states that these two facts lead to consistency.
  • Consistency: Compared with Hyvärinen’s result, the paper establishes a stronger notion than local consistency for standard score matching.Hyvärinen’s criterion establishes only J(θ) = 0 ⇔ θ = θ∗, which the paper characterizes as local consistency.
  • Asymptotic normality: Under an additional stronger Lipschitz assumption, the sliced score matching estimator is asymptotically normal as N →∞.The special-case theorem uses multivariate Rademacher projections; more general results are deferred to Appendix B.4.
  • Asymptotic normality: Larger M produces smaller asymptotic variance, while the variance gap relative to standard score matching decreases as M increases.Standard score matching has smaller asymptotic variance than sliced score matching with multivariate Rademacher projections.
  • Connections to other methods: For small perturbations, a suitable expectation of a specialized noise-contrastive estimation objective is equivalent to sliced score matching.This connection holds when the perturbation norm satisfies ∥v∥2 ≈ 0.
  • Connections to other methods: Hutchinson’s trick replaces the Hessian trace with an expectation of quadratic Hessian-vector products, yielding the variance-reduced sliced score matching objective.The identity uses a random vector distribution satisfying E_pv[vv⊺] = I.

5 RELATED WORK

Related work scales score matching by avoiding or approximating Hessian computations, while kernel-based methods estimate scores through Stein identities. These alternatives differ in computational cost, guarantees, and sensitivity to design choices.

  • Scalable score matching: Three prior approaches scale score matching to deep models on high-dimensional data: denoising score matching, approximate backpropagation, and curvature propagation.The paper introduces these as existing methods able to scale score matching beyond the standard computational regime.
  • Scalable score matching: Denoising score matching avoids Hessian computation by applying score matching to a noise-corrupted distribution.Its performance can be sensitive to the noise scale σ, whose selection is described as non-trivial.
  • Scalable score matching: Denoising score matching recovers the noise-corrupted data distribution rather than directly recovering the original data distribution.The method also requires choosing parameters of the noise distribution, with practical heuristics used for σ.
  • Scalable score matching: Approximate backpropagation limits computation to the Hessian diagonal but provides no theoretical guarantees for approximation errors.The diagonal-only computation is designed to match the cost of ordinary loss-gradient backpropagation.
  • Scalable score matching: Curvature propagation estimates the Hessian trace with an unbiased, complex-valued estimator, although its variance claim is based on pseudo-variance.The paper distinguishes the proof’s pseudo-variance analysis from the variance of the estimator.
  • Kernel score estimators: Kernel score estimators use generalized Stein identities to estimate score values from samples.Stein uses kernel feature maps, whereas Spectral expands the score in eigenfunctions and solves for coefficients.
  • Kernel score estimators: The Spectral method is argued to provide theoretical guarantees and principled out-of-sample estimation at unseen test points.The paper distinguishes this from Stein’s kernel-based construction in its experiments.

6 EXPERIMENTS

Experiments evaluate sliced score matching for density and score estimation, finding competitive or superior performance with improved scalability to high-dimensional and deep models.

  • Density estimation: SSM is comparable to SM, outperforms other computationally efficient approximations, and scales effectively to high-dimensional data.The comparison includes DSM, approximate backpropagation, and curvature propagation.
  • Deep Kernel Exponential Families: SSM-VR performs comparably to SM on DKEF models, while variance reduction substantially improves SSM performance.The models are evaluated using SM loss on held-out UCI test data.
  • Scalability to High Dimensional Data: SM runtime degrades linearly with input dimension and exceeds 12GB GPU memory beyond dimension 400, whereas SSM and other efficient methods remain scalable.The methods are compared using average minibatch runtime on multivariate standard-normal data.
  • Deep Flow Models: On NICE models for 784-dimensional MNIST, SSM-VR greatly outperforms other methods on SM loss, while DSM, approximate BP, and CP perform worse or require difficult tuning.The study uses a 20-layer NICE model with 1,000 units per layer; SM training is excluded because of its computational cost.
  • VAE with Implicit Encoders: For implicit encoders, SSM outperforms Stein and Spectral methods on MNIST, can outperform ELBO, and remains competitive when latent dimension increases.With M = 100, matching kernel-method computation, SSM also outperforms ELBO.
  • VAE with Implicit Encoders: On CelebA, SSM outperforms all baselines after 40k iterations for VAE experiments and outperforms kernel methods after 40k iterations for WAE experiments.The WAE results also report generally lower FID scores than the VAE results.

7 CONCLUSION

The paper presents sliced score matching as a scalable method for learning unnormalized models and estimating scores for implicit distributions. It reports theoretical guarantees and empirical advantages over competing methods.

  • Sliced score matching scales score-based estimation to deep models and high-dimensional data while remaining easy to implement with automatic differentiation.
  • The estimator is consistent and asymptotically normal under stated regularity conditions.
  • Experiments show improved performance for deep energy-based models and more accurate score estimates than kernel estimators for implicit VAEs and WAEs.

A Samples

This material introduces notation for data and model distributions, scores, projection vectors, and Hessians, then states assumptions used for the basic theoretical results.

  • The data distribution is denoted pd(x), while the model distribution is pm(x; θ), which may be an unnormalized energy-based model.
  • Projection vectors v have the same dimension as the input x and are used in the sliced objective.
  • The score functions are sm(x; θ) for the model and sd(x) for the data, with ∇xsm(x; θ) denoting the model-score Hessian.
  • The basic results assume differentiable score functions, finite projection-vector moments, vanishing boundary terms, identifiability, and positive model densities.
  • Under these conditions, the population sliced objective can be related to the score-matching objective up to a constant independent of θ.

B.3 CONSISTENCY

The consistency analysis establishes uniform convergence of the empirical objective and uses it to show that the sliced score matching estimator converges to the true parameter under regularity assumptions.

  • Consistency is analyzed for the empirical minimizer ˆθN,M of the sliced score matching objective.
  • The analysis assumes a compact parameter space, Lipschitz continuity of the model score and score outer product, and bounded projection-vector moments.
  • The function defining the empirical objective is Lipschitz in θ when the score model is sufficiently smooth and projection vectors have bounded higher-order moments.
  • Uniform convergence is obtained through symmetrization, chaining, and metric-entropy bounds.
  • Theorem 2 concludes that ˆθN,M converges in probability to the true parameter θ∗.

B.4 ASYMPTOTIC NORMALITY

The asymptotic-normality analysis adds smoothness and moment conditions, derives the estimator’s limiting variance, and gives simplified forms for Gaussian and Rademacher projections.

  • The asymptotic-normality proof requires additional smoothness conditions on second derivatives near the true parameter.
  • For Gaussian projection vectors, the matrices governing the asymptotic variance have simpler forms.
  • Theorem 3 establishes asymptotic normality under the stated assumptions.
  • For multivariate Rademacher projections, the asymptotic variance also admits a specialized expression.
  • The analysis uses a Taylor expansion of the empirical gradient together with consistency, the law of large numbers, and the central limit theorem.
  • Under similar assumptions, the original score matching estimator is likewise consistent and asymptotically normal.

C ADDITIONAL DETAILS OF EXPERIMENTS

The experiments specify kernel exponential-family and flow-model architectures, training procedures, preprocessing, and comparisons against existing score-matching methods. Log-likelihood evaluation uses AIS, while approximate backpropagation performs poorly on the reported DKEF comparison.

  • Comparisons: The study compares the proposed objective with DKEF models trained using exact score matching and evaluates sliced score matching, denoising score matching, approximate backpropagation, and curvature propagation.Denoising score matching uses a grid search over σ, while curvature propagation uses one noise sample.
  • Training procedure: The DKEF training procedure analytically minimizes the quadratic loss over α before computing validation loss.A closed-form solution is also derived for sliced score matching and related objectives.
  • DKEF experiments: RedWine and WhiteWine receive dequantization, PCA whitening, and preprocessing noise; training uses Adam with learning rate 10^-2 and batch size 200.The data split uses 10% for testing and 10% of the remainder for validation.
  • DKEF models: DKEF models use three Gaussian kernels with neural-network features and 200 trainable inducing points.Each feature extractor is a three-layer network with skip connections and 30 hidden neurons per layer.
  • Evaluation: Figure 3 reports DKEF log-likelihoods with higher values better, while approximate-backpropagation results are omitted because they were below −106.The log-likelihoods are estimated with AIS using N(0, 2I) and 1,000,000 samples.

D VARIANCE REDUCTION

The paper discusses variance reduction for sliced score matching through more random directions and control variates. Larger direction samples increase computation, whereas control variates can reduce variance with tractable expectations.

  • Variance reduction: Increasing M is the simplest variance-reduction strategy, but it requires more computation and can remove sliced score matching’s advantage.This occurs when M approaches the data dimension.
  • Control variates: Control variates use tractable-expectation random variables that are highly correlated with the target random variable.The paper defines c(θ; x, v) = 1/2(v^Tsm(x; θ))^2 for suitable normal or Rademacher directions.
  • Control variates: The theory guarantees a variance-reducing coefficient β(x), and the experiments often find β(x) ≡ 1 effective.The resulting control-variate objective exploits structural similarity to the sliced score-matching objective.
Loading 1905.07088v2…