Minimum Stein Discrepancy Estimators

1 Introduction

• Prohibitive multidimensional integration of normalizing constants makes likelihood computation difficult for complex models.The passage gives images, natural language, Markov random fields, and nonparametric density estimation as examples.Source: §1 · p. 1
• Score matching can be expensive in high dimensions and break down for non-smooth models or models with rapidly growing second derivatives.The passage also identifies poor behavior for heavy-tailed distributions and lack of robustness to outliers as distinct limitations.Source: §1 · pp. 1–2
• Prior work showed that Stein’s method can construct discrepancies controlling weak convergence of an empirical measure to a target [23] [25] [50] [24].Source: §1 · p. 2
• The paper unifies score matching, minimum probability flow, and contrastive divergence as special cases of minimum Stein discrepancy estimation.It establishes consistency, asymptotic normality, and robustness for kernel-based estimators and reports that they address non-smooth and heavy- or light-tailed distributions.Source: §1 · p. 2

2 Minimum Stein Discrepancy Estimators

• Minimum Stein discrepancy estimation chooses a parameter by minimizing a discrepancy between the data distribution and a parametric model.The discrepancy’s choice affects estimator consistency, efficiency, and robustness; Stein discrepancies can be evaluated when the model is known only up to normalization.Source: §2 · p. 2

2.1 Example 1: Diffusion Kernel Stein Discrepancy Estimators

• DKSD uses diffusion Stein operators and matrix-valued RKHS kernels, extending Langevin-based KSD through more flexible choices of diffusion and kernel.When K = Ik and m = Ih, DKSD reduces to KSD with scalar kernel h(x)k(x, y)h(y); h = 1 gives the usual Langevin-based KSD.Source: §2.1 · p. 3

2.2 Example 2: Diffusion Score Matching Estimators

• DSM generalizes score matching through a diffusion Stein operator and recovers standard score matching when m(x)m(x)⊤ = I.DSM is defined only for distributions with sufficiently smooth densities, and its sample-mean estimator requires m to be independent of θ.Source: §2.2 · p. 4

2.3 Further Examples: Contrastive Divergence and Minimum Probability Flow

• The diffusion Stein framework recovers contrastive divergence and minimum probability flow as special cases, alongside score matching.Contrastive divergence uses a Markov transition operator; minimum probability flow uses I − Pθ on a finite state space.Source: §2.3 · p. 5

2.4 Implementing Minimum SD Estimators: Stochastic Riemannian Gradient Descent

• The proposed stochastic Riemannian gradient descent uses information geometry induced by the discrepancy; for exponential families, DKSD and DSM approximations are convex quadratics with closed-form solutions when K and m are θ-independent.A diffusion-Stein discrepancy is convex in θ when m and the Stein class are θ-independent.Source: §2.4 · p. 5

3 Theoretical Properties for Minimum Stein Discrepancy Estimators

• DKSD estimators are strongly consistent and asymptotically normal under the stated regularity conditions, with a unique minimizer when the model map is injective.For misspecified models, the paper also assumes a unique minimizer.Source: §3 · p. 6
• The DKSD kernel assumptions hold for common Gaussian, inverse-multiquadric, and Matérn kernels with smoothness greater than 2; adapting the diffusion tensor can also ensure the stated guarantees.The paper describes the model assumptions as very weak given this adaptability.Source: §3 · p. 6
• DSM estimators have weak consistency and are asymptotically normal under their stated regularity conditions.The paper identifies convergence in probability as the form of DSM consistency.Source: §3 · pp. 6–7
• For natural exponential families, DKSD and DSM estimators are strongly consistent and asymptotically normal under the stated kernel, model, and integrability conditions.The listed conditions include an invertible diffusion matrix and linearly independent rows of the sufficient-statistic gradient.Source: §3 · p. 7
• The DKSD bias-robustness result gives bounded influence functions when its stated model and integrability conditions hold.The influence function measures the effect of an infinitesimal point-mass perturbation of the data distribution.Source: §3 · p. 7
• For a Gaussian location model, DKSD with a Gaussian kernel has bounded influence even with identity diffusion, whereas the score-matching influence function is unbounded.The paper also states that DSM influence can be bounded when both a scalar diffusion coefficient and its gradient decay at infinity.Source: §3 · p. 7

4.1 Rough densities: the symmetric Bessel distributions

• For rough Bessel densities with s = 0.6, KSD outperforms SM for some lengthscales, while both recover the parameter at s = 2.At s = 1, corresponding to the Laplace distribution, both obtain a reasonable location estimate.Source: §4.1 · p. 8, Figure 1 · p. 8

4.2 Heavy-tailed distributions: the non-standardised student-t distributions

• For ν = 5, DKSD estimates the Student-t location accurately and substantially improves scale estimation over SM, KSD, NNSM, and NNKSD.SM and KSD struggle to recover location, and their loss functions are far from convex; Riemannian SGD converges within a few dozen iterations, while SGD has not converged after 1,000.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 Robust estimators for light-tailed distributions: the generalised Gamma distributions

• With 80 observations corrupted to x = 8, DSM and DKSD are robust for the Gaussian case, and choosing α ≥ θ∗₂ − 1 guarantees a bounded influence function for the generalised Gamma model.For θ∗₂ = 5, α can be chosen as a function of θ₂ to obtain robustness for very light tails.Source: §4.3 · p. 9, Figure 3 · p. 9

4.4 Efficient estimators for a simple unnormalised model

• With n = 200, DKSD using m(x) = diag(1/(1 + x)) recovers θ∗ = −1 in the simple unnormalised model, while SM and KSD are less accurate.The example illustrates that a judicious diffusion matrix can improve estimator efficiency.Source: §4.4 · p. 9, Figure 4 · p. 9

• Minimum Stein discrepancy provides a general approach to constructing minimum distance estimators, with popular inference schemes recovered as special cases.Source: §5 · p. 9
• Choosing the Stein operator and Stein class offers flexibility to tailor inference schemes to trade off efficiency and robustness.Source: §5 · p. 9
• Identifying diffusion matrices for important machine-learning problems remains an open direction, including product-of-Student-t experts and graph-adapted kernels for large graphical models.Source: §5 · p. 9

Supplementary Material

• The appendices cover background, estimator derivations, information metrics, asymptotic results, and robustness.Appendix A supplies background, B derives minimum Stein discrepancy estimators, C derives their information metrics, D proves consistency and central limit results, and E discusses robustness.Source: Supplementary Material · p. 14

A Background Material

• The background covers vector calculus, norms, vector-valued RKHSs, separable kernels, and stochastic optimization on Riemannian manifolds.The optimization background describes gradient-flow updates using an exponential map or a first-order retraction that stays on the manifold.Source: A Background Material · p. 14, §A.1 · p. 14, §A.2 · p. 14

A.5 Background on Stochastic Optimisation on Riemmannian Manifolds

• A stochastic Riemannian update uses an unbiased loss-gradient estimator, with a retraction as a first-order alternative when the exponential map is costly.In a linear manifold, the retraction update reduces to an ordinary gradient step.Source: §A.5 · p. 15

B Derivation of Diffusion Stein Discrepancies

• The diffusion Stein operator and matrix-valued kernels provide the construction underlying DKSD and its corresponding Stein kernel.The derivations establish the operator's reproducing property and express the Stein kernel through score, kernel-derivative, and diffusion-field terms.Source: B Derivation of Diffusion Stein… · p. 16, §B.1 · p. 16, Proof · p. 16

B.3 The Stein Kernel Corresponding to the Diffusion Kernel Stein Discrepancy

• The diffusion field transforms a matrix kernel K into the kernel m(x)K(x,y)m(y)ᵀ.This identifies a way the choice of diffusion field changes the kernel used in the discrepancy.Source: §B.3 · p. 18

B.4 Special Cases of Diffusion Kernel Stein Discrepancy

• Setting m = I recovers the usual KSD, while scalar and matrix-valued choices yield special cases of DKSD.The appendix also expands expressions for diagonal kernels and diffusion matrices.Source: §B.4 · p. 19

B.5 Diffusion Kernel Stein Discrepancies as Statistical Divergences

• Under the stated conditions, DKSD is zero only when the compared distributions agree.The proof uses an integrally positive definite kernel, positive q, and invertibility of m to infer equality of the score functions.Source: §B.5 · p. 20, §B.5.1 · p. 20

B.6 Diffusion Score Matching

• DSM is introduced as another minimum Stein discrepancy, alongside the diffusion-kernel construction.The supplementary material presents it as a distinct example of a Stein discrepancy.Source: §B.6 · p. 20

B.6.1 Proof of Theorem 2: Diffusion Score Matching

• The diffusion score matching discrepancy is zero only when the compared distributions are equal almost everywhere.The proof uses positive densities and invertibility of the diffusion field to conclude that equality of transformed scores implies equality of distributions.Source: §B.6.1 · p. 21

B.6.2 Diffusion Score Matching Estimators

• DSM, like standard score matching, is defined only for distributions with sufficiently smooth densities.This is a scope condition on the distributions to which the estimator applies.Source: §B.6.2 · p. 21
• The θ-dependent part of the DSM objective does not depend on Q's density and can be estimated with an empirical measure.The passage identifies this as the basis for an unbiased estimator of that quantity.Source: §B.6.2 · p. 21

B.6.3 Proof of Theorem 10: DSM as a limit of DKSD

• As the kernel scale γ tends to zero, the DKSD construction approaches the DSM limit under the theorem's stated conditions.The theorem assumes positive q, a continuous square-integrable score difference, and a normalized positive smoothing function.Source: §B.6.3 · pp. 21–23
• The limit argument uses convergence of kernel integrals to the corresponding pointwise inner-product integral as γ tends to zero.The supplied derivation states this convergence for functions in C(R^d) ∩ L2(R^d).Source: §B.6.3 · p. 22

C Information Semi-Metrics of Minimum Stein Discrepancy Estimators

• The information semi-metric is defined from the mixed second derivatives of a discrepancy at matching parameters; if positive definite, it supports Riemannian gradient descent.The definition evaluates the derivatives at α = θ.Source: C Information Semi-Metrics of Minimum… · p. 23

C.2 Proof of Proposition 4: Information Semi-Metric of Diffusion Score Matching

• The DSM information form is semi-positive definite.The passage states this as the conclusion of the proof of Proposition 4.Source: §C.2 · p. 24

D.1 Diffusion Kernel Stein Discrepancies

• The consistency setup assumes correct specification; under misspecification, it additionally requires a unique minimizer.The stated setting is Q = Pθ* for a DKSD minimizer.Source: §D.1 · p. 24

D.1.1 Strong Consistency

• Under equicontinuity, integrability, and injectivity conditions, the DKSD estimator is strongly consistent.The assumptions include bounds on the score and diffusion-kernel terms over compact parameter subsets.Source: §D.1.1 · pp. 24–25

Proof

• The DKSD discrepancy vanishes only at its target distribution, yielding a unique minimizer when the model is identifiable.The proof uses injectivity of θ ↦ Pθ to obtain uniqueness.Source: Proof · p. 25
• Fréchet differentiability of k0 lets the mean value theorem establish equicontinuity, simplifying the conditions for strong consistency.The proof also uses integrable bounds and equicontinuity arguments.Source: Proof · p. 26
• Theorem 3 gives the main consistency result for minimum DKSD estimators.The supplied proof passage identifies Theorem 3 as the consistency result.Source: Proof · p. 26

D.2.1 Weak Consistency of DSM

• Under the stated regularity and parameter-space conditions, the DSM estimator is weakly consistent for θ*.The result applies when Θ is compact, or when Θ and Fθ are convex and θ* is interior.Source: §D.2.1 · p. 27

D.2.2 Asymptotic Normality of DSM

• With the theorem's smoothness, integrability, and invertible-information conditions, the DSM estimator is asymptotically normal.The information tensor is identified as gab(θ*).Source: §D.2.2 · p. 28

D.3.1 Strong Consistency of the Minimum Diffusion Kernel Stein Discrepancy Estimator

• For exponential-family models under the stated kernel, rank, invertibility, and integrability assumptions, the DKSD minimizer exists eventually and converges almost surely to its target.The proof establishes convergence of the estimator through convergence of the associated U-statistic matrix and its inverse.Source: §D.3.1 · p. 29, Proof · p. 29

D.3.2 Asymptotic Normality of the DKSD Estimator

• Under the stated L2 integrability conditions, the DKSD estimator is asymptotically normal.The proof proceeds by applying a central limit argument and the delta method.Source: §D.3.2 · p. 30

D.3.3 Diffusion Score Matching Asymptotics

• For exponential families with invertible m and linearly independent statistic gradients, the DSM estimator is strongly consistent; with A, v ∈ L2(Q), it is also asymptotically normal.The consistency result additionally requires A, v ∈ L1(Q).Source: §D.3.3 · p. 31

E.1 Robustness of Diffusion Kernel Stein Discrepancy

• Under additional assumptions bounding the relevant kernel integral, the DKSD influence function is bounded, giving B-robustness.The proof seeks boundedness of z ↦ ∥IF(z,Q)∥.Source: §E.1 · p. 32

E.2 Robustness of Diffusion Score Matching

• DSM is bias-robust if and only if its estimating-function term s(x, θ*_DSM) is bounded in x.The influence function is expressed as gDSM(θ)^−1s(x,θ).Source: §E.2 · p. 32

F.1 Efficiency of Minimum SD Estimators for Scale Parameters of Symmetric Bessel distributions

• In the scale-estimation experiment for a symmetric Bessel distribution, KSD can handle rougher densities, as illustrated at s = 0.6.The experiment uses n = 500 IID realizations in d = 1 and varies smoothness s.Source: §F.1 · p. 33, Figure 5 · p. 33

F.2 Bias Robustness of Minimum SD Estimators for the Symmetric Bessel and Non-standardised Student-t Distributions

• The robust estimator uses m(x) = 1/(1 + ∥x∥^α) with α = 1; the efficient DKSD uses choices selected to improve efficiency.The symmetric Bessel case has ν = 1000.Source: §F.2 · p. 33
• Robust DKSD keeps l1 error bounded as the Dirac value increases, whereas efficient DKSD error grows in both distributions.The comparison uses 500 data points, including 80 corrupted by a Dirac; the left panel is Student-t and the right is symmetric Bessel.Source: Figure 6 · p. 33, §F.2 · p. 33