PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Score Matching for Parameter Estimation on Convex Domains

Nishanth Shetty, Saisuchith Mahajan, Chandra Sekhar Seelamantula

arXiv:2609.11521v1cs.LGstat.ML

TL;DR

Unnormalized models require estimation methods that avoid explicit partition-function computation, while prior score-matching work on subsets of R^d lacked a formal objective derivation. This paper derives generalized score matching from MPF on convex domains, establishes theoretical guarantees, and finds that boundary attenuation choices affect finite-sample performance.

  • Problem

    Partition-function-free estimation is needed for unnormalized models, but prior score-matching work on subsets of R^d lacked a formal derivation of its objective.

  • Method

    The paper constructs generalized score matching as the infinitesimal limit of MPF and unifies classical and domain-adapted formulations on convex subsets of R^d.

  • Results

    The objective is a proper second-order local scoring rule, convex for exponential-family canonical parameters, and yields consistent empirical estimators under standard regularity conditions.

  • Takeaways & Limitations

    Different boundary attenuation rates produce different empirical performance, while a principled procedure for selecting the optimal attenuation function remains open.

  • Takeaways & Limitations

    Computing the generator G_ϕ is computationally expensive in very high dimensions, constraining the practical choice of ϕ.

Abstract

from arXiv · show

Maximum likelihood (ML) estimation is a principled and statistically efficient approach for learning probabilistic models. However, for unnormalized models, ML estimation requires evaluating the partition function and differentiating through it, which may not always be tractable. Score matching provides a practically viable alternative that circumvents this obstacle by fitting the score in a way that eliminates dependence on the normalizing constant. We derive the generalized score matching objective on a convex subset of $\mathbb{R}^{d}$ constructively starting from Minimum Probability Flow (MPF) learning, and show how classical score matching as well as domain-adapted variants for non-negative data arise naturally within the proposed framework. We show that the resulting objective is a {\it proper local scoring rule} of second-order, which provides the theoretical guarantee that the true density is recovered when the objective is minimized. Furthermore, for a model belonging to the exponential family, we establish convexity of the objective together with consistency of the finite-sample estimator under standard regularity conditions. Our derivation sheds new light on the scope and applicability of generalized score matching in various problem settings. We compare generalized score matching-based estimators on constrained domains, where the partition function is analytically intractable. We provide experimental results on parameter estimation for model densities belonging to the exponential family defined over convex subsets of $\mathbb{R}^{d}$, and a generative modeling use-case to demonstrate broader applicability of the proposed generalized score matching framework.

1 Introduction

The paper addresses the lack of a formal constructive derivation for generalized score matching on convex domains. It derives the objective from MPF and establishes theoretical guarantees, while evaluating constrained-domain estimation and generative modeling.

  • 1.1 Contributions: The paper derives generalized score matching objectives on convex subsets of R^d constructively as the infinitesimal limit of MPF.The framework includes the practically relevant case of non-negative data and differs from prior Fisher-divergence-based formulations.
  • 1.1 Contributions: Appropriate choices within the framework recover classical and domain-adapted score matching variants, including non-negative score matching.The associated operator is complete and tied to the convex function selected for the domain.
  • 1.1 Contributions: The resulting objective is a proper local scoring rule of second order whose minimization recovers the true data-generating distribution.
  • 1.1 Contributions: For exponential-family models, the objective is convex in canonical parameters and its empirical minimizer is consistent under standard regularity conditions.
  • Experiments: The paper evaluates parameter estimation on convex constrained domains and applies the proposed loss to implicit VAE training on MNIST and CelebA.

2 Related Work

Score matching avoids partition-function computation by matching model and data score functions, then using integration by parts to obtain a tractable objective. Generalized formulations replace the gradient with suitable operators and include boundary-adapted variants for non-negative data.

  • 2.1 Score Matching: Score matching fits the gradient of the log-density and, after integration by parts, depends only on model derivatives rather than the normalizing constant or target score.
  • 2.1 Score Matching: The operator-theoretic view generalizes score matching by replacing the gradient with a linear operator while retaining cancellation of normalization constants.Theoretical analyses connect generalized objectives to properties of associated Markov processes or diffusions.
  • 2.2 Non-negative Score Matching: For non-negative data, weighted objectives attenuate boundary effects near zero and extend to elementwise weights h(x).The tractable form includes weighted score, Hessian, and derivative terms.

3 Main Results

The paper derives generalized score matching on convex domains from small-neighborhood MPF limits using convex-function-induced local geometry. The construction yields complete operators and unifies existing score matching objectives under stated regularity and boundary assumptions.

  • 3.1 Generalized Score Matching: The derivation extends MPF-based recovery from R^d to generic convex supports, where constructing tractable conditional neighborhoods is otherwise challenging.
  • 3.1 Generalized Score Matching: Small-neighborhood limits of MPF recover generalized score matching objectives on convex domains through appropriately chosen neighborhoods and weights.The construction uses strictly convex functions and local Bregman balls to define domain-compatible neighborhoods.
  • 3.1 Generalized Score Matching: The derivation requires explicit regularity and boundary conditions, with bounded convex sets handled through Lipschitz-boundary assumptions and unbounded sets treated separately.
  • 3.1 Generalized Score Matching: The resulting operator is complete, so equality of its action on two supported densities implies equality of those densities almost everywhere.
  • 3.1 Generalized Score Matching: Classical score matching and non-negative score matching arise by selecting appropriate convex functions and associated matrix fields.The non-negative variant differs from an earlier formulation by an additional determinant-related term.
  • 3.1 Generalized Score Matching: The unified MPF derivation is presented as a principled and novel development that can extend to a broader class of objectives.

4 Generalized Score Matching is a Proper Scoring Rule

The paper places the proposed objective within the theory of proper local scoring rules of second order. It establishes that the resulting score is proper, supporting recovery of the true distribution through minimization.

  • 4 Generalized Score Matching is a Proper Scoring Rule: Proper scoring rules ensure that the expected loss is minimized when the model distribution equals the data distribution.
  • 4 Generalized Score Matching is a Proper Scoring Rule: The second-order local scoring-rule framework encompasses score matching and other losses through derivatives up to second order.
  • 4 Generalized Score Matching is a Proper Scoring Rule: The proposed score S_ϕ(x; θ) is proven to be a proper scoring rule with geometry determined by G_ϕ(x).

5 Analysis for Exponential Family

For exponential-family models, the empirical generalized score matching objective is convex under a positive-semidefiniteness condition and yields a consistent, asymptotically normal estimator under standard moment assumptions.

  • Positive semidefiniteness of Γ_N makes the finite-sample objective convex in θ.
  • Under finite-moment assumptions on Γ_0, g_0, and Σ_0, the unconstrained minimizer is almost surely unique and has a closed-form solution.
  • The resulting estimator is consistent and asymptotically normal.

6 Experiments

Experiments evaluate generalized score matching on constrained supports where maximum likelihood is intractable, including parameter estimation for truncated Gaussians and generative modeling applications.

  • Experimental setup: The experiments cover the positive orthant, simplex, and simplex polytope, with sample sizes N ∈ {200, 500, 800} and 50 independent trials.
  • Generative modeling: The framework also supports generative modeling through an implicit VAE trained with the proposed GSM loss on MNIST and CelebA.
  • Simplex-polytope results: ϕ1 achieves the lowest median MSE for μ at N = 800, with 0.0072 versus 0.0227 for ϕ2, 0.0828 for ϕ3, 0.0829 for h1, and 0.1050 for Truncated SM.
  • Simplex-polytope results: For precision-matrix estimation, h1 has MSE around 10^5–10^6 across sample sizes, roughly an order of magnitude higher than the other evaluated estimators.
  • Practical considerations: The framework uses convex barrier choices whose generators attenuate near domain boundaries, while different attenuation rates produce different empirical performance.

7 Conclusions and Outlook

The paper derives generalized score matching constructively from MPF for convex domains, establishes theoretical guarantees for exponential families, and identifies computational and methodological directions for future work.

  • Contributions: A small-neighborhood limit of the MPF objective yields generalized score matching on convex subsets of R^d, with classical and non-negative variants as special cases.
  • Theoretical guarantees: The generalized objective is a proper local scoring rule of second order, and exponential-family objectives are convex with consistent empirical estimators under standard regularity conditions.
  • Limitations and outlook: Computing the generator G_ϕ is computationally expensive in very high dimensions, constraining the practical choice of ϕ.
  • Connection to MPF: MPF measures probability flow from empirical data states to non-data states over infinitesimal time and avoids computing the intractable partition function.
  • Proof strategy: The derivation uses second-order Taylor expansion, Gaussian moments, and a divergence identity before taking the small-noise limit.

C Proof of Proposition 3.1

The proposition follows by applying integration by parts to the generalized score matching objective and removing a term independent of the model parameters.

  • The proof begins from Equation 4 and applies integration by parts under the stated regularity condition.
  • After substitution, the added term is independent of θ and can be omitted from the optimization objective.
  • Expanding the weighted norm yields the final equality and completes the proof.

D Proof of Theorem 3.2

The proof derives the generalized score matching objective by expanding the MPF objective over shrinking neighborhoods and evaluating the resulting integrals through symmetry, coordinate changes, and convergence arguments.

  • Small-radius expansion: An invertible linear transformation maps each local neighborhood to the unit Euclidean ball, simplifying the subsequent moment integrals.The transformation preserves the integration region up to the Jacobian factor because the relevant Hessian is symmetric and positive definite.
  • Small-radius expansion: A second-order Taylor expansion of the MPF objective, followed by normalization and the limit r(x) → 0, recovers the generalized score matching objective.The proof maps local Bregman neighborhoods to Euclidean balls, expands the energy and connectivity terms, and retains contributions of order r(x)^(d+2).
  • Moment calculations: Odd moments vanish by symmetry, while even moments produce the terms retained in the limiting objective.The proof uses symmetry lemmas for second-, third-, and fourth-order integrals, including the relation I1 = 3I2.
  • Technical conditions: The proof requires a radius function whose associated neighborhood remains inside the domain and can be chosen uniformly bounded.Bounding the radius preserves containment because reducing the radius only shrinks the corresponding neighborhood.
  • Limit identification: After removing terms independent of θ, the lowest parameter-dependent order is r(x)^(d+2), and dominated convergence justifies interchanging the limit and expectation.The resulting minimization is equivalent to minimizing Equation 10 with λ = C6.

E Proof of Proposition 3.2 and Its Extension to Unbounded Sets

The proof establishes the GSM form on bounded convex domains through integration by parts, then extends the argument to unbounded domains using smooth truncations and stronger regularity assumptions.

  • Bounded convex sets: On bounded convex domains, integration by parts converts the objective into the GSM formulation under boundary regularity conditions.The argument assumes a bounded convex set, a Lipschitz boundary, p ∈ C1, and boundary conditions specified in Proposition 3.2.
  • Unbounded sets: For unbounded domains, the proof uses smooth cutoff functions that equal one inside expanding balls and vanish outside slightly larger balls.The cutoff construction permits integration by parts on bounded convex intersections before taking the limit.
  • Unbounded sets: Dominated convergence and the cutoff derivatives’ vanishing limit allow the truncated identities to converge to the desired unbounded-domain identity.The proof then follows the bounded-case completing-the-squares argument.
  • Scope and assumptions: The unbounded-domain treatment requires stronger technical assumptions and may not apply directly in all practical settings.The authors note that additional structure can permit relaxation of some assumptions.
  • Identification: Under the connected-support argument, two densities with the same relevant score agree almost everywhere after normalization.Their ratio is constant almost everywhere, and because both are densities the constant must yield equality almost everywhere.

G Generalization of Theorem 3.1 and Theorem 3.2

The appendix generalizes the derivation by starting from a broader MPF objective and selecting connectivity functions that recover generalized score matching in small-neighborhood limits.

  • General starting objective: A strictly convex function f and a connectivity function g define a broader starting objective that contains the MPF objective as a special case.Choosing f(z) = −√z recovers the MPF objective, while symmetric conditional connectivity recovers another established formulation.
  • Taylor expansion: Second-order Taylor expansion of the generalized objective around nearby points yields the terms needed for the GSM limit.The derivation expands each component around y and collects terms through second order.
  • Soft neighborhoods: With conditional density connectivity qε(y | x), the generalized objective can be written in limiting form for sufficiently small ε.This is stated as Theorem G.1 under the assumptions of Theorem 3.1.
  • Hard neighborhoods: For hard neighborhoods, the analogous small-radius expansion uses the weighting function and generator Gϕ from the main derivation.Theorem G.2 states the corresponding limiting form under the assumptions of Theorem 3.2.
  • Interpretation: The framework connects score matching and generalized score matching by extending the same small-neighborhood perspective to the generalized setting.The appendix notes that the generalized results parallel prior score-matching derivations.

H.2 Proof of Proposition 4.1

The appendix derives estimator properties and reports experiments across exponential, truncated Gaussian, and Dirichlet models on constrained domains.

  • Proof of Theorem 5.1: The finite-sample objective has gradient Γ_Nθ + g_N, and positive-definite Γ_N yields an almost surely unique minimizer in closed form.The minimizer is obtained from Γ_Nθ = −g_N.
  • Proof of Theorem 5.1: The appendix also establishes almost sure convergence and asymptotic normality through sample-average convergence and standard limit theorems.The argument uses the strong law of large numbers, the continuous mapping theorem, and Slutsky’s theorem.
  • Exponential Model: Original score matching on the positive half-line produces the data-independent estimate 0, so it is not a proper scoring rule in this setting.The estimate does not depend on the observed samples.
  • Truncated Gaussian Model: For the positive-orthant truncated Gaussian, h1 achieves the lowest median MSE, while proposed ϕ1 is the strongest proposed estimator across sample sizes.At N = 800, h1 reaches 0.0016 for µ and 4.70 × 10^3 for K; ϕ1 reaches 8.47 × 10^3 for K versus 7.93 × 10^3 for h2.
  • Dirichlet Model: For the Dirichlet model on ∆9, ϕ2 has the lowest mean and median MSE, and all proposed choices beat the baselines in median MSE at every sample size.At N = 800, Truncated SM has median MSE 0.7314 and GSM CD has 1.3099.

K.4.1 Robustness Across Diverse Ground Truth Parameter Initializations

Across diverse parameter configurations, estimator performance varies by domain and model, with different potentials leading on the simplex, positive orthant, and Dirichlet settings. Additional experiments show sensitivity to the power-barrier exponent, non-convex boundaries, and candidate-selection heuristics, while generative modeling results remain competitive with SSM-VR.

  • Robustness Across Diverse Ground Truth Parameter Initializations: ϕ1 achieves the lowest error on the standard simplex, with a 100% win rate for K and a 90% win rate for µ across 50 configurations.Its average ranks are 1.00 for K and 1.16 for µ, while h1 ranks last for K.
  • Robustness Across Diverse Ground Truth Parameter Initializations: ϕ2 performs best for the Dirichlet model, obtaining a 96% win rate, average rank 1.04, and median MSE 0.0912.All three proposed estimators rank ahead of Truncated SM, GSM CD with h(x) = x, and Yu et al.
  • Robustness Across Diverse Ground Truth Parameter Initializations: On the positive orthant, h1 is strongest, winning 94% of K comparisons and 82% of µ comparisons, while proposed estimators trail it.ϕ3 ranks lowest for K, with average rank 5.00 and median MSE 1.83 × 10^5.
  • Effect of Power Barrier Exponent: The power-barrier exponent has a non-monotonic effect: p = 1.6 yields the lowest reported errors, whereas small exponents and p = 1.8 perform substantially worse.At p = 1.6, median MSE is 0.0021 for µ and 5.68 × 10^3 for K; at p = 1.8, these rise to 0.0149 and 1.17 × 10^4.
  • Evaluation Under Non-Convex Support: On the non-convex domain, K error plateaus near 150 across sample sizes, while µ error falls from 0.0959 at N = 50 to 0.0079 at N = 200.The analysis attributes this asymmetric behavior to non-vanishing internal-boundary contributions and a resulting asymptotic bias for K but not µ.
  • Data Driven Ranking of Candidate Choices of ϕ: Estimated total variance generally identifies stable potential functions, whereas the condition number performs well on truncated Gaussian settings but fails on the Dirichlet model.For Dirichlet, condition-number Top-1 accuracy is 8%, while estimated total variance reaches 58%; the latter still rejects highly unstable choices.
Loading 2609.11521v1…