PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretation and Generalization of Score Matching

Siwei Lyu

arXiv:1205.2629v1cs.LGstat.ML

TL;DR

Score matching lacks a fully established relation to maximum likelihood and originally applies only to continuous data with differentiable densities. The paper links the methods through Gaussian scale-space analysis, generalizes score matching with linear operators, and extends it to discrete data, interpreting the result as robustness to small noise perturbations.

  • Problem

    Score matching’s relation to maximum likelihood is unclear, and its differentiability requirements prevent direct application to discrete data.

  • Method

    The paper relates Fisher and KL divergences in scale space and generalizes score matching using linear operators, including marginalization for discrete data.

  • Results

    The Fisher divergence is the scale-space derivative of KL divergence, and generalized score matching yields an extension for discrete data.

  • Takeaways & Limitations

    Score matching can be interpreted as selecting parameters stable under small noise perturbations while retaining applicability to unnormalized models.

  • Takeaways & Limitations

    The paper leaves the study of appropriate complete linear operators for specific high-dimensional models as future work.

Abstract

from arXiv · show

Score matching is a recently developed parameter learning method that is particularly effective to complicated high dimensional density models with intractable partition functions. In this paper, we study two issues that have not been completely resolved for score matching. First, we provide a formal link between maximum likelihood and score matching. Our analysis shows that score matching finds model parameters that are more robust with noisy training data. Second, we develop a generalization of score matching. Based on this generalization, we further demonstrate an extension of score matching to models of discrete data.

1 Introduction

Score matching addresses partition-function intractability in high-dimensional density models. This paper studies its relation to maximum likelihood and extends it toward discrete data.

  • Motivation: Maximum likelihood can be computationally infeasible for high-dimensional models because partition functions may be intractable.Score matching avoids this bottleneck and has been applied to models such as natural images.
  • Motivation: Score matching offers consistent estimation and deterministic optimization without computing the model partition function.
  • Research questions: The paper identifies two unresolved issues: score matching’s relation to maximum likelihood and its restriction to continuous differentiable data.
  • Contributions: The paper formally links score matching to maximum likelihood and interprets score matching as favoring robustness to noisy training data.
  • Contributions: The paper generalizes score matching while preserving computational advantages and demonstrates an extension to discrete data.

2 Background

Maximum likelihood can fail computationally for high-dimensional unnormalized models because partition functions are difficult to compute. Score matching instead uses Fisher divergence and score functions, avoiding direct partition-function computation.

  • Maximum likelihood: Maximum likelihood is equivalent to minimizing KL divergence, with the data-dependent term approximated by averaging over training data.
  • Maximum likelihood: High-dimensional models such as Markov random fields often use unnormalized components whose partition functions are computationally intractable.Because maximum likelihood depends directly on normalized model densities, the partition function becomes a learning bottleneck.
  • Score matching: Score matching minimizes Fisher divergence between data and model densities using their score functions.The Fisher divergence is nonnegative and vanishes only when the densities agree almost everywhere, although it is not symmetric.
  • Score matching: Score matching avoids partition-function computation because normalization constants cancel in the model score function.Its objective can be computed as an expectation involving the unnormalized model.
  • Score matching: Under smoothness and fast-decay conditions, integration by parts supports rewriting the Fisher-divergence objective into a data-expectation form.

3 Score Matching and Maximum Likelihood

The paper establishes that Fisher divergence is the scale-space derivative of KL divergence under Gaussian smoothing. This relation explains score matching as emphasizing structures that remain stable under noise while discounting sampling artifacts.

  • Formal relation: The paper establishes a formal relation between Fisher divergence and KL divergence in a Gaussian scale space.The proof uses lemmas involving differential operators, the heat kernel, differentiation under the integral, and integration by parts.
  • Scale space: Gaussian smoothing convolves the data density with zero-mean white Gaussian noise whose variance is the scale factor.
  • Formal relation: The Fisher divergence equals the derivative of KL divergence with respect to the Gaussian scale factor.
  • Interpretation: Increasing the smoothing scale never increases KL divergence, because Fisher divergence is nonnegative.Larger scales smooth small local structures and reduce the signal-to-noise ratio.

4 Generalized Score Matching

Generalized score matching replaces the gradient in Fisher divergence with a linear operator, preserving computational advantages while enabling alternative formulations such as marginalization and posterior-mean operators.

  • Generalized score matching replaces the density gradient with a general linear operator L in Fisher divergence.The resulting generalized Fisher divergence is nonnegative and vanishes when the two densities agree almost everywhere.
  • Generalized score matching retains partition-function cancellation and can be computed from training data using expectations involving only the model density.The adjoint operator enables transformation to an expectation form, while the term independent of model parameters can be discarded.
  • The operator must be complete to preserve enough information for parameter learning; an operator mapping every function to zero is unusable.Completeness requires Lp and Lq equality to imply equality of the original densities.
  • The gradient operator recovers original score matching and is complete.Its adjoint is negative divergence, so the generalized formulation reduces to the original score-matching equations.
  • The marginalization operator compares singleton conditionals and remains complete under Brook’s Lemma.Its components correspond to reciprocals of conditional densities, with integration replaced by summation for discrete components.
  • Posterior-mean operators interpret generalized score functions as optimal latent-variable estimators, but completeness is difficult to verify generally.The additive Gaussian-noise case yields the Fisher divergence and original score matching, whereas marginalization lacks this posterior-mean interpretation.

5 Generalized Score Matching for Discrete Data

The paper extends score matching to discrete data by replacing continuous integration with summation and using a marginalization operator within generalized score matching. The resulting objective balances singleton conditional densities and agrees at optimum with ratio matching, while differing from maximum pseudo-likelihood.

  • Generalized Score Matching for Discrete Data: The original score matching method cannot directly handle discrete data because its continuous-data and differentiability requirements fail.For discrete data, ∇log p(x) is not well defined.
  • Generalized Score Matching for Discrete Data: The discrete extension minimizes generalized Fisher divergence over discrete vectors, replacing integration with summation.It uses the marginalization operator M within generalized score matching.
  • Generalized Score Matching for Discrete Data: The generalized objective is derived by substituting the marginalization formulation, applying its adjoint operator, and dropping terms independent of θ.Further rearrangement expresses the result using conditional probabilities.
  • Generalized Score Matching for Discrete Data: At optimum, minimizing generalized Fisher divergence balances the values of the singleton conditional densities.The relevant conditional-probability term reaches its minimum when the conditional value approaches a constant.
  • Relation with Ratio Matching: The discrete extension and ratio matching agree at optimum, although both differ from maximum pseudo-likelihood.Their objective functions are different from the original score-matching objective.

6 Conclusion

The conclusion presents two results: a formal relation between maximum likelihood and score matching, and a generalized formulation that extends score matching to discrete data. It interprets score matching as seeking parameters stable under small noise and identifies further applications and operator choices for future study.

  • 6 Conclusion: The Fisher divergence is the derivative of the KL divergence in scale space with respect to the scale factor.This establishes the formal link between maximum likelihood and score matching.
  • 6 Conclusion: Score matching therefore searches for parameters that are stable under small noise perturbations in training data.The conclusion presents this as an interpretation suggested by the divergence relation.
  • 6 Conclusion: The paper generalizes score matching using general linear operators in the Fisher divergence and instantiates the method for discrete data.The discrete formulation is presented as a natural extension of score matching.
  • 6 Conclusion: Future work includes other diffusion kernels, complete linear operators for high-dimensional models, and applications in bioinformatics and image modeling.These directions are described as areas the authors hope to explore.

Appendix

The appendix supplies proofs for the paper’s operator identities and conditional-probability results. Its derivations use integration-order changes, rearrangement, differentiation, and cancellation of joint-density terms.

  • Appendix: The proof of Lemma 2 starts from the definition of the transformed density and obtains the result by repeated differentiation.Combining the derived equations proves the lemma.
  • Appendix: Lemma 3 shows that ratios of joint probabilities for two assignments can be determined using only singleton conditionals.Normalization then implies uniqueness of the joint density given those singleton conditionals.
  • Appendix: The appendix proves an identity by introducing terms that cancel and reducing ratios to singleton conditionals.The resulting expression is completely determined by the singleton conditional probabilities.
  • Appendix: Lemma 4 is established by defining a reassigned vector and switching the integration order for x_i and ξ_i.Rearranging the resulting terms completes the proof.
Loading 1205.2629v1…