Density Estimation in Infinite Dimensional Exponential Families

1. Introduction

• The RKHS-indexed family P can approximate a broad class of probability densities arbitrarily well in KL divergence, motivating its use for density estimation.The kernel’s reproducing property makes k(x, ·) act as a sufficient statistic, and the paper treats both well-specified and misspecified estimation.Source: §1 · pp. 2–3
• Likelihood-based estimators can be impractical because handling the log-partition function is difficult, while KDE ignores P’s structure and performs poorly for moderate to large d.The cited sieve approach also requires constructing finite-dimensional subspaces and relies on assumptions that can be difficult to check.Source: §1 · pp. 3–4

1.1 Score Matching and Fisher Divergence

• Score matching replaces KL-based likelihood fitting with Fisher-divergence minimization, whose empirical objective can avoid the log-partition function during optimization.The paper notes that score matching still requires computing or approximating the normalizing constant once to obtain a normalized density.Source: §1.1 · pp. 4–5

1.2 Contributions

• The proposed Fisher-divergence estimator reduces well-specified estimation to a regularized finite-dimensional linear system and has consistency guarantees even when H is infinite-dimensional.The paper also establishes consistency in KL divergence, with rates in KL, Hellinger, and total-variation distances.Source: §1.2 · pp. 5–6
• Under a smoothness condition, the well-specified estimator is consistent with convergence rates in Fisher divergence, KL divergence, Hellinger distance, and total variation.The stated smoothness condition places f0 in the image of a fractional power of C; the paper notes that the rates can saturate for β > 1 under its regularizer.Source: §1.2 · p. 6
• In the misspecified case, the estimator’s Fisher divergence converges to the infimum over P, and simulations on Gaussian and Gaussian-mixture estimation show an advantage over KDE that grows with dimension.The paper presents these simulations as preliminary and clarifies that its principal goal is a flexible modeling technique, not uniform improvement over KDE.Source: §1.2 · pp. 6–7

2. Definitions & Notation

• The notation covers basic index sets, norms, function spaces, convolution, and Fourier transforms.It also defines the bounded, compact, adjoint, self-adjoint, positive, and fractional-power properties of operators on Hilbert spaces.Source: §2 · p. 7
• Trace-class and Hilbert-Schmidt operators, tensor products, and operator ranges are specified for use in the analysis.The tensor product x ⊗ y is viewed as an operator, and R(S) denotes the image of S.Source: §2 · p. 8
• A reproducing kernel belongs to its Hilbert space section-wise and reproduces function values through the inner product.The section inner products recover kernel values; Gaussian and Matérn kernels are among the examples discussed.Source: §2 · p. 8
• Kernel smoothness is expressed through continuous mixed derivatives, whose kernel sections represent derivatives of Hilbert-space functions.The derivative-section inner product gives the corresponding mixed derivative of the kernel.Source: §2 · pp. 8–9
• The section defines KL divergence, Hellinger distance, and total variation distance for comparing probability densities.Total variation is identified with the L1 distance between densities.Source: §2 · p. 9

3. Approximation of Densities by P

• Finite-dimensional RKHS-induced families reproduce finite-dimensional exponential families, while infinite-dimensional RKHSs give a strict generalization.The kernel section k(·, x) plays the role of a sufficient statistic in this construction.Source: §3 · p. 9
• The paper frames approximation richness as a key property of P when H is an infinite-dimensional RKHS.The section also presents finite-dimensional exponential families as generated by suitable finite-dimensional RKHSs.Source: §3 · p. 9
• Under the stated kernel conditions, P is dense in Pc for KL divergence, total variation, and Hellinger distance, and also in Lr when q0 lies in L1 ∩ Lr.The Lr conclusion applies for 1 < r ≤ ∞.Source: §3 · p. 10
• For continuous densities on a compact domain with uniform q0, P gives arbitrarily accurate approximation in KL, Hellinger, and Lr distances for 1 ≤ r ≤ ∞.Source: §3 · p. 10
• When H is sufficiently rich, elements of P approximate C1 densities with finite Fisher divergence, provided q0 is C1.The approximation is in Fisher divergence and relies on density of H in an appropriate function class.Source: §3 · p. 10

4. Density Estimation in P: Well-specified Case

• Minimizing empirical Fisher divergence yields an estimator computable by a finite-dimensional linear system, unlike the practically infeasible MLE with its log-partition function.The estimator is a Tikhonov-regularized solution to an ill-posed linear system; the system can still be computationally intensive when d and n are large.Source: §4 · pp. 11, 13

4.1 Consistency and Rate of Convergence

• For fλ,n, under f0 ∈ R(Cβ) for β > 0, the best convergence rate is n^-1/4 when β ≥ 1; in finite dimensions, the parametric rate is n^-1/2.In finite dimensions, λ is automatically chosen as n^-1/2.Source: §4.1 · pp. 14–15
• With bounded kernels and supp(q0) = Ω, the plug-in density estimator is consistent in Lr, Hellinger, KL, and Fisher divergence; Fisher-divergence rates are available even without smoothness.The cited results also report that convergence in Fisher divergence can use faster-decaying regularization and is faster than in the other distances.Source: §4.1 · pp. 15–16

4.2 Range Space Assumption

• The range-space condition f0 ∈ R(C^β) encodes smoothness relative to H, and the examples explain dimension-independent rates through that relative smoothness.For some RKHS choices, the n^-1/4 rate is described as minimax optimal.Source: §4.2 · pp. 16–17

4.3 Choice of Regularizer

• The standard Tikhonov-regularized estimator’s rate saturates at β = 1, motivating an appropriately chosen regularizer to improve rates for β > 1.The section frames the estimator as regularization of an ill-posed inverse problem.Source: §4.3 · p. 18, §4.2 · p. 18
• Finite qualification η0 causes rate saturation when β > η0, preventing the estimator from fully exploiting f0’s smoothness.The subsection introduces an alternate estimator using gλ to better capture smoothness, especially when β > 1.Source: §4.3 · pp. 18–19
• Unlike the earlier estimator, f_g,λ,n has no simple expression obtained by solving a linear system.Source: §4.3 · p. 19
• As β → ∞, infinite-qualification regularizers attain O_p0(n^-1/2) for ∥f_g,λ,n − f0∥H, versus n^-1/4 under Theorem 6.Improved rates for p_f_g,λ,n are also obtained in various distances.Source: §4.3 · p. 19
• Tikhonov regularization has qualification 1 and saturates at β = 1, whereas Showalter’s method and spectral cut-off have infinite qualification and improve rates for β > 1.Source: §4.3 · p. 20

5. Density Estimation in P: Misspecified Case

• J(p0∥p_fλ,n) converges to inf_{p∈P} J(p0∥p) as λ→0, λn→∞, and n→∞, assuming an attaining f* exists.The result is presented for bounded kernels and extends to unbounded kernels; its proof also extends to the alternative regularizer discussed in the paper.Source: §5 · p. 20
• The analysis extends H through the map Ik into W2(Ω, p0), where functions differing by a constant are identified because constants do not change the density.The resulting estimation problem is to approximate [f*]∼ in W2(Ω, p0) with f∈F.Source: §5 · pp. 21–22
• Theorem 12 removes the requirement for an attaining f* and establishes consistency and a convergence rate for the estimator under its stated assumptions.Its construction treats estimation as approximating the target [f*]∼ in W2(Ω, p0) by f in F.Source: §5 · p. 22
• If Sk is injective and Ik(H) is dense in W2(Ω, p0), some p0 outside P can be consistently estimated even when no f* attains zero Fisher divergence.In this case inf_{p∈P} J(p0∥p)=0, and the results of Theorem 12 coincide; Theorem 10 does not cover this situation.Source: §5 · p. 22
• For β≥1, Theorem 12(ii)'s rates match the well-specified rates in Theorem 7, whereas Theorem 10 gives a rate only when f* is sufficiently smooth.The comparison range is β≥1 in Theorem 7; β=0 and β=1 correspond to different spaces in the two theorems.Source: §5 · p. 23
• If Ik(H) is dense in W2(Ω, p), every p∈PFD can be approximated by some p̃∈P to arbitrary Fisher-divergence accuracy.The proposition states that for every ε>0, J(p∥p̃)≤ε.Source: §5 · p. 23

6. Numerical Simulations

• The proposed estimator requires solving an nd × nd linear system, compared with KDE's O(n) computational requirement.The paper notes that KDE is simple to compute but performs poorly at moderate to large dimensions.Source: §6 · p. 23
• The paper presents the estimator as a modeling technique for approximating unknown densities through a rich infinite-dimensional family, not primarily as a way to improve on KDE.The simulations are described as preliminary empirical investigation.Source: §6 · pp. 23–24
• The experiments estimate a standard normal and a Gaussian mixture; the normal case is well-specified, while the mixture case is misspecified.The selected kernel places Gaussian distributions in the family, making the normal task well-specified and the mixture task misspecified.Source: §6 · p. 24
• Accuracy is assessed with the score-matching objective and correlation with the true density, using an empirical reference based on 10,000 samples from the target.The proposed method's normalization constant is difficult to estimate accurately, motivating these two evaluation measures.Source: §6 · p. 24
• The proposed estimator generally beats KDE on score-objective values and correlation, with low-dimensional exceptions, and its advantage grows as dimension increases.The score-objective comparison has low-dimensional exceptions for the Gaussian and Gaussian-mixture tasks; correlation also has exceptions in some mixture cases.Source: §6 · pp. 24–25, Figure 1 · p. 25

7. Summary & Discussion

• The proposed estimator minimizes regularized empirical Fisher divergence and yields a finite-dimensional linear system, providing an alternative to maximum-likelihood estimators hindered by log-partition computation.The family is indexed by functions in an RKHS and can approximate a broad class of densities arbitrarily well in KL divergence.Source: §7 · p. 25
• Consistency and convergence rates are established for well-specified and misspecified settings under smoothness assumptions such as log p0 ∈ R(C^β).The paper identifies understanding the source condition and minimax optimality as open questions.Source: §7 · p. 26
• The source condition log p0 ∈ R(C^β) is not well understood, and choosing regularization for unknown β motivates future adaptive estimators.The paper also raises minimax optimality as an open question.Source: §7 · p. 26
• Solving a linear system of size nd × nd makes computational efficiency a practical concern; the paper calls for alternate estimators or more efficient implementations.The authors identify improving applicability as the reason to study these alternatives.Source: §7 · pp. 26–27

8.1 Proof of Proposition 1

• Under kernel condition (5), H is dense in C0(Ω) in the uniform norm, yielding density in L1, KL, Hellinger, and Lr distances.For r > 1, the proof uses a bound on ∥pf − pg∥Lr(Ω) in terms of ∥f − g∥∞.Source: §8.1 · p. 27

8.2 Proof of Corollary 2

• For any 1 ≤ r ≤ ∞, the proof approximates densities in Pc by elements of P in Lr, and thereby obtains Hellinger density.It constructs positive approximants pθ and uses approximation in P0; the Hellinger conclusion follows from h(p, q) ≤ ∥p − q∥L1(Ω).Source: §8.2 · p. 27
• The proof also establishes KL density by choosing positive approximants pθ with KL(p∥pθ) arbitrarily small and approximating them from P.It uses the density of H in C0(Ω) to complete the approximation.Source: §8.2 · pp. 27–28

8.3 Proof of Theorem 4

• The proof identifies C as trace-class and compact, establishes its Bochner integral representation (6), and gives the unique regularized minimizer fλ = −(C + λI)^−1ξ.The minimizer is for Jλ(f).Source: §8.3 · pp. 28–29

8.4 Proof of Theorem 5

• The general representer theorem yields the finite-dimensional form used for fλ,n; the standard representer theorem gives the same solutions when K is nonsingular.When K is not invertible, the standard representer formulation has many solutions.Source: §8.4 · pp. 29–30

8.5 Proof of Theorem 6

• Theorem 6’s proof decomposes estimator error into three empirical terms and regularization bias, and establishes convergence under its stated range condition.The proof notes that S2 cannot be improved in its analysis and can dominate the other terms, preventing the bound from capturing kernel smoothness.Source: §8.5 · pp. 30–32

8.6 Proof of Theorem 7

• Under supx∈Ω k(x, x) < ∞ and supp(q0) = Ω, the proof finds a representative in R(C) for each f0 ∈ H with the same density, enabling consistency results to transfer to density distances.The proof transfers convergence to Lr for 1 ≤ r ≤ ∞, Hellinger, and KL distances.Source: §8.6 · p. 32

8.7 Proof of Proposition 8

• The proof of Proposition 8 establishes R(C) ⊂ G.It derives this inclusion using generalized Young’s and Hausdorff–Young inequalities.Source: §8.7 · p. 35

8.8 Proof of Theorem 9

• Theorem 9’s proof bounds error terms by O_p0(λ^β) and O_p0(n^-1/2), then uses the resulting H-norm bound to establish the stated conclusions.The proof says the claims involving Lr, h, and KL follow the same approach as Theorem 7.Source: §8.8 · p. 36

8.9 Proof of Theorem 10

• Theorem 10’s proof establishes convergence toward f* as λ→0 and n→∞, with an additional bound when C has a bounded inverse.The proof invokes the earlier bound on ∥fλ,n−f*∥H.Source: §8.9 · p. 38

8.10 Proof of Proposition 11

• Proposition 11’s proof shows that Ik maps H continuously into W2(Ω,p0), and that Ik and its adjoint Sk are Hilbert–Schmidt and compact.It also states that the other assertions about SkIk and IkSk are straightforward.Source: §8.10 · p. 38

8.11 Proof of Theorem 12

• As λ→0, λn→∞, and n→∞, the estimator’s Fisher divergence converges to inf_{p∈P} J(p0∥p).Under the stated smoothness condition, the proof derives rates using a bound on B(λ).Source: §8.11 · p. 41

8.12 Proof of Proposition 13

• The proof of Proposition 13 uses density of Ik(H) in W2(p) to approximate [f]~ arbitrarily closely in W2(p).For any ε>0, it chooses g∈H with ∥[f]~−Ikg∥W2≤ε.Source: §8.12 · p. 41

A. Appendix: Technical Results

• The appendix presents technical results used in the paper’s proofs.Source: A. Appendix: Technical Results · p. 42

A.1 Bounds on Various Distances Between pf and pg

• The subsection derives bounds between p_f and p_g in P∞ from earlier results and an elementary inequality.Claims (iii) and (iv) are quoted from Lemma 3.1 of van der Vaart and van Zanten (2008), and claim (ii) follows from (A.2) with r = 1.Source: §A.1 · pp. 42–43

• A minimizer of an objective depending on finitely many inner products is represented in the span of the corresponding vectors.The theorem considers a real Hilbert space and a convex differentiable function of the inner products with φ_1,…,φ_m.Source: §A.2 · p. 43
• The span coefficients satisfy a possibly nonlinear equation involving the Gram matrix of the vectors.The Gram matrix has entries ⟨φ_i, φ_j⟩_H, and the adjoint of the associated map is defined in the theorem’s setup.Source: §A.2 · p. 43

A.3 Bounds on Approximation Errors, A0(λ) and A 1

• For f_λ = (C + λI)^−1Cf, the approximation error A_θ(λ) tends to zero as λ tends to zero for θ > 0.When f lies in R(C^β) and β + θ > 0, the proposition gives an additional approximation-error bound.Source: §A.3 · p. 44

A.4 Bound on the Norm of Certain Operators and Functions

• Proposition A.4 provides asymptotic bounds for norms of certain operators and functions, with the proof using Chebyshev’s inequality.The result is presented for a separable Hilbert space and an integrability condition on a Hilbert-Schmidt-operator-valued function.Source: §A.4 · p. 45

A.5 Interpolation Space

• The real-method interpolation spaces are defined through norms built from the K-functional for two continuously embedded Banach spaces.The K-functional minimizes the sum of the E0 norm of one component and t times the E1 norm of the other.Source: §A.5 · p. 47

B. Appendix: Miscellaneous Results

• The appendix presents proofs of claims made in Sections 1, 4, and 5.Source: B. Appendix: Miscellaneous Results · p. 48

B.1 Relation between Fisher and Kullback-Leibler Divergences

• Proposition B.1 relates Fisher and Kullback–Leibler divergences for densities smoothed by Gaussian convolution, under stated conditions.The smoothed densities are p_t and q_t, formed by convolution with N(0, tI_d).Source: §B.1 · p. 48
• The proof uses an identity that generalizes de Bruijn’s identity and integrates it with respect to t.Source: §B.1 · p. 48

B.2 Estimation of p0: Unbounded k

• The unbounded-kernel construction assumes a known bound M on the RKHS norm of f0.The estimator is clipped, and its construction requires M to be known a priori.Source: §B.2 · pp. 48–49
• The clipped estimator is constructed through a quadratically constrained quadratic program for its coefficients.The coefficients are obtained by solving the stated QCQP.Source: §B.2 · pp. 48–49
• Consistency in Lr, Hellinger, and KL distances matches the bounded-kernel case under additional integrability conditions, while consistency in J is weaker.The discussion notes that the weaker J result may reflect the proof technique.Source: §B.2 · p. 51

B.3 R(Cβ) and Interpolation Spaces

• For non-integer β > 0, R(C^β) is interpreted as an interpolation space between R(C^⌈β⌉) and R(C^⌊β⌋).The proposition applies to any self-adjoint, bounded, compact operator on a separable Hilbert space.Source: §B.3 · p. 52
• The proof also uses an interpolation result for weighted spaces whose norms are equivalent under the stated conditions.Source: §B.3 · p. 52
• For 0 < β < 1, the section identifies R(C^β) with the interpolation space [H, R(C)]_{β,2}.Source: §B.3 · p. 54

B.4 Denseness of IkH in W2(Rd, p)

• Under conditions on a bounded continuous translation-invariant kernel, the section establishes injectivity of the restriction of S_k to the specified one-dimensional subspace.The result is stated for d = 1; the corresponding general d > 1 result is open.Source: §B.4 · pp. 54–55
• Characterizing injectivity of S_k on W2(R^d, p), equivalently denseness of I_kH there, remains open.Even the stated one-dimensional restricted result does not guarantee injectivity on the full space.Source: §B.4 · p. 54
• The listed examples satisfying Proposition B.5’s kernel conditions include Gaussian, Matérn with β > 1, and inverse multiquadric kernels on R.Source: §B.4 · p. 55