Source-linked AI summary
Universality, Characteristic Kernels and RKHS Embedding of Measures
Bharath K. Sriperumbudur, Kenji Fukumizu, Gert R. G. Lanckriet
TL;DR
The paper generalizes RKHS embeddings from probability measures to finite signed Borel measures and studies when the embedding is injective. It shows that injectivity holds if and only if the kernel is universal, providing a measure-embedding characterization and relating universal and characteristic kernels.
Problem
Applications of RKHS mean embeddings require injectivity to distinguish probability measures, while universality has traditionally been studied through function approximation and kernel-based Bayes-risk results.
Method
The paper embeds finite signed Borel measures as mean elements in an RKHS and analyzes kernel conditions for injectivity.
Results
The embedding is injective if and only if the kernel is universal, yielding a novel measure-embedding characterization of universal kernels and a relation between universal and characteristic kernels.
Takeaways & Limitations
Universality can be viewed through injective embedding of finite signed measures, connecting function-approximation theory with characteristic-kernel measure embeddings.
Takeaways & Limitations
c0-universality approximates only C0(X), so it cannot handle functions in C(X)\C0(X).
Abstract
from arXiv · showhide
A Hilbert space embedding for probability measures has recently been proposed, wherein any probability measure is represented as a mean element in a reproducing kernel Hilbert space (RKHS). Such an embedding has found applications in homogeneity testing, independence testing, dimensionality reduction, etc., with the requirement that the reproducing kernel is characteristic, i.e., the embedding is injective. In this paper, we generalize this embedding to finite signed Borel measures, wherein any finite signed Borel measure is represented as a mean element in an RKHS. We show that the proposed embedding is injective if and only if the kernel is universal. This therefore, provides a novel characterization of universal kernels, which are proposed in the context of achieving the Bayes risk by kernel-based classification/regression algorithms. By exploiting this relation between universality and the embedding of finite signed Borel measures into an RKHS, we establish the relation between universal and characteristic kernels.
1. Introduction
The paper extends RKHS embeddings from probability measures to finite signed measures and uses injectivity to characterize universality and relate it to characteristic kernels.
- Main contributions: Injectivity of the finite signed-measure embedding is linked to universal kernels, providing a measure-embedding characterization distinct from function approximation.The paper presents this as a main contribution and connects it to kernel-based classification and regression.
- Universality characterizations: For compact Hausdorff spaces, c-universality is characterized exactly by injectivity for finite signed Radon measures.Both necessary and sufficient conditions are provided.
- Universality characterizations: For non-compact Hausdorff spaces, cc-universality is characterized by injectivity for compactly supported finite signed Radon measures.This measure-based characterization differs from an integral-operator characterization.
- Universality characterizations: c0-universality is characterized by injectivity for finite signed Radon measures on locally compact Hausdorff spaces.This characterization also clarifies its relationship with characteristic kernels.
- Relations among universality notions: c0-universality approximates only C0(X), whereas cb-universality extends the target space to bounded continuous functions Cb(X).The broader target class is described as more applicable in learning theory.
- Relations among universality notions: On compact spaces, c-, cc-, c0-, and cb-universality are equivalent; on non-compact spaces, cb- is stronger than c0-, which is stronger than cc-universality.The relationships among these notions are summarized in Figure 1.
- Characteristic kernels: The paper relates universality to characteristic kernels and shows that c0-universality makes MMD induce the usual weak topology on Radon probability measures.Characteristicness alone is sufficient for MMD to be a metric, but the topology result requires the stronger c0-universality condition.
2. Definitions & Notation
The paper establishes notation for continuous-function spaces, dual spaces, Radon measures, positive-definite kernels, Fourier transforms, and analytic functions.
- Function spaces: C(X) denotes continuous real-valued functions, Cb(X) bounded continuous functions, and C0(X) continuous functions vanishing at infinity.These spaces use the uniform norm ∥f∥u = sup_x∈X |f(x)|.
- Measures: The support of a finite signed Radon measure consists of points whose every neighborhood has nonzero total variation.The support is defined using |µ|(U) ≠ 0 for every open neighborhood U.
- Measures: Mbc(X) is the space of compactly supported finite signed Radon measures, while Mb(X) contains all finite signed Radon measures.M1+(X) denotes Radon probability measures.
- Kernels: A positive-definite kernel satisfies the associated quadratic-form condition, while strict positive definiteness requires equality only for zero coefficients on distinct points.The conditional version additionally imposes coefficients summing to zero.
- Fourier analysis: For f ∈ L1(Rd), the paper denotes its Fourier transform and inverse by f-hat and f-check; finite Borel-measure Fourier transforms are bounded and uniformly continuous.The notation is introduced for Fourier analysis on Rd.
- Additional notation: The interior of a set A is denoted A◦, and a holomorphic function is complex differentiable at every point of its domain.An entire function is holomorphic on all of Cd.
3. Characterization of Universal Kernels
The paper characterizes several forms of kernel universality through injective RKHS embeddings of finite signed measures, yielding necessary and sufficient conditions and relations among universality notions.
- Relations among notions: On compact spaces, c-, cc-, c0-, and cb-universality coincide because the relevant continuous-function spaces are identical.In particular, c-universality is equivalent to injectivity for finite signed Radon measures.
- Main characterization: Theorem 6 makes injectivity of finite signed-measure embeddings equivalent to c-, cc-, c0-, or cb-universality under appropriate assumptions on X and the measure class.This provides a measure-embedding perspective complementary to function approximation.
- Novelty: The characterizations improve on earlier results by providing necessary and sufficient conditions, including a complete characterization of c-universality where Steinwart gave only sufficiency.They replace certain integral-operator criteria with injectivity of measure embeddings into the RKHS.
- Relations among notions: c0-universality implies cc-universality, but the converse generally fails; the two notions become equivalent for radial kernels on R^d.The implication follows from the inclusion of compactly supported measures in the broader finite-measure class.
- Checkable conditions: Strict positive definiteness is necessary for c-, cc-, and c0-universality, but generally is not sufficient for c0-universality.For translation-invariant kernels on R^d, c0-universality is characterized by supp(Λ) = R^d; compact support of ψ is sufficient.
- Checkable conditions: The paper derives easily checkable universality criteria for kernel classes including translation-invariant kernels, with results summarized in Figure 1.Examples include Gaussian, Laplacian, and exponential kernels, while some sinc-type kernels are cc-universal but not c0-universal.
4. Characteristic Kernels and Universality
The paper connects characteristic kernels, which make probability-measure embeddings injective, with universality through RKHS measure embeddings. It establishes equivalences and one-way implications across several kernel classes, while also identifying conditional strict positive definiteness as necessary for characteristicness.
- Motivation: The embedding of probability measures is useful for higher-order statistical tasks but must be injective to distinguish measures.Characteristic kernels are introduced to guarantee this injectivity.
- Universal and characteristic kernels: Universal kernels imply characteristic kernels for the corresponding classes of probability measures.This follows from Proposition 20 for c-, cc-, and c0-universal kernels.
- Universal and characteristic kernels: For kernels satisfying (A1) on R^d, c0-universality is equivalent to characteristicness on all Borel probability measures.The result provides both necessary and sufficient conditions in this setting.
- Universal and characteristic kernels: On translation-invariant and radial kernels over R^d, universal and characteristic properties are equivalent, whereas the converse is not generally true.The paper also gives specialized equivalences involving c-, cc-, and c0-universality under additional assumptions.
- Characteristic and strictly pd kernels: Every characteristic kernel is conditionally strictly positive definite, but conditional strict positive definiteness alone does not generally imply characteristicness.The converse to Proposition 23 is explicitly stated to be false.
- Metrization of weak topology: Characteristicness suffices for MMD to be a metric, but c0-universality is required for MMD to metrize the weak topology.For locally compact Hausdorff spaces, c0-universality makes the MMD topology coincide with the weak topology.
5. Conclusions & Discussion
The paper generalizes RKHS embeddings from probability measures to finite signed Borel measures and uses injectivity to characterize universality. It then relates universal and characteristic kernels, while noting that Lp-universality is outside the paper’s direct treatment.
- Conclusions: The paper shows that injective RKHS embedding of finite signed Borel measures is equivalent to kernel universality.This supplies a measure-embedding characterization distinct from function-approximation characterizations.
- Conclusions: The measure-embedding result establishes relations between universal and characteristic kernels, including equivalence for translation-invariant and radial kernels on R^d.Characteristicness concerns injective embeddings of Borel probability measures.
- Discussion: The discussion focuses on universality notions defined through RKHS density in subsets of continuous functions under the uniform norm.The scope includes several uniform-approximation formulations of universality.
- Discussion: Lp-universality is not directly considered because its relation to measure embeddings is not straightforward through the Hahn–Banach theorem.The paper notes that later results establish equivalence between Lp-universality and c0-universality.
Appendix A. Supplementary Results
The appendix establishes the RKHS representation of finite signed Borel measures by treating integration against a measure as a bounded linear functional.
- Supplementary result: For a measurable bounded kernel, integration of any RKHS function against a finite signed Borel measure defines a linear functional.The functional is written as Tµ[f] = ∫X f(x) dµ(x).
- Supplementary result: The Riesz representation theorem yields a unique RKHS element representing this functional through the inner product.For every f in H, Tµ[f] = ⟨f, λµ⟩H.
- Supplementary result: The representing element is the kernel mean embedding λµ = ∫X k(·, x) dµ(x).This follows by evaluating the representation at kernel sections k(·, u).