A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings

1 Introduction

• Kernel embeddings of distributions have been applied to independence and two-sample testing, learning on distributions, goodness-of-fit testing, and probabilistic programming.Source: §1 · p. 1
• Conditional distributions can represent more complicated relations between random variables than unconditional distributions.The paper notes prior CME applications in dynamical systems, graphical models, probabilistic inference, reinforcement learning, state-space models, and causal inference.Source: §1 · p. 1
• The measure-theoretic CME approach uses drastically weaker assumptions than prevalent operator-based definitions and provides an explicit expression.The paper also derives vector-valued regression estimates and analyzes their properties, including universal consistency.Source: §1 · p. 1
• The approach yields conditional analogues of MMD and HSIC, named MCMD and HSCIC, whose properties are demonstrated through simulations.Source: §1 · p. 2

2 Preliminaries

• The paper takes random variables X, Y, and Z on a probability space, with Z serving as the conditioning variable.Source: §2 · p. 2

2.1 Positive definite kernels and RKHS embeddings

• Under an integrability condition, a distribution embeds as a Bochner integral in an RKHS, and joint distributions embed in a tensor-product RKHS.The tensor-product kernel is the product of the component kernels.Source: §2.1 · p. 2
• The centered joint embedding represents covariance, while characteristic kernels make the embedding-based MMD a metric and allow HSIC to characterize independence.Source: §2.1 · p. 2

2.2 Conditioning

• Conditional expectation of a Banach-space-valued variable is defined through its integrals over events in the conditioning σ-algebra; a regular version permits integration against the conditional measure.Source: §2.2 · p. 3

2.3 Vector-valued RKHS regression

• Vector-valued RKHS regression maps inputs to a Hilbert-space output, and its regularized empirical minimizer has a representer form with coefficients solving linear equations.Source: §2.3 · p. 3

3 Conditional mean embedding

• The CME of X given Z is the conditional expectation of the kernel feature map kX(X, ·), and is a Z-measurable random variable valued in HX.Its inner product with any f in HX gives the conditional expectation of f(X) given Z almost surely.Source: §3 · p. 4
• The conditional joint embedding yields conditional cross-covariance by subtracting the product of conditional mean embeddings before applying the tensor-to-operator map.Its inner products express conditional covariance almost surely.Source: §3 · p. 4

3.1 Comparison with existing definitions

• The prevalent operator-based CME requires a kernel on Z, conditional expectations to lie in its RKHS, and an inverse of C_ZZ; these conditions can be restrictive.The paper instead defines the CME using conditional expectation given σ(Z), without a kernel on Z.Source: §3.1 · p. 4
• Unlike the regression interpretation in, which establishes existence without an explicit expression, this definition gives the CME as a conditional expectation.Source: §3 · p. 4, §3.1 · p. 5
• The prior operator Σ_YX|Z integrates conditional covariance over Z, unlike the paper’s conditional cross-covariance relation.Source: §3 · p. 4, §3.1 · p. 5

4 Empirical estimates

• The conditional mean embedding is a measurable function of Z, so estimating it becomes vector-valued regression from Z into HX.For continuous Z, conditioning the Riesz representation theorem on the measure-zero event Z = z is not measure-theoretically rigorous.Source: §4 · p. 5
• The regularized empirical-loss minimizer has a kernel expansion whose coefficients solve (KZ + nλI)f = kX.The estimator is obtained by substituting f = (KZ + nλI)^-1kX into its kernel expansion.Source: §4 · p. 5

4.1 Surrogate loss, universality and consistency

• The true conditional mean embedding almost surely uniquely minimizes both the original and surrogate losses over L2(Z, PZ; HX).The consistency theorem is stated for the surrogate loss; the original loss is asymptotically bounded by the surrogate loss plus the expected conditional variance η.Source: §4.1 · pp. 6–7
• Under bounded kernels and a C0-universal operator-valued kernel, the estimator is universally consistent for the surrogate loss without distributional assumptions.The regularisation parameter must decay more slowly than O(n^-1/2).Source: §4.1 · p. 6
• When λn = O(n^-1/4), the conditional expected surrogate-loss difference between the estimated and true conditional distributions is Op(n^-1/4).The boundedness assumption holds for commonly used Gaussian and Laplacian kernels. Some smoothness or other distributional assumptions are necessary for universal convergence rates; without them, rates can be arbitrarily slow.Source: §4.1 · pp. 6–7
• The surrogate loss’s excess over the minimum represents expected conditional variance, which is zero when X is measurable with respect to Z.When X and Z are independent, η is the variance of kX(X, ·) in HX; in general it represents irreducible loss due to noise in X.Source: §4.1 · p. 7

5.1 Maximum conditional mean discrepancy

• MCMD compares conditional mean embeddings at each conditioning value, and its norm gives the discrepancy between the embeddings.The section describes it as the conditional analogue of MMD and notes that its empirical estimation cannot use U- or V-statistics because there are not multiple samples from each conditional distribution.Source: §5.1 · pp. 7–8

5.3 Experiments

• Simulations illustrate MCMD near zero for matching conditional distributions and nonzero where their dependence on Z differs; HSCIC is near zero for noise or independent cases and positive for dependent cases.The MCMD simulations also display conditional witness functions; Figure 3 compares additive and multiplicative noise settings.Source: Figure 2 · p. 8, §5.1 · p. 8, Figure 3 · p. 9

5.1 Maximum conditional mean discrepancy

• MCMD is zero almost everywhere exactly when the conditional distributions agree almost everywhere, given a characteristic kX, mutually absolutely continuous conditioning-variable distributions, and regular conditional versions.The theorem compares PX|Z=z and PX′|Z′=z, including when the conditioning variables differ.Source: §5.1 · p. 8

5.2 Hilbert-Schmidt conditional independence criterion

• HSCIC is zero almost surely exactly when X and Y are conditionally independent given Z, assuming a characteristic product kernel and a regular conditional version.The criterion is a random quantity, unlike the single-value operator-based ICOND criterion; the section states that operator-based definitions may fail to exist or be exact under their assumptions.Source: §5.2 · pp. 8–9

• The paper proposes a CME approach based on Bochner conditional expectation that avoids the stringent assumptions of previous operator-based approaches.Source: §6 · p. 9
• Natural vector-valued regression yields empirical estimates, with universal consistency under no distributional assumptions and an Op(n^-1/4) convergence rate in the well-specified case.Source: §6 · p. 9
• The paper extends MMD, witness functions, and HSIC to the conditional setting.Source: §6 · p. 9

• The authors describe the work as theoretical and do not discuss broader societal impact.Source: Broader Impact · p. 10

• The paper applies the generalized inequality in the RKHS setting, using convexity of the squared norm and Bochner integrability to meet its conditions.Source: A Generalised Jensen’s Inequality · pp. 14–15
• Generalized Jensen’s inequality bounds a convex function of an integrable random variable’s expectation by the expectation of that function.Source: A Generalised Jensen’s Inequality · p. 14
• The conditional form bounds f(E[V | E]) by E[f(V) | E] under the stated integrability and convexity conditions.Source: A Generalised Jensen’s Inequality · p. 14

B Generalisation Error Bounds

• Earlier consistency-rate results cannot guarantee consistency here: their lower bound requires finite-dimensional H_X, while the upper-bound Hilbert-Schmidt condition is not met for the chosen kernel.The cited lower-rate result and results used by therefore do not apply in this context.Source: B Generalisation Error Bounds · p. 15
• The analysis applies uniform algorithmic stability to the squared-loss estimator and uses bounds for stability and generalisation under stated assumptions.The estimator is trained from D, with D_i denoting the sample after removing observation i.Source: B Generalisation Error Bounds · pp. 15–16
• Bounded kernels and measurable point embeddings verify several assumptions; a replacement condition for Assumption 3 yields stability and lets the generalisation bound hold with the new β.The squared-loss Assumption 3 is not generally satisfied, but restricting the output space through bounded kX addresses this issue.Source: B Generalisation Error Bounds · pp. 16–17

C Proofs

• The tensor-product embedding satisfies ⟨f ⊗ g, µPXY⟩ = EXY[f(X)g(Y)], and the stated map between tensor-product RKHS elements and Hilbert-Schmidt operators is an isometric isomorphism.Source: C Proofs · pp. 17–18
• For a regular version Q of P(· | E), integrating any Bochner P-integrable H against Q yields a version of E[H | E].Source: C Proofs · p. 19
• Conditional inner products with µPX|Z and µPXY|Z give versions of the conditional expectations of f(X) and f(X)g(Y), respectively.The identities hold almost surely for f ∈ HX and g ∈ HY.Source: C Proofs · p. 20
• The conditional mean embedding has a measurable deterministic representation, and its associated function minimizes both stated objectives with almost-sure uniqueness.The representation requires separability of HX; uniqueness is with respect to PZ.Source: C Proofs · pp. 20–21
• For bounded kernels with C0-universal lXZ and λn decaying slower than O(n^-1/2), the learning algorithm is universally consistent in surrogate loss ˜EX|Z for any joint distribution PXZ.The guarantee holds for every ϵ > 0 and δ > 0 once n is large enough.Source: C Proofs · p. 22
• In the setting of Theorem 4.4, if FPX|Z ∈ GXZ and λn decays to 0 at rate O(n^-1/4), then ẼX|Z(F̂PX|Z,n,λn) − ẼX|Z(FPX|Z) = OP(n^-1/4).Source: C Proofs · p. 22
• MCMDPX|Z,PX′|Z′ = 0 almost everywhere if and only if the conditional distributions agree for almost every z, assuming kX is characteristic, PZ and PZ′ are mutually absolutely continuous, and both conditionals have regular versions.Source: C Proofs · p. 22
• HSCIC(X, Y | Z) = 0 almost surely if and only if X and Y are conditionally independent given Z, when kX ⊗ kY is characteristic and P(· | Z) has a regular version.Source: C Proofs · p. 24
• If the conditional embedding difference is zero almost surely, characteristic kernels and countable intersection establish X ⊥⊥ Y | Z.The conditional joint distribution then equals the product of the conditional marginals almost surely.Source: C Proofs · p. 24
• If X ⊥⊥ Y | Z, then HSCIC(X, Y | Z) = 0 almost surely.The proof assumes the conditional joint distribution equals the product of the conditional marginals almost surely.Source: C Proofs · p. 24