Source-linked AI summary
Implicit Kernel Learning
Chun-Liang Li, Wei-Cheng Chang, Youssef Mroueh, Yiming Yang, Barnabás Póczos
TL;DR
Kernel selection materially affects performance, but existing approaches rely on predefined kernels or explicit spectral-density and feature-weight models. IKL learns kernel spectral distributions with an implicit deep generative model and evaluates the resulting kernels through sampled random Fourier features. Across MMD GAN image and text generation and supervised Random Kitchen Sinks, IKL outperforms predefined kernels and improves over existing kernel-learning approaches.
Problem
Method
IKL transforms samples from a known base distribution through a neural generator to learn samples from a task-specific kernel spectral distribution.
Source: §2.1 · p. 2, §5 · p. 8Results
IKL outperforms predefined kernels on image and text generation and improves over existing kernel-learning approaches in supervised learning.
Source: §3.3.1 · pp. 4–5Takeaways & Limitations
Learning spectral distributions through deep generative models provides a data-driven route to more effective kernels across generative and supervised tasks.
Source: §5 · p. 8With limited random features, OPT-IKL may perform worse than IKL in high-dimensional settings, and IKL adds sampling and parameterization overhead.
Source: §3.3.1 · p. 5, §4.1 · p. 7
Abstract
from arXiv · showhide
Kernels are powerful and versatile tools in machine learning and statistics. Although the notion of universal kernels and characteristic kernels has been studied, kernel selection still greatly influences the empirical performance. While learning the kernel in a data driven way has been investigated, in this paper we explore learning the spectral distribution of kernel via implicit generative models parametrized by deep neural networks. We called our method Implicit Kernel Learning (IKL). The proposed framework is simple to train and inference is performed via sampling random Fourier features. We investigate two applications of the proposed IKL as examples, including generative adversarial networks with MMD (MMD GAN) and standard supervised learning. Empirically, MMD GAN with IKL outperforms vanilla predefined kernels on both image and text generation benchmarks; using IKL with Random Kitchen Sinks also leads to substantial improvement over existing state-of-the-art kernel learning algorithms on popular supervised learning benchmarks. Theory and conditions for using IKL in both applications are also studied as well as connections to previous state-of-the-art methods.
1 Introduction
Kernel selection remains important despite extensive kernel research, motivating data-driven approaches that learn spectral distributions with implicit generative models. IKL applies this idea to MMD GANs and supervised learning, with competitive or improved performance over existing kernel choices and learning methods.
1 Introduction
- Kernel selection strongly influences practice, motivating alternatives to heuristic choice and extensions beyond predefined-kernel combinations.Prior work includes multiple kernel learning, explicit spectral-density models, optimized random features, and learned feature maps.Source: §1 · p. 1
- IKL models kernel spectral distributions with deep implicit generative models that learn sampling processes rather than explicit densities.The method is presented as a data-driven route to spectral-distribution modeling that previous work had underexplored.Source: §1 · pp. 1–2
- IKL is demonstrated for MMD GANs on image and text generation and for Random Kitchen Sinks in standard supervised learning.The paper also studies training conditions, theoretical guarantees, and connections to existing approaches in both applications.Source: §1 · p. 2
- In both applications, IKL achieves competitive or better performance than heuristic kernel selections and existing methods for modeling kernel spectral densities.The paper frames these results as evidence of the potential for learning more powerful kernels with deep generative models.Source: §1 · p. 2
2 Kernel Learning
For shift-invariant kernels, kernel learning can be formulated as learning a spectral distribution, and IKL performs this through an implicit generator trained for a task-specific objective. Symmetry constraints enforce real-valued kernels, while random Fourier features provide practical kernel evaluation.
2 Kernel Learning
- Shift-invariant kernels can be represented through spectral distributions, making kernel learning equivalent to learning the distribution used in that representation.Bochner’s theorem characterizes continuous, real-valued, symmetric, shift-invariant positive-definite kernels through positive finite spectral measures.Source: §2 · p. 2, §2.1 · p. 2
2.1 Implicit Kernel Learning
- IKL samples ν from a known base distribution and transforms it with h_ψ into samples from the task-learned spectral distribution.The task-specific objective can include kernel alignment and MMD, and optimization uses sampled data points and base-distribution samples to estimate gradients.Source: §2.1 · pp. 2–3
- A symmetric base distribution and sign-based parameterization enforce a symmetric spectral density, producing real-valued kernels.The construction uses h_ψ(ν)=sign(ν)◦h̃_ψ(abs(ν)) when the base distribution is symmetric.Source: §2.1 · pp. 2–3
- Finite random Fourier features approximate IKL kernel evaluations when the learned generator has no closed-form kernel expression.The approximation is computed as an inner product of feature vectors evaluated at transformed samples h_ψ(ν_i).Source: §2.1 · p. 3
3 MMD GAN with IKL
This section introduces IKL for MMD GANs, establishes theoretical conditions for a continuous and differentiable training objective, and evaluates learned kernels on image and text generation. IKL achieves the best reported performance across both tasks while variance control supports the theoretical assumptions.
3.1 Training MMD GAN with IKL
- IKL learns the MMD GAN base kernel through a neural spectral transformation composed with the learned feature embedding, replacing hand-selected kernels.Kernel evaluations are represented through the MMD objective and optimized over the kernel parameters.Source: §3.1 · p. 3
3.2 Property of MMD GAN with IKL
- The maximized IKL MMD objective is continuous everywhere and differentiable almost everywhere in generator parameters under the stated assumptions.The assumptions include local Lipschitzness and differentiability of the generator, Lipschitzness of the feature map, compact parameter sets, and integrability conditions.Source: §3.2 · pp. 3–4
3.3 Empirical Study
- IKL is evaluated on CIFAR-10 image generation and length-32 character-level Google Billion Words text generation using inception/FID and JS-4 metrics.Each algorithm is trained for 10,000 iterations, with fixed-kernel comparisons and learned-kernel baselines.Source: §3.3 · p. 4
3.3.1 Results and Discussion
- Predefined kernels are task-specific: RQ reaches 6.785 inception score on images, whereas Gaussian reaches 0.381 JS-4 on text compared with RQ's 0.463.The text result is slightly worse than WGAN-GP, while the image result favors RQ over Gaussian.Source: §3.3.1 · pp. 4–5
- IKL achieves the best performance in both image and text generation, including a CIFAR-10 inception score of 6.876 versus 6.785 for RQ kernels.The difference from predefined kernels is significant at the 95% confidence level.Source: Table 1 · p. 4, §3.3.1 · p. 5
- The variance constraint bounds E[∥hψ(ν)∥2] to guarantee that kψ is Lipschitz; without it, variance rises, causing exploded MMD values and increasing JS-4 divergence.Source: §3.3.1 · p. 5
4 Random Kitchen Sinks with IKL
IKL learns kernel spectral distributions through an implicit sampler, then uses sampled random features in a two-stage Random Kitchen Sinks procedure. On synthetic and benchmark classification tasks, it improves over representative kernel-learning methods while retaining consistency and generalization guarantees.
4 Random Kitchen Sinks with IKL
- IKL learns a sampler for kernel spectral distributions in stage 1, then solves a convex Random Kitchen Sinks problem using sampled features in stage 2.The features used in the two stages can differ, allowing fewer features during kernel learning and more during classification.Source: §4 · p. 6
4.1 Empirical Study
- IKL adapts the spectral distribution to data, producing slower error degradation with dimension than RFF when the standard-normal distribution is ill-suited.RFF is competitive at d ≤6 but degrades rapidly as d increases, whereas IKL and OPT-KL adapt to the task.Source: §4.1 · p. 7
- IKL improves over OPT-KL and SM on benchmark binary classification tasks and nearly reaches NN performance on some MNIST settings.The comparison uses challenging MNIST and CIFAR10 label pairs with 10000 training and 2000 test examples; Figure 5 reports test error as the second-stage basis count varies.Source: §4.1 · p. 7, Figure 5 · p. 8
- IKL's separate feature sampling avoids OPT-KL's requirement to reuse the same random features for training and testing, reducing computation concerns when the basis count is large.OPT-KL trains on M features, while IKL samples m = 64 features during the first stage and varies M in the second stage.Source: §4 · p. 6, §4.1 · p. 7
4.2 Consistency and Generalization
- Consistency holds as training data and random-feature counts increase, with guarantees governed by the induced function-class complexity and feature count.The generalization bound depends on n, m, and the Rademacher complexity of the IKL kernel; more powerful neural kernels introduce a consistency/generalization trade-off.Source: §4.2 · pp. 7–8
- IKL's guarantees cover the two-stage supervised-learning procedure, while its learned transformation allows training and evaluation to use different random features.This flexibility supports simple SGD training without retaining learned features, and empirical studies find performance competitive with state-of-the-art kernel-learning methods.Source: §4.2 · p. 8
5 Discussion
IKL is presented as a generic kernel-learning framework that can complement existing embedding and spectral-mixture methods. Its learned sampler may also extend kernel-based Bayesian inference through sampling.
5 Discussion
- IKL learns sampling processes for spectral distributions by transforming samples from a base distribution P(ν) into samples for another kernel.The framework is described as generic rather than tied to a single objective or application.Source: §5 · p. 8
- IKL can combine with kernel learning via embeddings and mixtures of spectral distributions, making it applicable to several existing kernel-learning approaches.The paper specifically demonstrates combinations with Wilson et al. (2016) and Wilson and Adams (2013).Source: §5 · p. 8
- The learned sampler may provide an easier route to Bayesian inference via sampling.This is proposed as a possible extension under a framework similar to Saatchi and Wilson (2017).Source: §5 · p. 8
A Proof of Theorem 2
The proof establishes Theorem 2 by showing the MMD objective is locally Lipschitz and differentiable almost everywhere under Lipschitzness, compactness, and differentiability assumptions. It then uses these properties to obtain the stated convergence result.
A Proof of Theorem 2
- The proof bounds the kernel difference after composing the kernel with the learned feature map, using Lipschitz assumptions on the kernel and generator.The resulting bound supports Lipschitz control of the MMD objective in the generator parameter.Source: A Proof of Theorem 2 · p. 11
- The convergence argument uses identity choices for hψ and fϕ, which recover the Gaussian kernel and reduce the result to weak convergence under Gaussian-kernel MMD.The proof assumes the parameter spaces are compact and the relevant objective is differentiable.Source: A Proof of Theorem 2 · p. 11, §A.1 · p. 11
A.1 Continuity
- The MMD objective in equation (10) is shown locally Lipschitz in θ, and hence differentiable almost everywhere under the auxiliary maximum-function lemma.The argument applies the local-Lipschitz property term by term before invoking compactness and differentiability assumptions.Source: §A.1 · p. 11
B Proof of Lemma 3
The lemma establishes a bounded gradient for the implicitly generated kernel when the transformed spectral samples have finite second moment. This gradient bound yields Lipschitzness of the kernel.
B Proof of Lemma 3
- Writing the kernel through its spectral expectation and assuming bounded inputs, finite E[∥hψ(ν)∥2] gives a uniform bound on ∥∇t kψ(t)∥.The proof uses |sin(x)| < |x| and the boundedness of t.Source: B Proof of Lemma 3 · p. 11
- The bounded gradient implies a Lipschitz bound between kernel values through the mean value theorem.For any t and t′, the intermediate point lies on the line segment joining them.Source: B Proof of Lemma 3 · pp. 11–12
C.1 Additional Quantitative Results
Table 2 reports full MMD GAN results across inception score, FID, and JS-4 for different kernels. IKL is best among the compared predefined and learned kernels, although FID differences are less significant.
C.1 Additional Quantitative Results
- IKL is best among Gaussian, RQ, and SM kernels across the reported MMD GAN tasks in Table 2.The comparison reports means and standard errors.Source: §C.1 · p. 12, Table 2 · p. 12
- FID differences are less significant than inception-score and JS-4 differences, and FID is noted as a biased evaluation metric.Source: §C.1 · p. 12
C.2 Computational Issues of GAN trainings with IKL
IKL adds little computational overhead because random-feature sampling is parallelizable and the discriminator network dominates runtime. The variance constraint is sufficient, but weaker conditions may suffice on CIFAR-10.
C.2 Computational Issues of GAN trainings with IKL
- IKL adds about 2,000 parameters through its three-layer MLP, negligible compared with the discriminator networks.The discriminator has roughly 0.8–10 million parameters across the described architectures.Source: §C.2 · p. 12
- IKL's per-iteration runtime is not significantly different from WGAN-GP or MMD GAN with Gaussian kernels.The experiments use 1,024 random features per iteration; discriminator computation dominates the cost.Source: §C.2 · p. 12
C.3 Detailed Discussion of Variance Constraints
- Alternative variance-constraint implementations, including an L2 penalty or Lagrange method, show no significant practical difference.Source: §C.3 · p. 12
- The variance constraint is sufficient for the stated theoretical assumptions, while CIFAR-10 can train without it and reach 6.731 ± 0.034 inception score.The unconstrained result is slightly worse than IKL, and weaker necessary conditions remain future work.Source: §C.3 · p. 13
C.4 IKL with and without Neural Networks on GAN training
IKL-NN generates random features through a learned neural transform, whereas IKL-RFF directly learns a finite feature pool. IKL-NN achieves comparable or better performance with fewer parameters and greater sampling flexibility.
C.4 IKL with and without Neural Networks on GAN training
- IKL-NN can sample arbitrarily many random features, while IKL-RFF is limited by its learned pool size m.This makes IKL-RFF increasingly memory-inefficient when more features are required.Source: §C.4 · p. 13
- With equal parameter counts, IKL-NN scores 6.876 versus 6.801 for IKL-RFF with m = 128 on CIFAR-10.IKL-RFF requires m = 4096 to reach the same or better inception score, using more parameters.Source: §C.4 · p. 13, Figure 6 · p. 13
- IKL-NN uses 2,048 parameters, compared with 2,048 and 65,536 for IKL-RFF at m = 128 and m = 4096, respectively.Source: §C.4 · p. 13
- On Google Billion Word, IKL-NN achieves 0.372 versus 0.383 for IKL-RFF despite IKL-RFF using more parameters.Source: §C.4 · p. 13, Table 3 · p. 13
- IKL-RFF needs more updates to achieve performance competitive with IKL-NN, suggesting a more difficult optimization problem.The underlying theory and whether better optimization can close the gap remain open questions.Source: §C.4 · pp. 13–14
D Proof of Theorem 5
The appendix proves Theorem 5 using data-consistency and random-feature-sampling lemmas, then states a generalization result for Random Kitchen Sinks. It also records experimental hyperparameters for GAN training.
D Proof of Theorem 5
- Theorem 5 is established by combining lemmas controlling consistency with respect to data and random-feature sampling, followed by a union bound.The proof uses McDiarmid's and Hoeffding's inequalities.Source: D Proof of Theorem 5 · pp. 14–15
- Lemma 8 defines the auxiliary kernel t_h(x, x′) = s(x, x′)k_h(x, x′) used in the proof's expectation and Rademacher analysis.Source: D Proof of Theorem 5 · pp. 14–15
E Generalization of Random Kitchen Sinks with IKL
- Theorem 10 states a generalization result using true and empirical misclassification risks for a classifier under Random Kitchen Sinks.Source: E Generalization of Random Kitchen… · p. 16
F Hyperparameters
- GAN experiments use Gaussian or RQ bandwidth grids, Adam optimization, batch size 64, and critic-update counts of 5 for CIFAR-10 and 10 for Google Billion Word.The IKL learning rate is 10^-6 in the listed hyperparameter settings.Source: F Hyperparameters · p. 16, §F.1 · p. 16
F.2 Random Kitchen Sinks with IKL
The Random Kitchen Sinks with IKL experiments use standardized evaluation procedures, including validation-based hyperparameter selection, Logistic Regression classifiers, and early stopping.
F.2 Random Kitchen Sinks with IKL
- Experiments tune hyperparameters on validation data, select Logistic Regression C by 3-fold training cross-validation, evaluate test error, normalize CIFAR-10 and MNIST features, and use Adam learning rate 10^-6 with early stopping.OPT-KL additionally tunes ρ over {1.25, 1.5, 2, 4, 16, 64}.Source: §F.2 · p. 16