Source-linked AI summary
The Kernel Mixture Network: A Nonparametric Method for Conditional Density Estimation of Continuous Random Variables
Luca Ambrogioni, Umut Güçlü, Marcel A. J. van Gerven, Eric Maris
TL;DR
Conditional density estimation needs flexible models for complex continuous distributions, while quantized softmax discretizes the target. The paper introduces KMN, which uses neural-network weights over kernels centered on training points. Across Bayesian filtering and generative modeling, KMN compares favorably with quantized softmax, and the filtering experiments also report better likelihood than the EKF.
Problem
Quantized softmax discretizes continuous targets, while mixture density networks constrain the form of conditional densities; flexible continuous conditional-density estimation is therefore a central challenge.
Source: §1 · pp. 1–2Method
KMN represents conditional densities as neural-network-weighted combinations of kernels centered on training points and trains the weights by minimizing negative log likelihood.
Source: §1 · p. 2Results
KMN outperforms quantized softmax in generative-model likelihood and image realism and variability, and its filtering model outperforms quantized CNN and EKF in almost all validation trials.
Source: §1 · p. 2, §4.1 · p. 6Takeaways & Limitations
KMN supports Bayesian filtering of complex densities on manifolds and, with the same generative architecture, produces less-overfit, more realistic and diversified face images than quantized softmax.
Source: §5 · p. 8The KMN assumes nonnegative output-node activations to ensure its weights are nonnegative.
Source: §3 · p. 4
Abstract
from arXiv · showhide
This paper introduces the kernel mixture network, a new method for nonparametric estimation of conditional probability densities using neural networks. We model arbitrarily complex conditional densities as linear combinations of a family of kernel functions centered at a subset of training points. The weights are determined by the outer layer of a deep neural network, trained by minimizing the negative log likelihood. This generalizes the popular quantized softmax approach, which can be seen as a kernel mixture network with square and non-overlapping kernels. We test the performance of our method on two important applications, namely Bayesian filtering and generative modeling. In the Bayesian filtering example, we show that the method can be used to filter complex nonlinear and non-Gaussian signals defined on manifolds. The resulting kernel mixture network filter outperforms both the quantized softmax filter and the extended Kalman filter in terms of model likelihood. Finally, our experiments on generative models show that, given the same architecture, the kernel mixture network leads to higher test set likelihood, less overfitting and more diversified and realistic generated samples than the quantized softmax approach.
1 Introduction
The paper introduces the kernel mixture network (KMN), which uses a neural network to weight kernels centered on training points for nonparametric conditional density estimation. It applies KMN to Bayesian filtering and generative modeling, and distinguishes it from related kernel and deep-learning methods.
1 Introduction
- KMN models conditional densities as neural-network-weighted combinations of kernels centered on a subset of training points, combining quantized softmax flexibility with continuous density modeling.The kernels can be defined over real numbers, circles, other manifolds, or discrete topological objects such as graphs.Source: §1 · p. 2
- With the same LSTM-PCA architecture, KMN significantly outperforms quantized softmax in model likelihood and generated-image realism and variability.The paper also reports less overfitting and more realistic, diversified images for KMN.Source: §1 · p. 2
- KMN filtering estimates complex posterior densities from prior-distributed synthetic samples and is presented as a CNN-based approach for nonlinear dynamical systems.The paper relates this approach to ε-free approximate Bayesian inference and contrasts it with methods that use variational Bayes.Source: §1 · p. 2, §1.1 · p. 2
1.1 Related work
- LSTM-PCA models each principal component conditioned on higher-variance components and is proposed as a simpler alternative to computationally more expensive pixelRNN.The authors use it to generate human-face images; PCA also supports faster training and sampling by focusing on high-variance components.Source: §1 · p. 2, §1.1 · p. 3
- Unlike hybrid kernel-DNN methods, KMN fixes its kernels and uses a DNN to determine their mixing weights; its kernels need not be positive semidefinite.It is trained with standard gradient descent rather than stochastic variational learning.Source: §1.1 · p. 2
- KMN can be combined with geometric deep learning because its output density can be defined on an arbitrary output manifold by choosing appropriate kernels.The paper also suggests possible use with graph-distribution models and other generative methods, but describes these extensions as prospective.Source: §1.1 · p. 2
2 Background
Conditional density estimation maps predictive variables to a probability density over a target variable. The section reviews mixture and quantized-softmax approaches, then introduces kernel density estimation and its extension to a family of kernels with fixed bandwidths.
2 Background
- A conditional density estimator maps predictive variables y to a probability density over possible values of x.In machine learning, this mapping is typically learned from training pairs (x, y).Source: §2 · p. 3
2.1 Mixture density and quantized softmax networks
- Mixture density networks combine parameterized densities whose mixing coefficients and parameters come from a DNN, while quantized softmax assigns probabilities to intervals treated independently.The mixture approach exploits continuous values but constrains density form; quantized softmax yields a piecewise-constant density.Source: §2.1 · p. 3
2.2 Kernel density estimation
- Kernel mixture density estimation avoids bandwidth optimization by fixing bandwidths and placing a family of kernels at each sample point.The kernel family can be defined beyond R × R; the von Mises kernel is given as an example for densities on a circle.Source: §2.2 · p. 4
- Traditional conditional kernel-density methods estimate joint and marginal densities separately, a strategy often infeasible in machine learning because it requires density estimation in high-dimensional spaces.Kernel density estimation centers symmetric, normalized kernels on samples, with weights determining sample importance.Source: §2.2 · p. 4
3 The kernel mixture network
The kernel mixture network generalizes quantized softmax by using a broader family of kernels, with a deep neural network determining the weights. Its kernel choices support conditional densities on arbitrary output spaces, while center-point selection addresses the cost of using many kernels.
3 The kernel mixture network
- Quantized softmax is a special case of the kernel mixture model, using rectangular kernels and weights from the softmax output.The KMN instead uses a broader family of kernels to exploit the topology of continuous variables, with a DNN determining the weights.Source: §3 · p. 4
- Non-negative output activations ensure that the network’s mixture weights are non-negative.Source: §3 · p. 4
- Center points come from training-set values, so the variable range need not be known in advance; nearby centers are recursively removed using a distance threshold δ.Using as many kernels as training points can be impractical on large datasets.Source: §3 · p. 4
- The model’s loss is obtained by substituting its density expression into the negative log likelihood and can be optimized by standard back-propagation.Source: §3 · p. 5
- Any valid probability-density kernels can be used, enabling conditional density estimation on arbitrary manifolds or discrete objects such as graphs.Kernels need not be differentiable because gradients are computed with respect to the weights.Source: §3 · p. 5
4 Experiments
The experiments evaluate KMN in Bayesian filtering and face generation against quantized alternatives and, for filtering, the extended Kalman filter. KMN outperforms the filtering alternatives on almost all validation trials and shows better test likelihood and sample quality than softmax in both face-generation experiments.
4.1 Applications to Bayesian filtering
- The method uses KMN to estimate the filtering density p(xt|y1, ..., yt−1) from latent series sampled from the prior, followed by synthetic observations sampled from the likelihood.In the reported experiment, 499800 simulated training time series and 200 validation time series were used to train and evaluate a deep CNN.Source: §4.1 · p. 5
- For the nonlinear oscillator, posterior densities tightened as observations accumulated, and KMN recovered the skewness of the posterior.Figure 1 shows the filter density alongside past observations and the underlying ground-truth signal.Source: §4.1 · p. 5, Figure 1 · p. 6
- Figure 2 compares filtering validation loss and likelihood: KMN converged faster to a better local minimum than quantized CNN and outperformed quantized CNN and EKF in almost all validation trials.The comparison used the same CNN architecture for KMN and quantized CNN.Source: §4.1 · pp. 5–6, Figure 2 · p. 6
- For an an-harmonic wave, the von Mises-kernel filter quickly tracked phase, with conditional densities defined on a circle where −π and π are equivalent.Figure 3 shows posterior densities for the phase.Source: §4.1 · p. 7, Figure 3 · p. 7
4.2 Applications to generative modeling
- In grayscale face generation, Figure 4 compares generated faces and test losses; across grayscale and color experiments, KMN had higher test likelihood, less overfitting, and more realistic, diverse outputs than softmax.KMN faces were sharper, less blurry, and had fewer artifacts; softmax models began overfitting after about 50 epochs, while KMN models continued learning through 100 epochs.Source: Figure 4 · p. 8, §4.2 · p. 8
5 Discussion
The discussion reports that KMN generates more realistic and diversified face images with less overfitting than quantized softmax under the same architecture. It also highlights applications to graph distributions and scalar-function reconstruction on manifolds.
5 Discussion
- With the same architecture, KMN is less likely to overfit and generates more realistic, diversified face images than quantized softmax.Figure 5 shows generated color faces and the models' test-loss progress during training.Source: §5 · p. 8, Figure 5 · p. 9
- KMN combines quantized softmax flexibility with the regularizing properties of kernel density estimation for neural-network conditional density estimation.Source: §5 · p. 8
- The authors suggest KMN may improve sample quality and diversity with PixelCNN, PixelRNN, and WaveNet.They also describe using diffusion kernels to model distributions over discrete graphs, with kernels encoding average distances between nodes.Source: §5 · pp. 8–9
- KMN can reconstruct arbitrary scalar functions on manifolds by replacing the density-estimation loss with a loss such as mean squared error.Source: §5 · p. 9