PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortized Bandwidth Learning for Kernel Density Estimation under Logarithmic Score

Junyi Liang, Hailiang Du

arXiv:2608.20445v1cs.LG

TL;DR

KDE bandwidth selection traditionally relies on prescribed rules or sample-specific optimization, motivating a reusable alternative for converting finite samples into density estimates. The paper learns a sample-to-bandwidth mapping across density-estimation tasks under the logarithmic score, using bounded-support truncation and affine transfer. Across Gaussian, heterogeneous multi-family, and Gaussian-mixture benchmarks, the amortized selector consistently outperforms the classical selectors considered, especially for small and heterogeneous samples.

  • Problem

    KDE turns finite samples into densities, but bandwidth selection remains dependent on prescribed rules or repeated sample-specific optimization.

  • Method

    The framework learns a reusable sample-to-bandwidth mapping across tasks under the logarithmic score, with truncated-and-renormalized bounded support and affine standardization for interval transfer.

  • Results

    The amortized selector consistently outperforms the classical selectors across Gaussian, multi-family, and Gaussian-mixture experiments, with especially large gains for small and heterogeneous samples.

  • Takeaways & Limitations

    Finite Gaussian mixtures can generate effective training tasks, and selectors trained across task distributions remain effective across varied density structures.

  • Takeaways & Limitations

    The study is restricted to one-dimensional Gaussian-kernel KDE with a single global bandwidth and a small set of interpretable sample features.

Abstract

from arXiv · show

Kernel density estimation converts finite samples into probability densities, but its performance depends critically on bandwidth selection. Classical selectors prescribe the sample-to-bandwidth rule analytically or asymptotically, or solve a new optimization for each sample. An amortized framework is proposed that instead learns this mapping across a distribution of density-estimation tasks by optimizing the logarithmic score. A truncated-and-renormalized bounded-support formulation enables stable learning across heterogeneous tasks, while affine standardization allows a selector trained on a single reference interval to transfer across bounded intervals. Experiments under Gaussian sampling, a multi-family benchmark, and randomized Gaussian-mixture training show that the amortized selector consistently and substantially outperforms Silverman's rule, the Sheather--Jones selector, and least-squares cross-validation, with especially large gains in small and heterogeneous samples. Finite Gaussian mixtures provide a generic training mechanism supported by their $L^1$ approximation property. Selectors trained in this way generalize strongly across different density structures, allowing the same trained selector to be applied directly to finite samples from unknown densities without specifying or fitting a distributional family. This combination of broad applicability and strong empirical performance makes the framework attractive for a wide range of applications in which finite samples or ensembles must be converted into continuous probability densities.

1 Introduction

The paper reframes KDE bandwidth selection as learning a reusable sample-to-bandwidth mapping across density-estimation tasks, trained under the logarithmic score. A bounded-support formulation stabilizes learning, and experiments evaluate performance across Gaussian, heterogeneous multi-family, and Gaussian-mixture tasks.

  • Motivation: KDE converts finite samples into density estimates, but bandwidth selection decisively controls smoothing and estimation quality.The setting considered is one-dimensional Gaussian-kernel KDE, with possible extension to other kernel families.
  • Motivation: Classical selectors use analytical, asymptotic, plug-in, or cross-validation rules, while many bandwidth criteria rely on integrated squared error.The paper instead motivates probabilistic assessment through the logarithmic score, which is local for continuous densities.
  • Contribution: The amortized framework learns a shared sample-to-bandwidth mapping from a distribution of density-estimation tasks rather than prescribing it analytically or reoptimizing for every sample.The learned mapping is intended to be reusable like a classical selector while adapting to task structure.
  • Contribution: Truncation and renormalization place the target density and KDE on a common bounded interval, concentrating logarithmic-score learning on the practical region of interest.This addresses excessive smoothing driven by rare realizations in remote low-density regions.
  • Experiments: The evaluation uses Gaussian, bounded multi-family, and Gaussian-mixture benchmarks to test finite-sample behavior, structural heterogeneity, and flexible task generation.The experiments assess whether one learned selector improves on classical selectors across these settings.

2 Bandwidth-Learning Framework

The framework learns a positive bandwidth rule from permutation-invariant sample features across tasks, using a logarithmic-score objective with bounded-support truncation and renormalization. Affine standardization transfers the learned rule from a reference interval to other bounded intervals while preserving the task-specific minimizing bandwidth under scale transformation.

  • Amortized bandwidth prediction: The selector retains KDE as the final estimator while learning only a reusable bandwidth-selection rule across density-estimation tasks.A predictor maps permutation-invariant sample features to bandwidths without task-specific reoptimization.
  • Amortized bandwidth prediction: The learned mapping uses sample size, mean, standard deviation, skewness, and kurtosis as low-dimensional, interpretable features.These features represent size, location, scale, asymmetry, and tail behavior; the predictor is implemented as an MLP in the proof-of-concept.
  • Logarithmic score criterion: The logarithmic score aligns bandwidth training with probabilistic density assessment, with task averages approximating expected performance over a task distribution.Its sample average is the negative log-likelihood per observation, while its expectation is cross-entropy.
  • Bounded-support formulation: Truncation and renormalization rewrite both the target density and KDE as probability densities on a common bounded interval.The interval is a working domain chosen from physical, operational, historical, or reference information, with margins intended to avoid losing substantial mass.
  • Cross-interval transfer: Affine standardization maps samples from [A, B] to [−1, 1], applies the reference-interval predictor, and rescales the resulting bandwidth to the original interval.The rescaling is exactly compatible with the bounded logarithmic-score criterion, with h_[A,B] = a h_[−1,1].

3 Experimental Demonstration

The experiments compare amortized bandwidth selection with classical selectors under Gaussian, heterogeneous multi-family, and randomized GMM task distributions. Across these settings, the amortized approach achieves lower logarithmic scores, with especially strong advantages for heterogeneous and small-to-moderate samples.

  • Experimental design: The experiments evaluate Gaussian, bounded multi-family, and randomized finite-GMM task distributions using logarithmic score.The Gaussian benchmark isolates sample-size effects; the latter two test robustness across density shapes and training generators.
  • Gaussian sampling: Under Gaussian sampling, the amortized selector has the lowest empirical logarithmic score across the displayed sample-size range.It remains closer to the standard-normal reference than Silverman’s rule, Sheather–Jones, and least-squares cross-validation.
  • Gaussian sampling: The amortized selector retains a clear advantage through at least n = 50, although all four selectors converge toward the true standard-normal reference as n increases.Classical selectors perform poorly at very small sample sizes because occasional undersmoothing can assign very low density to plausible outcomes.
  • Bounded multi-family sampling: Across ten heterogeneous density families, the amortized selector again achieves the lowest empirical logarithmic score and shows a more pronounced advantage than under Gaussian sampling.The benchmark includes skewed, heavy-tailed, boundary-concentrated, multimodal, and multi-scale structures.
  • Bounded GMM sampling: On bounded GMM tasks, GMM-trained selectors substantially outperform classical selectors, while the multi-family-trained selector also performs well and cross-distribution generalization is observed.On the multi-family test distribution, the directly matched selector is best among learned selectors; GMM-trained selectors remain close and outperform classical methods, especially at small and moderate sample sizes.
  • Bounded GMM sampling: Across GMM and multi-family tests, GMM-trained selectors have very similar scores across K = 8, 16, and 32, supporting randomized finite GMMs as flexible training distributions.For K = 32, only 6.0% of generated densities have six or more modes, so increasing component count does not proportionally increase modal complexity.

4 Summary and Discussion

The paper learns a reusable bandwidth-selection rule across density-estimation tasks, retaining classical KDE while optimizing the logarithmic score. Bounded-support training and Gaussian-mixture task generation support strong performance across heterogeneous density structures, within a one-dimensional scope.

  • Summary and Discussion: The amortized framework learns the sample-to-bandwidth mapping across tasks while retaining Gaussian-kernel KDE as the density estimator.Training uses permutation-invariant sample features and reuses the learned mapping without task-specific reoptimization.
  • Summary and Discussion: The bounded-support formulation places target and estimated densities on a common interval, preventing remote low-density realizations from driving excessive smoothing.Affine standardization and bandwidth rescaling enable transfer from a reference interval to other bounded intervals.
  • Summary and Discussion: The amortized selector attains the lowest empirical logarithmic score across the examined Gaussian sample sizes, with largest gains at small samples, and performs even better relative to classical selectors across density families.These comparisons include Silverman’s rule, the Sheather–Jones selector, and least-squares cross-validation.
  • Summary and Discussion: Finite Gaussian mixtures provide a flexible training distribution, and GMM-trained selectors substantially outperform classical selectors while remaining close to matched multi-family-trained selectors.The result is supported by the L1 approximation property and holds on independently generated GMM tasks.
  • Summary and Discussion: Once trained, the same selector can be applied directly to samples from unknown densities without specifying, identifying, or fitting a distributional family.Cross-distribution experiments associate this broad applicability with strong performance across diverse density structures.
  • Summary and Discussion: The study is restricted to one-dimensional Gaussian-kernel KDE with a single global bandwidth and a small collection of interpretable sample features.The paper identifies richer features, alternative kernels, locally adaptive bandwidths, and multivariate estimation as extensions.

A Classical Bandwidth Selectors

The experiments compare three classical bandwidth selectors: Silverman’s closed-form normal-reference rule, Sheather–Jones plug-in selection, and least-squares cross-validation. They differ in their criteria and computational procedures, from direct formulas to numerical optimization.

  • Silverman’s Rule: Silverman’s rule combines the sample standard deviation with a robust interquartile-range scale estimate in a closed-form normal-reference prescription.Using the minimum of the two scale estimates limits extreme-observation influence while retaining n−1/5 dependence.
  • Sheather–Jones Selector: The Sheather–Jones selector is derived from asymptotic mean integrated squared error and estimates density-derivative functionals through a multistage plug-in procedure.Its solve-the-equation implementation uses pilot estimation and numerical root finding.
  • Implementation Caveats: Numerical implementations can deviate from exact selector identities because of floating-point arithmetic, discretization, root-finding tolerance, binning, or tied minima.The Sheather–Jones value is specifically affected by pilot estimation and numerical root finding.
  • Least-Squares Cross-Validation: Least-squares cross-validation selects bandwidth by minimizing a sample-based estimate of integrated squared error using leave-one-out KDE terms.For Gaussian kernels, the criterion can be evaluated from pairwise sample differences.
  • Least-Squares Cross-Validation: The LSCV implementation evaluates 60 logarithmically spaced candidate bandwidths relative to Silverman’s bandwidth.Constant multiplicative spacing provides comparable relative resolution across bandwidth scales.

A.4 Use in the Bounded-Support Experiments

In bounded-support experiments, classical selectors are applied on the sample’s original scale and then used within the same truncated-and-renormalized density representation. This preserves a common evaluation framework while leaving selector algorithms unchanged.

  • Bounded-Support Evaluation: For samples observed on [A, B], each classical selector is applied directly on the original sample scale before bounded-support evaluation.The resulting bandwidth is then inserted into the bounded estimator.
  • Bounded-Support Evaluation: All methods use the same truncated-and-renormalized density representation and logarithmic score criterion.The bounded transformation changes the evaluated density but not the specified bandwidth-selection algorithms.
  • Transformation Properties: Bandwidth selectors are translation invariant and scale equivariant when bandwidths transform proportionally with the data scale.Dimensionless ratios such as h_b(S)/sd(S) remain scale invariant.
  • Transformation Properties: The derivations concern exact selector constructions, whereas numerical implementations may differ because of floating-point arithmetic, root finding, binning, or tied minima.This caveat applies to the transformation identities used for comparisons.

B.1 Affine Transformation of the Gaussian KDE and Logarithmic Score

Affine transformations preserve the relevant ordering of bandwidth selectors and alter logarithmic scores only through bandwidth-independent terms. Consequently, Gaussian location–scale comparisons can be reduced to standard-normal sampling when selectors are translation invariant and scale equivariant.

  • Affine KDE Transformation: Under x ↦ δ + cx, the Gaussian KDE transforms by shifting its arguments and scaling bandwidth h to ch.The sample and bandwidth undergo the same affine transformation.
  • Logarithmic Score: The transformed logarithmic score equals the original score plus log2 c, so the minimizing bandwidth rescales without changing the optimizer’s ordering.The additive term is independent of bandwidth.
  • Amortized Selector: The amortized selector is parameterized through a dimensionless bandwidth ratio depending only on sample size in the Gaussian comparison.Sample size is unchanged by affine transformation, and empirical quantiles scale proportionally.
  • Classical Selectors: Silverman’s rule and Sheather–Jones preserve scale equivariance because their scale and pilot derivative-functional constructions transform homogeneously.For Sheather–Jones, whenever h solves the equation for S, ch solves the corresponding equation for Sδ,c.
  • Classical Selectors: Least-squares cross-validation preserves candidate ordering because all criterion values are multiplied by the same positive scale factor.The minimizing grid index is therefore unchanged, including under deterministic tie breaking.
  • Gaussian Comparison: Absolute logarithmic scores for N(δ, c2) differ from those for N(0, 1) by a common additive log2 c term, leaving selector rankings and pairwise differences unchanged.Standard-normal sampling is sufficient for comparisons across the Gaussian location–scale family.

C.1 Gaussian Benchmark

The Gaussian benchmark uses a scale-equivariant amortized bandwidth selector whose prediction depends on sample size, trained online with Gaussian tasks and logarithmic-score evaluation.

  • The selector uses a scale-equivariant parameterization for the Gaussian benchmark.
  • The bandwidth predictor is a fully connected multilayer perceptron with ReLU hidden activations and a softplus output transformation.
  • Training draws sample sizes uniformly from 5 through 256 and generates batches of 256 independent samples from N(0,1)^n.
  • The expected logarithmic score for each task is evaluated using 256-point Gauss–Hermite quadrature.
  • Optimization uses Adam with learning rate 10^-3, gradient clipping at 1, and a maximum of 10,000 steps, with model selection based on a fixed validation set.

C.2 Bounded Multi-Family Benchmark

The bounded multi-family benchmark represents tasks with five features and trains a positive-bandwidth network across ten distribution families using independently sampled bounded-density tasks.

  • The amortized selector uses a five-dimensional feature representation for the bounded multi-family benchmark.
  • A fully connected network with ReLU hidden activations and a softplus output transformation predicts a positive bandwidth.
  • Training samples equally among ten distribution families, draws parameters from family-specific pools, and generates a fresh bounded-distribution sample for each task.
  • The sample size is drawn independently and uniformly from 5 through 256, with 256 tasks in each training batch.
  • Training uses Adam with learning rate 5 × 10^-4, gradient clipping at 1, and an exponential moving average with decay 0.999.
  • Validation contains 20 tasks for every integer sample size from 5 through 256, and training reaches the maximum of 40,000 optimization steps.

C.3 GMM Benchmarks

The GMM benchmarks train separate amortized selectors from freshly generated bounded Gaussian-mixture tasks, while retaining the multi-family benchmark’s architecture and training procedure.

  • The K = 8, K = 16, and K = 32 GMM-trained selectors share the multi-family selector’s features, architecture, bounded logarithmic-score objective, and training procedure.
  • Each training task independently generates a fresh GMM instead of sampling from a fixed parameter pool.
  • For every task, the sample size is drawn uniformly from 5 through 256, and both observed and evaluation samples come from the same bounded GMM.
  • The evaluation sample contains 1024 independent Monte Carlo draws.
  • Separate networks are trained for each K value, and all three runs reach 40,000 steps without triggering early stopping.

C.4 Evaluation and Bootstrap Procedure

Bandwidth comparisons use specified sample-size grids, independent test tasks, shared task-specific evaluation draws, and paired bootstrap intervals to quantify uncertainty across methods.

  • The Gaussian benchmark uses 30,000 independent test samples at each sample size.
  • The bounded multi-family benchmark uses 3000 independent test tasks from each of ten families at each sample size.
  • The bounded GMM benchmark uses 30,000 independently generated K = 32 GMM tasks at each sample size.
  • Each bounded bandwidth-comparison task uses 1024 independent Monte Carlo evaluation draws, shared across all selectors.
  • Uncertainty bands are 90% paired nonparametric bootstrap intervals based on 1000 replicates and shared resampled task indices across methods.
Loading 2608.20445v1…