Source-linked AI summary
A Review of Kernel Density Estimation with Applications to Econometrics
Adriano Zanin Zambom, Ronaldo Dias
TL;DR
Econometricians need density estimates to model the probabilistic structure of data. This review synthesizes kernel density estimation theory and bandwidth-selection methods, then introduces SiZer for examining features across smoothing levels. It also discusses estimator behavior and examples spanning bandwidth choices and boundary estimation.
Problem
Modeling the uncertainty and probabilistic structure of economic data requires density estimation for identifying underlying structures and inferring unknown models.
Method
The paper reviews kernel density estimation theory, classical and modern bandwidth-selection approaches, and SiZer analysis across different smoothing levels.
Results
The review presents bandwidth-selection and density-estimation examples, including undersmoothing from likelihood cross-validation, boundary-unbiased estimation, and SiZer feature assessments.
Takeaways & Limitations
Kernel density estimation and bandwidth choice provide tools for analyzing economic densities and their visible features across smoothing levels.
Takeaways & Limitations
No bandwidth-selection procedure is considered best in every situation.
Abstract
from arXiv · showhide
Nonparametric density estimation is of great importance when econometricians want to model the probabilistic or stochastic structure of a data set. This comprehensive review summarizes the most important theoretical aspects of kernel density estimation and provides an extensive description of classical and modern data analytic methods to compute the smoothing parameter. Throughout the text, several references can be found to the most up-to-date and cut point research approaches in this area, while econometric data sets are analyzed as examples. Lastly, we present SIZer, a new approach introduced by Chaudhuri and Marron (2000), whose objective is to analyze the visible features representing important underlying structures for different bandwidths.
1 Introduction
Econometric density estimation models the probabilistic structure of data, supporting identification of underlying structures and inference about unknown models. The paper reviews kernel density estimation, bandwidth selection, and SiZer across traditional and modern approaches.
- Motivation: Density estimation helps econometricians model the uncertainty and probabilistic structure underlying observed data.These estimates can support applied economic analysis and describe the mechanism generating the data.
- Motivation: Parametric models restrict densities to finitely many parameters, whereas nonparametric models allow flexible forms from infinite collections of curves.The nonparametric setup typically assumes smoothness properties such as differentiability and square-integrable second derivatives.
- Kernel density estimation: Kernel smoothing is a widely used approach to density estimation, with foundations associated with Rosenblatt (1956) and Parzen (1962).The paper reviews kernel-weight assumptions and estimator properties including bias and variance.
- Bandwidth selection: The smoothing parameter is crucial because it controls the smoothness of nonparametric density estimates.The paper discusses bandwidth selection in detail, covering classical and modern approaches.
- Paper organization: The review progresses from histograms to kernel density estimation, bandwidth selection, and SiZer analysis of features across smoothing levels.SiZer is introduced as a modern approach for assessing visible features that may represent underlying structures at different bandwidths.
2 The Histogram
Histograms estimate densities by grouping observations into bins, with bin heights determined by empirical proportions and bin width. Their appearance depends strongly on bandwidth and bin placement, which can produce jagged, oversmoothed, or misleading features.
- Limitations: Histograms are usually not smooth and may display bumps caused only by noise.Despite these disadvantages, the histogram has played a fundamental role in nonparametric statistics and data presentation.
- Construction: A histogram is a step-function density estimate whose bin heights equal observations per bin divided by bin width.The construction is presented for iid observations from an absolutely continuous distribution.
- Construction: Histogram bins are intervals determined by a starting point x0 and width h, and observations within each bin estimate its probability.The density approximation divides estimated bin probabilities by the bin width.
- Smoothing: The bandwidth h controls histogram smoothness: small bandwidths create jagged estimates, whereas large bandwidths tend to oversmooth them.Bandwidth and the number of bins are linked, so choosing one determines the other.
- Smoothing: For the same data, a small bandwidth produces many bins while a large bandwidth produces fewer bins.The paper illustrates this contrast with two histograms of randomly generated data.
- Limitations: Changing the starting point of bin edges can change the apparent shape and therefore the resulting estimate.This bin-edge problem is identified as a disadvantage of histograms.
3 Kernel Density Estimation
Kernel density estimation estimates a density by weighting observations around each evaluation point, with bandwidth controlling smoothness. The review develops estimator properties, kernel choices, practical examples, and key limitations including boundary bias and tail-sensitive smoothing.
- Estimator construction: Kernel density estimation uses each evaluation point x as the center of a bin of width 2h rather than relying on fixed bin edges.The method is rooted in histogram methodology but adapts the bin location to the point being estimated.
- Estimator construction: The naive kernel can produce ragged estimates with misleading bumps and constant regions, especially when few observations are present.In the CEO compensation example, the naive estimate displays several small bumps attributed to noise rather than the underlying density.
- Kernel choices: The Epanechnikov kernel produces a smoother estimate than the naive kernel, avoiding the small-bump issue illustrated in the CEO compensation data.Figure 2 contrasts the naive solid line with the Epanechnikov dashed line.
- Kernel choices: Suitable kernel weights help overcome discontinuities and bumps; Gaussian weighting yields a smooth estimated density with derivatives of all orders.Common alternatives include Epanechnikov, Uniform, Gaussian, and Triweight kernels.
- Theoretical properties: Kernel density estimators balance bias and variance through bandwidth h, with consistency requiring h → 0 and nh →∞ as n →∞.Under these conditions, MSE(f̂(x)) → 0.
- Bandwidth and limitations: The optimal bandwidth depends on the unknown roughness quantity f′′, while long-tailed densities create a trade-off between tail noise and oversmoothing of central features.Adaptive bandwidth methods make bandwidth size depend on the estimation location.
4 The choice of the smoothing parameter h
Choosing the smoothing parameter h is crucial because the purpose of estimation and the number of required estimates affect which selection strategy is practical. The review surveys subjective, classical, plug-in, reference-density, and adaptive approaches without identifying one procedure as best in every situation.
- Motivation: Bandwidth selection is crucial, and no procedure has yet been considered best in every situation.The appropriate choice can depend on the purpose of the estimation.
- Subjective selection: Subjective selection examines densities across bandwidths, starting with substantial smoothing and decreasing it until a reasonable estimate is reached.This approach becomes impractical when many estimates are needed.
- Automatic selection: Automatic bandwidth selection is essential when a large number of estimates forms part of a broader analysis.The paper presents automatic methods as a response to the impracticality of repeated subjective inspection.
- Automatic selection: Automatic methods are divided broadly into classical and plug-in categories, with classical methods including cross-validation, Mallows’s Cp, and AIC.Classical methods extend procedures used in parametric modeling.
- Automatic selection: Plug-in methods estimate f, sometimes using a pilot bandwidth, and substitute that estimate into MISE to compute the optimal bandwidth.The review also discusses bandwidth estimates based on a reference density and compares method advantages and disadvantages through prior work.
4.1 Reference to a Distribution
Reference-density bandwidth selection substitutes a chosen density family for the unknown curvature term in the optimal-bandwidth formula. A robust modification uses the interquartile range to reduce sensitivity to outliers, which can otherwise make MISE-based bandwidths smooth away important modes.
- Reference-density bandwidth selection: Reference-density methods estimate f′′ by assuming a specified density family, then substitute that estimate into the optimal-bandwidth formula.A Gaussian reference density with a Gaussian kernel is one example.
- Reference-density bandwidth selection: Using an estimate of σ produces a data-based optimal-bandwidth estimate.
- Robust bandwidth selection: The robust version replaces the spread estimate σ with the interquartile range R to improve resistance to outliers.
- Robust bandwidth selection: For CO2 per capita in 2008, the robust bandwidth captures the modal peak, whereas the MISE-minimizing bandwidth smooths it out.Tail outliers make hMISE larger than hrobust, producing greater smoothing.
- Scope: Reference-density methods have limited practical use when the prespecified density family is selected incorrectly, unlike plug-in and classical methods.
4.2 Plug-in Methods
Plug-in methods estimate the unknown curvature quantities entering asymptotically optimal bandwidth formulas, with later procedures combining plug-in and cross-validation ideas. The literature develops sequential, higher-order, and hybrid estimators for this purpose.
- Core plug-in strategy: Plug-in bandwidth selection estimates the unknown R(f′′) term required by the asymptotically optimal bandwidth formula.The resulting estimator hMISE is obtained after estimating the only unknown part of the formula.
- Sequential procedures: Scott, Tapia, and Thompson propose iterating curvature estimation and bandwidth substitution until the bandwidth converges.
- Higher-order procedures: Higher-order procedures estimate integrated derivatives such as R(f^(p)) and use additional Taylor-expansion terms.
- Hybrid procedures: Modern plug-in research combines plug-in ideas with classical approaches such as cross-validation.Biased cross-validation is cited as an example of this hybrid direction.
4.3 Classical Methods
Classical bandwidth selection includes least-squares, likelihood, and biased cross-validation, each replacing an inaccessible loss or likelihood with a data-based criterion. These methods are intuitive and widely used but can be highly variable and undersmooth.
- Least Squares Cross-Validation: Least-squares cross-validation is described as intuitive and fundamental in recent bandwidth-selection research.
- Least Squares Cross-Validation: Least-squares cross-validation estimates the integrated squared error from the data and minimizes the resulting criterion over h.Its construction follows the ISE expansion after removing the term independent of h.
- Biased Cross-Validation: Biased cross-validation replaces the unknown R(f′′) in asymptotic MISE with a cross-validated kernel estimate, making it a hybrid selector.Its bandwidth is hBCV = arg minh BCV(h).
- Likelihood Cross-Validation: Likelihood cross-validation omits each observation in turn, averages log leave-one-out density estimates, and selects the bandwidth minimizing CV(h).The selected bandwidth is stated to minimize the Kullback-Leibler distance between the estimate and the density.
- Limitations: Cross-validation bandwidths are highly variable and usually undersmooth density estimates.Different samples from the same distribution may produce bandwidths with large variance.
- Indirect Cross-Validation: Indirect cross-validation bandwidths can have relative error converging at n^1/4, compared with n^1/10 for least-squares cross-validation.The review also reports slightly lower mean integrated squared error for indirect cross-validation.
4.4 Other Methods
Other approaches vary bandwidths across locations or observations, use binning and bootstrap procedures, address boundary bias with asymmetric kernels, and estimate distribution functions rather than densities. These methods expand kernel estimation beyond a single global smoothing parameter and standard support assumptions.
- Variable Bandwidth: Balloon estimators vary bandwidth according to the evaluation point x, while sample-point estimators vary it with each observed data point.
- Variable Bandwidth: Sample-point estimators always integrate to 1, ensuring that the result is a density.Their bandwidth h(Xi) is itself random because it depends on random variables.
- Binned Estimation: Binned kernel density estimation uses equally spaced bins, their centers, and bin counts to construct an estimator from finitely many non-empty bins.
- Bootstrap Methods: Bootstrap bandwidth procedures replace MSE with a bootstrapped MSE* that can be minimized directly, using subsamples or pilot densities for resampling.
- Estimating Densities on R+: Gamma kernels replace symmetric kernels for densities supported on R+, and their boundary bias is O(b) near the origin and in the interior.The construction assumes b → 0 and nb → ∞.
- Distribution Function Estimation: Kernel distribution function estimation targets F(x) instead of f(x), with quality measures including integrated criteria based on weighted squared error.
4.5 Example of Bandwidth Selection Methods
The example contrasts bandwidth selectors, showing that different choices produce markedly different density estimates. Plug-in methods can oversmooth, while classical cross-validation can undersmooth; the hybrid method selects the largest bandwidth and smooths away the mode.
- Plug-in selectors can produce oversmoothed results, whereas classical methods such as cross-validation tend to select smaller bandwidths and undersmooth.
- Bandwidth selection aims to determine from the data which features are important and which should be smoothed away.
- The example uses exports of goods and services for countries in 2011 to compare classical and plug-in bandwidth selectors.
- The rule-of-thumb and Sheather–Jones plug-in estimators produce very smooth fits, while unbiased cross-validation selects a small bandwidth and yields a highly variable estimate.
- Hybrid biased cross-validation selects the largest bandwidth, producing a very smooth estimate that smooths away the peak.
5 SiZer
SiZer analyzes density features across a family of bandwidths rather than targeting one true curve. Its scale-space display uses derivative significance to distinguish increasing, decreasing, indistinguishable, and data-insufficient regions.
- Bandwidth selection is difficult because estimates range from wiggly curves at small bandwidths to nearly simple least-squares fits at large bandwidths.
- Scale-space analysis examines significant aspects of an underlying function because different smoothing levels may reveal different intrinsic features.
- Chaudhuri and Marron introduced SiZer to analyze visible features representing important underlying structures across different bandwidths.
- The scale-space surface represents the family of smooth curves indexed by bandwidth h and location x, capturing structures under different smoothing levels.
- SiZer displays derivative-significance information as a color map, with blue for significant increases, red for significant decreases, purple for indistinguishable directions, and gray for insufficient data.
- In the GDP-per-person example, large bandwidths suggest one mode near 16000, while mid-level bandwidths suggest two modes near 10000 and 43000.
Referˆencias
The references document research on kernel density estimation, bandwidth selection, smoothing, cross-validation, scale-space analysis, and related applications.
- The bibliography includes work on bandwidth selection and smoothing methods for kernel density estimation.
- The references cover variable-bandwidth, local, gamma-kernel, and bias-correction approaches to density estimation.
- Applications include disease-cluster identification, typhoon-trajectory tracking, automotive shape modeling, and cosmological parameter estimation.
- Chaudhuri and Marron are cited for SiZer and scale-space views of curve estimation.
- Several cited studies address cross-validation, bootstrap selection, and data-based smoothing parameters.