PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel density estimation based sampling for imbalanced class distribution

Firuz Kamalov

arXiv:1910.07842v2cs.LGstat.ML

TL;DR

Class imbalance is common in applications with rare events, where minority cases may be important. This paper uses KDE to generate minority samples and reports that it can outperform standard sampling methods across datasets and classifiers.

  • Problem

    Class imbalance occurs in applications with rare events, and resampling approaches face a tradeoff: undersampling may lose information while oversampling may overfit.

    Source: §1 · pp. 1–2

  • Method

    The paper estimates the minority-class density with KDE and generates new minority samples from that estimated distribution.

    Source: §1 · p. 2

  • Results

    Across 3 simulated and 12 real-life datasets, KDE outperformed four standard benchmarks on F1-score and G-mean, with performance robust across three base classifiers.

    Source: §5 · p. 15

  • Takeaways & Limitations

    The authors conclude KDE should be considered a potent tool for problems involving imbalanced class distribution.

    Source: §5 · p. 15

    For some datasets where KDE outperformed other methods, its performance was not overwhelming; the authors suggest the single-hidden-layer, 32-node network architecture may have contributed.

    Source: §4.2 · p. 13

Abstract

from arXiv · show

Imbalanced response variable distribution is a common occurrence in data science. In fields such as fraud detection, medical diagnostics, system intrusion detection and many others where abnormal behavior is rarely observed the data under study often features disproportionate target class distribution. One common way to combat class imbalance is through resampling the minority class to achieve a more balanced distribution. In this paper, we investigate the performance of the sampling method based on kernel density estimation (KDE). We believe that KDE offers a more natural way of generating new instances of minority class that is less prone to overfitting than other standard sampling techniques. It is based on a well established theory of nonparametric statistical estimation. Numerical experiments show that KDE can outperform other sampling techniques on a range of real life datasets as measured by F1-score and G-mean. The results remain consistent across a number of classification algorithms used in the experiments. Furthermore, the proposed method outperforms the benchmark methods irregardless of the class distribution ratio. We conclude, based on the solid theoretical foundation and strong experimental results, that the proposed method would be a valuable tool in problems involving imbalanced class distribution.

1. INTRODUCTION

Class imbalance can bias classifiers toward the majority class even when minority cases are important. The paper proposes KDE sampling to generate minority instances and reports better performance than other resampling techniques.

  • In binary imbalance, majority negatives vastly outnumber minority positives, biasing models toward the negative class.Source: §1 · p. 1
  • Minority positives can matter more despite their scarcity, as cancer detection illustrates.Source: §1 · p. 2
  • Resampling balances classes by undersampling the majority or oversampling the minority, but undersampling may lose information and oversampling may overfit.Source: §1 · p. 2
  • The paper proposes KDE sampling, estimating the minority density from existing samples to generate new instances, and reports better performance than other resampling techniques.Source: §1 · p. 2

2. LITERATURE

Prior work addresses class imbalance through multiple sampling and modeling strategies, each with tradeoffs. This paper distinguishes its contribution as a systematic comparison of KDE sampling with standard methods.

  • Undersampling can lose information, while random minority oversampling can overfit; SMOTE instead generates examples by interpolation between minority samples.Source: §2 · p. 3
  • ADASYN concentrates generated examples near the class boundary; other approaches use kernel-SVM feature-space interpolation or Mahalanobis distance.Source: §2 · p. 3
  • Parametric density estimation assumes a fixed density function and may introduce specification bias or overfit, whereas nonparametric methods estimate density directly from data.Source: §2 · pp. 3–4
  • KDE is a popular nonparametric density-estimation method with applications including breast-cancer analysis, image annotation, wind-power forecasting, and outlier detection.Source: §2 · p. 4
  • Earlier KDE sampling paired oversampling with an RBF classifier; this paper instead systematically compares KDE with other standard sampling methods.The prior study tested six datasets.Source: §2 · p. 4

3. KDE SAMPLING

KDE estimates a density from sample points, then uses the estimate to generate new instances; kernel choice and bandwidth determine how the estimate is formed. The section also describes bandwidth selection and the multivariate extension.

3. KDE SAMPLING

  • KDE estimates density by averaging kernel contributions around sample points, then uses the estimated density to generate new instances.The bandwidth parameter controls estimate smoothness and the bias–variance tradeoff.Source: §3 · pp. 4–5
  • Bandwidth h controls estimate smoothness: larger values yield smoother, lower-variance but higher-bias estimates, while smaller values yield higher-variance but lower-bias estimates.The paper notes that h affects the approximation more than the choice of kernel.Source: §3 · p. 5
  • Because MISE depends on the unknown density, bandwidth selection uses rule-of-thumb methods such as Scott’s or cross-validation, which searches for the h minimizing sample MISE.The section identifies Scott’s rule and cross-validation as frequently used approaches.Source: §3 · p. 5
  • For multivariate samples, KDE uses a bandwidth matrix H; this study applies multivariate Scott’s rule and a multivariate normal kernel.The bandwidth matrix can be selected in different ways, and the data covariance matrix is denoted S.Source: §3 · pp. 5–6
  • Figure 1 contrasts KDE’s Gaussian spraying around existing minority points with SMOTE’s convex-hull interpolation, ROS repetition, ADASYN’s edge emphasis, and NearMiss majority undersampling.The figure description says NearMiss loses majority-class information through undersampling.Source: §3 · p. 6

4. NUMERICAL EXPERIMENTS

The experiments compare KDE sampling with standard oversampling and undersampling methods on simulated and real-life datasets. KDE performs strongly across several simulated conditions and for KNN and SVM, while its advantage is less pronounced with the tested MLP.

4.1. Simulated Data.

  • Across simulated imbalance ratios, KDE outperforms other methods on G-mean and F1-score, and is best on AUC at 80% imbalance and second at 70% and 90%.The varying-ratio experiment uses a feedforward neural network with one hidden layer.Source: Figure 2 · p. 9, §4.1 · p. 9
  • For the nearly linearly separable simulation, KDE significantly outperforms the other sampling techniques on AUC.A feedforward neural network was trained on each resampled dataset.Source: Table 1 · p. 9, §4.1 · p. 9

4. NUMERICAL EXPERIMENTS

  • The real-life evaluation covers 12 datasets with imbalance ratios from 1.86:1 to 42:1, comparing KDE with ROS, SMOTE, ADASYN, and NearMiss using KNN, SVM, and MLP.Source: §4 · p. 6, §4.2 · pp. 10–11

4.1. Simulated Data.

  • In the 3D simulation, KDE achieves the best AUC and F1-score and the second-best G-mean.The compared resampled data are generated from majority and minority samples in a cube with a central sphere removed.Source: §4.1 · p. 10, Table 2 · p. 11

4.2. Real Life Data.

  • On ecoli, KDE with KNN achieves G-mean 0.753 (5% above SMOTE) and F1-score 0.691 (6% above SMOTE).These results are reported for the KNN classifier.Source: §4.2 · p. 11, Table 4 · p. 12
  • On spectrometer, KDE with SVM achieves G-mean 0.924 (12% above SMOTE) and F1-score 0.878 (14% above SMOTE).The paper reports that SVM results are similar to KNN results overall.Source: §4.2 · p. 12, Table 5 · p. 13
  • With MLP, KDE sometimes outperforms other sampling methods, but its performance is not overwhelming in the reported experiments.The authors suggest the single-hidden-layer, 32-node network architecture may affect performance and note that another architecture may perform better.Source: §4.2 · pp. 12–13, Table 6 · p. 14

5. CONCLUSION

Across simulated and real-life datasets, KDE outperformed standard sampling methods in a number of scenarios and remained robust across base classifiers. The paper concludes that KDE is a potent tool for imbalanced-class problems.

5. CONCLUSION

  • KDE offers a flexible, statistically grounded way to generate minority samples with minimal overfitting, adjustable through kernel choice and bandwidth.The paper describes KDE as a natural approach and notes its established statistical foundation.Source: §5 · pp. 14–15
  • Implementations of KDE are available in Python, R, Julia, and other languages, which the authors cite as making it appealing for oversampling.Source: §5 · p. 15
  • Across 3 simulated and 12 real-life datasets, KDE outperformed four standard methods by F1-score and G-mean in a number of scenarios and remained robust across three base classifiers.The authors conclude KDE should be considered a potent tool for imbalanced-class problems.Source: §5 · p. 15
Loading 1910.07842v2…