PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning

Justin Engelmann, Stefan Lessmann

arXiv:2008.09202v1cs.LG

TL;DR

Class imbalance and the limits of conventional oversampling motivate a GAN-based approach for mixed-type tabular data. The paper proposes conditional Wasserstein GAN oversampling and benchmarks it across real-world credit-scoring datasets, finding competitive performance against standard methods while noting that no method consistently dominates.

  • Problem

    Class imbalance challenges classification, while prior GAN-based oversampling largely omitted recent tabular-generation advances and categorical variables.

  • Method

    The paper develops cWGAN-based oversampling for numerical and categorical tabular data and evaluates it against five benchmark oversampling techniques.

  • Results

    The method outperforms Random Oversampling on five of seven datasets and outperforms SMOTE and its variants on four, tying SMOTE on a fifth.

  • Takeaways & Limitations

    GAN-based oversampling is a competitive addition to the oversampling toolbox for mixed-type credit-scoring data.

  • Takeaways & Limitations

    No oversampling method performs generally better than its alternatives, and not altering class distribution often delivers competitive results.

Abstract

from arXiv · show

Class imbalance is a common problem in supervised learning and impedes the predictive performance of classification models. Popular countermeasures include oversampling the minority class. Standard methods like SMOTE rely on finding nearest neighbours and linear interpolations which are problematic in case of high-dimensional, complex data distributions. Generative Adversarial Networks (GANs) have been proposed as an alternative method for generating artificial minority examples as they can model complex distributions. However, prior research on GAN-based oversampling does not incorporate recent advancements from the literature on generating realistic tabular data with GANs. Previous studies also focus on numerical variables whereas categorical features are common in many business applications of classification methods such as credit scoring. The paper propoes an oversampling method based on a conditional Wasserstein GAN that can effectively model tabular datasets with numerical and categorical variables and pays special attention to the down-stream classification task through an auxiliary classifier loss. We benchmark our method against standard oversampling methods and the imbalanced baseline on seven real-world datasets. Empirical results evidence the competitiveness of GAN-based oversampling.

1 Introduction

Class imbalance complicates classification, while conventional oversampling can struggle with complex tabular data. The paper develops and evaluates a conditional GAN-based oversampling method for mixed-type credit-scoring data.

  • Class imbalance can make classifier training challenging and lead to poor predictive performance.
  • Nearest-neighbour interpolation may perform poorly on high-dimensional datasets and can generate minority cases in majority-class space.
  • Prior GAN-based oversampling research largely used purely numerical data despite tabular applications commonly containing categorical variables.
  • The proposed method models numerical and categorical variables, incorporates the target variable for downstream classification, and follows recent tabular-GAN advances.
  • The method is evaluated on seven real-world credit-scoring datasets against benchmark oversampling methods and no oversampling using five classifiers.
  • It outperforms four SMOTE variants and Random Oversampling on the majority of datasets, while no oversampling can also deliver competitive results.

2 Background

Imbalanced learning includes algorithm-level and data-level strategies, with oversampling preserving minority information but risking overfitting and noise. GANs may address limitations of neighbour-based interpolation in complex mixed-type data.

  • Data-level approaches balance classes through undersampling, oversampling, or combinations of both and can be used with any classification algorithm.
  • Oversampling is attractive when minority samples are scarce because it uses available information, although it can introduce overfitting and noise.
  • SMOTE generates minority samples by linearly interpolating between neighbouring observations.
  • Nearest-neighbour selection is problematic when tabular variables are heterogeneous and high-dimensional spaces are sparse.
  • Interpolating conceptually different neighbours can create synthetic minority cases inside majority-class space.
  • GANs can learn complex, high-dimensional distributions and could therefore perform better in such settings.

3 Related literature

Prior GAN oversampling studies provide limited and mixed evidence, especially for mixed-type tabular data. Newer tabular-generation architectures address categorical variables and distributional modeling, motivating the paper’s broader benchmark.

  • Earlier work with cGANs reported better performance than other methods, but omitted detailed study-design and dataset-level results.
  • A prior mixed-type study used one dataset, lacked SMOTE and Random Oversampling comparisons, and reported mixed results against no oversampling.
  • Most surveyed GAN oversampling studies used numerical-only datasets and did not incorporate advances in realistic tabular-data generation.
  • Existing evaluations often had limited datasets, classifiers, metrics, or baseline comparisons, producing mixed or difficult-to-interpret evidence.
  • CTGAN combines one-hot categorical outputs, Gaussian-mixture modeling for numerical columns, and a WGANGP loss.
  • The paper aims to use tabular-GAN advances, especially categorical-variable treatment, while comparing performance across multiple real-world mixed-type credit datasets.

4.1 Generative Adversarial Networks

GANs learn data distributions through adversarial generator–discriminator training, but Vanilla GANs are difficult to train. WGAN and WGANGP modify the objective to provide more useful training behavior, while conditional GANs target class-specific distributions.

  • Vanilla GAN: GANs train a generator to produce realistic samples and a discriminator to distinguish synthetic from real data.The generator maps latent noise into data space, while the discriminator classifies samples as real or fake.
  • Vanilla GAN: Vanilla GAN training can be difficult because equilibrium is not guaranteed and loss values do not necessarily track sample quality.Mode collapse occurs when the generator maps many noise inputs to the same output.
  • WGAN: WGAN replaces the standard objective with Wasserstein-1 distance, which remains meaningful for distributions with disjoint supports.The Wasserstein distance can compare candidate distributions even when they do not overlap, unlike the Jensen–Shannon divergence.
  • WGANGP: WGANGP is easier to train and generally works better than Vanilla GAN, although its gradient-penalty calculation is computationally demanding.The paper notes that WGANGP has become preferred over Vanilla GAN for tabular data generation.
  • WGANGP: WGANGP enforces the Lipschitz constraint with a gradient penalty instead of discriminator weight clipping.The penalty is evaluated on interpolations between real and synthetic samples and acts as a regularizer.
  • Conditional GANs: Conditional GANs append class labels to generator and discriminator inputs, enabling class-specific sampling and empirically more stable training.The generator estimates pX|y, while conditioning the discriminator prevents it from ignoring the class label.

4.2 GANs for tabular data

Tabular GANs must handle heterogeneous numerical and categorical variables whose values, categories, and distributions differ by column. The paper discusses differentiable categorical generation and techniques for modelling discrete or multimodal numerical data.

  • Tabular-data characteristics: Tabular datasets combine numerical and categorical variables, each presenting distinct modelling challenges.Possible categories and value ranges usually differ across columns, and the work distinguishes only numerical from categorical columns.
  • Modelling categorical variables: Categorical outputs are difficult for GANs because discrete sampling is not differentiable and soft one-hot outputs can reveal synthetic data.Straight-through estimation passes hard samples forward but produces biased gradients.
  • Modelling categorical variables: Gumbel-softmax provides fully differentiable categorical outputs that more closely emulate hard one-hot samples.Its temperature controls output diversity and the similarity of logits.
  • Modelling categorical variables: Embedding categorical representations is another strategy: each one-hot categorical variable is passed through its own embedding layer before discrimination.Label encoding followed by rounding is conceptually questionable because distances in label space lack conceptual meaning.
  • Modelling numerical variables: Numerical columns can contain frequent exact modes or fixed increments, making exact-value generation difficult for neural networks.A discriminator may reject values near frequent modes even when they are plausible.
  • Modelling numerical variables: Adding small zero-mean Gaussian noise to numerical columns during discriminator training addresses the difficulty of matching exact numerical modes.With min-max scaling to [0, 1], the authors found a standard deviation of 0.01 worked well, although it was not extensively tuned.
  • Modelling numerical variables: Self-conditioning explicitly conditions numerical generator outputs on the generator’s categorical outputs to model cross-variable patterns more easily.The authors introduce this mechanism to help the generator model structured numerical patterns.

4.3 Our method: cWGAN-based oversampling

The proposed cWGAN oversamples minority classes by combining conditional generation, WGANGP, and an auxiliary classifier loss for class-recognisable tabular samples. On the Adult dataset, it models numerical and categorical distributions realistically, with specific difficulties around neighbouring numerical modes.

  • GAN objective: After training a conditional GAN, the generator produces additional samples explicitly conditioned on the minority class for oversampling.The method estimates pX|y and samples xnew = G(z, y = yminority).
  • GAN objective: The cWGAN combines WGANGP with an auxiliary classifier loss that encourages generated samples to recognisably belong to the conditioned class.Unlike the original AC-GAN setup, the method uses separate discriminator and auxiliary-classifier networks, allowing discriminator conditioning as well.
  • GAN objective: The auxiliary-classifier loss is dynamically scaled and capped so Wasserstein training remains primary and the generator is not overfit to the classifier.The scale is set to λAC = 0.1|D(G(z))| and the loss is capped at 0.3, corresponding to at least 74% classifier confidence.
  • GAN objective: On Adult data, the auxiliary-classifier loss makes synthetic samples clearly recognisable as their conditioned class, especially for the harder minority class.Synthetic minority samples become easier to recognise than real minority samples, while majority samples remain just recognisable enough to avoid the loss.
  • Network structures: Using no activation for numerical generator outputs achieved the best generative performance in limited Adult-dataset testing.This choice permits values outside the training range, requiring the generator to learn to constrain outputs mostly within that range.
  • Network structures: Crosslayers model feature interactions in both generator and discriminator, increasing noise variation in the generator and discriminative power in the discriminator and classifier.Stacking n crosslayers enables efficient computation of (n + 1)-th degree feature interactions.
  • Generative performance of our method: The cWGAN approximates individual numerical distributions well and captures multiple modes, but struggles to differentiate neighbouring modes.Evaluation uses Gaussian-kernel density estimates for numerical variables and log-normalised category counts for categorical variables.

5 Experimental design

The experiment compares cWGAN-based oversampling with standard alternatives and no oversampling across tabular credit-scoring settings. It evaluates both numerical and categorical data using multiple classifiers and classification metrics.

  • Datasets and classifiers: The study compares methods on seven real-world datasets using five classification algorithms.The classifiers include Random Forest, Logistic Regression, Gradient Boosting, k-Nearest-Neighbours, and Decision Trees.
  • Benchmark methods: SMOTE generates minority samples by interpolating between a selected case and a randomly chosen nearest neighbour.The interpolation coefficient epsilon is sampled uniformly from [0, 1], and the neighbour count is a hyperparameter.
  • Benchmark methods: SMOTENC handles nominal and continuous data by combining SMOTE for continuous variables with neighbour-based nominal values.Nominal differences are incorporated into the distance calculation through a penalty based on continuous-feature variability.
  • Benchmark methods: ADASYN and B-SMOTE target difficult minority examples using neighbourhood information.ADASYN creates more samples near majority cases, while B-SMOTE focuses on minority cases considered in danger of misclassification.
  • Benchmark methods: The benchmark includes no oversampling, Random Oversampling, SMOTE, SMOTENC, ADASYN, B-SMOTE, and cWGAN.cWGAN is based on a conditional Wasserstein GAN.
  • Evaluation: Performance is measured with AUC-ROC, AUC-PR, and Brier score.The Brier score evaluates probability errors, whereas AUC-ROC and AUC-PR are ranking-based metrics.

6 Results

Across seven datasets, cWGAN is competitive with benchmark oversampling methods but does not dominate universally. Its strongest results occur on the two strongly non-linear datasets, whereas no oversampling is usually best on the remaining datasets.

  • Statistical analysis: The Friedman test rejects equivalent-performance hypotheses for most classifier-metric combinations, except AUC-ROC and AUC-PR for KNN and Decision Tree.The Brier score appears more sensitive to oversampling differences than the two ranking metrics.
  • Overall comparison: cWGAN outperforms all four SMOTE variants on five of seven datasets and ties with SMOTE on a sixth.It also outperforms Random Oversampling on five of seven datasets.
  • Overall comparison: No oversampling achieves the best mean rank on all but two datasets, while cWGAN performs best overall on those two.The two exceptions are Home Equity and Kaggle.
  • Dataset linearity: On the two strongly non-linear datasets, cWGAN is the best-performing technique; on the remaining datasets, no oversampling performs best.The paper defines strong non-linearity using a Random Forest versus Logistic Regression AUC-ROC difference of at least 0.1.
  • Classifier effects: cWGAN performs well with Random Forest and Gradient Boosting but badly with Logistic Regression.This pattern is consistent with the distinction between complex and mostly linear decision boundaries.
  • Ablation study: Ablations generally perform worse, supporting the usefulness of the method’s major components.Removing both the WGANGP and auxiliary-classifier losses performs worse on most classifier-metric combinations and has a worse mean rank.

7 Conclusion

The paper proposes cWGAN-based oversampling and evaluates it on seven real-world credit-scoring datasets using five classification algorithms and three performance metrics. Results are competitive overall, with especially strong performance on the two strongly non-linear datasets, but no oversampling method is consistently best.

  • Conclusion: The study evaluates cWGAN-based oversampling against five benchmark techniques and no oversampling on seven real-world credit-scoring datasets.The evaluation uses five classification algorithms and three classification-performance metrics.
  • Conclusion: cWGAN oversampling outperforms Random Oversampling on five of seven datasets and outperforms SMOTE and its variants on four of seven.On a fifth dataset, cWGAN ties with SMOTE and outperforms the SMOTE variants.
  • Conclusion: Not oversampling performs better than every considered oversampling method on five of seven datasets, indicating that addressing class imbalance is not always beneficial in credit scoring.The paper suggests this may relate to the mostly linear structure of many credit-scoring datasets.
  • Conclusion: On the two strongly non-linear datasets, oversampling improves performance and cWGAN is the best-performing method.The paper also reports that cWGAN performs well with high-performance non-linear classifiers such as Random Forest or Gradient Boosting.
  • Conclusion: The findings position cWGAN as a useful addition to the oversampling toolbox, particularly for strongly non-linear datasets, but require broader validation.Future work should test more strongly non-linear and more heavily imbalanced datasets, improve hyperparameter defaults, and evaluate other domains.

A.1 Ranks

Table 10 ranks oversampling methods for each dataset across all classifiers, with results sorted by best overall performance.

  • Table 10 reports per-dataset rankings of each oversampling method across all classifiers.Best ranks are shown in bold, and the table is sorted by best overall performance.

A.2 Raw scores

Tables 11–13 report numerical oversampling results across methods, datasets, and classifiers using three evaluation metrics. Results are averaged over six runs with standard deviations reported.

  • AUC-ROC: AUC-ROC results are averaged over six runs, with standard deviations in brackets and higher values preferred.The best value for each classifier and dataset is bolded.
  • AUC-PR: AUC-PR results are averaged over six runs, with standard deviations in brackets and higher values preferred.The best value for each classifier and dataset is bolded.
  • Brier score: Brier score results are averaged over six runs, with standard deviations in brackets and lower values preferred.The best value for each classifier and dataset is bolded.
Loading 2008.09202v1…