Source-linked AI summary
SMOTE-ENC: A novel SMOTE-based method to generate synthetic data for nominal and continuous features
Mimi Mukherjee, Matloob Khushi
TL;DR
Class imbalance makes minority-class prediction difficult; SMOTE-ENC encodes nominal labels by their association with the minority class and yielded better results than SMOTE-NC on 4 of 5 datasets. Its advantage was especially evident where categorical features were numerous or associated with the target.
Problem
Class imbalance can make classifiers inefficient at identifying under-represented instances, especially when minority-class identification is the focus.
Source: §1 · p. 1Method
Results
4 of 5 datasets showed better results with SMOTE-ENC; gains were substantial on banking and credit-card datasets, while the others showed no significant improvement.
Source: §4 · p. 12Takeaways & Limitations
Unlike SMOTE-NC, SMOTE-ENC can also be applied to datasets containing only nominal features.
Source: §3.3 · p. 8, §5 · p. 13SMOTE-NC requires at least one continuous feature, so it cannot handle datasets whose attributes are all categorical, such as the car dataset.
Source: §3.3 · p. 8
Abstract
from arXiv · showhide
Real world datasets are heavily skewed where some classes are significantly outnumbered by the other classes. In these situations, machine learning algorithms fail to achieve substantial efficacy while predicting these under-represented instances. To solve this problem, many variations of synthetic minority over-sampling methods (SMOTE) have been proposed to balance the dataset which deals with continuous features. However, for datasets with both nominal and continuous features, SMOTE-NC is the only SMOTE-based over-sampling technique to balance the data. In this paper, we present a novel minority over-sampling method, SMOTE-ENC (SMOTE - Encoded Nominal and Continuous), in which, nominal features are encoded as numeric values and the difference between two such numeric value reflects the amount of change of association with minority class. Our experiments show that the classification model using SMOTE-ENC method offers better prediction than model using SMOTE-NC when the dataset has a substantial number of nominal features and also when there is some association between the categorical features and the target class. Additionally, our proposed method addressed one of the major limitations of SMOTE-NC algorithm. SMOTE-NC can be applied only on mixed datasets that have features consisting of both continuous and nominal features and cannot function if all the features of the dataset are nominal. Our novel method has been generalized to be applied on both mixed datasets and on nominal only datasets. The code is available from mkhushi.github.io
1. Introduction
Class imbalance makes minority-class prediction difficult, motivating methods that rebalance skewed data. SMOTE-ENC addresses limits of prior SMOTE methods for nominal features and is reported to perform better than SMOTE-NC when datasets contain numerous multi-label nominal features.
1. Introduction
- In heavily skewed datasets, models tend to predict the over-represented class, making them inefficient at identifying minority instances.This is especially critical when identifying minority instances is the point of interest.Source: §1 · p. 1
- Random over-sampling can overfit by duplicating minority instances, while random under-sampling can underfit after discarding potentially valuable data.Both approaches are described as having poor generalization on unseen data.Source: §1 · p. 2
- SMOTE creates synthetic minority examples between selected minority instances and their nearest neighbors rather than replicating existing instances.The paper describes this approach as improving minority-class classifier performance across a range of datasets.Source: §1 · p. 2
- Continuous-feature SMOTE variants can mishandle categorical attributes, including by creating new labels without physical significance.The paper gives Borderline-SMOTE, SMOTEENN, SMOTE-Tomek, ADASYN, and SMOTE-D as examples of variants developed for continuous variables.Source: §1 · p. 2
- SMOTE-NC preserves categorical labels but treats multi-label associations alike and requires at least one continuous feature.These limitations motivate a method that distinguishes nominal labels by their affinity toward the minority class.Source: §1 · p. 2
- SMOTE-ENC is reported to perform better than SMOTE-NC on datasets with numerous multi-label nominal features.The method is presented as usable whether or not continuous features are present.Source: §1 · p. 2
2. Materials and Methods
SMOTE-ENC changes the distance calculation for categorical labels by encoding their association with the minority class. The section explains this encoding and its scaling, then describes the evaluation setup and metrics.
2. Materials and Methods
- The worked example uses three continuous features and one categorical feature in an imbalanced dataset with two minority and three majority instances.Its sample-table caption identifies the example as an explanation of SMOTE-ENC.Source: §2 · p. 3, Table 1 · p. 3
- SMOTE-NC adds the same fixed distance penalty whenever categorical labels differ, whereas SMOTE-ENC uses encoded values to reflect changes in minority-class association.The paper illustrates the distinction with a life-insurance example in which single and divorced labels should be closer than either is to married.Source: §2 · pp. 2–4
- SMOTE-ENC assigns each nominal label a numeric χ value, with higher values indicating stronger association with the minority class.When continuous features are present, χ is multiplied by the median standard deviation of those features; for nominal-only data, χ is used directly.Source: §2 · pp. 4–5
- The comparison covers five datasets with diverse minority-to-majority ratios and evaluates precision, recall, F-beta, PR-AUC, and ROC-AUC.Accuracy is excluded because the authors say it does not capture performance on highly imbalanced data.Source: §2 · p. 5
- The evaluation uses stratified cross-validation, a random-forest classifier, and two-tailed t-tests on precision, recall, and F1-score results.The stated threshold for statistical significance is p = 0.05.Source: §2 · p. 5
3. Results
Across five datasets, the evaluations compare SMOTE-ENC with SMOTE-NC using precision, recall, F1-score, ROC-AUC, and PR-AUC. Results vary by dataset, with improvements reported for banking, credit-card, and car data and a mixed result for rain data.
3.1. Evaluation on banking tele-marketing dataset
- On banking data, SMOTE-ENC improved minority-class precision and F1 at threshold 0.5 and had better ROC and PR curves than SMOTE-NC.The precision, recall, and F1 improvements were statistically significant (two-tailed t-test p < 0.002).Source: Figure 1 · p. 6, §3.1 · p. 6
3.2. Evaluation on credit card dataset
- On credit-card data, SMOTE-ENC outperformed SMOTE-NC on minority-class precision and F1 at threshold 0.5 and achieved higher ROC and PR scores.Differences in precision, recall, and F1 were statistically significant (two-tailed t-test p < 0.004).Source: Figure 3 · p. 7, §3.2 · p. 7
3.3. Evaluation on car dataset
- For car data, SMOTE-ENC achieved an F1-score of 83.33 at threshold 0.5 and produced high ROC and PR scores.SMOTE-NC could not handle this dataset because it contains no continuous features; SMOTE-ENC supports nominal-only datasets.Source: §3.3 · p. 8
3.4. Evaluation on forest cover dataset with class 2 and 6
- On forest-cover classes 2 and 6, precision was consistent across methods, while SMOTE-ENC increased minority-class recall and marginally improved F1 at threshold 0.5.The PR curve also showed SMOTE-ENC marginally ahead of SMOTE-NC.Source: §3.4 · p. 9, Figure 6 · p. 9
3.5. Evaluation on rain dataset
- On rain data, SMOTE-ENC improved precision but reduced recall enough to lower F1, while ROC and PR curves marginally favored SMOTE-NC.The three nominal features remained outside the random-forest top five importance ranking under SMOTE-ENC.Source: §3.5 · p. 11, Figure 9 · p. 11, Figure 10 · p. 11
4. Discussion
The discussion reports that SMOTE-ENC’s advantage is concentrated in datasets with many nominal features and some feature-target association, and describes its feature-specific nominal distances and applicability to categorical-only data.
4. Discussion
- One-hot encoding can inflate dimensionality for multi-label features, while continuous-feature SMOTE variants may generate nominal labels without physical significance.Source: §4 · pp. 11–12
- SMOTE-ENC calculates each nominal feature’s inter-label distances separately from that feature’s distribution, unlike SMOTE-NC’s shared inter-label distance.Source: §4 · p. 12
- SMOTE-ENC yielded better results on 4 of 5 datasets; it was not better than SMOTE-NC when categorical-target association was poor.Source: §4 · p. 12
- Despite similar ROC and precision-recall curves, SMOTE-ENC improved precision on banking, credit-card, and rain datasets and recall on forest-cover; F-beta is a better evaluator than F1.Source: §4 · p. 12
- SMOTE-ENC substantially outperformed SMOTE-NC on banking and credit-card datasets, where nominal features exceeded 25% of features versus under 20% elsewhere.Source: §4 · p. 12
- SMOTE-NC’s distance requires continuous-feature standard deviations, so it fails on nominal-only datasets; SMOTE-ENC handles mixed and categorical-only data.Source: §4 · pp. 12–13
5. Conclusion
The authors conclude that SMOTE-ENC can outperform the existing method when datasets have many categorical features and can also handle nominal-only datasets. They identify alternative association measures and combinations with other SMOTE variants as directions for future work.
5. Conclusion
- SMOTE-ENC over-performs the existing method when datasets contain a substantial number of categorical features.Source: §5 · p. 13
- Unlike SMOTE-NC, SMOTE-ENC is equipped to handle datasets with only nominal features as well as mixed datasets.Source: §5 · p. 13
- The authors propose evaluating other ways to measure nominal-feature association and extending SMOTE-ENC with other SMOTE variants.Source: §5 · p. 13