Source-linked AI summary
Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features
Florian Pargent, Florian Pfisterer, Janek Thomas, Bernd Bischl
TL;DR
High-cardinality unordered categorical features require effective numerical encodings, and evidence about which strategy works best across algorithms and task types has been limited. The paper benchmarks encoding strategies with five machine-learning algorithms across regression and classification datasets, finding that regularized target encoding—especially GLMM encoding with five-fold cross-validation—performed best overall. The authors conclude that this approach is a reasonable default, while noting limitations from minimal learner tuning and feature-wise encoding.
Problem
The paper examines how to encode high-cardinality categorical features effectively and whether the best encoding depends on the downstream machine-learning algorithm.
Method
The study benchmarks encoding strategies with five machine-learning algorithms across regression, binary-classification, and multiclass-classification datasets.
Results
Regularized target encoding was superior or competitive across datasets, with GLMM encoding using 5-fold cross-validation ranking first for every algorithm except SVM.
Takeaways & Limitations
GLMM-based regularized target encoding with 5-fold cross-validation is suggested as a reasonable default strategy across machine-learning algorithms and dataset types.
Takeaways & Limitations
The benchmark used minimal tuning for the machine-learning algorithms and feature-wise encodings cannot represent interactions between levels of different features.
Abstract
from arXiv · showhide
Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and -- if possible -- derive best practices on when to use which technique. We conducted a large-scale benchmark experiment, where we compared different encoding strategies together with five ML algorithms (lasso, random forest, gradient boosting, k-nearest neighbors, support vector machine) using datasets from regression, binary- and multiclass- classification settings. In our study, regularized versions of target encoding (i.e. using target predictions based on the feature levels in the training set as a new numerical feature) consistently provided the best results. Traditionally widely used encodings that make unreasonable assumptions to map levels to integers (e.g. integer encoding) or to reduce the number of levels (possibly based on target information, e.g. leaf encoding) before creating binary indicator variables (one-hot or dummy encoding) were not as effective in comparison.
1 Introduction
High-cardinality categorical features are difficult to encode efficiently because one-hot encoding becomes inefficient as levels increase. The study benchmarks encoding strategies across algorithms and task types to identify effective defaults.
- Motivation: High-cardinality categorical features have many unordered levels, making standard one-hot encoding inefficient.The paper focuses on transforming such features into numerical representations for subsequent machine-learning models.
- Encoding strategies: Target-based encodings use target information associated with feature levels, whereas target-agnostic methods do not.Target-agnostic methods include one-hot and frequency encoding; target-based methods include target or impact encoding.
- Encoding strategies: Rare levels can make simple target encoding overfit because their encoded means closely reflect individual training observations.Regularization shrinks level effects toward the global mean, while cross-validation provides an alternative safeguard against overfitting.
- Study motivation: Prior small-scale studies produced inconclusive or mixed evidence about target encoding for high-cardinality features.Earlier benchmarks differed in scope, dataset count, and algorithm coverage, motivating a broader comparison.
- Study design: The benchmark evaluates seven encoding techniques with LASSO, random forests, gradient boosting, KNN, and SVM across regression, binary-classification, and multiclass-classification datasets.It also varies feature-cardinality thresholds and considers runtime and model complexity.
2 Encodings
The paper describes how categorical encoders handle unseen levels during prediction and outlines integer encoding as a simple baseline.
- General considerations: Encoding methods must specify how levels absent from training are handled during prediction.The paper identifies treatment of new levels as an important implementation detail.
- Integer encoding: Integer encoding maps observed training levels to integers from 1 to L, despite the absence of meaningful order among categories.Unseen levels are encoded as missing values and then mode-imputed to the most frequent training level.
2.2 Frequency Encoding
Frequency encoding represents each categorical level by how often it appears in the training data, implicitly grouping levels with similar frequencies.
- Frequency encoding: Frequency encoding maps each level to its observed training-set frequency.New levels are assigned a frequency of 1 during prediction.
- Frequency encoding: The method assumes that a level’s frequency has a functional relationship with the target.The subsequent model can distinguish most effectively between levels with different frequencies.
2.3 Indicator Encoding
Indicator encoding converts categorical levels into binary columns using either one-hot or dummy encoding, with different treatment of reference and unseen levels.
- Dummy encoding: Dummy encoding creates L−1 indicator columns by representing one reference level with zeros across all columns.The reference level is chosen from the observed categories.
- Unseen levels: One-hot encoding represents unseen prediction-time levels with a zero vector, whereas dummy encoding does not collapse them into the arbitrary reference category.The paper uses the first alphabetically ordered level as the dummy-encoding reference category.
2.4 Hash Encoding
Hash encoding transforms categorical feature levels into indicator representations using a hash function and a fixed number of indicator columns. Different levels may collide, with collisions becoming more likely as the hash size decreases.
- Hash encoding transforms each feature level l into an integer hash(l) based on its label.
- The hashed integer determines the active indicator column through (hash(l) mod hash.size) + 1.
- Some feature levels receive the same indicator representation, and smaller hash.size values increase the number of such collisions.
2.5 Leaf Encoding
Leaf encoding fits a decision tree to predict the target from a categorical feature, then represents each feature level by its terminal-node number. This groups levels that lead to similar target values.
- Leaf encoding fits a decision tree on the training set to predict the target from the categorical feature.
- Each feature level is encoded by the number of the terminal node reached by observations with that level.
- The encoding combines feature levels with similar target values.
- The implementation uses CARTs with categorical-feature support that can be pruned using internal performance estimates from 10-fold CV.
2.6 Impact Encoding
Impact encoding represents each categorical feature level using target information: conditional target means for regression or conditional class frequencies for classification. For classification, the encoder uses a logit link and produces one numeric feature per target class.
- Impact encoding represents each feature level by its conditional target mean in regression.
- For classification, impact encoding uses the conditional relative frequency of one or more target classes.
- The classification impact encoder uses a logit link and transforms the original feature into C numeric features, one for each target class.
- A smoothing parameter ϵ is introduced in the impact-encoding procedure.
2.7 GLMM Encoding
GLMM encoding interprets smoothed target encoding as a mixed model with level-specific random intercepts and a global fixed intercept. The implementation supports regression and classification, automatically regularizes the encoding, handles unseen levels, and can combine encoding with cross-validation.
- Smoothed target encoding can be interpreted as a generalized linear mixed model with a random intercept for each feature level and a fixed global intercept.
- The implementation provides GLMM encoders for regression, binary classification, and multiclass classification.
- For multiclass classification, the method fits C one-vs-rest GLMMs, producing one encoded feature per class.
- The fixed-intercept estimate can encode feature levels not observed during training.
- Using a GLMM determines a reasonable amount of regularization automatically, so tuning the complete machine-learning pipeline is unnecessary.
- Cross-validation trains encoders on independent observations while retaining data for training the downstream machine-learning model.
2.8 Control Conditions
The benchmark includes three control conditions to contextualize encoder effectiveness, including a featureless learner baseline and random-forest controls.
- The featureless learner predicts the training-set target mean for regression and the most frequent training-set class for classification.
- The controls help distinguish gains from categorical encoding from performance achievable without using informative features.
- The section also considers random-forest controls to assess whether explicit encoding improves an algorithm that can process categorical variables through other strategies.
3 Benchmark Setup
The benchmark evaluates categorical encoders across diverse high-cardinality datasets, five machine-learning algorithms, and multiple threshold settings using cross-validated predictive-performance estimates.
- The study uses datasets containing categorical variables with many levels, including established benchmark datasets downloadable from OpenML.
- High-cardinality thresholds of 10, 25, and 125 define alternative encoder configurations for assessing when advanced methods are useful.
- Five algorithms are benchmarked: LASSO, random forest, gradient boosting, k-nearest neighbors, and support vector machine.
- The benchmark minimizes hyperparameter tuning because its focus is the effect of encoding strategies rather than algorithm comparison.
- Analyses use a common mlrCPO and mlr workflow with HPC scaling and 5-fold cross-validation for predictive-performance estimates.
- Performance is reported with RMSE for regression, AUC for binary classification, and a classification metric for multiclass tasks.
4 Benchmark Results
Across regression and classification benchmarks, regularized target encoding—especially GLMM encoding with cross-validation—generally outperforms or matches alternatives, although results vary by algorithm, dataset, and computational setting.
- Meta-rankings: GLMM-based approaches with cross-validation outperform other approaches across algorithms in stable meta-rankings.
- Algorithm-specific findings: For random forests, every strategy except removing the feature beat the no-encoding control, suggesting that algorithm-native categorical handling is not necessarily optimal.
- Dataset patterns: Dataset clusters were ambiguous and not determined by problem type, suggesting similar encoder-ranking patterns across regression and classification settings.
- Overall encoder performance: Regularized target encoding was superior or at least competitive on all datasets, with GLMM encoding using 5-fold cross-validation ranking first for every algorithm except SVM.
- Algorithm-specific findings: Regularized target encoding outperformed or matched unregularized impact encoding, while integer encoding performed poorly with gradient boosting.
- Algorithm-specific findings: Indicator encoding ranked next best for LASSO, but its maximum of 125 indicator variables per feature may have limited its observed performance.
- Algorithm-specific findings: Target encoding suited distance-based KNN and SVM by mapping categories to a single smooth numerical feature, though SVM results require caution because some conditions failed computationally.
- Runtime: Runtime analysis suggests target encoding can accelerate subsequent models, but regularized GLMM with gradient boosting substantially increased runtime in the tested pipeline.
5 Discussion
Regularized target encoding performed consistently well across algorithms and datasets, while the benchmark also identifies important scope limits and opportunities for further encoder research.
- Main findings: Target encoding was never outperformed, although other encoding strategies achieved comparable performance under some conditions.
- Scope and selection: The benchmark could not identify consistent relationships between encoder performance and dataset characteristics.The authors therefore suggest that hyperparameter tuning may be needed to select the level threshold for target encoding.
- Limitations: Minimal tuning likely produced suboptimal performance for gradient boosting, k-nearest neighbors, and support vector machine learners.The study assumes encoder rankings remain comparable under more extensive tuning.
- Future work: Joint feature-and-level encoders that learn vector-valued representations are identified as an avenue for future comparison with classical optimal-scaling methods.
- Future work: The study concludes that refining statistically regularized target encoders and studying their theoretical properties is a valuable research direction.