Source-linked AI summary
The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks
João Manoel Herrera Pinheiro, Suzana Vilas Boas de Oliveira, Thiago Henrique Segreto Silva, Pedro Antonio Rabelo Saraiva, Enzo Ferreira de Souza, Ricardo V. Godoy, Leonardo André Ambrosio, Marcelo Becker
TL;DR
Feature scaling lacks comprehensive evaluation despite its potential to affect model performance and reproducibility. This study benchmarks 12 scaling techniques across 14 machine-learning models and 16 classification and regression datasets, finding that ensembles are robust while several other models are scaler-sensitive.
Problem
Feature scaling is often applied indiscriminately or incorrectly, and insufficient evaluation can affect performance, reproducibility, and generalizability across datasets or domains.
Method
The study evaluates 12 scaling techniques plus an unscaled baseline across 14 tabular machine-learning models and 16 classification and regression datasets.
Results
Ensemble methods and Naive Bayes largely maintain high performance irrespective of scaling, whereas Logistic Regression, SVMs, MLPs, KNN, and TabNet are highly sensitive to scaler choice; scaling also affects computational costs.
Takeaways & Limitations
Model-specific scaling selection and more rigorous experimental design can help avoid preprocessing errors such as data leakage and unsupported scaling choices.
Takeaways & Limitations
The study relies on standard algorithm implementations from well-known libraries rather than variations or newer advances within those algorithm families.
Abstract
from arXiv · showhide
This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and $R^2$) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques.
I. INTRODUCTION
The study addresses limited and fragmented evidence on how feature scaling affects different machine-learning algorithms and datasets. It evaluates scaling choices systematically to support reproducible, model-specific assessment.
- Feature scaling maps attributes to comparable weights, but applying it without evaluating model suitability may harm performance, validity, and generalization.
- The study benchmarks 12 scaling techniques, including VAST, Pareto, Logistic Sigmoid, and Hyperbolic Tangent, across 14 supervised algorithms and 16 classification and regression datasets.
- Prior studies usually examined few algorithms or datasets, leaving comprehensive comparisons of scaling methods across models scarce.
- The evaluation is designed to address gaps in earlier work that often used narrow algorithm, dataset, or task selections and sometimes introduced data leakage.
- Open-source code, experimental data, and model parameters are provided to support reproducibility and comparison with the reported findings.
III. BACKGROUND
The background describes a deliberately broad but manageable scaler benchmark combining established and less common transformations. It also introduces normalization procedures and their intended data transformations.
- The benchmark selects 12 scaling techniques to balance breadth, diversity, experimental feasibility, and interpretability across 14 algorithms and 16 datasets.
- The selected methods include standard Min–Max, Standardization, and Robust Scaling alongside less commonly benchmarked VAST, Pareto, Logistic Sigmoid, and Hyperbolic Tangent transformations.
- 1) Min-Max Normalization (MM): Min-Max normalization maps values to a fixed range, typically [0, 1], using each feature’s minimum and maximum.
- Max normalization divides each feature by its maximum absolute value and is advantageous for strictly non-negative data.
- 3) Z-score Normalization (ZSN): Z-score normalization, or Standardization, transforms features to mean 0 and unit variance using the feature mean and standard deviation.
4) Variable Stability Scaling (VAST)
Variable Stability Scaling adjusts data according to feature stability and incorporates the coefficient of variation as a scaling factor. The supplied passages also contrast related scaling approaches, including Pareto scaling, mean centering, robust scaling, and quantile transformation.
- 4) Variable Stability Scaling (VAST): Variable Stability Scaling adjusts data based on the stability of each feature.It is particularly useful for high-dimensional datasets.
- 4) Variable Stability Scaling (VAST): The method incorporates the coefficient of variation as a scaling factor.The coefficient of variation is represented as CV = σ/µ in the supplied passage.
- Related scaling methods: Pareto scaling centers each feature and divides it by the square root of its standard deviation.It is intended to preserve relative differences while reducing the impact of large variances.
- Related scaling methods: Mean centering subtracts each feature’s mean and is often used before Principal Component Analysis.The supplied passage identifies PCA as a common preprocessing use.
- Related scaling methods: Robust scaling uses the median and interquartile range, making it robust to outliers.The supplied passages identify the median and IQR as its scaling basis.
- Related scaling methods: Quantile transformation maps data to a uniform or normal distribution and is useful for non-linear data.
9) Decimal Scaling Normalization (DS)
Decimal Scaling Normalization rescales attribute values by adjusting their decimal points to fit within (−1, 1). The supplied passages also list related transformations and the machine-learning models considered in the study.
- 9) Decimal Scaling Normalization (DS): Decimal scaling adjusts decimal points so normalized attribute values fall within (−1, 1), excluding the endpoints.The exponent uses the smallest integer j such that max(|Xnorm|) < 1.
- Related transformations: A tanh-normalization variant replaces Hampel estimators with each feature’s mean and standard deviation.
- Related transformations: Logistic sigmoid-based transformation applies the sigmoid function to the data.The supplied equation expresses Xnorm as 1/(1 + e^-q), with q defined using X and µ.
- Related transformations: The hyperbolic tangent transformation is described as similar to tanh transformation but applied differently in certain contexts.
- Models evaluated: The study includes linear, logistic, nearest-neighbor, tree, boosting, Bayesian, neural-network, support-vector, forest, and TabNet models.The supplied model descriptions cover both classification and regression where applicable.
- Models evaluated: Logistic Regression estimates binary-class probabilities using the logistic function, while Linear Regression fits a linear relationship to a continuous target.
- Models evaluated: SVM/SVR use an RBF kernel for classification margins and regression within a tolerance margin.
- Models evaluated: MLP is a feedforward neural network trained by backpropagation, whereas Random Forest uses bootstrap-aggregated decision trees.
C. METRICS
The study evaluates classification with accuracy and regression with MAE, MSE, and R2. Accuracy is retained despite its known limitations for imbalanced datasets.
- Classification metrics: Accuracy measures the proportion of correctly predicted instances in classification tasks.Its formula uses TP, TN, FP, and FN.
- Classification metrics: Accuracy can be misleading for imbalanced datasets, although the study includes it for its binary and multiclass datasets.
- Regression metrics: Mean Absolute Error measures the average absolute difference between predicted and actual values.It provides an intuitive measure of error magnitude.
- Regression metrics: Mean Squared Error averages squared prediction differences and penalizes larger errors more heavily.
- Regression metrics: R2 indicates the proportion of dependent-variable variance predictable from the independent variables.Higher R2 indicates a better model fit.
3) Computational Metrics
The study complements predictive-performance evaluation with memory, training-time, and inference-time measurements. It uses numeric UCI datasets, splits data before preprocessing, and adopts a 70/30 train–test division.
- Computational metrics: Memory usage measures the increase during dataset loading and feature scaling.
- Computational metrics: Training time records how long each model takes to train on a given dataset.
- Computational metrics: Inference time measures the time required for trained models to predict unseen data.These computational metrics matter for real-time or resource-constrained environments.
- Datasets and setup: The experiments use UCI datasets selected for standardized formats and diverse real-world benchmarking.The supplied dataset descriptions cover classification and regression collections.
- Datasets and setup: Datasets are split into training and test sets before feature scaling to avoid data leakage.This preserves separation between the sets for robust and unbiased evaluation.
- Datasets and setup: A 70/30 train–test split provides the study’s practical balance between training data and performance evaluation.The paper notes that no universal optimal split exists.
C. EXPERIMENT WORKFLOW
The workflow standardizes datasets, creates reproducible train/test splits, applies each scaler separately to every dataset–model combination, and records evaluation outputs and parameters.
- Traceability: Configuration files record parameters used by each model, enabling traceability of the experimental setup.
- Data preparation: Datasets are imported, cleaned, encoded, and standardized before being split into saved training and testing subsets.
- Outputs: The pipeline records validation metrics, training time, inference time, memory usage, and model parameters for each experiment.
- Scaling and evaluation: Each scaler is fitted on training data, transformed onto both splits, and followed by model training and test evaluation.
- Automation: Dedicated scripts automate dataset import, cleaning, splitting, training, result storage, and end-to-end experiment orchestration.
E. SOURCE CODE OF THIS EXPERIMENTS
The experiments compare scaling effects across classification and regression models using statistical tests and evaluate predictive performance, computational time, and memory consumption. Results distinguish scaling-robust ensembles from models whose outcomes vary with preprocessing, while also identifying memory differences among scalers.
- Validation metrics: Ensemble methods, including Random Forest, LightGBM, CatBoost, and XGBoost, consistently achieved high validation performance regardless of preprocessing strategy or dataset.
- Validation metrics: Logistic Regression, SVM, KNN, TabNet, and MLP showed strong dependence on scaler choice, indicating pronounced preprocessing sensitivity.
- Statistical analysis: Friedman tests found significant variation for Logistic Regression, SVM, MLP, and KNN, but not for Random Forest, Naive Bayes, CART, AdaBoost, CatBoost, LightGBM, XGBoost, or TabNet.
- Regression results: Regression experiments showed a similar scaling-sensitivity pattern, although KNN regression had no statistically significant Wilcoxon differences across scaling methods.
- Computational cost: Tree-based models showed no significant speed benefit from scaling, whereas inference times for KNN, SVM, and SVR were more sensitive to scaler choice.
- Memory usage: RobustScaler, StandardScaler, Tanh Transformer, and Hyperbolic Tangent were most memory intensive, while MaxAbsScaler, MinMaxScaler, and Decimal Scaler used the least memory.
VI. LIMITATION
The study’s broad empirical scope is accompanied by limitations involving hyperparameter tuning, dataset diversity, classification metrics, validation strategy, and algorithm implementation scope.
- Hyperparameter Optimization: Default hyperparameters were used, so tuning each model, scaler, and dataset combination could reveal different pairings or improve performance.The authors identify comprehensive hyperparameter optimization as beyond the current scope.
- Scope and Diversity of Datasets: The 16 datasets could be expanded with higher-dimensional data, different underlying distributions, and more specialized domains.The authors frame broader dataset coverage as a direction for strengthening the findings.
- Evaluation Metrics for Classification: Classification relied primarily on accuracy, which may be misleading for imbalanced datasets; future work could add F1, precision, recall, AUC, or balanced precision.These additional metrics are proposed to provide more nuanced assessment on skewed class distributions.
- K-Fold Cross-Validation: The study used hold-out validation rather than k-fold cross-validation because the full experimental setup would substantially increase computational cost.Cross-validation or repeated runs are proposed to further validate robustness.
- Dataset Size and Algorithm Implementations: Some smaller datasets were not augmented with synthetic data, and the study mainly relied on standard library implementations rather than newer algorithm variants.Both dataset augmentation and investigation of algorithm-family variations are identified as future directions.
APPENDIX A TABLES & FIGURES RESULTS
The appendix reports regression performance, computational costs, memory usage, and statistical comparisons across datasets, models, and scaling methods. Its figures and tables organize results by metric and task, including ensemble-model comparisons.
- Regression Performance: R2 results are reported for 7 datasets, with RF, LGBM, CatBoost, and XGBoost identified as ensemble models.The figure presents regression R2 outcomes across the evaluated datasets and ensemble methods.
- Regression Performance: MAE and MSE results are reported for 7 datasets, including comparisons involving RF, LGBM, CatBoost, and XGBoost.The appendix separates these regression metrics into dedicated figures and tables.
- Computational Cost: Training and inference times are reported separately for classification and regression tasks across datasets, models, and scaling methods.The appendix includes figures, tables, and statistical tests for computational timing.
- Memory Usage: Memory usage is reported in kB per dataset and scaling method.The appendix includes both a memory-usage figure and a corresponding table.
- Statistical Comparisons: Statistical tables compare scaled versus unscaled features and compare all scaling methods for accuracy, R2, MSE, MAE, training time, and inference time.These tests provide metric-specific comparisons across the scaling configurations.