PaperlayerView exact revision on arXiv

Source-linked AI summary

The choice of scaling technique matters for classification performance

Lucas B. V. de Amorim, George D. C. Cavalcanti, Rafael M. O. Cruz

arXiv:2212.12343v1cs.LG

TL;DR

Scaling is widely used in classification, but the choice among scaling techniques is not generally examined carefully despite its potential performance consequences. This paper compares five techniques across 20 monolithic and ensemble models and 82 binary datasets with varying imbalance ratios. The results show significant effects of scaling choice, relationships between ensemble and base-model variation, and deployment trade-offs without a single universally appropriate model.

  • Problem

    Few studies broadly compare scaling techniques across multiple classification algorithms and datasets, although choosing the wrong technique can be worse than not scaling.

  • Method

    The authors evaluate five scaling techniques on 82 binary datasets and 20 classification models, using fold-specific scaling to avoid test-set leakage.

  • Results

    Scaling choice significantly affects classification performance, can be worse than no scaling when inadequate, and ensemble performance variation relates to that of base models.

  • Takeaways & Limitations

    Model deployment has no one-size-fits-all scaling choice because application constraints and scaling feasibility may determine the appropriate model.

  • Takeaways & Limitations

    Further work is needed to explain ensemble–base-model variability and identify data characteristics that support scaling-technique selection.

Abstract

from arXiv · show

Dataset scaling, also known as normalization, is an essential preprocessing step in a machine learning pipeline. It is aimed at adjusting attributes scales in a way that they all vary within the same range. This transformation is known to improve the performance of classification models, but there are several scaling techniques to choose from, and this choice is not generally done carefully. In this paper, we execute a broad experiment comparing the impact of 5 scaling techniques on the performances of 20 classification algorithms among monolithic and ensemble models, applying them to 82 publicly available datasets with varying imbalance ratios. Results show that the choice of scaling technique matters for classification performance, and the performance difference between the best and the worst scaling technique is relevant and statistically significant in most cases. They also indicate that choosing an inadequate technique can be more detrimental to classification performance than not scaling the data at all. We also show how the performance variation of an ensemble model, considering different scaling techniques, tends to be dictated by that of its base model. Finally, we discuss the relationship between a model's sensitivity to the choice of scaling technique and its performance and provide insights into its applicability on different model deployment scenarios. Full results and source code for the experiments in this paper are available in a GitHub repository.\footnote{https://github.com/amorimlb/scaling\_matters}

1. Introduction

This paper investigates whether scaling-technique choice affects classification performance across models, datasets, and imbalance ratios. It broadens prior comparisons to examine monolithic and ensemble models, ensemble–base-model relationships, and performance variation under different imbalance conditions.

  • Motivation: Scaling is an essential preprocessing step because unscaled feature ranges can bias analysis toward less informative attributes.Scaling places features within comparable ranges and may improve classification performance.
  • Motivation: Choosing an inappropriate scaling technique can harm classification performance more than leaving data unscaled.The paper identifies scaling-technique selection as a methodological decision requiring careful attention.
  • Motivation: The study addresses limited prior evidence by comparing multiple scaling techniques across different algorithm types and larger datasets.Earlier studies often tested only two techniques, one technique with broader models, or three classification algorithms.
  • Research questions: The paper examines whether scaling sensitivity varies by model, whether homogeneous ensembles follow their base models, and whether imbalance ratio affects performance variation.It also analyzes how sensitivity relates to model performance and how ensemble construction might manipulate that relationship.
  • Study design: The experiment uses 82 datasets, five scaling techniques, and 20 classification models, including 11 monolithic and nine ensemble models.The datasets span different domains and imbalance ratios; performance is measured with F1 and G-Mean using 5-fold cross-validation and hypothesis tests.
  • Contributions: The main findings are that scaling choice matters, ensemble scaling rankings relate to base-model rankings, and scaling-induced variation is more salient at higher imbalance ratios.These findings support analyzing scaling choices as part of model selection and deployment considerations.

2. Scaling techniques

Scaling techniques transform feature values to reduce scale-driven dominance, but they differ in how they handle offsets, variances, outliers, and distribution shapes.

  • Scaling adjusts features so wider numerical ranges do not dominate narrower ones during model learning.This can mitigate bias toward less informative attributes and improve classification performance.
  • Common scaling techniques combine translation and rescaling, although Mean Centering only subtracts each vector component's mean.Translation moves data along the X-axis, while scaling concentrates or spreads it horizontally.
  • The study compares Standard, Min-max, Maximum Absolute, Robust, and Quantile Transformer techniques as diverse, commonly used representatives.Figure 1 illustrates their effects on opposing means, outliers, and differing distribution shapes.
  • Standard Scaler produces zero-mean, unit-variance distributions but makes inliers too narrow when outliers are present.It subtracts the attribute mean and divides by the standard deviation.
  • Min-max Scaler maps values to [0, 1], while Maximum Absolute Scaler divides by the maximum absolute value without translating.Both can handle ordinary distributions similarly to Standard Scaler, but outliers can prevent effective variance equalization.
  • Robust Scaler uses the median and interquartile range to reduce outlier effects, effectively equalizing variances in the illustrated outlier case.Quantile Transformer consistently produces similar distributions and converts the uniform example into a standard normal distribution, though its nonlinear transformation may distort linear correlations.

3. Classification Algorithms

The experiment covers a broad range of classification algorithms, including monolithic and ensemble models, to support conclusions across algorithm categories.

  • The study evaluates monolithic and ensemble classifiers spanning eight subcategories.The selection is intended to support analysis whose results may generalize across a wide range of classification algorithms.
  • Monolithic models include instance-based, probabilistic, discriminant-analysis, rule-based, and neural-network methods.
  • Ensemble models are grouped into Static, DCS, and DES subcategories.The paper includes ensembles because these methods, particularly DCS and DES, have shown promise in various scenarios.

3.1. Monolithic Models

The paper covers monolithic classifiers across instance-based, probabilistic, discriminant-analysis, rule-based, and neural-network categories, including their defining classification mechanisms.

  • Instance-based algorithms: Instance-based algorithms classify queries using selected training instances or learned abstractions, including KNN, GLVQ, and SVM.KNN uses nearby labels, GLVQ uses learned prototypes, and SVM uses a separating hyperplane.
  • Probabilistic algorithms: Probabilistic algorithms estimate class-membership probabilities; the study includes Gaussian Naive Bayes and Gaussian Process classifiers.Gaussian Naive Bayes assumes Gaussian feature likelihoods.
  • Discriminant Analysis: Discriminant-analysis methods seek decision surfaces separating classes, with LDA using linear and QDA using quadratic surfaces.Both methods can also be used for dimensionality reduction.
  • Rule-based and neural-network models: The study includes Decision Tree as a rule-based classifier and Perceptron and MLP as neural-network models.Perceptron uses a single layer and linear decision surface, whereas MLP uses multiple layers and nonlinear decision surfaces.

3.2. Ensemble Models

The ensemble-model coverage includes homogeneous static and dynamic ensembles, whose decisions combine or select outputs from base classifiers.

  • Ensemble foundations: Ensemble models combine monolithic-classifier outputs to produce a joint decision, and the study restricts analysis to homogeneous ensembles.Homogeneous ensembles generate base classifiers using a single classifier algorithm.
  • Static ensembles: Static ensembles commonly combine base-classifier outputs through majority voting or weighted majority voting.
  • Dynamic ensembles: Dynamic ensembles select one classifier or a subset using local performance within a query’s Region of Competence.DCS selects a single classifier, while DES selects a subset.
  • Dynamic ensembles: Dynamic selection methods are investigated because scaling can alter the local competence region used during classifier selection.
  • Static ensembles: The study includes four static ensembles, including Bagging, Random Forests, AdaBoost, and XGBoost.Bagging uses bootstrap training replicates and majority voting; Random Forests additionally randomize feature selection.
  • Dynamic ensembles: Five dynamic ensembles are included: three Dynamic Classifier Selection strategies and two Dynamic Ensemble Selection strategies.The dynamic methods include OLA, LCA, MCB, KNORA-E, and KNORA-U.

4. Experiment Methodology

The experiment evaluates five scaling techniques across 20 classifiers and 82 datasets with varied imbalance, using stratified cross-validation and imbalance-aware metrics.

  • Dataset selection: The 82 datasets were selected from 91 binary KEEL datasets by requiring no more than 30% categorical attributes.The selection preserves varied imbalance ratios and focuses scaling analysis on predominantly numerical data.
  • Dataset selection: The datasets span diverse domains and use predefined 5-fold stratified cross-validation splits to support reproducibility.The KEEL preprocessing converted multiclass problems to binary ones and supplied fixed folds.
  • Datasets Scaling: Each dataset received Standard, Min-max, Maximum Absolute, Robust, and Quantile Transformer variants, plus an unscaled baseline.
  • Datasets Scaling: Scaling parameters were estimated independently from each training fold and then applied to its test fold to avoid look-ahead bias.For Min-max scaling, each fold’s training minimum and maximum transform the corresponding test data.
  • Performance metrics definitions: Performance was measured with F1 and G-Mean because accuracy can be overly optimistic on highly imbalanced datasets.Both metrics are defined from confusion-matrix elements.

5. Results and Discussion

Scaling technique choice materially affects classification performance across monolithic and ensemble models, with sensitivity varying by model, imbalance ratio, and ensemble base learner.

  • Careless scaling can perform worse than leaving data nonscaled, so technique selection must match the dataset-model pair.
  • 14 of 20 models showed relevant performance variation across scaling techniques; the exceptions were LDA, QDA, DT, and three DT-based ensembles.
  • 28 of 40 tests rejected equal-performance hypotheses across all datasets, compared with 15 and 11 rejections for low- and medium-imbalance strata.
  • For all 82 datasets, scaling differences were significant for most monolithic models, except DT and partially KNN.
  • The SVM linear model reached a 0.991 F1 range on segment-0; 26 of 82 datasets exceeded 0.5 and 6 exceeded 0.8.
  • SVMs, GP, Perceptron, and MLP showed larger mean ranges, while DT and DT-based ensembles were nearly insensitive to scaling.
  • Quantile Transformer won most often overall and in medium- and high-imbalance strata, whereas Standard Scaler was best for low imbalance.
  • More scale-sensitive ensembles tended to follow their Perceptron base model’s ranking pattern, suggesting base-model testing can reduce ensemble scaling evaluations.

6. Related Works

Prior studies compared scaling techniques using narrower combinations of techniques, datasets, or classifiers. This paper addresses that limited diversity with a broader comparative design.

  • Jain et al. and Dzierżak et al. each examined only Min-max and Z-score normalization.
  • Only Singh et al. among the reviewed studies used a relevant number of datasets when considering the generality of findings.
  • Raju et al. used seven scaling techniques on one diabetes dataset with KNN and two SVM variants, reporting 5%–10% gains over nonscaled data.
  • Singh et al. tested 14 techniques on 21 datasets but only evaluated KNN, while Mishkov et al. tested 16 techniques on four datasets with MLP.
  • Earlier studies lacked sufficient diversity simultaneously in scaling techniques, datasets, and classification algorithms.
  • Some prior technique sets included highly similar or equivalent transformations, including up to seven variants of one technique.

7. Lessons learned

Scaling technique choice materially affects classification performance for scale-sensitive models, with implications for selecting models and preprocessing strategies across deployment settings.

  • The best-to-worst scaling-technique performance difference is relevant and statistically significant in most cases, except for models inherently insensitive to scale.
  • Choosing the wrong scaling technique can hurt performance more than leaving data unscaled.For SVM lin, the F1 difference exceeded 0.5 on 32% of datasets and 0.8 on 7%.
  • Scaling-technique performance differences occur across all imbalance-ratio levels but appear more salient at higher imbalance ratios.
  • Standard Scaler performed best on low-imbalance-ratio datasets, whereas Quantile Transformer performed better in all other imbalance-ratio strata.
  • SVMs, GLVQ, Gaussian Process, Perceptron, MLP, Bagging, OLA, LCA, and KNORA-U are highly sensitive to scaling-technique choice.
  • Decision Trees and tree-based ensembles are virtually insensitive, while Perceptron ensembles show a wider sensitivity range with varying F1 and G-Mean performance.
  • For scale-sensitive ensembles, scaling-technique rankings tend to resemble those of their base models, enabling cheaper ensemble screening through a single base-model instance.
  • For real-time or online applications, DT or AdaBoost are preferable because they combine low scale sensitivity with reasonable performance.

8. Conclusion

The study evaluates how scaling-technique choice affects monolithic and ensemble classifiers across diverse datasets. It finds that scale choice significantly influences performance, relates ensemble variability to base models, and makes deployment constraints relevant to model selection.

  • The experiment tested five scaling techniques with 82 binary datasets spanning imbalance ratios and 20 classification models, including monolithic and ensemble algorithms.
  • Scaling-technique choice significantly affects classification performance, and an unsuitable technique can perform worse than not scaling the data.
  • Analysis of scale sensitivity and model performance revealed a relationship between ensemble performance variation and that of its base models.
  • The findings indicate that model deployment has no one-size-fits-all solution because application constraints affect model selection.
  • Future work will examine how ensembles reduce performance ranges across scaling techniques relative to base models and how ensemble configurations appear in the Figure 7 analysis.
Loading 2212.12343v1…