PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Predictive Modelling under Imbalanced Distributions

Paula Branco, Luis Torgo, Rita Ribeiro

arXiv:1505.01658v2cs.LG

TL;DR

Imbalanced domains arise when user-important target values are rare in training data, challenging predictive modelling and evaluation. This paper surveys solutions for classification and regression, organizes them into a taxonomy, and discusses related problems and scope boundaries.

  • Problem

    Imbalanced domains make important, poorly represented target cases difficult to model and evaluate with standard criteria.

  • Method

    The paper provides a survey covering definitions, performance measures, and approaches for imbalanced classification and regression, organized into a three-part taxonomy.

  • Results

    The survey presents a taxonomy grouping existing approaches into data pre-processing, special-purpose learning methods, and prediction post-processing.

  • Takeaways & Limitations

    Handling imbalanced domains requires attention to user relevance and to methods suited to both classification and regression tasks.

  • Takeaways & Limitations

    The optimal data distribution is domain dependent, and perfectly balanced classification data does not always provide optimal results.

Abstract

from arXiv · show

Many real world data mining applications involve obtaining predictive models using data sets with strongly imbalanced distributions of the target variable. Frequently, the least common values of this target variable are associated with events that are highly relevant for end users (e.g. fraud detection, unusual returns on stock markets, anticipation of catastrophes, etc.). Moreover, the events may have different costs and benefits, which when associated with the rarity of some of them on the available training data creates serious problems to predictive modelling techniques. This paper presents a survey of existing techniques for handling these important applications of predictive analytics. Although most of the existing work addresses classification tasks (nominal target variables), we also describe methods designed to handle similar problems within regression tasks (numeric target variables). In this survey we discuss the main challenges raised by imbalanced distributions, describe the main approaches to these problems, propose a taxonomy of these methods and refer to some related problems within predictive modelling.

1 Introduction

Imbalanced predictive modelling arises when user-important target values are rare in training data, creating challenges for model construction and evaluation. The paper surveys solutions across classification and regression and proposes a broader taxonomy of methods.

  • Motivation: Rare target values can be especially important to users, yet they are often poorly represented in available training samples.Examples include rare diseases, extreme financial returns, and catastrophes.
  • Motivation: User preferences and sparse representation of important cases create problems for predictive modelling approaches at several levels.The paper frames imbalance as both a data-distribution problem and a preference-sensitive modelling problem.
  • Scope and contributions: The survey defines imbalanced domains for both classification and regression tasks and reviews performance-assessment measures and modelling approaches.This extends coverage beyond prior surveys focused only on classification.
  • Scope and contributions: The paper proposes a broader taxonomy of methods that includes post-processing strategies alongside other approaches.The paper also describes important techniques for addressing imbalanced domains.
  • Organization: The paper covers problem definition, evaluation metrics, modelling approaches, related problems, and conclusions across its six sections.Its organization mirrors the survey’s progression from defining imbalance to reviewing solutions and related issues.

2 Problem Definition

Imbalanced data sets combine a user preference for accuracy on important target values with few training examples representing those values. Standard criteria may ignore these preferences, producing sub-optimal models and misleading evaluations.

  • Formalization: An importance function φ maps target values to user relevance, with 1 representing maximal importance and 0 representing minimum relevance.A relevance threshold t_R separates relevant and normal cases.
  • Formalization: The relevant subset D_R contains cases with φ(y_i) > t_R, while D_N contains cases with φ(y_i) ≤ t_R.The two subsets partition the training data according to the user-defined relevance threshold.
  • Problem definition: Imbalance occurs when the number of relevant examples in D_R is much smaller than the number of normal examples in D_N.The rare cases are those to which the user assigns greater importance.
  • Problem definition: Standard learning and evaluation criteria assume uniform relevance and therefore may be insensitive to user preferences over the target domain.This can make the resulting models sub-optimal with respect to those preferences.
  • Approach categories: Solutions include modifying learning algorithms, changing the data before learning, and transforming predictions after learning.These strategies are presented as three categories for addressing inadequate models.

3 Performance Metrics for Imbalanced Domains

Imbalanced-domain evaluation must reflect user preference for rare, important cases rather than allowing frequent cases to dominate model assessment. The survey reviews classification and regression metrics designed to expose relevant-case performance and error behavior.

  • Motivation: Standard evaluation criteria focus on frequent cases, conflicting with user preferences for poorly represented but important cases.
  • Classification metrics: Accuracy can reach 99% by predicting the majority class when only 1% of examples are minority cases, while missing every minority case.
  • Classification metrics: Fβ combines precision and recall, with β controlling their relative weighting; larger β emphasizes recall, whereas β < 1 emphasizes precision.
  • Classification metrics: ROC curves show the trade-off between true-positive and false-positive rates, while AUC summarizes average performance but is not biased toward the minority class.
  • Regression metrics: For regression, MSE and MAD are inadequate because they measure error magnitude without accounting for relevance or error location across the target domain.

4 Modelling Strategies for Handling Imbalanced Domains

The survey groups approaches for imbalanced domains into data pre-processing, special-purpose learning, and prediction post-processing, with hybrid methods combining categories. These strategies address models that otherwise focus on normal examples and neglect rare events.

  • Overview: Imbalanced-domain approaches are grouped into data pre-processing, special-purpose learning, and prediction post-processing.The survey also discusses hybrid methods that combine approaches from different categories.
  • Data Pre-processing: Data pre-processing changes the training distribution so standard learning algorithms focus on cases relevant to the user.Its advantages include broad compatibility with existing tools and models aligned with user goals, although the strategy has stated disadvantages.
  • Special-purpose Learning Methods: Special-purpose learning modifies algorithms to incorporate user goals directly into models.This can improve comprehensibility but restricts users to modified algorithms or requires developing new ones.
  • Prediction Post-processing: Prediction post-processing keeps the original data and standard learner, then adjusts model predictions according to user preferences and data imbalance.The resulting model can support different deployment loss functions without retraining or retaining the training data.
  • Hybrid Solutions: Hybrid methods combine approaches from different strategy types to exploit their respective advantages.The survey presents these approaches separately and uses Figure 7 to synthesize the categories and techniques.

4.1 Data Pre-processing

Data pre-processing changes the training data or sampling process so standard learners focus more on user-relevant, underrepresented cases. The survey covers re-sampling, active learning, weighting, recognition-based learning, synthetic data generation, and adaptive combinations.

  • Pre-processing modifies the training data distribution according to user goals, after which any standard learning algorithm can be applied.
  • Re-sampling, active learning, and data-space weighting are the three main pre-processing types.
  • Re-sampling is effective for addressing imbalance, but the optimal distribution is domain-dependent and need not be perfectly balanced.
  • Random under-sampling can discard useful majority examples, whereas random over-sampling can increase overfitting through duplicated minority examples.
  • Recognition-based methods learn the target class without counterexamples, while one-class learning classifies using a threshold on similarity.
  • As imbalance increased, binary-classifier performance decreased while one-class-classifier performance remained relatively stable.
  • Synthetic-data methods reduce overfitting risk from replicated examples and improve generalisation; SMOTE interpolates minority examples with minority neighbours.
  • VIRTUAL generates minority synthetic examples during training and outperforms competing over-sampling techniques in SVMs for generalisation performance and computational complexity.

4.2 Special-purpose Learning Methods

Special-purpose learning methods modify algorithm preference criteria or objectives so models directly account for imbalanced data, user preferences, or prediction costs. The survey covers cost-sensitive and non-cost-based adaptations across classifiers, ensembles, and regression.

  • Special-purpose methods modify existing algorithms to learn from imbalanced data by changing their preference criteria.
  • Cost-sensitive algorithms incorporate prediction costs to minimise total cost when misclassified examples have different costs.
  • Cost-sensitive adaptations have been studied for decision trees, support vector machines, neural networks, boosting ensembles, and Random Forests.
  • AdaBoost adaptations such as RareBoost, AdaC1, AdaC2, AdaC3, and BABoost modify weight updates to treat classes unequally using costs.
  • Other adaptations alter kernels, objectives, constraints, splitting criteria, or class-weighted distances without directly requiring a cost matrix.
  • The regression method ubaRules is a utility-based rule ensemble designed to produce accurate and interpretable predictions under non-uniform utility.
  • These methods may require a cost matrix, deep learner knowledge, and algorithm-specific modification, limiting portability across learning systems.

4.3 Prediction Post-processing

Prediction post-processing keeps the original data and standard learner, then changes predictions according to user preferences and imbalance. The survey distinguishes threshold adjustment from cost-sensitive post-processing and notes limited evaluation, especially for regression.

  • Post-processing manipulates model predictions rather than the training data or learning algorithm.
  • Threshold methods vary a class-membership threshold over a score to produce different classifiers.
  • Cost-sensitive post-processing associates costs with prediction errors and minimises expected cost.
  • Moving the decision threshold, applying sampling, and adjusting the cost matrix produced classifiers with the same performance in one study.
  • Post hoc cost-sensitive methods have mainly been explored for classification and had not been applied or evaluated on imbalanced data distributions according to the survey.
  • Regression post-processing costs had few limited solutions, with no progress yet reported for evaluating them in imbalanced domains.
  • Reframing converts crisp regression into soft conditional-density estimation and optimises expected loss for new contexts.

4.4 Hybrid Methods

Hybrid methods combine re-sampling with specialised learners or ensemble techniques to combine advantages of multiple imbalance-handling strategies. Examples include mixtures of experts, multi-learner voting, ensemble-of-ensembles, and SMOTE with biased SVMs.

  • Hybrid methods combine basic imbalance-handling approaches to capitalise on their different advantages.
  • Existing hybrid approaches commonly combine re-sampling with special-purpose learning algorithms.
  • Mixture-of-experts architectures address uncertainty about the optimal amount of over- or under-sampling through multiple processing levels.
  • A facilitator-agent system filters features, sends data to Naive Bayes, C4.5, and 5NN learners, and combines predictions by majority voting.
  • EasyEnsemble and BalanceCascade build balanced bags by majority under-sampling and train each bag with AdaBoost.
  • A SMOTE–Biased-SVM approach over-samples support vectors or the entire minority class before final classification with biased SVM.
  • A clustering-based method assigns minority examples to non-pure clusters and builds decision trees for clusters, using a neural network for final minority predictions.

5 Related Problems

Imbalanced classification often coexists with overlap, small samples, rare cases, high dimensionality, noise, and small disjuncts. These related problems can further degrade model performance and require attention alongside imbalance.

  • Class overlap: Class overlap makes distinguishing classes harder because overlapping regions contain similar numbers of training cases from each class.Imbalance and overlap were often treated separately, although later work examined their relationship.
  • Small samples and rare cases: Too few minority-class examples prevent learners from capturing minority characteristics and hinder algorithm generalisation.Small samples may also trigger rare cases that are extremely scarce and difficult to detect or use for generalisation.
  • Sampling: Non-random sampling that favours the minority class can improve classifier performance, whereas a balanced training set is not always the most favourable setting.The target distribution may differ from the class distribution in the available training data.
  • High dimensionality: High-dimensional imbalanced domains require selecting features that contain key information, particularly in applications such as text classification.Feature selection is recommended as a response to both high dimensionality and class imbalance.
  • Noise: Noise has greater impact on least-represented examples, and studies report that class noise generally affects learners more significantly than imbalance.The interaction between imbalance and class noise was also examined in intrinsically imbalanced data.
  • Small disjuncts: Small disjuncts have low coverage, bias learners toward larger subclusters, and increase misclassification of cases in the smaller subclusters.Under class imbalance, error concentration toward small disjuncts increases; balancing the training distribution decreases that concentration.
  • Combined effects: The conjunction of related problems with imbalanced domains tends to further degrade classifier performance, so their relationship should not be ignored.The related problems considered include small disjuncts, class overlap, and small sample size.

6 Conclusions

The paper formulates imbalanced predictive modelling for both classification and regression and surveys methods for obtaining and evaluating models. It also introduces a taxonomy of approaches and examines related problems connected to imbalance.

  • Scope: The survey covers solutions for obtaining and evaluating predictive models under imbalanced distributions in both classification and regression tasks.It presents a formulation of the problem intended for both task types.
  • Taxonomy: The paper’s taxonomy groups existing approaches into data pre-processing, special-purpose learning methods, and prediction post-processing.These three categories organize the surveyed approaches to imbalanced domains.
  • Regression coverage: Because most existing solutions and previous surveys focus on classification, this survey additionally addresses approaches to imbalanced data sets within regression tasks.The paper describes this broader coverage as an innovative aspect and the first survey to include regression approaches.
  • Related problems: The paper also describes problems strongly related to imbalanced distributions and highlights research examining their relationships with imbalanced data sets.The related problems are treated as part of the survey’s broader analysis of imbalanced predictive modelling.
Loading 1505.01658v2…