PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework

Gabriel Aguiar, Bartosz Krawczyk, Alberto Cano

arXiv:2204.03719v2cs.LG

TL;DR

Imbalanced data streams lack standardized, holistic evaluation procedures despite combining class skew with drift and other evolving difficulties. The paper introduces a reproducible benchmark framework and compares state-of-the-art methods across diverse scenarios, finding distinct trade-offs between imbalance robustness and adaptation while identifying leading architecture patterns and practical boundaries.

  • Problem

    Imbalanced data-stream research lacks standardized procedures and benchmarks for holistic evaluation across algorithms, data difficulties, and binary and multi-class settings.

  • Method

    The paper builds a reproducible framework with varied benchmark problems, standardized evaluation tools, and comparisons of streaming classifiers across evolving scenarios.

  • Results

    The study finds trade-offs between adapting to change and robustness to imbalance; ROSE and SRP show balanced performance, while bagging or hybrid architectures with effective drift adaptation perform strongly.

  • Takeaways & Limitations

    The framework provides a reusable basis for fair comparisons, recommendations, and future method development in imbalanced data streams.

  • Takeaways & Limitations

    Existing resampling methods work mainly in binary streams, while some classifiers degrade as the number of classes increases and often assume a fixed class set.

Abstract

from arXiv · show

Class imbalance poses new challenges when it comes to classifying data streams. Many algorithms recently proposed in the literature tackle this problem using a variety of data-level, algorithm-level, and ensemble approaches. However, there is a lack of standardized and agreed-upon procedures and benchmarks on how to evaluate these algorithms. This work proposes a standardized, exhaustive, and comprehensive experimental framework to evaluate algorithms in a collection of diverse and challenging imbalanced data stream scenarios. The experimental study evaluates 24 state-of-the-art data streams algorithms on 515 imbalanced data streams that combine static and dynamic class imbalance ratios, instance-level difficulties, concept drift, real-world and semi-synthetic datasets in binary and multi-class scenarios. This leads to a large-scale experimental study comparing state-of-the-art classifiers in the data stream mining domain. We discuss the advantages and disadvantages of state-of-the-art classifiers in each of these scenarios and we provide general recommendations to end-users for selecting the best algorithms for imbalanced data streams. Additionally, we formulate open challenges and future directions for this domain. Our experimental framework is fully reproducible and easy to extend with new methods. This way, we propose a standardized approach to conducting experiments in imbalanced data streams that can be used by other researchers to create complete, trustworthy, and fair evaluation of newly proposed methods. Our experimental framework can be downloaded from https://github.com/canoalberto/imbalanced-streams.

1 Introduction

Data streams require learning methods that adapt to high-volume, high-velocity, evolving data, while simultaneous class imbalance and concept drift complicate evaluation. This paper addresses the lack of holistic standards with a reproducible framework and large comparison of streaming classifiers.

  • Data stream challenges: Data streams differ from static datasets through their volume, velocity, and non-stationarity, requiring incremental and adaptive learning.Concept drift can degrade classifiers when previously learned knowledge becomes outdated.
  • Imbalanced-stream challenges: Simultaneous class imbalance and concept drift can change class definitions, imbalance ratios, and majority–minority roles over time.Fixed-property solutions are unsuitable for these changing conditions.
  • Evaluation gap: Existing evaluations often cover limited algorithms, data difficulties, and binary datasets, lacking standardized holistic procedures and benchmarks.The paper identifies a need to translate diverse stream difficulties into meaningful benchmark problems.
  • Framework: The proposed framework standardizes metrics, statistical tests, and visualization tools for transparent and reproducible evaluation of binary and multi-class streams.It is designed as a reusable template for evaluating new classifiers against relevant methods.
  • Empirical study: The study compares 24 state-of-the-art algorithms across 515 benchmark problems covering dynamic imbalance, instance difficulties, real-world and semi-synthetic data, and multiple class settings.The framework also supports recommendations about classifier strengths and weaknesses and identifies open challenges.

3 Imbalanced data

Imbalanced learning concerns skewed class distributions that bias standard classifiers toward majority classes. Difficulty is further shaped by minority sample scarcity, class overlap, and heterogeneous instance-level challenges.

  • Class distribution: Higher imbalance ratios and minority scarcity can bias training toward majority classes and reduce minority-class generalization.Small minority samples may also fail to represent the actual distribution, producing data shift.
  • Class topology: Class overlap creates uncertainty regions where majority examples can dominate decision-boundary learning.Skewed distributions intensify the difficulty caused by overlapping class regions.
  • Instance-level difficulty: Minority classes may form small disjuncts or subconcepts, reducing local sample sizes and creating instance-level differences in difficulty.A neighborhood-based taxonomy distinguishes safe, borderline, rare, and outlier instances.

4 Imbalanced data streams

Imbalanced data streams combine skewed distributions with temporal changes such as drift, shifting class roles, and emerging or disappearing classes. The paper surveys data-level and algorithm-level responses, while highlighting limits of direct static-data techniques.

  • Stream-specific challenges: Stream imbalance may evolve with concept drift, changing class roles, class definitions, instance difficulties, and per-class arrival frequencies.Monitoring class properties alone is insufficient when incoming class frequencies also change.
  • Approach taxonomy: Data-level approaches rebalance streams through resampling, whereas algorithm-level approaches modify learners using cost sensitivity or one-class classification.Data-level methods are more classifier-agnostic; algorithm-level methods are more specialized and less flexible.
  • Resampling: Static resampling cannot be transferred directly to streams because concept drift can make resampled data obsolete or reinforce the wrong class after role switches.Streaming methods therefore need to track which classes should be resampled over time.
  • Resampling: Informed streaming resampling methods commonly extend SMOTE or use adaptive windows to track changing stream conditions.Blind methods are computationally efficient for ensembles but generally perform poorly alone.
  • Alternative data-level methods: Dynamic low-dimensional embeddings can use imbalance and separability information to seek discriminative projections, potentially reducing the need for resampling.This approach is motivated especially by high-dimensional and difficult streams.
  • Algorithm-level methods: Training modifications include skew-insensitive Hoeffding trees, online prior adjustment, and loss functions designed for imbalanced streams.These methods alter predictions, training criteria, or neural-network losses rather than the incoming dataset.
  • Algorithm-level methods: Cost-sensitive streaming methods adapt costs or thresholds online, sometimes combining misclassification costs with accuracy criteria.Examples include cost-sensitive trees, perceptrons, and extreme learning machines.
  • Related domains: One-class classifiers model minority classes or decompose multiclass problems and can react effectively to concept drift.The surveyed literature also connects imbalanced streams with continual learning and long-tailed recognition.

5 Ensembles for imbalanced data streams

Ensembles address stream imbalance and drift by combining diverse or resampled classifiers and adapting their composition over time. Their designs vary by pool generation, feature space, line-up management, and combination strategy.

  • Role of ensembles: Ensembles can improve predictive performance, generalization, and robustness while managing drift through classifier updates or additions.For imbalance, ensemble methods combine base learners with dedicated skew-handling mechanisms.
  • Classifier pool generation: Classifier pools are heterogeneous when they use different model types and homogeneous when they diversify one model type through training-set modifications.Both strategies seek complementary decision boundaries or classifier diversity.
  • Feature space modification: Feature-space design may use all features, feature subspaces, or newly constructed embeddings for base classifiers.Feature modification is one independent axis of ensemble construction.
  • Ensemble line-up: Ensemble line-ups may be fixed, dynamically pruned and replaced, or hybrid, with voting used to adjust base-learner importance.Dynamic replacement targets responsiveness to concept drift.
  • Imbalance-aware ensembles: Resampling is frequently combined with Online Bagging and related architectures such as Adaptive Random Forest, Online Boosting, and Dynamic Weighted Majority.The taxonomy connects ensemble construction with approaches for drifting and imbalanced streams.
  • Adaptive combination: Combination schemes can update classifier weights according to minority-class competence or use reinforcement learning to favor stronger minority-class performers.Other surveyed ensembles address limited labels by selecting instances or inducing classifiers dynamically.
  • Neural-network ensembles: Neural-network ensembles include balanced-subset training, cost-sensitive initialization, and incremental improvement for imbalanced streams.Most surveyed ensembles instead use Hoeffding trees or their variants as base classifiers.

6 Experimental setup

The experiments evaluate stream classifiers across controlled binary and multi-class imbalance scenarios, using reproducible protocols, diverse generators, and multiple performance metrics. The setup combines test-then-train evaluation with metrics that capture agreement, class-balanced performance, and ranking ability.

  • The study addresses six questions covering imbalance severity, static versus dynamic ratios, instance-level difficulty, concept drift, real-world streams, and accuracy–resource trade-offs.
  • The benchmark suite spans binary and multi-class streams with static and dynamic imbalance, instance-level difficulties, concept drift, and real-world datasets.
  • The evaluation compares 24 state-of-the-art stream algorithms, with taxonomies describing their cost-sensitive, training-modification, resampling, base-learner, and hybrid properties.
  • Controlled experiments use ten generators, including nine MOA generators and one proposed by Brzeziński, with generator-specific attributes, classes, and drift capabilities.
  • Each instance is tested before online training, and performance is measured with Accuracy, Kappa, G-Mean, AUC, PMAUC, WMAUC, and EWMAUC.
  • Kappa adjusts agreement for chance and class distributions, whereas AUC is invariant to class-distribution changes and PMAUC extends AUC to multi-class streams.

7 Results

The results examine binary imbalanced streams across static and dynamic imbalance, instance difficulties, drift, and real-world settings, using both detailed and aggregated comparisons. Under static imbalance, methods often trade strong performance on one metric for more balanced behavior across G-Mean and Kappa.

  • The results are presented at three levels: detailed comparisons of the top five methods, aggregated comparisons of the top ten, and summaries covering all methods.
  • The binary experiments cover static and dynamic imbalance, instance-level difficulties, concept drift, and real-world imbalanced datasets.
  • Static imbalance ratio: Blind resampling methods generally lose performance as static imbalance increases; UOB maintains G-Mean but its Kappa deteriorates substantially.The passage attributes this discrepancy to aggressive majority-instance removal under extreme imbalance, which can increase false positives.
  • Static imbalance ratio: CSARF achieves the best G-Mean among algorithm-level methods but does not preserve that advantage on Kappa, while ROSE and CALMID offer more balanced and robust performance.
  • Static imbalance ratio: Algorithm selection is nontrivial because some methods excel on one metric, whereas others provide more even performance across G-Mean and Kappa.
  • Static imbalance ratio: LB is the strongest standard ensemble in the reported comparison, while SRP and ARF show limited robustness as imbalance ratios increase.

Discussion

Under dynamic imbalance and instance-level difficulty, performance depends strongly on adaptation, ensemble design, and the structure of minority instances. Rare instances and evolving minority clusters are especially challenging, while bagging, hybrid, and general-purpose ensembles often remain robust.

  • Dynamic class imbalance: ROSE provides balanced performance under dynamic imbalance, while CSARF leads G-Mean but suffers under Kappa.ROSE’s adaptation mechanisms and dynamic sliding windows improve robustness under high imbalance ratios and class-role flipping.
  • Dynamic class imbalance: Boosting-based ensembles perform worse under evolving class proportions because their members may be trained on different class ratios.This can reinforce the small-sample problem for minority classes.
  • Dynamic class imbalance: Sudden and gradual imbalance drift produce similar long-run G-Mean and Kappa performance, although some methods recover more slowly without explicit adaptation.Class-role flipping is substantially more damaging for ARF and CSARF because their adaptation mechanisms do not handle changing class roles effectively.
  • Instance-level difficulty: SMOTE-based methods generally fail on borderline and rare instances because non-homogeneous neighborhoods make oversampling amplify uncertainty.The problem is most severe for rare instances, which may lack spatial and temporal coherence in streams.
  • Instance-level difficulty: LB, ARF, and KUE robustly handle difficult instances and evolving minority clusters, often outperforming dedicated imbalance methods even at IR = 100.Their instance subsampling can reduce neighborhood overlap and uncertainty, while their drift mechanisms help with evolving clusters.
  • Instance-level difficulty: Rare instances and splitting or merging minority clusters cause the largest performance drops because they combine small sample sizes, overlap, noise, and loss of coherent structure.Most methods are not insensitive to both changing imbalance ratios and instance-level difficulty.

Discussion

Concept drift exposes a trade-off between handling changing decision boundaries and handling class imbalance. The strongest approaches combine skew-insensitive mechanisms with explicit or implicit adaptation, while boosting remains comparatively weak.

  • Approach to class imbalance: OOB and UOB lose substantial performance under concept drift, while informed resampling offers only slight improvement over blind resampling.The results indicate that adaptation mechanisms matter more than the resampling strategy for evolving decision boundaries.
  • Approach to class imbalance: ARF, ARFR, LB, and SRP perform decently under concept drift, but their performance declines as the imbalance ratio increases.This exposes the limitation of drift-focused methods when class skew becomes more severe.
  • Ensemble architecture: Boosting methods perform worst under concept drift because drift destabilizes classifier chains and makes earlier errors less informative for subsequent updates.The paper identifies improved drift adaptation as necessary for boosting ensembles to become competitive with bagging.
  • Ensemble architecture: Hybrid architectures increasingly dominate because they combine bagging with dynamic base-classifier weighting and dynamic ensemble line-ups.The combination is presented as necessary for handling class imbalance and concept drift simultaneously.
  • Relationship between drift and imbalance: ROSE, CSARF, and SMOTE-OB are the strongest all-round methods because they combine drift adaptation with mechanisms that reduce sensitivity to class skew.ARF, LB, and SRP cannot handle increasing imbalance ratios, whereas UOB and OOB cannot handle non-stationarity.

Discussion

When concept drift and dynamic imbalance occur together, successful methods require both adaptation and skew robustness. Real-world streams further show that memory, delayed or missing classes, and nonuniform evolution change which methods perform well.

  • Dynamic imbalance with concept drift: ROSE and CALMID remain robust under simultaneous concept drift and dynamic imbalance, while CSARF excels on G-Mean but has Kappa shortcomings.ROSE balances both metrics through drift detection and class-specific balanced buffers.
  • Dynamic imbalance with concept drift: Bagging and hybrid ensembles outperform boosting in drifting streams with evolving imbalance, especially when they dynamically replace ensemble members.Replacing weak members can be more effective than adapting existing members after rapid changes.
  • Concept-drift speed: Gradual drift is harder than sudden drift because the new concept arrives slowly, prolonging performance degradation and encouraging costly false adaptations.Sudden drift causes a sharper drop but provides stable new-concept data for faster recovery.
  • Real-world streams: Real-world streams expose different weaknesses from synthetic scenarios because instances are not stratified and classes may disappear temporarily.The experiment evaluates 19 real-world imbalanced data streams with uncontrolled combinations of learning difficulties.
  • Real-world streams: Blind-resampling ensembles such as OOB and UOB perform poorly on real-world streams, whereas SMOTE-OB provides balanced and stable results across G-Mean and Kappa.Online resampling can suffer catastrophic forgetting when recent arrivals temporarily overrepresent one class.
  • Real-world streams: ROSE performs especially well on real-world data, while ARF ranks third on Kappa, suggesting that concept-drift adaptation can matter more than class-imbalance handling in these datasets.Bagging and hybrid ensembles remain superior overall, although boosting narrows its performance gap in real-world conditions.

Discussion

Multi-class imbalance differs substantially from binary imbalance and requires mechanisms designed for multiple class relationships. ARF-based ensembles and class-aware buffering perform best, while blind resampling and several specialized methods remain weak.

  • Class-imbalance approach: Multi-class performance differs significantly from binary performance, showing that binary imbalance mechanisms do not transfer reliably to many classes.The paper identifies multi-class streams as requiring dedicated mechanisms, particularly when the number of classes is large.
  • Class-imbalance approach: Blind resampling degrades in multi-class streams: OOB becomes mediocre and UOB loses acceptable predictive power.These methods struggle to capture complex class relationships and can intensify overlap and noise.
  • Classifier performance: ARFR ranks first on Kappa and second on PMAUC, while CSARF leads G-Mean but incurs more false positives under Kappa.ARFR’s weighting assigns importance to every tree according to class distribution, independent of the number of classes.
  • Classifier performance: The three best classifiers are ARF-based, and ROSE is the only training-modification method with good multi-class results.ROSE’s class-specific buffers appear well suited to multi-class imbalance, whereas HDVFDT and GHVFDT rank among the weakest.
  • Ensemble architecture: Bagging ensembles dominate multi-class performance, while KUE and SRP handle static multi-class imbalance satisfactorily.LB is the only underperforming bagging method noted in this comparison.

Discussion

Under multi-class imbalance and concept drift, classifier performance depends on how methods handle changing class distributions and evolving decision boundaries. Bagging, hybrid designs, and suitable drift adaptation generally perform better, while blind resampling is vulnerable.

  • Class imbalance approach: Dynamic imbalance can yield higher average classifier performance than static skew because class roles rotate, exposing each class as majority over time.This rotation can counteract the small-sample problem in multi-class streams.
  • Class imbalance approach: Blind resampling is unreliable: OOB remains acceptable, whereas UOB loses predictive power because undersampling worsens small-sample difficulty.UOB therefore cannot exploit stream statistics when a minority class becomes majority.
  • Class imbalance approach: ARFR, ROSE, and CALMID show strong robustness under evolving imbalance, while CSARF performs well on PMAUC but less satisfactorily on Kappa.ROSE provides the most balanced results across both metrics in this scenario.
  • Ensemble architecture: Bagging-based ensembles cluster among the strongest methods, while CALMID and MICFOAL improve performance through hybrid architectures designed for multi-class streams.These results identify ensemble architecture as an important factor under evolving imbalance.
  • Drift speed: Imbalance-change speed has a marginal effect on most classifiers; PMAUC distinguishes gradual from sudden changes more clearly than Kappa.Methods without drift handling respond more slowly, but this does not produce major performance differences.
  • Concept drift: Concept drift increases multi-class difficulty because classifiers must adapt to complex, changing relationships among several classes.The experiment evaluates classifiers under concept drift with static imbalance using generated streams and additional concatenated streams.

Discussion

Combining concept drift with dynamic imbalance and many classes exposes substantial weaknesses in blind resampling and some cost-sensitive methods. ROSE is especially robust, while effective adaptation and ensemble architecture remain central to performance.

  • Concept drift and shifting imbalance: Concept drift combined with evolving imbalance severely degrades OOB and makes UOB the worst-performing classifier in this scenario.Blind oversampling cannot adapt well to simultaneous changes in feature space and class characteristics.
  • Concept drift and shifting imbalance: ROSE is the clear winner under concept drift and dynamic class imbalance, adapting to new concepts even when class roles change.Its adaptation and skew-insensitive mechanisms support robustness to both types of change.
  • Ensemble architecture: SRP and LB can outperform specialized multi-class imbalance methods, indicating that drift adaptation may matter more than skew-specific design in this difficult setting.KUE also reaches performance similar to dedicated skew-insensitive ensembles.
  • Concept drift speed: Most classifiers recover quickly from sudden concept drift, so drift speed produces local differences without lasting effects on final performance.The analysis finds no clear relationship between drift speed and classifier performance.
  • Concept drift and shifting imbalance: Only ROSE and SRP balance adaptation to concept changes with robustness to shifting class imbalance.Most existing methods trade off one capability against the other.
  • Number of classes: ARFR maintains performance as the number of classes increases, whereas CSARF, OOB, and UOB lack robustness in high-class-count settings.ROSE and MICFOAL also show strong robustness, with ROSE a safe choice particularly under Kappa.
  • Ensemble architecture: Hybrid and bagging-based architectures dominate many-class experiments, with ROSE benefiting from feature and instance subspaces plus per-class memory buffers.The experiment evaluates classifiers under concept drift and dynamic imbalance across varying class counts.
  • Number of classes: Performance deteriorates as class counts rise, with a clear break point above 20 classes and highly unsatisfactory results for most methods at 30 classes.The decline reflects difficulty memorizing new concepts and generalizing across all classes.

Discussion

Real-world and semi-synthetic multi-class streams reveal that no classifier dominates every benchmark, although hybrid, resampling, and general-purpose ensembles each show strengths. Semi-synthetic streams particularly separate methods under rapidly changing imbalance and drift.

  • Real-world datasets: On real-world multi-class streams, no evaluated classifier is consistently better than its peers, and PMAUC is similar across classifiers while Kappa differentiates them more.Real-world data combine multiple challenges beyond multi-class imbalance.
  • Real-world datasets: UOB performs unsatisfactorily on real-world data, while OOB works better mainly when the number of classes is relatively small.These findings reinforce the weakness of blind resampling for high-class-count problems.
  • Real-world datasets: CSARF excels on PMAUC but trails on Kappa, whereas ROSE, CALMID, and MICFOAL provide highly satisfactory real-world performance.ROSE performs less strongly than in more difficult controlled scenarios, where its mechanisms are especially useful.
  • Semi-synthetic datasets: Semi-synthetic streams preserve real-world characteristics while adding rapidly evolving imbalance, producing two clusters of classifiers by predictive performance.These benchmarks combine real-world data with critical class-ratio changes and concept drift.
  • Semi-synthetic datasets: Oversampling outperforms undersampling on semi-synthetic streams, while ARFR remains close to standard ARF, highlighting the strength of its core design.This result agrees with the benchmark authors’ conclusion that smart oversampling is effective.
  • Semi-synthetic datasets: CSARF achieves the best PMAUC on semi-synthetic streams, while ROSE, CALMID, and MICFOAL handle rapid imbalance changes effectively.The hybrid mechanisms remain effective when real-world data contain rapidly changing class ratios.
  • Semi-synthetic datasets: Adding sudden and extreme changes widens the gap between best and worst methods, while SRP performs exceptionally well on these real-world benchmarks.The result suggests potential in low-dimensional representations for imbalanced streams.

Discussion

Across binary and multi-class benchmarks, ROSE offers the strongest overall balance, while CSARF is highly predictive but less balanced across classes. Practical selection must jointly consider predictive metrics, runtime, and memory because lightweight and accurate methods do not always coincide.

  • Classification metrics: G-Mean and PMAUC are strongly correlated, whereas Kappa adds information about whether performance is balanced across all classes.Kappa helps distinguish classifiers favoring minority classes from those providing balanced performance.
  • Overall performance: ROSE and CSARF are the two best overall classifiers, with ROSE strongest on Kappa and CSARF strongest on selected G-Mean or PMAUC evaluations.CSARF’s rank drops under Kappa and it becomes unsuitable when the number of classes is very high.
  • Overall performance: ARFR ranks among the strongest multi-class classifiers, while SMOTE-OB and OOB are particularly strong or competitive in binary settings.SMOTE-OB does not support multi-class problems, and OOB is close to average there.
  • Overall performance: OADA performs worst on classification metrics in both settings, suggesting limitations of boosting-based ensembles under multiple learning difficulties.The study attributes low predictive power to destabilization of the boosting procedure.
  • Computational and memory complexity: HDVFDT and GHVFDT use little memory and run quickly, but lightweight single-tree classifiers are generally less competitive than ensembles.Resource-efficient methods may therefore sacrifice predictive performance.
  • Computational and memory complexity: Oversampling-based methods require the highest computational resources because they enlarge the stream with artificial instances, while SMOTE adds nearest-neighbor cost.These costs can make high-performing resampling methods impractical under resource constraints.
  • Overall trade-off: The strongest classifiers are often opposed to the most resource-efficient ones, so algorithm choice depends on available memory and computation.SMOTE-OB can require more than 256GB of RAM per run despite excellent classification results.
  • Overall trade-off: ROSE provides the best overall trade-off between predictive performance and complexity under equal weighting of these criteria.OOB, UOB, ROSE, and CALMID offer strong trade-offs, but ROSE achieves the best combined result.

9 Recommendations

The recommendations emphasize matching algorithms and evaluation practices to the specific dynamics and difficulties of imbalanced data streams rather than relying on one universal solution. They also advocate reproducible, holistic comparisons using complementary metrics and appropriately selected benchmarks.

  • Algorithm selection: ROSE, CSARF, OOB, ARFR, and CALMID are recommended as effective off-the-shelf classifiers, with ROSE especially recommended for balanced performance.These recommendations are based on the exhaustive experimental study.
  • Imbalance dynamics: Static and evolving imbalance ratios favor different strategies: oversampling and training modification work well for static ratios, while dynamic ratios benefit from adaptive ensembles with pruning.ROSE is identified as an example of an algorithm-level solution for evolving imbalance ratios.
  • Concept drift: Concept drift adaptation should be examined explicitly because skew-insensitive classifiers can suffer, while general-purpose classifiers may perform well in specific non-stationary cases.The recommendation is to embed an efficient concept drift adaptation mechanism in new methods and use general-purpose ensembles as references.
  • Instance-level difficulties: Instance-level difficulties should be analyzed because oversampling methods tend to handle them well, although existing methods do not directly exploit such instances to improve adaptation and robustness.These difficulties include challenging instance structures that can substantially affect classifier behavior.
  • Number of classes: Binary and multi-class streams require different considerations: some methods degrade as class counts rise, resampling methods translate poorly, and emerging or disappearing classes need dedicated mechanisms.ROSE is reported to work well across class counts, whereas CSARF is sensitive to the number of classes.
  • Evaluation practice: New methods should use fair, holistic, and reproducible evaluations with popular and recent reference classifiers, complementary metrics, and benchmarks selected for the study rather than applied universally.The framework provides code, classifiers, datasets, metrics, statistical tests, and visualization tools; not every benchmark is necessary or suitable for every study.

10 Open challenges and future directions

The paper identifies open directions spanning faster resampling, proactive adaptation, evolving classes, fairness, feature selection, and broader imbalanced-stream tasks. It also emphasizes standardized, reproducible evaluation as a foundation for future research.

  • Informative and fast resampling: Current undersampling methods underperform, especially for multi-class streams, while SMOTE-based oversampling improves predictive power at high computational cost.Future resampling methods should adapt to concept drift and dynamic imbalance while reducing computational complexity and avoiding nearest-neighbor dependence.
  • Proactive instead of reactive tackling of dynamic class imbalance: Proactive methods could anticipate how class imbalance evolves instead of adapting only after changes occur.The paper argues that anticipation could reduce recovery time and produce more robust classifiers.
  • Improving boosting-based ensembles: Boosting-based ensembles perform poorly for imbalanced streams, motivating architectures that adapt quickly and handle small minority samples.The identified weaknesses are intended to guide development of more suitable boosting classifiers.
  • Handling evolving number of classes: Evolving class counts require models that detect new classes, forget outdated ones, and remember recurring classes under dynamic imbalance.The paper connects this challenge with continual and lifelong learning approaches.
  • Broader future directions: Future work should address fairness, online skew-insensitive feature selection, and imbalanced tasks beyond binary and multi-class classification.These directions include reducing majority-class bias, supporting multi-label outputs and rare-value regression, and handling other stream-learning tasks.
  • Reproducible research: The reproducible framework evaluates 24 algorithms across 515 benchmarks and provides a standardized template for future classifier comparisons.Its scenarios combine static and dynamic imbalance, concept drift, data-level difficulties, real-world problems, and binary or multi-class settings.
  • Reproducible research: The framework is intended to grow through community additions of algorithms, problems, and benchmarks, while many domain questions remain open.The authors present community growth as part of advancing reproducible research in data stream mining.

Acronyms

The paper uses a broad set of acronyms for adaptive forests, online resampling and boosting, synthetic oversampling, streaming ensembles, and specialized multiclass or concept-drift methods.

  • Adaptive forests: ARF and ARFR denote Adaptive Random Forest and Adaptive Random Forest with Resampling, respectively.
  • Specialized imbalance methods: CALMID, CSARF, and C-SMOTE denote multiclass active learning, cost-sensitive adaptive forests, and continuous synthetic minority oversampling.
  • Specialized learners: ESOS-ELM denotes Ensemble of Subset Online Sequential Extreme Learning Machine.
  • Trees and stream learners: GHVFDT and HDVFDT denote Gaussian Hellinger and Hellinger Distance Very Fast Decision Trees, while IRL, KUE, LB, and MICFOAL denote incremental, updated-ensemble, leveraging-bagging, and online active-learning methods.
  • Online ensembles: OADA, OADAC2, OBA, OOB, ORUB, OSMOTE, OUOB, and UOB denote online boosting, bagging, resampling, and undersampling methods.
  • Oversampling and ensemble methods: ROSE, SMOTE, SMOTE-OB, SRP, and VFC-SMOTE denote self-adjusting ensembles, synthetic oversampling, online bagging, random patches, and continuous oversampling methods.
Loading 2204.03719v2…