Challenges in Benchmarking Stream Learning Algorithms with Real-world Data

• Stream algorithm comparisons are hindered by inconsistent recommendations, evaluation procedures, datasets, assumptions, and limited public non-stationary real-world data.Unlike batch learning, stream mining lacks a reasonably sized public repository of real-world streams with detailed descriptions, and dataset suitability for incremental or abrupt changes is often unclear.Source: §1 · p. 2
• The article explains data-stream concepts and the characteristics, causes, and issues underlying different types of distribution change.Source: §1 · p. 2
• It reviews commonly used real-world stream datasets and critically discusses data issues and the absence of a benchmark standard.Source: §1 · p. 3
• The authors build 11 real-world datasets from an optical-sensor public-health application, with controlled concept changes whose types, patterns, and occurrence times are identifiable.The application concerns recognition of disease-vector insects from evolving optical-sensor data, and the datasets include evaluations with state-of-the-art classifiers and drift detectors.Source: §1 · p. 3
• They develop and make available a repository containing 27 real-world datasets for benchmarking stream classifiers and change detectors.Source: §1 · p. 3

2.1 Data Stream

• Data streams are continuously observed ordered instances from real-world applications, where classification associates each feature vector with a class label.Their potentially infinite length, real-time processing requirements, and changing distributions make conventional batch methods unsuitable.Source: §2.1 · pp. 3–4

2.2 Concept Drift

• Concept drift is a change in the underlying data distribution, appearing as abrupt, gradual, incremental, or recurring transitions, while isolated outliers and noise are excluded.Sliding windows provide an indirect way to analyze drift, but their size and overlap determine which distributions and concepts are visible.Source: §2.2 · p. 5

2.3 Independent Distribution

• Real-world streams generally violate i.i.d. assumptions because distributions change over time and temporal dependence may connect current labels or attributes to historical observations.Distinguishing temporal dependence from independent-example streams is essential because time-series change detection and stream drift detection require different approaches.Source: §2.3 · pp. 5–6

2.4 Types of Concept Drifts

• Dataset shifts distinguish changes in features, class proportions, or feature-label relationships, with concept shifts potentially combining these changes and often remaining difficult to detect without labels.Temporal overlap can exaggerate class overlap when concepts are not separated by time, whereas intermediate distributions can make some otherwise undetectable shifts traceable.Source: §2.4 · pp. 7–9

2.5 Data Stream Realization

• Stream realizations differ between materialization sequences, where observations depend strongly on prior values, and observational sequences, where sampled instances are individually important under a shared evolving concept.Sequences may be temporal, spatial, or logical, and irregular spacing between observations creates additional complications for stream analysis.Source: §2.5 · p. 11
• Temporal or spatial marks are essential for interpreting concept drift when observations follow environmental progression; unordered sequences lack drift, making forgetting mechanisms unnecessary.Mosquito activity illustrates how missing timestamps can make behavior changes appear abrupt, whereas timestamps reveal prolonged observation gaps and changing conditions.Source: §2.5 · p. 12

2.6 Stream Classification

• Stream classification predicts labels online as examples arrive, requiring models to evolve or adapt as distributions change; evolving methods forget continuously, whereas adaptive methods update when drift is detected.VFDT incrementally updates tree statistics without adapting to distribution changes, while CVFDT grows and replaces alternative subtrees when new splits become more accurate.Source: §2.6 · pp. 13–14

2.7 Drift Detection

• Drift detection is categorized by label availability and method: supervised versus unsupervised, distribution comparisons, sequential analysis, and statistical process control.Detection performance depends strongly on window sizing: windows smaller than the change rate can miss drifts, while larger windows delay detection; ADWIN and SEED use dynamic sub-windows.Source: §2.7 · pp. 14–15

• More than 150 papers show that stream-learning research relies on approximately 15 real-world datasets, limiting benchmark diversity compared with batch learning.The review was conducted over the last two decades to identify the most-used datasets.Source: §3 · p. 16
• The benchmark collection includes electricity-price, forest-cover, poker-hand, network-intrusion, flight-delay, gas-identification, and social-survey classification datasets, covering varied prediction tasks and domains.Electricity has 45,312 instances and eight attributes; Forest Covertype has 581,012 instances and 54 attributes; Airlines has 539,383 examples and seven attributes.Source: §3 · pp. 16–17
• Several datasets provide plausible temporal or environmental change, but others lack logical sampling order or were not originally designed as streaming data.The Gas Sensor Array data were independently sampled, divided into batches without consistent logical order, and associated drift with changing gas concentrations.Source: §3 · pp. 17–18
• Because public benchmark data are scarce, researchers simulate drift by switching features, swapping or joining classes, reordering data by hidden features, or using synthetic generators.Hidden-feature reordering can produce abrupt drift for nominal features and incremental drift for ordinal features.Source: §3 · pp. 19–20
• Simulating drift in static data or relying on generators can introduce data bias and produce incorrect conclusions in experimental evaluations.The paper defines data bias as consciously or unconsciously selecting data to confirm a desired finding.Source: §3 · p. 20

4 Criticisms to Current Datasets for Stream Learning

• The commonly used real-world stream datasets contain too few events and concept drifts, so evaluations based on them may be biased or incorrect.This shortage motivates scrutiny of benchmark dataset suitability before comparing stream classifiers and drift detectors.Source: §4 · p. 20

4.1 Uncertainty about Changes

• Existing datasets often lack verified drift presence, drift type, change pattern, timing, and annotations, limiting valid evaluation of detectors and potentially producing incorrect conclusions.Performance degradation may instead reflect poor generalization or noise; consequently, methods suited to gradual or abrupt changes often require artificial data for analysis.Source: §4.1 · p. 21

4.2 Temporal Dependence

• No-Change outperforms Naive Bayes with DDM on Electricity (85.33% vs 81.23%), Forest Covertype (95.07% vs 88.04%), and Poker-hand (74.51% vs 61.96%).These results show that temporal dependence can make simple label-persistence baselines appear stronger than adaptive classifiers, without demonstrating generalization or adaptation to change.Source: Figure 7 · p. 23, §4.2 · p. 23

4.3 Data Bias

• Choosing different small dataset subsets reverses conclusions about DDM and CUSUM: DDM wins in one scenario, they are similar in another, and CUSUM wins in a third.The paper therefore calls for sufficiently large benchmarks covering diverse stream properties.Source: Figure 8 · p. 24, §4.3 · p. 24

4.4 Insufficient Amount of Instances

• Among 16 popular stream datasets, only Poker-hand and Sensor Stream exceed one million examples, while more than half contain fewer than 50,000.Such sizes are manageable for batch algorithms and generally do not meaningfully challenge stream processing, storage, or memory constraints.Source: §4.4 · p. 25

4.5 Lack of Complex Distributions

• Real stream benchmarks rarely represent changing class ratios and complex distributions involving both P(Y) and P(Y|X) or P(X), limiting realistic evaluation of imbalance.Electricity’s class distribution barely changes, while under-sampling a class modifies the real problem; existing imbalanced datasets also retain uncertainty or insufficient-size drawbacks.Source: §4.5 · pp. 26–27

4.6 Streaming as an afterthought

• Poker-hand, Gas Sensor Array, and Forest Covertype were not inherently designed as streams, with challenges arising from unexplained or potentially unnatural instance ordering.Poker-hand’s original data are independently and identically distributed, Gas Sensor Array examples are independently collected, and Forest Covertype ordering likely reflects unspecified geography.Source: §4.6 · pp. 27–28

5 A Real-world Streaming Application with Concept Drifts

• The proposed benchmark uses optical sensors to recognize flying insect species in real time under concept drift.Environmental conditions alter insect behavior and sensor measurements, motivating adaptive classification models.Source: §5 · p. 28

5.1 Smart Trap for Insects

• Smart traps classify and count target insects in real time without expert analysis, avoid capturing non-target species, and reduce power consumption.The trap combines optical sensing, signal filtering, recording, feature extraction, and supervised classification.Source: §5.1 · p. 29

5.2 Sensor to Measure Insect Flying Data

• The sensor records wing-induced infrared-light variations as audio signals while remaining largely insensitive to agents that do not cross its optical path.Wing-beat frequency and harmonic components provide information for distinguishing insect species.Source: §5.2 · pp. 29–30

5.3 Data Collection in Changing Environment

• Data were collected for approximately three months across controlled temperature and humidity conditions, yielding around one million instances from 17 species and 21 class labels.Temperature varied from 20°C to 40°C and humidity from 20% to 90%; the final stream datasets use six labels from three species and both sexes.Source: §5.3 · pp. 30–32

5.4 Features Extraction

• Because wing-beat frequency distributions overlap across species, the benchmark represents each signal with 33 features capturing spectral peaks, harmonics, and temporal statistics.Wing-beat frequency is distinctive but insufficient alone for accurate multi-species classification.Source: §5.4 · pp. 33–34

5.5 Insect Stream Data

• The benchmark contains 11 datasets formed from incremental, abrupt, gradual, recurrent, and out-of-control temperature or class-proportion changes, with balanced and imbalanced variants.The Out-of-control dataset has 24 labels and severe imbalance, including majority classes with 170,220 and 168,819 instances and minority classes with 17 and 157.Source: §5.5 · pp. 34–36

5.6 Temporal Overlap

• For two insect classes, overlap in wing-beat frequency was 36% across all temperatures but averaged 23% when each temperature was isolated.This temperature-conditioned reduction demonstrates temporal overlap caused by evolving temperature concepts and supports concept-specific modeling.Source: §5.6 · pp. 37–38, Table 3 · p. 38
• Individual temperature-specific classifiers averaged 90% accuracy versus 84% for one classifier trained across all temperatures, using 10-fold cross-validation and Random Forests with 200 trees.All 33 insect-dataset features were used; Table 4 reports the accuracy rates.Source: §5.6 · p. 38, Table 4 · p. 39
• At 24°C, the conjoint classifier caused a 20% accuracy reduction relative to the individual classifier, making it the most difficult and most affected temperature.The magnitude of temporal-overlap effects varied across temperatures.Source: §5.6 · p. 38

USP Data Stream Repository

• The repository provides 27 datasets spanning real problems, including 16 previously studied datasets and 11 new optical-sensor datasets for automatic insect recognition.The authors intend to expand the repository regularly through collaborative contributions.Source: §6 · p. 39
• Stream classifiers and drift detectors should be evaluated across broad datasets, especially real-world data, to reduce biased conclusions.The recommendation follows established practices in machine learning and time-series research.Source: §6 · p. 39
• The repository includes results from two simple baseline methods for every dataset to support performance analysis of new proposals.Comparisons with baseline methods such as those proposed by Bifet et al. (2013) are presented as essential.Source: §6 · p. 39

7 Evaluation and Discussion

• The experiments used MOA implementations, prequential evaluation with 1,000-instance windows, and representative classifiers including No-Change, Majority-Class, Naive Bayes, VFDT, Leveraging Bagging, and ARF.The baseline classifiers used only past labels, with Majority-Class relying on a 1,000-instance moving window.Source: §7 · p. 39, §7.1 · pp. 39–40

7.1 Classification

• Adaptive Random Forest achieved the best overall classification results, followed by Leveraging Bagging, with approximately 70–80% accuracy across drift patterns.Incremental VFDT and Naive Bayes were outperformed because they lack explicit concept-drift strategies.Source: Table 5 · p. 40, §7.1 · p. 40
• Classification performance varied by stream pattern: gradual drift caused sharp temporary declines, recurring concepts produced similar behavior across datasets, and incremental changes were hardest to adapt to.Incremental streams were more stable under slow changes, whereas imbalanced streams showed instability at points associated with changes in P(Y).Source: §7.1 · pp. 40–42
• On Out-of-control data, classifiers were relatively stable despite many undefined changes, but performance remained limited, with ARF reaching about 70% overall prequential accuracy.Source: §7.1 · p. 42, Figure 27 · p. 43

7.2 Drift Detection

• ADWIN and STEPD achieved the best general drift-detection results across the insect-stream datasets when evaluated with standardized Naive Bayes base classifiers and detector settings.Detectors used 1,000-example windows, at least 100 examples for detection, and MOA default values for remaining parameters.Source: §7.2 · pp. 43–44, Table 6 · p. 44
• STEPD detected all five abrupt changes in Abrupt data but also produced false detections during stable concepts; it identified 61 and 59 points in Incremental-abrupt-reoccurring and Incremental-reoccurring data.Only the final detections within concepts B, D, and F generally improved accuracy.Source: §7.2 · pp. 44–45

Challenges

• The lack of real-world benchmarks encourages synthetic-data-heavy evaluations, risking biased conclusions about stream classifiers and drift detectors.Existing real datasets often lack knowledge of change types, patterns, and occurrence times.Source: §8 · p. 46

New datasets

• The 11 optical-sensor datasets capture non-stationary insect behavior with incremental, abrupt, gradual, and recurring changes, change indicators, and complex class-distribution shifts.They support a public-health Smart Trap application for recognizing disease-vector insects and contain a significant number of instances.Source: §8 · p. 46

Limitations

• The proposed datasets have limitations because manual arrival-order manipulation and window shuffling may make them semi-real, despite preserving application meaning under simulated environmental changes.These procedures were used to identify drift points and types while avoiding temporal dependence.Source: §8 · p. 46
• The absence of timestamps restricts evaluation of time-constrained learning, including whether models update between arrivals and how algorithms use idle periods for updates.This limitation is also present in most datasets from the literature.Source: §8 · p. 47

Repository

• The USP Data Stream Repository provides 27 datasets: 16 previously used in literature and 11 new insect-recognition datasets, with baseline results and planned future expansion.The repository covers real problems and is intended for the broader machine-learning community.Source: §8 · p. 47