Source-linked AI summary
Challenges in Benchmarking Stream Learning Algorithms with Real-world Data
Vinicius M. A. Souza, Denis M. dos Reis, Andre G. Maletzke, Gustavo E. A. P. A. Batista
TL;DR
Stream-learning research needs more consistent evaluation because publicly available non-stationary real-world datasets are limited and experimental practices vary. The paper introduces the USP Data Stream Repository, combining literature datasets with new optical-sensor datasets whose change patterns are characterized, and reports benchmark results to support comparison and reproducibility. Across the datasets, Adaptive Random Forest achieved the best overall results, while the repository provides 27 datasets and accompanying baseline or literature-method results.
Problem
Stream-learning algorithms are difficult to compare because non-stationary real-world datasets and consistent evaluation practices are limited.
Method
The paper develops a public repository combining 27 real-world datasets, including 11 optical-sensor datasets with characterized concept changes, and provides benchmark evaluations.
Results
Adaptive Random Forest achieved the best overall results across the datasets, followed by Leveraging Bagging, with overall results around 70-80% for different drift patterns.
Takeaways & Limitations
The repository and accompanying results aim to make stream-learning experiments more comparable and reproducible.
Takeaways & Limitations
The evaluation can be affected by data bias, especially when few real datasets are combined with many synthetic datasets or when simulated drifts are used.
Abstract
from arXiv · showhide
Streaming data are increasingly present in real-world applications such as sensor measurements, satellite data feed, stock market, and financial data. The main characteristics of these applications are the online arrival of data observations at high speed and the susceptibility to changes in the data distributions due to the dynamic nature of real environments. The data stream mining community still faces some primary challenges and difficulties related to the comparison and evaluation of new proposals, mainly due to the lack of publicly available non-stationary real-world datasets. The comparison of stream algorithms proposed in the literature is not an easy task, as authors do not always follow the same recommendations, experimental evaluation procedures, datasets, and assumptions. In this paper, we mitigate problems related to the choice of datasets in the experimental evaluation of stream classifiers and drift detectors. To that end, we propose a new public data repository for benchmarking stream algorithms with real-world data. This repository contains the most popular datasets from literature and new datasets related to a highly relevant public health problem that involves the recognition of disease vector insects using optical sensors. The main advantage of these new datasets is the prior knowledge of their characteristics and patterns of changes to evaluate new adaptive algorithm proposals adequately. We also present an in-depth discussion about the characteristics, reasons, and issues that lead to different types of changes in data distribution, as well as a critical review of common problems concerning the current benchmark datasets available in the literature.
1 Introduction
Stream-learning research lacks standardized, publicly available non-stationary real-world datasets, making algorithm comparisons difficult. The paper addresses this gap through a repository combining existing and new datasets for benchmarking classifiers and drift detectors.
- The lack of publicly available non-stationary real-world datasets hinders consistent comparison and evaluation of stream-learning proposals.
- The paper also reviews stream-learning concepts, existing benchmark datasets, and challenges caused by inconsistent evaluation practices.
- The repository includes 11 new datasets from an optical-sensor application involving recognition of disease-vector insects.
- The paper proposes a repository containing 27 real-world datasets for benchmarking stream classifiers and change detectors.
- These new datasets have controlled concept changes whose types, patterns, and occurrence times can be identified for evaluation.
2 Background
Data streams are ordered, continuously arriving observations whose distributions can change over time, challenging conventional batch-learning assumptions and requiring models to adapt online. Concept drift varies in form and detectability, while temporal dependence, observation windows, and mixed time-series features complicate stream analysis and evaluation.
- Data Streams: Data streams are continuously observed sequences used in applications such as sensors, satellites, stock markets, and financial systems.Classification streams associate each feature vector with a class label.
- Data Stream Constraints: Potentially infinite length, real-time processing, limited memory, and changing distributions make traditional batch methods unsuitable for many streams.These restrictions motivate techniques such as sampling, sketching, histograms, wavelets, and sliding windows.
- Concept Drift: Concept drift is a change in the parameters of a non-stationary data distribution and may occur abruptly, gradually, or incrementally.Incremental drift passes through intermediary concepts, whereas gradual drift transitions smoothly between concepts.
- Concept Drift: Sliding windows provide an indirect view of drift, but their size and overlap affect which distributions are observable.Large windows can hide inner distributions, while small windows may contain too few examples to recognize concepts.
- Distributional Assumptions: Stream data often violate i.i.d. assumptions because distributions change over time and historical labels may exhibit temporal dependence.This dependence directly affects the design and evaluation of stream-learning approaches.
- Data Stream Realization: Combining time-series and independently distributed features can make statistical changes misleading because detected feature changes may not affect the target behavior.The paper notes that time-series change detection and drift detection require different approaches.
3 Stream Datasets from Literature
A review of stream-learning datasets shows that commonly used benchmarks span diverse applications but often provide limited scale and may not adequately represent challenging real-world drift. The paper also warns that simulated or selectively chosen data can introduce bias into evaluation.
- Dataset review: A literature review found that stream-learning benchmarks cover applications including electricity, environmental sensing, cybersecurity, flight delays, and gas detection.The reviewed datasets include electricity-market prediction, sensor identification, intrusion detection, flight-delay prediction, and chemical-gas measurements.
- Potential distribution changes: Several datasets encode temporal or environmental variation as possible sources of distribution change, including internet use, chess ratings, weather, lighting, seasons, and user preferences.Examples include survey responses collected over five years, chess games over multiple years, sensor readings, electricity demand, email interests, and image conditions.
- Dataset diversity: The reviewed datasets include diverse modalities and tasks, such as survey, game, sensor, image, keystroke, electricity, forest-cover, poker-hand, intrusion, and flight data.The collection ranges from binary classification to multiclass identification and prediction tasks with different feature types and collection periods.
- Evaluation concerns: Simulating drift or relying on generated data can introduce data bias and lead to incorrect conclusions in experimental evaluation.The paper defines data bias as consciously or unconsciously selecting data to confirm a desired finding.
4 Criticisms to Current Datasets for Stream Learning
Current stream-learning benchmarks are limited by uncertain drift characteristics, biased dataset selection, short streams, simple distributions, and datasets not designed for streaming. These issues can make algorithm comparisons and conclusions unreliable.
- 4.1 Uncertain Changes: Existing datasets often have unknown drift types, patterns, and locations, limiting evaluation of detection accuracy, false alarms, and detection delay.Performance degradation may also arise from poor generalization or noise rather than concept drift.
- 4.3 Data Bias: Reduced dataset choices can reverse which drift detector appears superior, with DDM winning in one scenario, similar performance in another, and CUSUM winning in a third.The scenarios use different groups of three datasets, showing that conclusions depend strongly on dataset selection.
- 4.4 Insufficient Amount of Instances: Only two of 16 popular stream datasets contain more than one million examples, while over half contain fewer than 50,000 and pose little storage challenge.These sizes are often manageable by batch-learning algorithms, weakening the practical basis for memory-focused stream evaluation.
- 4.5 Lack of Complex Distributions: Many real datasets lack complex, time-varying class distributions, including changes in P(Y) and P(Y|X), which limits realistic evaluation of stream methods.Under-sampling can introduce class imbalance but may alter the original problem characteristics.
- 4.6 Streaming as an afterthought: Poker-hand is poorly suited to streaming because independently drawn hands lack an evolving hidden concept and temporal dependence in the original dataset.Its small size also permits batch processing, further weakening its suitability as a stream benchmark.
- 4.6 Streaming as an afterthought: The MOA Poker-hand version imposes temporal dependence through a different instance ordering, so the reordering should be disclosed and its limited challenge recognized.The authors characterize temporal dependence as the only introduced challenge and one of the least interesting streaming problems.
5 A Real-world Streaming Application with Concept Drifts
The paper presents a real-world optical-sensor application for recognizing flying insects under changing environmental conditions. It constructs insect stream datasets with known distribution patterns, including class overlap, imbalance, and evolving classes.
- 5 A Real-world Streaming Application with Concept Drifts: The proposed benchmark uses optical sensors to recognize flying insect species in real time for stream-classification and drift-detection research.The sensor records light variations caused by insects crossing an infrared light window and converts them into audio signals for feature extraction.
- 5 A Real-world Streaming Application with Concept Drifts: Environmental changes such as temperature, air pressure, and humidity can alter insect behavior, making a stationary data-generating process inappropriate.These conditions influence insect flight behavior and therefore the measured sensor data.
- 5 A Real-world Streaming Application with Concept Drifts: The collected stream contains around one million instances from 17 insect species, represented by 21 sex-specific or species-level class labels.Data were collected for approximately three months in a non-stationary environment.
- 5.6 Temporal Overlap: 90% versus 84%: temperature-specific Random Forest classifiers average 90% accuracy, compared with 84% for one classifier trained across all temperatures.At 24°C, the joint classifier is 20% less accurate than the individual classifier.
6 USP Data Stream Repository
The USP Data Stream Repository addresses inconsistent stream-learning evaluation by providing public real-world datasets and reference results. It combines established literature datasets with new optical-sensor insect datasets for more comparable and reproducible experiments.
- 6 USP Data Stream Repository: The USP Data Stream Repository provides 27 datasets for evaluating stream classifiers and drift detectors.The collection includes 16 datasets previously used in the literature and 11 new optical-sensor insect datasets.
- 6 USP Data Stream Repository: The repository is intended to reduce dataset-related flaws in future stream-learning evaluations caused by the scarcity of non-stationary real-world data.The paper recommends testing methods across a wide range of real datasets and comparing them with baseline methods.
- 6 USP Data Stream Repository: Reference results from state-of-the-art methods are reported on the proposed data to make experiments more comparable and reproducible.The repository therefore supplies both benchmark data and performance results from methods in the literature.
7 Evaluation and Discussion
The evaluation benchmarks stream classifiers and drift detectors on the insect datasets using prequential accuracy and change-point detection. Adaptive Random Forest and Leveraging Bagging perform best overall, while drift-detector results favor ADWIN and STEPD.
- 7.1 Classification: ARF achieves the best overall classification results across all datasets, followed by Leveraging Bagging, with results around 70–80% for different drift patterns.VFDT and Naive Bayes are outperformed because they do not explicitly address concept drift.
- 7.1 Classification: Gradual drift causes a drastic performance fall immediately before 15,000 instances while two concepts coexist during the transition.The balanced Incremental-gradual stream contains overlapping concepts until the new concept becomes complete.
- 7.1 Classification: Incremental changes generally make adaptation more difficult than abrupt changes when concepts recur in the stream.Results on Incremental-reoccurring data are similar to those on Incremental-abrupt-reoccurring data, suggesting abrupt changes are not the main source of performance impact in recurring concepts.
- 7.1 Classification: ARF reaches around 70% overall prequential accuracy on the Out-of-control dataset despite its many classes and undefined change patterns.The classifiers are relatively stable over time, but the best overall accuracy remains limited.
- 7.2 Drift Detection: ADWIN and STEPD achieve the best general drift-detector results on the insect stream data.The evaluation reports both overall prequential accuracy and the total number of detected changes.
- 7.2 Drift Detection: STEPD identifies 59 change points in the balanced Incremental-reoccurring dataset, where incremental changes make exact change-point localization difficult.The figure uses background gradients to represent incremental changes over time.
8 Conclusions
The paper addresses evaluation bias in stream learning by proposing a public repository that combines existing real-world datasets with 11 optical-sensor insect datasets. It also identifies important benchmark limitations, including manipulated arrival order and missing timestamps.
- 8 Conclusions: Using synthetic data and few real datasets can introduce bias and lead to incorrect conclusions about stream-algorithm behavior.The paper links this problem to the limited availability of real-world non-stationary data.
- 8 Conclusions: The proposed insect datasets include incremental, abrupt, gradual, and recurring changes, change indicators, complex class distributions, and many instances.These characteristics support evaluation of stream classifiers and drift detectors under varied non-stationary conditions.
- 8 Conclusions: The benchmark manually changes example arrival order and shuffles similar examples within windows to identify drift points and reduce temporal dependence.The authors state that these procedures do not affect the application’s meaning but acknowledge that they make the data semi-real or not entirely real.
- 8 Conclusions: The absence of timestamps restricts evaluation of time-constrained learning behaviors, such as updating models between arrivals or during idle periods.This limitation is also present in many datasets from the literature.
- 8 Conclusions: The repository provides 27 datasets, including 16 previously evaluated datasets and 11 new optical-sensor datasets for automatic insect recognition.It also reports results from two baseline methods for all datasets.