PaperlayerView exact revision on arXiv

Source-linked AI summary

Active learning for data streams: a survey

Davide Cacciarelli, Murat Kulahci

arXiv:2302.08893v4stat.MLcs.LGstat.ME

TL;DR

Obtaining labeled observations can be costly and time-consuming, while data streams call for online selection rather than choosing from a fixed pool. This survey reviews online active-learning strategies and concludes that research has concentrated on classification, with comparatively limited work on online active linear regression and regression models generally.

  • Problem

    Obtaining labeled observations can be costly and time-consuming, motivating methods that select informative data for labeling.

    Source: §1 · p. 1

  • Method

    The survey reviews online active-learning strategies, including uncertainty, diversity, query-by-committee, and reinforcement-learning approaches.

    Source: §7 · pp. 33–34

  • Results

    Existing research has predominantly focused on online classification for stationary and drifting streams, with comparatively limited work on online active linear regression or online regression models generally.

    Source: §6 · p. 32

  • Takeaways & Limitations

    The survey identifies potential for model-agnostic and single-pass online active-learning strategies suited to practical settings.

    Source: §7 · p. 34

    Deep active-learning methods proposed so far are not easily adapted to stream-based settings.

    Source: §5.1 · p. 30

Abstract

from arXiv · show

Online active learning is a paradigm in machine learning that aims to select the most informative data points to label from a data stream. The problem of minimizing the cost associated with collecting labeled observations has gained a lot of attention in recent years, particularly in real-world applications where data is only available in an unlabeled form. Annotating each observation can be time-consuming and costly, making it difficult to obtain large amounts of labeled data. To overcome this issue, many active learning strategies have been proposed in the last decades, aiming to select the most informative observations for labeling in order to improve the performance of machine learning models. These approaches can be broadly divided into two categories: static pool-based and stream-based active learning. Pool-based active learning involves selecting a subset of observations from a closed pool of unlabeled data, and it has been the focus of many surveys and literature reviews. However, the growing availability of data streams has led to an increase in the number of approaches that focus on online active learning, which involves continuously selecting and labeling observations as they arrive in a stream. This work aims to provide an overview of the most recently proposed approaches for selecting the most informative observations from data streams in real time. We review the various techniques that have been proposed and discuss their strengths and limitations, as well as the challenges and opportunities that exist in this area of research.

1 Introduction

The introduction frames online active learning as a way to reduce the cost of obtaining labeled data by selecting informative observations from streams. It positions the survey as a synthesis of newer query strategies and related evaluation, application, and research questions.

1 Introduction

  • Because labeling can be costly and time-consuming, active learning selectively queries informative observations to reduce labeling costs and improve model performance.Source: §1 · p. 1
  • Existing surveys largely cover pool-based learning or online learning with complete labels, leaving newer online active-learning strategies without a comprehensive synthesis.The survey aims to address that gap by reviewing methods for selecting stream observations to label online.Source: §1 · p. 2
  • The review considers query strategies, query timing, model updates under distribution change, scalability, and evaluation, alongside applications and future directions.It also outlines the paper's coverage of active-learning foundations, state-of-the-art methods, evaluation strategies, and applications.Source: §1 · p. 2

2 Preliminaries on active learning

The preliminaries establish how active learning selects labels under a query budget, distinguish common selection criteria and learning scenarios, and connect active learning with semi-supervised methods. They also contrast static-pool selection with online decisions over arriving data.

2 Preliminaries on active learning

  • Active learning selects a subset of unlabeled observations for labeling under a query budget, while the survey identifies its literature-search procedure.The authors searched SCOPUS and Google Scholar and reviewed papers individually for relevance.Source: §2 · p. 2

2.1 Instance selection criteria

  • Selection criteria include uncertainty, expected error or variance, expected model change, disagreement, diversity or density, and hybrids.Examples include margin-based selection near a decision boundary, query by committee, and combining margin sampling with clustering.Source: §2.1 · p. 3

2.2 Active learning scenarios

  • Active-learning scenarios differ in how unlabeled instances are supplied to the learner and selected for labeling.Source: §2.2 · p. 3

2.2.1 Membership query synthesis active learning

  • Membership query synthesis lets the learner request labels for arbitrary input-space points, including generated examples that a human may be unable to label.The process continues until the label budget is met or a performance stopping criterion is satisfied.Source: §2.2.1 · p. 3

2.2.2 Pool-based active learning

  • Pool-based active learning selects informative observations from a fixed unlabeled set, with batch size k trading timely criterion updates against computational efficiency.The cited examples include querying 5–30 points per round or adding 3% of observations at a time.Source: §2.2.2 · p. 4

2.2.3 Online active learning

  • Online active learning makes decisions as data arrives, using single-pass evaluation or window-based selection of the top k observations from a chunk.Window-based methods suit high-volume streams without tight decision deadlines, whereas single-pass methods are needed when decisions must follow individual observations immediately. The review also identifies applications including manufacturing, clinical trials, text classification, fraud detection, customer service, and marketing.Source: §2.2.3 · pp. 5–6
  • Online strategy design must account for whether streams drift, how long labels are delayed, and whether models are updated incrementally or retrained.Stationary-stream strategies may perform suboptimally under drift, while intermediate label latency can create redundant requests for similar instances.Source: §2.2.3 · p. 6

2.3 Connection between active learning and semi-supervised learning

  • Semi-supervised learning uses labeled and unlabeled data but does not address the active-learning question of when to query; the two can also be combined.Its approaches include unsupervised preprocessing, wrapper methods, and graph-based methods.Source: §2.3 · p. 7

3 Online active learning approaches

This section surveys online active-learning strategies for data streams, organizing them by strategy family and covering methods for stationary and drifting distributions. It describes approaches that allocate labels using confidence, uncertainty, representativeness, novelty, drift adaptation, experimental design, and bandit or reinforcement-learning criteria.

3 Online active learning approaches

  • The taxonomy covers stationary and drifting stream classification, evolving fuzzy systems, and experimental-design or bandit methods, including regression and sequential decision-making.The categories span stable versus changing distributions, adaptive fuzzy rules, and real-time selection of informative data or promising options.Source: §3 · pp. 7–8

3.1 Stationary data stream classification approaches

  • In online active learning, Cesa-Bianchi et al (2004) Bernoulli-based selective sampling has query probability converging to 1 as bpt approaches 0, while Dasgupta et al (2005) thresholded examples with |bpt| below Γ.Sculley (2007) compared fixed thresholding, Bernoulli-based sampling, and logistic margin sampling; the latter differs from traditional b-sampling in the shape of the resulting sampling distribution.Source: §3.1 · p. 9
  • Passive-aggressive updates learn from any nonzero-loss example, while minimax-gradient selection offers a more conservative response to noisy labels (Lu et al, 2016; Bordes et al, 2005).Second-order sampling also accounts for model confidence to mitigate misleading margin decisions during cold starts (Hao et al, 2018b).Source: §3.1 · pp. 10–11
  • For class imbalance, Hao et al (2018b) add cost sensitivity, while Qin et al (2021) combine diversity and uncertainty but require batch-wide similarity and uncertainty calculations.Qin et al’s strategy is therefore unsuitable for single-pass active learning.Source: §3.1 · pp. 11–12
  • Differentially private online selection randomizes hyperplane-based sampling, and private training uses mini-batch strategies to address slow convergence (Ghassemi et al, 2016).The selection mechanism randomizes labeling decisions within and outside a sampling region.Source: §3.1 · p. 12
  • In Huang et al.’s online disagreement-based active learning, the learner queries labels for points in the disagreement region and evaluates its policy by label complexity—the expected queried-label count—and regret.A related method uses weak labels for sampling and combines pseudo-labels with collected true labels to estimate empirical risk.Source: §3.1 · pp. 13–14
  • Hao et al (2018a) extend expert forecasters to active learning, requesting labels under confidence conditions and adding robustness when expert advice may be noisy.Their active EWAF and active GF use limited queried labels to measure loss; the sampling decision compares active and fully supervised predictions.Source: §3.1 · p. 15
  • Chae and Hong (2021) combine random-feature online multiple-kernel learning with confidence-based sampling, optimizing kernels locally and combining them globally.Kernel weights minimize regret over labeled instances, and selection uses a confidence condition on similarity between learned kernel functions.Source: §3.1 · pp. 15–16

3.2 Drifting data stream classification approaches

  • Drifting streams may exhibit covariate, real concept, or label shifts, alongside abrupt, gradual, incremental, or recurring changes in distribution.Covariate shift changes p(x) but not p(y | x), real concept drift changes p(y | x), and label shift changes p(y); virtual and real drift may coincide. Figure 7 illustrates the four change-rate patterns.Source: §3.2 · pp. 16–17, Figure 7 · p. 17
  • Drift-aware active learning combines sampling with detectors, but real concept drift may require labels; Krawczyk et al (2018) use ADWIN and raise labeling rates around drift.Their threshold falls by r at a warning and by 2r at detected drift; Castellani et al (2022) vary sampling rates around drift while preserving the nominal overall rate.Source: §3.2 · pp. 17–18
  • To address verification latency, Castellani et al (2022) use spatial and propagated pseudo-label information to avoid oversampling nearby high-utility points.Missing labels are estimated by a weighted majority vote of k-nearest neighbors, with weights depending on label arrival time.Source: §3.2 · pp. 18–19
  • Pham et al (2022) address delayed and redundant queries with forgetting and simulated labels, using bagging-based utility estimates or fuzzy labeling.Forgetting removes labeled observations that will be outside the training window when delayed labels arrive; simulation helps avoid repeatedly querying similar instances.Source: §3.2 · p. 19
  • Zhang et al (2020a) combine ADWIN with uncertainty sampling and class-specific thresholds, while Cheng et al (2023) and Martins et al (2023) adapt sampling by chunks.Zhang et al replace the ensemble classifier with the highest error at drift and favor minority-class examples; Cheng et al select relevant and diverse points after drift, whereas Martins et al tune uncertainty thresholds by window.Source: §3.2 · p. 19
  • Zhu et al (2007) use a chunk-trained ensemble to select points with the largest ensemble variance, while Shan et al (2019) extend pairwise classifiers for multiple drift timescales.The pairwise approach combines a stable classifier with dynamic classifiers; the single reactive-classifier version struggles with gradual drift, while multilevel sliding windows help capture gradual and abrupt changes.Source: §3.2 · p. 20
  • Yin et al. (2023) proposed adaptive data-stream classification using microclustering: after initializing micro-clusters from initial training data, they collect labels through random sampling and a class-weighted margin score, then dynamically update the model to adapt to concept drift.Source: §3.2 · p. 21
  • Liu et al (2021) combine local-density and uncertainty thresholds in a cognition window to select representative, difficult instances while adapting to gradual and abrupt drift.The window uses memory strength to determine when it is full and should be updated.Source: §3.2 · p. 22

3.3 Evolving fuzzy systems approaches

  • Evolving fuzzy systems update model structure and parameters as labels arrive; active selection includes conflict and ignorance for classification, and output or parameter uncertainty for regression.Structure schemes generate, merge, or prune rules; Lughofer and Pratama (2018) assess predictive uncertainty with locally adaptive error bars and parameter uncertainty with A-optimality, while Lughofer and ˇSkrjanc (2023) use D-optimality, rule overlap, and novelty.Source: §3.3 · p. 23
  • Other fuzzy-model strategies use spherical-potential novelty thresholds, buffered queries for visual inspection, or Hotelling T^2 and SPE statistics to flag informative observations.Subramanian et al (2014) also use prediction-error thresholds when labels are available; Cernuda et al (2014) retrain a TS fuzzy model with a sliding window.Source: §3.3 · pp. 23–24

3.4 Experimental design and bandit approaches

  • A-, D-, and E-optimality target parameter-estimation quality, while G-optimality protects against worst-case prediction variance but can be sensitive to anomalies.I- and V-optimality instead integrate or average prediction variance over the region of interest.Source: §3.4 · pp. 24–25
  • Online design methods use norm thresholding or conditional D-optimality to select stream observations; the latter estimates its threshold from passively collected warm-up data.Fontaine et al (2021) study online optimal design under heteroskedasticity to allocate labeling budget across covariates; Cacciarelli et al (2022b) initialize conditional D-optimality using a warm-up set and kernel density estimation.Source: §3.4 · pp. 25–26
  • Cacciarelli et al (2023) report that outliers can degrade design-optimality sampling, and bounding the search area trades outlier protection against exploration.The choice of the second threshold controls that trade-off.Source: §3.4 · p. 26
  • Riquelme et al (2017a) use trace UCB to estimate model difficulty and allocate a shared labeling budget proportionally, balancing exploration and exploitation.The approach frames simultaneous model estimation as a multi-armed bandit problem.Source: §3.4 · pp. 26–27
  • Bandit objectives differ between maximizing cumulative reward through exploration and exploitation and finding the best arm with confidence or a fixed label budget.Pure exploration can suit safety-constrained settings focused on understanding the system, such as drug discovery or soft sensor development (Fortuna et al, 2007; Shi and Xiong, 2018; Chan et al, 2018; Tang et al, 2018).Source: §3.4 · p. 27
  • Linear-bandit active learning selects informative arms to learn an unknown reward model, with fixed-confidence and fixed-budget methods and generalized linear extensions.The reward is modeled as r(x) = x⊤θ∗ + ε, where ε is zero-mean i.i.d. noise; cited work also studies offline G-optimality and XY-allocation.Source: §3.4 · p. 27
  • Transductive designs pull arms from X to identify the best arm or improve predictions on a separate, pre-given unlabeled set Z.Online work by Camilleri et al (2021) balances time complexity—the number of unlabeled observations spanned—and label complexity—the number of labels queried before stopping and declaring the best arm.Source: §3.4 · p. 27
  • Reinforcement-learning approaches optimize query decisions using feedback on whether a requested label was useful, with contextual-bandit experts rewarded or penalized over time.In Wassermann et al (2019), experts recommend querying based on prediction-certainty thresholds Γ; positive ρ+ rewards indicate a query prevented a wrong prediction, while negative ρ− rewards indicate querying was unnecessary.Source: §3.4 · pp. 27–28
  • Woodward and Finn (2017) use an LSTM-based reinforcement-learning policy to choose between predicting a label and paying to request it, maximizing long-term reward.The policy accounts for label-request cost and incorrect-prediction cost, with rewards assigned to requests, correct predictions, and incorrect predictions; the latter two are negative.Source: §3.4 · p. 28

4 Evaluation strategies

Evaluation of stream-based active learning uses several protocols, with prequential evaluation favored for drifting streams and holdout tests used when methods are not suited to drift. Learning curves and statistical tests help compare strategies, while evaluation choices vary across studies.

Evaluation strategies

  • Learning curves plot holdout performance against labeled-example count; their slope indicates sample efficiency, and statistical tests can assess whether strategy differences are significant.The review names non-parametric tests for comparing two or more strategies.Source: §4 · p. 28
  • A holdout test set can assess prediction improvement over time on a stationary stream, whereas prequential evaluation monitors prediction error as the stream evolves.Prequential evaluation tests each incoming instance before deciding whether to query its label and update the model.Source: §4 · pp. 28–29
  • For drifting streams, studies generally favor prequential evaluation, while holdout tests tend to be used for methods ill-suited to concept drift.Other studies use varied or unspecified protocols, including testing on data encountered during training, window-based accuracy, and theoretical analyses.Source: §4 · p. 29, Table 1 · p. 29
  • Evaluation protocols differ across studies, including fixed-confidence bandit evaluations based on label complexity and regret or error measured against trials.The review also describes per-round regret as the loss difference between a forecaster and the best expert.Source: §4 · p. 29

5 Real-world applications and challenges

Applications span spam filtering, network identification, computer vision, object detection, autonomous driving, and industrial optimization. Across these settings, practical challenges include stream-rate scalability, label quality and delay, distribution drift, interpretability, evaluation, and annotation workflows.

5.1 Applications

  • Online active learning has been applied to spam filtering and network protocol identification, selecting informative emails or representative instances as data arrive.The network-identification approach adapts to dynamic data distributions.Source: §5.1 · p. 30, Figure 13 · p. 30
  • In computer vision, clustering and pairwise embedding comparisons hinder single-pass use, so most online applications rely on traditional models with uncertainty-based sampling.Examples include 3D-object classification with a Mondrian forest and process monitoring with image features and streaming classifiers.Source: §5.1 · p. 30
  • For streaming deep-network training, selected examples are sampled according to their contribution to the covariance matrix of examples already chosen for the current batch.The approach uses gradient representations and a probability proportional to covariance contribution.Source: §5.1 · p. 30
  • Object-detection applications include SBAD, which combines active learning with self-supervision and uses a warm-up threshold to query high-confidence images, and budget allocation toward underperforming scenario slices.Autonomous-car work also uses diversity-based sampling to reduce false alarms and learn unseen faults.Source: §5.1 · p. 31, Figure 14 · p. 31
  • A compressor application combines a thermodynamics-based model, a Gaussian Process surrogate, and online active learning to choose new experimental points in real time.Source: §5.1 · p. 31

5.2 Challenges

  • Real-world deployment requires sampling decisions faster than the process feed rate and attention to annotation errors, bias, and latency.Slow decisions may require buffering, while labels may be delayed for physical inspection, medical treatment, or batch updates.Source: §5.2 · pp. 31–32
  • Drifting distributions can make selection criteria less effective, while interpretability, evaluation under label scarcity, and human-computer interaction remain challenges.The review distinguishes real-time annotation from postponed annotation workflows.Source: §5.2 · p. 32

6 Summary and future directions

The survey reviews online active learning approaches and finds that existing work focuses predominantly on online classification. It identifies regression, single-pass sampling for dynamic streams, and online reinforcement learning as directions for further research.

6 Summary and future directions

  • Existing research predominantly develops online classification models for stationary and drifting streams, with comparatively limited work on online active linear regression and online regression models generally.Table 2 summarizes approaches by data-processing capabilities, stream assumptions, task, and model characteristics.Source: §6 · p. 32, Table 2 · p. 33
  • Future work should develop online regression strategies beyond linear models and linear bandits, including nonlinear and model-agnostic methods.The review notes emerging interest in Bayesian optimization for nonlinear regression.Source: §6 · p. 32
  • Single-pass sampling for dynamic streams could address cases where time or memory constraints prevent batching or maintaining and updating multiple models.The survey also identifies strategies covering multiple types of distribution shift and online reinforcement learning for optimizing sampling as research directions.Source: §6 · p. 32

7 Conclusion

Online active learning is relevant to real-world applications, where obtaining annotated data remains a major roadblock. The survey reviews strategies and contexts, highlights challenges and opportunities, and points to potential for model-agnostic, single-pass methods.

7 Conclusion

  • The survey identifies annotated-data collection as a major roadblock to integrating AI models with real-world applications.It names healthcare, autonomous driving, and industrial production as examples.Source: §7 · p. 33
  • Reviewed strategies span online classification, online regression, and online semi-supervised learning.They include uncertainty sampling, diversity sampling, query by committee, and reinforcement learning.Source: §7 · p. 34
  • The survey sees potential for model-agnostic, single-pass online active learning strategies in practical settings.Source: §7 · p. 34

Related papers

Loading 2302.08893v4…