Source-linked AI summary
Online Deep Learning: Learning Deep Neural Networks on the Fly
Doyen Sahoo, Quang Pham, Jing Lu, Steven C. H. Hoi
TL;DR
The paper addresses how to train DNNs online when data arrives sequentially, may drift, and cannot be retained in full. It introduces Hedge Backpropagation, which adapts network depth using multiple classifiers and shared representations. The framework enables online DNN learning across stationary and concept-drifting settings.
Problem
Online DNN learning is difficult because streaming data may be too large to store, concept drift may occur, and fixed-depth backpropagation has convergence limitations.
Method
Hedge Backpropagation uses an overcomplete network, multiple output classifiers, and hedging to adapt network depth online while sharing feature representations.
Results
The framework enables DNNs to learn online by combining shallow and deep predictors, with HBP adapting quickly in concept-drifting scenarios through hedging.
Takeaways & Limitations
HBP provides an online deep-learning approach that continuously adapts model capacity as more data arrives.
Takeaways & Limitations
Online DNN learning remains constrained by convergence issues, including vanishing gradients and diminishing feature reuse, especially for deep networks.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) are typically trained by backpropagation in a batch learning setting, which requires the entire training data to be made available prior to the learning task. This is not scalable for many real-world scenarios where new data arrives sequentially in a stream form. We aim to address an open challenge of "Online Deep Learning" (ODL) for learning DNNs on the fly in an online setting. Unlike traditional online learning that often optimizes some convex objective function with respect to a shallow model (e.g., a linear/kernel-based hypothesis), ODL is significantly more challenging since the optimization of the DNN objective function is non-convex, and regular backpropagation does not work well in practice, especially for online learning settings. In this paper, we present a new online deep learning framework that attempts to tackle the challenges by learning DNN models of adaptive depth from a sequence of training data in an online learning setting. In particular, we propose a novel Hedge Backpropagation (HBP) method for online updating the parameters of DNN effectively, and validate the efficacy of our method on large-scale data sets, including both stationary and concept drifting scenarios.
Introduction
Online Deep Learning addresses the challenge of learning DNNs from sequential, potentially drifting data without storing the full stream. The proposed framework adapts network capacity over time using Hedge Backpropagation.
- Streaming data may be too large to store and may exhibit concept drift, motivating online model learning.
- Existing online learning methods mainly optimize shallow linear or kernel models with online convex optimization.
- Online Deep Learning seeks to learn DNNs from data streams despite non-convex optimization and practical limitations of standard online backpropagation.
- The framework adapts DNN capacity from shallow to complex models over time, combining online learning with deep learning.
- Hedge Backpropagation attaches classifiers to hidden-layer representations and uses their online performance to train DNNs of adaptive capacity.
Related Work
Prior online-learning methods primarily target shallow models, while existing approaches for online-compatible deep learning remain limited by convergence, architecture selection, or batch-style training. These limitations motivate adaptive, genuinely streaming deep-learning methods.
- Online learning algorithms are scalable and memory efficient but are primarily designed for linear or kernel-based models.
- Applying online backpropagation to DNNs faces convergence problems and makes optimal network depth difficult to validate during streaming.
- Sliding-window approaches add mini-batch training stages, making them unsuitable for a fully streaming data setting.
- Shallow-to-deep approaches exploit faster shallow convergence, but the reviewed methods are designed around deepest-layer outputs.
- Architecture-learning methods incorporate network hyperparameters into optimization and can use regularization to reduce network width.
Online Deep Learning
Online Deep Learning learns DNNs from sequential data while adapting network depth during training. Hedge Backpropagation combines classifiers at multiple depths and updates shared representations online to address convergence and capacity-selection challenges.
- Online learning setup: Online deep learning learns a prediction function from sequentially arriving examples, updating the model after each revealed label.Performance is evaluated by cumulative prediction error over the stream.
- Limitations: Standard online backpropagation requires fixed depth and can suffer from vanishing gradients, slow early convergence, and diminishing feature reuse.Shallow networks may suit early instances, whereas deeper networks may suit longer streams.
- Hedge Backpropagation: HBP uses an overcomplete network with classifiers at every depth and automatically adapts depth online through hedging.Its final prediction is a weighted combination of classifiers based on representations from h(0) through h(L).
- Hedge Backpropagation: Classifier weights are discounted according to suffered loss, normalized each round, and used to weight the combined prediction.A smoothing parameter prevents deeper classifiers from receiving vanishingly small weights after poor initial performance.
- Hedge Backpropagation: Feature parameters receive gradients from every dependent classifier, weighted by the classifiers’ hedge weights, rather than only from the deepest output.The summation starts at the current layer because shallower classifiers do not depend on deeper representations.
- Properties: HBP provides online model selection over network depths, shares representations between shallow and deep classifiers, and is designed to adapt under concept drift.The authors characterize it as competing and collaborating multi-depth networks, with a regret bound of RT ≤ T ln N.
Experiments
The experiments evaluate HBP on large-scale stationary and concept-drifting streams, showing that adaptive depth addresses changing convergence and capacity needs in online DNN learning. HBP consistently combines shallow networks’ early advantages with deeper networks’ later performance.
- Datasets: The evaluation uses large-scale Physics, Infinite MNIST, synthetic, and concept-drift datasets, including 5 million-instance Higgs and Infinite MNIST streams.CD1 contains concepts C1-C2-C1, with each segment comprising one third of the stream.
- Model Selection: Shallow DNNs perform best early, while 4-layer and 8-layer networks perform better in later stream segments.The first 0.5% favors the shallowest network; [10-15]% favors 4 layers in most cases, and [60-80]% favors 8 layers.
- Model Selection: The optimal DNN depth is unknown before learning, and final error alone does not conclusively identify the most suitable depth.Deeper networks converge more slowly but may achieve better overall performance when more stream instances become available.
- Overall Performance: HBP outperforms DNNs trained with Online Gradient Descent at different depths in cumulative online error.HBP combines faster initial convergence with deeper representations later, enabling automatic online model selection.
- Stationary and Drifting Streams: HBP consistently outperforms baselines on stationary data and adapts fairly quickly to concept drift, although shallower baselines win during the earliest online stages.Simple 16- and 20-layer networks have difficulty exploiting deep representations on concept-drifting data.
- Robustness to Base-Network Depth: Across 12-, 16-, 20-, and 30-layer base networks, HBP performance changes little, whereas substantial depth increases hurt simple Online BP.This experiment evaluates robustness on Higgs against traditional Online BP.
Conclusion
The paper presents Hedge Backpropagation (HBP) as an online deep learning method that enables DNNs to learn from streaming data. HBP adapts effective network capacity through multiple classifier outputs and knowledge sharing across depths.
- HBP enables DNN usage in online settings by addressing model-selection and convergence difficulties that hinder streaming-data learning.The method targets appropriate network depth, vanishing gradients, and diminishing feature reuse.
- HBP uses multiple hidden-layer outputs with a hedging strategy and modified backpropagation to share knowledge among shallow and deep classifiers.
- HBP automatically changes effective network capacity according to observed data complexity.
- Figure 2 reports convergence behavior for DNNs in online learning on stationary and concept drifting data.
- Figure 3 reports how HBP classifier weight distributions evolve over time on the HIGGS dataset.