PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis

Tyler L. Hayes, Christopher Kanan

arXiv:1909.01520v3cs.LGcs.CVstat.ML

TL;DR

The paper addresses the difficulty of learning immediately from non-stationary streams without catastrophic forgetting or batch accumulation. It combines deep CNN representations with streaming linear discriminant analysis, and reports stronger performance than incremental batch and streaming alternatives on ImageNet and CORe50.

  • Problem

    Conventional DNNs catastrophically forget under incremental non-iid updates, while incremental batch learning delays inference until large batches are accumulated.

  • Method

    Deep SLDA freezes CNN feature layers and incrementally trains a linear output layer using streaming class statistics without storing previous data.

  • Results

    Deep SLDA exceeds recent incremental batch-learning methods on ImageNet and CORe50, including 6% and over 11% higher Ωall than iCaRL, respectively.

  • Takeaways & Limitations

    Streaming SLDA provides a lightweight approach that mitigates catastrophic forgetting across different data orderings and remains effective despite training only the output layer.

  • Takeaways & Limitations

    The proposed method trains only the output layer; future work would combine it with rehearsal to update the entire CNN when compute and memory are available.

Abstract

from arXiv · show

When an agent acquires new information, ideally it would immediately be capable of using that information to understand its environment. This is not possible using conventional deep neural networks, which suffer from catastrophic forgetting when they are incrementally updated, with new knowledge overwriting established representations. A variety of approaches have been developed that attempt to mitigate catastrophic forgetting in the incremental batch learning scenario, where a model learns from a series of large collections of labeled samples. However, in this setting, inference is only possible after a batch has been accumulated, which prohibits many applications. An alternative paradigm is online learning in a single pass through the training dataset on a resource constrained budget, which is known as streaming learning. Streaming learning has been much less studied in the deep learning community. In streaming learning, an agent learns instances one-by-one and can be tested at any time, rather than only after learning a large batch. Here, we revisit streaming linear discriminant analysis, which has been widely used in the data mining research community. By combining streaming linear discriminant analysis with deep learning, we are able to outperform both incremental batch learning and streaming learning algorithms on both ImageNet ILSVRC-2012 and CORe50, a dataset that involves learning to classify from temporally ordered samples.

1. Introduction

Streaming learning targets real-time, resource-constrained agents that learn individual instances and remain testable during training. The paper combines deep CNN features with SLDA to address catastrophic forgetting and outperform batch and streaming alternatives.

  • Motivation: Real-time agents must learn each instance immediately in a single pass under severe memory and compute constraints.Inference must remain possible at any point during training.
  • Motivation: Conventional DNN updates can catastrophically forget previously learned tasks when non-iid data changes critical weights.This reflects the stability-plasticity dilemma.
  • Learning settings: Incremental batch learning repeatedly processes large labeled batches before testing, limiting its suitability for real-time applications.Streaming learning instead supports immediate use of newly learned information.
  • Approach: The paper adapts streaming linear discriminant analysis to train a CNN output layer incrementally without storing previous data.The resulting classifier is lightweight and deployable on embedded platforms.
  • Results: Deep SLDA surpasses recent incremental batch-learning methods on ImageNet ILSVRC-2012 and CORe50 while being over 100 times faster and using 1,000 times less memory.The contribution passage explicitly compares these methods despite their updating hidden layers.

2. Problem Formulations & Related Work

The paper distinguishes streaming learning from incremental batch learning and reviews methods for mitigating forgetting in both settings. It motivates deep SLDA as a lightweight alternative for immediate inference on non-stationary data streams.

  • 2.1. Streaming Learning: Streaming learning processes one example at a time in a single pass, allowing evaluation at any point without revisiting data.Incremental batch learning instead permits repeated passes over batches and tests between batches.
  • 2.1. Streaming Learning: Progress in mitigating forgetting is greater for incremental batch learning than for streaming learning, leaving a gap with offline models.The paper identifies streaming learning as the less developed paradigm.
  • 2.2. Methods for Incremental Batch Learning: Incremental batch methods mitigate forgetting through weight regularization, sparse updates, classifier ensembles, partial rehearsal, and pseudo-rehearsal.Rehearsal methods have recently shown the strongest success among these approaches.
  • 2.2. Methods for Incremental Batch Learning: Rehearsal methods consume memory and computation because they retain examples or generative models and process additional samples during updates.These costs contribute to the mismatch between batch learning and real-time deployment.
  • 2.3. Methods for Streaming Learning: Streaming classifiers include Hoeffding Decision Trees, ensembles, and shallow ARTMAP networks, but ARTMAP is order-sensitive and lacks representation learning.The reviewed streaming methods primarily come from data mining.
  • 2.3. Methods for Streaming Learning: GEM-family methods constrain updates using stored samples but require task labels during inference, limiting embedded deployment when labels are unavailable.Performance significantly degrades without task labels at testing.
  • 2.3. Methods for Streaming Learning: ExStream updates CNN fully connected layers and combats forgetting with class-specific prototype buffers that merge nearby exemplars when full.It therefore uses partial rehearsal in the streaming setting.
  • 2.3. Methods for Streaming Learning: SLDA maintains one running mean per class and a shared covariance matrix, predicting with the closest Gaussian defined by those statistics.SQDA similarly assumes Gaussian classes but does not share the same covariance assumption.

3. Deep Streaming LDA

Deep SLDA freezes a CNN feature extractor and incrementally trains a linear decoder using class statistics and a shared covariance. Its independent class means limit forgetting, while covariance updates provide an optional adaptation mechanism.

  • Architecture: The CNN is decomposed into a fixed feature extractor G and a trainable final fully connected layer F.The fixed early layers exploit transferable filters across natural image datasets.
  • Architecture: SLDA adapts streaming LDA to train a linear decoder for CNN feature vectors.The decoder uses weight matrix W and bias vector b over K categories.
  • Streaming statistics: For each class, SLDA stores a running mean and count, alongside one shared covariance matrix.These statistics are updated as each new feature-label pair arrives.
  • Covariance handling: Shrinkage computes the precision matrix as Λ = [(1 − ε)Σ + εI]^-1 with ε = 10^-4, using either frozen or online-updated covariance.The frozen variant inverts covariance once, whereas online updating requires inversion before inference.
  • Covariance handling: The covariance update uses an online rule, after which predictions derive decoder weights and biases from the learned statistics.The cited passages identify the update and prediction stages but do not provide the complete displayed formulas.
  • Forgetting behavior: Independent running class means make SLDA resistant to catastrophic forgetting, while covariance changes cause at most gradual forgetting.Covariance remains sensitive to class ordering.

4.1. Baseline & Comparison Models

The study compares deep SLDA with streaming, incremental batch, regularization, and offline baselines under a common ResNet-18 setup. The comparisons distinguish methods by streaming capability, task-label requirements, replay, and which network layers they update.

  • Comparison setup: All models incrementally train a ResNet-18 CNN, while comparisons include methods that do and do not require task labels.SLDA is evaluated mainly against models operating without task labels; GEM-family methods require task labels at test time.
  • Streaming models: Deep SLDA updates the CNN classification layer using either a fixed or incrementally updated covariance matrix.The two variants differ only in whether the covariance matrix remains fixed after base initialization or changes during streaming learning.
  • Streaming models: Fine-Tuning updates either only the output layer or the entire network sample-by-sample in one pass without a buffer.The baseline is described as suffering from catastrophic forgetting.
  • Streaming models: ExStream stores and merges prototype vectors in a buffer before training the network’s fully connected layers.It is a streaming method that updates fully connected layers and previously achieved state-of-the-art performance on CORe50.
  • Incremental batch and reference models: iCaRL and End-to-End are incremental batch methods that use stored examples, rehearsal-related mechanisms, and multiple passes over batches.iCaRL uses distillation and a Nearest Class Mean classifier, whereas End-to-End uses the CNN output layer and augmentation strategies.
  • Incremental batch and reference models: Offline models provide normalization references and compare updating only the feature extractor with updating both feature extractor and classifier.A-GEM and other regularization models are additionally compared in a small-scale experiment because GEM requires task labels.

4.2. Datasets, Data Orderings, & Evaluation

Experiments use ImageNet and CORe50 to evaluate incremental classification across different data-ordering conditions. Performance is normalized against optimized offline learning using the Ω_all metric, whose interpretation depends on incremental and offline performance over time.

  • Datasets: ImageNet contains over one million images across 1,000 categories, with models initialized on 100 classes and evaluated using top-5 accuracy every 100 learned classes.Performance is computed on all classes learned so far.
  • Datasets: CORe50 contains temporally ordered video sequences of objects recorded across 10 categories, five objects per category, and 11 environmental conditions.The videos are sampled at 1 fps using 128×128 bounding-box crops, producing 600 training and 225 test images per class.
  • Data orderings: CORe50 is evaluated under iid, class iid, instance, and class instance orderings because temporal presentation order affects final results.These orderings vary whether frames are shuffled, grouped by class, or organized temporally by object instances.
  • Evaluation: Ω_all normalizes incremental performance against optimized offline performance trained on all data available up to each time t.The metric is usually between 0 and 1, but can exceed 1 when the incremental learner outperforms the offline baseline.

4.3. Network Initialization

The experiments use shared base initializations so model comparisons are fair across ImageNet and CORe50. SLDA additionally initializes its covariance matrix from the same base data using a shrinkage estimator.

  • ImageNet initialization: ImageNet models initialize F(·) and G(·) on 100 fixed, randomly selected classes, leaving 900 classes to be learned incrementally.The same initialization procedure is used for the compared models.
  • CORe50 initialization: CORe50 models initialize F(·) and G(·) with pretrained ImageNet weights, replace the final layer with 10 outputs, and fine-tune on 1,200 CORe50 samples.The selected samples depend on data ordering but remain fixed across models within the comparison.
  • Evaluation comparison: Table 1 reports Ω_all results for ImageNet and CORe50, identifying updated parameters and streaming methods and averaging CORe50 results over 10 runs.All models use ResNet-18, and the best streaming model is highlighted for each dataset and ordering.
  • SLDA initialization: SLDA initializes its covariance matrix on the shared base data using the Oracle Approximating Shrinkage estimator.This covariance initialization is performed after establishing the common model base initialization.

4.4. Main Results

Deep SLDA performs strongly against both streaming and incremental batch baselines on ImageNet and CORe50 despite updating only the CNN output layer. Its covariance-update benefit depends on the dataset and initialization, while performance remains sensitive to data ordering for some baselines.

  • ImageNet results: SLDA outperforms iCaRL overall and ends with higher ImageNet accuracy than End-to-End despite not training the CNN hidden layers.Updating the SLDA covariance matrix adds only about 0.5% on ImageNet, likely because the features are already well represented after 100-class initialization.
  • ImageNet results: Streaming models without replay buffers suffer catastrophic forgetting and achieve poor overall ImageNet performance.This contrasts with SLDA’s stronger performance under the same streaming setting.
  • CORe50 results: On CORe50, plastic-covariance SLDA outperforms fixed-covariance SLDA, ExStream, iCaRL, and the streaming model without a replay buffer.It performs almost as well as the full offline learner for iid ordering and remains close to offline performance for the other orderings.
  • CORe50 results: Updating the covariance matrix produces a large CORe50 performance boost across all four orderings, unlike the small ImageNet improvement.The difference is attributed to CORe50’s smaller base initialization, which makes the initial covariance less representative of the full training set.
  • Regularization comparisons: Regularization baselines are evaluated on CORe50 with and without task labels, reflecting their dependence on task information in the experimental setup.The comparison includes methods that constrain parameter changes during incremental training.
  • CORe50 results: ExStream and both SLDA variants outperform iCaRL on the class iid and class instance CORe50 orderings.These orderings are among the conditions where class presentation structure differentiates the methods.

4.5. Additional Experiments and Analysis

Additional experiments show that deep SLDA is computationally and memory efficient, remains effective with fewer base classes, and can transfer from ImageNet features to CORe50 streaming learning.

  • Compute: SLDA requires 30 minutes on ImageNet, compared with 31 hours for ExStream and 62 hours for iCaRL.These comparisons use the same hardware.
  • Memory: SLDA stores only 0.001 GB for its covariance matrix, versus 0.041 GB for ExStream prototype vectors and 3.011 GB for rehearsal images.The 3.011 GB figure applies to End-to-End and iCaRL storing 20 images per class on ImageNet.
  • Base Initialization: Using 50 initialization classes lets SLDA outperform iCaRL, while 75 classes lets it outperform both iCaRL and End-to-End on ImageNet.The standard ImageNet initialization uses 100 classes.
  • Domain Transfer: Initializing G(·) and Σ on CORe50 performs best for iid and instance orderings, whereas initializing only G(·) on CORe50 performs best for both class orderings.Table 3 reports Ω_all classification results for fixed and plastic covariance matrices under these initialization choices.
  • Domain Transfer: ImageNet-initialized features with a plastic covariance matrix remain within 5.6% of CORe50 initialization for iid ordering and within 1.0% for class iid ordering.The passage presents this as evidence that SLDA can transfer domains without a CORe50 base initialization phase.

5. Discussion & Conclusion

The discussion concludes that deep SLDA is a lightweight streaming approach that mitigates forgetting across data orderings and outperforms recent incremental batch methods, while identifying rehearsal-based extensions as future work.

  • Conclusion: SLDA outperforms iCaRL by 6% on ImageNet and over 11% on CORe50 in Ω_all despite training only the output layer.iCaRL updates the entire network and therefore uses more computational time and resources.
  • Contribution: Deep SLDA combines CNN features with streaming linear discriminant analysis for large-scale classification without storing previous data.The discussion characterizes the approach as lightweight and effective for streaming learning.
  • Implications: The authors recommend testing output-layer-only training after base initialization because offline results indicate that hidden-layer training can achieve greater performance.This recommendation concerns future incremental-learning evaluations.
  • Future Work: A proposed future direction combines SLDA's rapid learning with rehearsal that updates the entire CNN when compute and memory are available.The authors identify feature drift and choosing between the main output layer and SLDA as challenges.

Appendix

The appendix specifies the offline optimization settings and training schedules used for ImageNet and CORe50 experiments.

  • Optimization: The offline model uses SGD with momentum=0.9, weight decay=1e-4, and standard image augmentation.Augmentation includes random flips and random resized crops to 224×224 pixels.
  • ImageNet: ImageNet training runs for 90 epochs with batch size 128 and learning rate 0.1, decayed at 30 and 60 epochs.The ImageNet schedule uses the iCaRL parameters from the cited reference.
  • CORe50: CORe50 training runs for 40 epochs with batch size 256 and learning rate 0.01, decayed at 15 and 30 epochs.The schedule specifies the CORe50 training configuration used in the experiments.
Loading 1909.01520v3…