PaperlayerView exact revision on arXiv

Source-linked AI summary

REMIND Your Neural Network to Prevent Catastrophic Forgetting

Tyler L. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, Christopher Kanan

arXiv:1910.02509v3cs.LGcs.CVcs.NE

TL;DR

Conventional neural networks forget previous knowledge during incremental learning, while existing CNN replay methods store raw images and commonly rely on batch-based training. REMIND uses compressed intermediate representations for one-example-at-a-time replay, achieving strong incremental classification results and extending to streaming VQA.

  • Problem

    Incrementally updating neural networks on non-iid lifelong data can cause catastrophic forgetting, while CNN replay commonly stores raw pixels and uses batch-based training.

  • Method

    REMIND compresses hidden CNN representations with tensor quantization, stores them for replay, reconstructs selected memories, and updates plastic layers online.

  • Results

    REMIND outperforms existing models on ImageNet ILSVRC-2012 and CORe50 under the same memory, achieves state-of-the-art object-classification results, and extends to VQA with almost no changes.

  • Takeaways & Limitations

    Compressed feature replay supports brain-inspired streaming learning across iid and instance-ordered data and extends beyond classification to VQA.

  • Takeaways & Limitations

    REMIND initializes its quantization model during base initialization, while selective replay strategies tested by the authors did not outperform uniform selection.

Abstract

from arXiv · show

People learn throughout life. However, incrementally updating conventional neural networks leads to catastrophic forgetting. A common remedy is replay, which is inspired by how the brain consolidates memory. Replay involves fine-tuning a network on a mixture of new and old instances. While there is neuroscientific evidence that the brain replays compressed memories, existing methods for convolutional networks replay raw images. Here, we propose REMIND, a brain-inspired approach that enables efficient replay with compressed representations. REMIND is trained in an online manner, meaning it learns one example at a time, which is closer to how humans learn. Under the same constraints, REMIND outperforms other methods for incremental class learning on the ImageNet ILSVRC-2012 dataset. We probe REMIND's robustness to data ordering schemes known to induce catastrophic forgetting. We demonstrate REMIND's generality by pioneering online learning for Visual Question Answering (VQA).

1 Introduction

Lifelong learning exposes neural networks to non-iid streams and catastrophic forgetting. REMIND addresses gaps in existing replay and training paradigms by storing compressed representations and learning online.

  • Catastrophic forgetting occurs when new learning overwrites existing neural representations.
  • Replay mitigates forgetting by mixing new instances with old ones during network fine-tuning.The approach is motivated by biological memory consolidation.
  • Existing CNN replay methods store raw pixels, unlike the compressed representations associated with hippocampal memory.
  • Common incremental CNN training uses repeated passes over large class batches, whereas animal learning is streaming and resource-constrained.This limits applications such as immediate on-device learning.
  • REMIND introduces streaming replay using tensor quantization to store hidden CNN representations efficiently.It is influenced by biological replay and hippocampal indexing theory.

3 Related Work

Related work primarily mitigates catastrophic forgetting through replay, regularization, or network expansion. REMIND differs from prior CNN replay methods by storing quantized mid-level features instead of raw images.

  • Replay, regularization, and network expansion are the three main mechanisms identified for mitigating catastrophic forgetting.
  • Recent ImageNet incremental-class methods use raw-pixel replay with distillation loss and shared prototype-selection and memory conventions.
  • REMIND is the first described model to demonstrate effective forgetting mitigation by replaying quantized mid-level CNN features.
  • Regularization methods constrain parameter updates according to parameter importance, while some combine regularization with replay.
  • Network-expansion approaches can require test-time task labels, growing memory, and may not scale to thousand-category datasets.

4 REMIND: Replay using Memory Indexing

REMIND performs streaming replay by compressing intermediate CNN features, storing their quantization indices, reconstructing selected memories, and updating only the network’s plastic layers. Its design uses product quantization, bounded memory, and feature-level augmentation while keeping lower-level features fixed.

  • REMIND compresses current CNN features and reconstructs previously compressed representations to train plastic weights with the current example.This allows more instances to be stored under a smaller memory budget than raw-image replay.
  • A 20K-example ImageNet memory budget can hold over 1M compressed instances with REMIND.
  • The CNN is split into fixed lower layers G and trainable upper layers F, with predictions computed as y_i = F(G(X_i)).The lower layers are kept fixed because early CNN features are considered highly transferable.
  • REMIND stores quantization indices and labels in a replay buffer, uniformly reconstructs r instances, and updates F using those instances plus the current input.During inference, features are also quantized and reconstructed before reaching F.
  • Product Quantization partitions each feature element into subvectors and assigns separate codebook indices to compress high-dimensional tensors.
  • When the replay buffer reaches capacity, REMIND inserts the new sample and randomly removes an example from the largest class.
  • Replay augmentation uses random resized crops and manifold mixup directly on quantized tensors before updating F.Mixed features combine representations and labels from two replay sets using a beta-distributed coefficient.
  • REMIND’s fixed-feature assumption requires low-level visual representations to transfer across image datasets.The CNN is jointly initialized offline on an initial subset before streaming updates begin.

5 Experiments: Image Classification

The experiments evaluate instance-by-instance streaming classification under shared data orders and memory constraints. REMIND is compared with replay, streaming, and offline baselines on ImageNet and CORe50, where it achieves strong performance across settings.

  • Compared methods: REMIND uses Product Quantization and replay augmentation, while Fine-Tuning uses no buffer and suffers from catastrophic forgetting.ExStream rehearses prototypes but trains only fully connected layers; SLDA uses running class means and a tied covariance matrix.
  • Experimental setup: All models are trained instance-by-instance with shared example orders and the same base CNN initialization.ImageNet is evaluated after each batch on all trained classes, while CORe50 is evaluated on all test data after each batch.
  • ImageNet: REMIND outperforms all comparison models on ImageNet under the same memory budget.The ImageNet replay budget is 1.51 GB for 10,000 raw-image prototypes, while REMIND stores indices for 959665 compressed examples; SLDA is second best.
  • Incremental batch comparison: REMIND achieves comparable performance to the best methods in incremental batch learning despite targeting streaming learning.The incremental-batch variant trains more slowly than streaming methods.

6 Experiments: Incremental VQA

The VQA experiments extend REMIND to streaming multimodal learning with temporally ordered image-question-answer data. REMIND outperforms streaming baselines on TDIUC and CLEVR under both evaluated orderings.

  • Streaming VQA setup: Streaming VQA presents temporally ordered image, question, and answer triplets, with replay storing quantized features alongside questions and answers.The method can be combined with attention-based, compositional, and bi-modal VQA systems.
  • Datasets: TDIUC and CLEVR test streaming VQA across diverse question types and underlying reasoning tasks.TDIUC contains over 1.7 million question-answer pairs, while CLEVR contains over 700000 pairs from 70000 synthetic images.
  • Evaluation: The experiments compare REMIND with ExStream, SLDA, an offline baseline, and buffer-free fine-tuning under iid and question-type orderings.Models are evaluated progressively as training data or question types are encountered.
  • Results: REMIND outperforms streaming baselines on both TDIUC and CLEVR with a 50% buffer.Performance is strong with SAN on TDIUC and MAC on CLEVR; CLEVR results are higher under question-type ordering than iid ordering.

7 Additional Classification Experiments

Additional experiments examine REMIND’s components and replay-buffer size on ImageNet. Product Quantization and augmentation provide the clearest gains, while enlarging the buffer or using real features has limited benefit relative to memory cost.

  • Component analysis: Product Quantization is critical: 32 codebooks outperform one-codebook k-means by 7.7% in Ωall.The main variant uses 32 codebooks of size 256 and a 959665-example buffer.
  • Component analysis: Data augmentation improves performance by 3.7% in Ωall.Augmentation is the next most helpful component after Product Quantization.
  • Buffer and memory: Storing the entire dataset does not yield significant improvements over the main buffer setting.The comparison evaluates buffer size as a component of REMIND’s ImageNet performance.
  • Feature representation: Using real features yields only a marginal 1.3% improvement while requiring nearly 16 times more memory.The result highlights a performance-memory trade-off in the feature representation used to train F(·).
  • Replay buffer size: More replay memory generally improves streaming ImageNet performance across equivalent buffer sizes.The study compares 0.75 GB, 1.51 GB, and larger or full-dataset budgets for REMIND and competing models.

8 Discussion & Conclusion

REMIND combines brain-inspired replay with compressed representations for online learning and achieves strong results across supported tasks. The authors identify several avenues for improving quantization, replay selection, and architecture design.

  • Results: REMIND achieved state-of-the-art results for object classification and can handle iid and instance-ordered streams without modification.The comparison specifically contrasts REMIND with iCaRL, Unified, and BiC.
  • Generality: REMIND also extends to Visual Question Answering with almost no changes, supporting its generality beyond object classification.
  • Discussion: REMIND uses lossy compressed representations rather than raw pixels, aligning replay more closely with memory consolidation in the brain.The method stores compressed representations and replays them during waking-hours-style learning.
  • Future directions: Potential improvements include pretraining or enlarging the quantization codebook, selective replay, and co-designing the CNN architecture with REMIND.The authors report that tested selective-replay rules did not outperform uniform selection, while architecture co-design could improve results.

S1 Parameter Settings

The supplementary settings describe training configurations, memory measurements, and storage requirements for REMIND and comparison models. They include dataset-specific learning schedules, buffer constraints, and feature-storage analysis.

  • Training settings: Table S1 provides training parameter settings for REMIND and offline models.
  • ImageNet: ImageNet REMIND uses per-class learning-rate decay from 0.1 until the learning rate reaches 0.001 after all examples are seen.
  • Memory settings: Incremental-batch ImageNet training gives REMIND the same buffer size as streaming learning, equivalent to 1.51 GB of compressed storage.
  • Baselines: ExStream uses 20 prototype vectors per class, while SLDA uses shrinkage regularization of 10^-4.
  • Storage analysis: Figure S1 reports auxiliary storage for quantized CNN features across ResNet-18 top-layer parameter percentages on CORe50 and ImageNet.The star marks the parameter setting used in the main experiments.
  • Baseline settings: Comparison models use specified ImageNet and CORe50 settings, including class-by-class, instance-by-instance learning and dataset-specific batch schedules.

S2 Where Should ResNet-18 be Quantized?

Quantizing earlier in ResNet-18 can provide a more biologically sensible representation, but larger spatial feature maps substantially increase auxiliary storage. Architectural changes could make earlier quantization feasible while complicating comparisons and search.

  • Storage constraint: Earlier ResNet-18 quantization increases feature-map spatial dimensions and auxiliary storage requirements.Quantizing layer 3 in the ImageNet experiments would require 129 GB to store a representation.
  • Biological consideration: The chosen quantization layer is constrained by storage efficiency, although an earlier layer may better reflect hippocampal connectivity to visual processing areas.
  • Architectural trade-off: Reducing spatial dimensions earlier while increasing feature dimensions would allow earlier quantization, but could prevent direct comparison with prior work.
  • Architectural trade-off: Such architectural modification may require considerable search to find a suitable compromise between representation design and evaluation comparability.

S3 Additional Image Classification Experiments

Additional ImageNet and CORe50 experiments examine buffer size, training architecture, replay settings, and average accuracy. REMIND remains competitive or superior under matched memory constraints, while performance depends on reconstruction and replay choices.

  • S3.1 Buffer Size Comparisons: At 0.75 GB, REMIND and BiC are top performers; at 1.51 GB and 3.01 GB, BiC is best, while REMIND trails by 4% and 6.6%.REMIND requires less than 12 hours versus nearly 65 hours for BiC at 1.51 GB, and its Ωall changes by only 1.3% across the tested buffer range.
  • S3.1 Buffer Size Comparisons: REMIND outperforms all comparison models on CORe50 across streaming and incremental-batch settings at every tested buffer size.At 200 exemplars, REMIND uses one-third as much memory as comparison models and does not require their additional distillation cache.
  • S3.3 Changing F(·) and G(·): Adding trainable layers improves CORe50 accuracy with diminishing returns, while earlier feature extraction increases memory because intermediate features have larger spatial dimensions.This experiment concerns the choice of features used to train the plastic component θF.
  • S3.4 Product Quantization Settings: REMIND’s performance improves as PQ codebook count and codebook size increase, but memory efficiency decreases.Replay quality depends on tensor reconstruction quality from the product-quantization codebooks.
  • S3.5 Altering Replay: Increasing replay samples beyond 20 degrades CORe50 performance, whereas increasing ImageNet replay from 20 to 50 samples improves Ωall by 0.6%.The authors hypothesize that larger replay sets overfit CORe50 because it contains fewer samples.
  • S3.6 Average Accuracy for ImageNet and CORe50: Using average accuracy over testing intervals, REMIND remains the top streaming performer and is competitive in incremental-batch ImageNet experiments.The metric uses top-5 accuracy for ImageNet and top-1 accuracy for CORe50.
  • S3.6 Average Accuracy for ImageNet and CORe50: CORe50 average accuracy is lower for class orderings because evaluation includes unseen classes, whose low accuracies reduce μall.This evaluation choice affects the reported aggregate metric rather than indicating a separate model mechanism.
  • S3.6 Average Accuracy for ImageNet and CORe50: On CORe50, iid and instance orderings are easiest, class orderings are hardest, and REMIND differs significantly from every comparison model across all four orderings.The results average performance and standard deviations over 10 dataset permutations and use a Student’s t-test at a 99% confidence interval.

S4 Additional VQA Experiments

Additional VQA experiments test REMIND across buffer sizes and data orderings on TDIUC and CLEVR. The results indicate strong performance with limited memory and learning curves that closely track offline behavior for q-type ordering.

  • S4 Additional VQA Experiments: A 25% buffer rivals, and sometimes outperforms, a 100% buffer for REMIND on streaming VQA.The experiments vary buffer sizes from 25% to 100% of the complete training set on TDIUC and CLEVR.
  • S4 Additional VQA Experiments: REMIND’s q-type-ordering learning curves closely follow the offline curves on both TDIUC and CLEVR.The reported behavior indicates learning new question types without forgetting old ones.
  • S4 Additional VQA Experiments: Figure S6 provides qualitative VQA examples from REMIND trained with a 50% buffer on q-type ordering for TDIUC and CLEVR.The top row shows TDIUC examples and the bottom row shows CLEVR examples.
Loading 1910.02509v3…