Source-linked AI summary
FearNet: Brain-Inspired Model for Incremental Learning
Ronald Kemker, Christopher Kanan
TL;DR
Incremental class learning challenges standard DNNs because classes arrive sequentially and prior examples may be unavailable, causing catastrophic forgetting and making exemplar storage difficult to scale. FearNet addresses this with a generative, brain-inspired dual-memory system that consolidates recent memories into long-term storage without storing previous examples. It achieves state-of-the-art performance on large image and audio incremental-learning benchmarks while maintaining a relatively small memory footprint.
Problem
Incremental class learning exposes standard DNNs to catastrophic forgetting, while scalable alternatives should avoid storing the entire training dataset.
Method
FearNet uses HC- and mPFC-inspired memory systems, a BLA-inspired recall selector, and generative pseudorehearsal to consolidate memories without storing previous training examples.
Results
FearNet outperforms existing methods on large image and audio classification benchmarks and achieves state-of-the-art results with a relatively small memory footprint.
Takeaways & Limitations
FearNet demonstrates that dual-memory models can retain and consolidate recently learned information while retaining older information with lower memory requirements.
Takeaways & Limitations
The evaluated incremental-class-learning setting assumes that every post-base study session contains examples from only one class.
Abstract
from arXiv · showhide
Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall. FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.
1 INTRODUCTION
Incremental class learning requires sequentially learning classes without necessarily revisiting earlier examples, exposing standard DNNs to catastrophic forgetting and scalability challenges. FearNet addresses this with a brain-inspired, memory-efficient dual-memory architecture that consolidates recent memories into long-term storage without retaining previous training examples.
- Problem: Incremental classification presents sequential batches and may prevent learners from re-studying previously seen examples.The base session contains exemplars from multiple classes, while subsequent sessions contain only one class.
- Problem: Standard DNNs suffer catastrophic forgetting when learning new data, while offline retraining with old examples is costly and difficult to scale.The motivation includes on-device platforms with limited computing power, storage, and memory.
- FearNet: FearNet combines recent-memory, long-term-storage, and recall-selection subsystems inspired by HC, mPFC, and BLA.HC supports recent recall, mPFC stores long-term memories, and BLA determines which system to use.
- FearNet: FearNet uses pseudorehearsal with a generative autoencoder to replay previously learned examples during consolidation without storing prior training data.The generated examples are replayed alongside novel information as recent memories are consolidated into long-term storage.
- Results: FearNet achieves state-of-the-art results on large image and audio datasets with a relatively small memory footprint.The result is presented as evidence that dual-memory models can be scaled.
2 RELATED WORK
Related work addresses catastrophic forgetting through rehearsal, architectural constraints, self-organizing memories, and exemplar-based retention. These approaches reduce forgetting but often require storing prior data or trade off retention against learning new information.
- Architectural methods: Other approaches mitigate forgetting through separate memory paths, sparse updates, frozen weights, or permutations of training data.These include the dual-memory proposal of French, FEL, PathNet, and methods for permuted datasets.
- Rehearsal: Rehearsal mixes old and new examples to reduce forgetting under non-iid training, but requires storing previous training data.Pseudorehearsal instead generates new examples for previously learned classes.
- Architectural methods: GeppNet uses a self-organizing map as long-term memory and updates it only when inputs are sufficiently novel.GeppNet+STM adds a fixed-size buffer, improving base-knowledge retention while the STM-free version learns new data better.
- Exemplar methods: iCaRL’s older-information recall is severely impaired when exemplars per class decrease from 20 to 1.The figure contrasts EPC = 20 with EPC = 1.
- Exemplar methods: iCaRL rehearses stored exemplars selected by herding, retaining J/T exemplars per class after learning the dataset.For CIFAR-100, the paper reports J = 2,000 and J/T = 20.
3 MAMMALIAN MEMORY: NEUROSCIENCE AND MODELS
FearNet draws on a mammalian dual-memory theory in which HC supports recent memories and mPFC supports remote memories. Neuroscience findings further motivate sleep-based consolidation and BLA-mediated shifts in memory retrieval as memories mature.
- Dual-memory theory: The mammalian dual-memory model assigns recent-memory recall to HC and remote-memory recall to mPFC.The theory has considerable experimental support from neuroscience.
- Hippocampal complex: HC is associated with forming new memories through highly plastic neurons in the dentate gyrus.Adult neurogenesis is described as one mechanism supporting new memory creation.
- Memory consolidation: mPFC contributes strongly to long-term memory consolidation during REM sleep.The proposed mechanism involves HC reactivating recent memories that then replay in mPFC during sleep.
- Recall selection: BLA can shift memory retrieval between HC and mPFC as memories are consolidated over time.This role was observed in contextual fear-conditioning experiments in mice and motivated FearNet’s recall-selection module.
4 THE FEARNET MODEL
FearNet combines complementary short- and long-term memory systems with a BLA controller that selects the memory source during prediction. During sleep, an autoencoder generates pseudo-examples to consolidate recent memories into long-term storage without retaining original examples.
- Dual-memory architecture: FearNet combines HC for immediate recent-memory recall, mPFC for remote-memory storage, and BLA for selecting the memory center during prediction.BLA determines whether an input should be classified using HC or mPFC.
- mPFC training: mPFC jointly optimizes supervised classification and unsupervised reconstruction losses.The classification term uses softmax loss, while reconstruction uses weighted mean squared errors across layers.
- Pseudo-example generation: mPFC stores class-specific encoder statistics, including mean feature vectors and covariance matrices, to generate pseudo-examples during consolidation.For each class, FearNet samples a Gaussian and decodes the sample into a pseudo-example, balancing the consolidation set across classes.
- Memory consolidation: During sleep, FearNet consolidates HC memories into mPFC through intrinsic replay using pseudo-examples generated by an autoencoder.The pseudo-examples replace the original HC inputs during transfer.
- Prediction routing: BLA outputs a value between 0 and 1 indicating whether mPFC should be used, while FearNet combines this signal with HC and mPFC predictions.The predicted class is selected from HC when its confidence-weighted probability exceeds the mPFC alternative; otherwise mPFC supplies the prediction.
5 EXPERIMENTAL SETUP
FearNet is evaluated for incremental class learning using normalized image and audio embeddings from three benchmark datasets, compared with established incremental-learning methods and offline performance. Evaluation tracks retention of base knowledge, recall of new classes, and accuracy across all classes seen so far.
- Metrics: The evaluation computes accuracy on the newest class, the base-knowledge classes, and all test data seen up to each study session.These quantities are denoted αnew,t, αbase,t, and αall,t.
- Metrics: Ωbase and Ωall are normalized against an offline MLP baseline, with 1 indicating similar performance to that baseline.Values above 1 occur only when the incremental method is more accurate than the offline model.
- Datasets: FearNet is evaluated on CIFAR-100, CUB-200, and AudioSet, covering object, fine-grained bird, and audio classification.CIFAR-100 contains 100 object categories, CUB-200 contains 200 bird species, and AudioSet provides ten-second sound-clip embeddings.
- Implementation: The models use pre-extracted embeddings, while FearNet’s mPFC and BLA are trained with NAdam and tuned across architecture and sleep-frequency hyperparameters.The reported search varies layer depth, unit count, and how often consolidation occurs.
- Comparison models: FearNet is compared with FEL, GeppNet, GeppNet+STM, iCaRL, and 1-NN using the same input feature embeddings for each dataset.The comparisons include iCaRL as a state-of-the-art incremental-class-learning method and 1-NN because of its similarity to HC.
- Results visualization: Figure 4 reports mean-class test accuracy for all classes seen so far.This figure is used to compare incremental models with the offline MLP baseline over the learning sequence.
6 EXPERIMENTAL RESULTS
FearNet is evaluated on standard incremental-learning benchmarks, memory-consolidation frequency, and multimodal learning. It achieves the strongest reported overall retention metrics, while performance depends on sleep frequency and the quality of initial base-knowledge.
- 6.1 STATE-OF-THE-ART COMPARISON: FearNet achieves the best Ωbase and Ωall scores on CIFAR-100, CUB-200, and AudioSet among the six evaluated methods.Ωbase measures retention of base knowledge, while Ωall measures integration of new memories over time; both are normalized to the offline MLP baseline.
- 6.1 STATE-OF-THE-ART COMPARISON: FearNet’s BLA generally predicts which memory network to use, but its lower Ωnew indicates occasional selection of mPFC when HC is more appropriate.The comparison uses an oracle that knows the associated memory location as a reference.
- 6.2 ADDITIONAL EXPERIMENTS: Sleeping less often improves retention of base-knowledge but reduces recall of newly learned information.On CUB-200, increasing the number of classes learned before sleep improves base-knowledge retention while reducing new-information recall; repeated sleep can also gradually weaken older mPFC memories.
- 6.2 ADDITIONAL EXPERIMENTS: FearNet can incrementally learn multimodal CIFAR-100 and AudioSet inputs when its initial base-knowledge is strong.When starting with lower base-knowledge performance, such as AudioSet, the model struggles to learn new information incrementally.
- 6.2 ADDITIONAL EXPERIMENTS: Increasing the number of base-knowledge classes produces a noticeable increase in overall model performance on CUB-200.The paper attributes this to better mPFC representations from more data and fewer remaining incremental-learning steps.
7 DISCUSSION
The discussion emphasizes FearNet’s low memory footprint and examines trade-offs in its class-statistics representation. It also identifies open questions and boundaries for multimodal and broader adaptations.
- 7 DISCUSSION: FearNet’s memory footprint is comparatively small because it stores class statistics rather than some or all raw training data.The memory comparison considers fixed model capacity and extrapolates requirements from CIFAR-100 to 1,000 classes.
- 7 DISCUSSION: Updating class statistics when a class appears in multiple study sessions remains an open question.Possible approaches include running updates for class means and covariances or favoring more recent data because of autoencoder learning.
- 7 DISCUSSION: FearNet assumes normally distributed mPFC encoder outputs for each class, although a more complex model such as a Gaussian Mixture Model may be appropriate.The assumption concerns the class statistics used to represent memories.
- 7 DISCUSSION: Replacing each full class covariance matrix with a diagonal matrix reduces model performance but preserves a functioning FearNet variant.The full covariance matrix is identified as the largest contributor to model size.
- 7 DISCUSSION: Extensions to unsupervised learning and regression are possible, but regression requires changing mPFC’s loss function and may require grouping similar input vectors.A supervised data-permutation experiment would additionally require storing statistics from prior permutations and classes.
8 CONCLUSION
FearNet is a brain-inspired framework for incremental learning across modalities and object classes. It outperforms existing methods while retaining older information and using less memory.
- FearNet incrementally learns data across different modalities and object classes.
- FearNet outperforms existing incremental class-learning methods on large image and audio classification benchmarks.
- FearNet recalls and consolidates recently learned information while retaining older information.
- FearNet is more memory efficient for platforms with limited size, weight, and power budgets.
A.1 MODEL HYPERPARAMETERS
The appendix reports FearNet, iCaRL, GeppNet, and FEL training parameters, including searched hyperparameter ranges and implementation choices.
- Training parameters are reported for FearNet separately for each dataset.
- Training parameters for iCaRL and GeppNet variants are also documented for the experiments.
- The appendix identifies hyperparameter search spaces when values are given as ranges.
- The FEL layer uses a number of units defined by the accompanying training-parameter specification.
A.2 ICARL PERFORMANCE WITH MORE EXEMPLARS
Additional experiments examine iCaRL with more stored exemplars, alternative BLA variants, multimodal base knowledge, and the effect of base-knowledge size on FearNet.
- A.2 ICARL PERFORMANCE WITH MORE EXEMPLARS: 100 exemplars per class improve iCaRL performance but still do not outperform FearNet.CUB-200 has about 30 training samples per class, so 100 exemplars store the entire training set there; main results use 20.
- A.3 BLA VARIANTS: The binary MLP BLA variant works best at selecting whether recall should use HC or mPFC.Isolation Forest and Elliptic Envelope tend to prefer HC, while one-class SVM prefers mPFC.
- A.5 MULTI-MODAL LEARNING EXPERIMENT: FearNet works well when it adequately learns base knowledge, but poor base-knowledge learning deteriorates incremental performance.The multimodal experiments evaluate mean-class accuracy on both the base-knowledge and entire test sets.
- A.6 BASE-KNOWLEDGE EFFECT ON PERFORMANCE: As base-knowledge size increases, base and overall performance increase, while new-class performance remains relatively stable.The slight decrease in new-class performance corresponds to occasional erroneous mPFC choices by BLA.