Source-linked AI summary
Less-forgetting Learning in Deep Neural Networks
Heechul Jung, Jeongwoo Ju, Minju Jung, Junmo Kim
TL;DR
Catastrophic forgetting causes DNNs to lose previously learned information when learning data from new environments. This paper proposes a source-data-free less-forgetting method, finds forgetting between mini-batches, and reports improved recognition and generalization performance.
Problem
Catastrophic forgetting causes DNNs to forget source-domain information during target-domain learning, even when source and target domains are nearly homogeneous.
Method
The paper proposes a less-forgetting learning method that maintains the source-domain feature space without accessing source-domain training data.
Results
The method is reported to reduce forgetting in domain adaptation and general learning while improving generalization and recognition performance.
Takeaways & Limitations
Less-forgetting learning preserves previously learned information while adapting to target data and can also improve deep-network performance.
Abstract
from arXiv · showhide
A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.
I. INTRODUCTION
The paper introduces less-forgetting learning to reduce catastrophic forgetting during domain adaptation without using source-domain data. It also identifies forgetting between mini-batches and reports improved generalization performance.
- Motivation: Catastrophic forgetting occurs when adapting a network to target data, including when source and target domains are nearly homogeneous.Traditional transfer learning copies source weights and fine-tunes on target data, causing previously learned source information to be forgotten.
- Related work: Prior methods alleviate forgetting but may require source-domain information or lack an explicit guarantee of unforgetting.Some approaches compute source-domain cluster centroids, limiting applicability to pre-trained models without accessible source data.
- Proposed method: The proposed method maintains the original source-domain feature space after retraining only with target data.Features from the same source class and target data remain well clustered under the proposed method.
- General learning: Forgetting also occurs between mini-batches during stochastic-gradient-descent training because mini-batches are small subsets of the full dataset.The proposed method is applied to this general learning setting as well.
- Results: The authors report that less-forgetting learning alleviates forgetting and provides better generalization performance.The method is presented as a way to improve recognition performance while retaining previously learned information.
A. Less-forgetting Problem
The less-forgetting problem is formulated for a pre-trained source network adapted with target data when source data are unavailable. The target network should preserve the source network's outputs on source-domain inputs.
- Problem setting: The setting assumes access to pre-trained source-model weights and target-domain training data, but not source-domain data.This captures adaptation to a new domain under a no-source-data constraint.
- Datasets: The source and target datasets are mutually exclusive and each is divided into training and validation subsets.The source dataset contains source-domain examples, while the target dataset contains target-domain examples.
- Source network: The source network F(x; θ(s)) is trained on the source dataset, producing source-domain weight parameters θ(s).The weights are initialized randomly before source-domain training.
- Target network: The target network F(x; θ(t)) is trained using the target dataset to obtain updated parameters θ(t).The target-learning process uses the target-domain data after source-model training.
- Less-forgetting condition: Less-forgetting requires the target network to remain approximately equivalent to the source network for source-domain inputs.The condition is expressed as F(x; θ(t)) ≈ F(x; θ(s)) for x ∈ D(s).
II. LESS-FORGETTING LEARNING
The method preserves source-domain decision boundaries and feature representations while learning target-domain data without source examples. It combines classification and feature-similarity objectives using source-network weights as initialization.
- The method targets two properties: unchanged decision boundaries and source-like feature locations in the target network.
- Target-domain training uses target data to encourage features similar to those extracted by the source network, despite inaccessible source data.
- It initializes the target network with source-network weights and freezes the softmax-layer weights to preserve classifier boundaries.
- The total objective combines cross-entropy classification loss and Euclidean feature loss, weighted by λc and λe.Usually, λe is smaller than λc.
- The Euclidean loss compares layer L −1 feature vectors from the source and target networks.
III. LESS-FORGETTING FOR GENERAL LEARNING CASES
The paper identifies forgetting between mini-batches during ordinary stochastic-gradient training and adapts less-forgetting learning to this setting. The modified method periodically updates source parameters and unfreezes classifier boundaries.
- Forgetting can arise between mini-batches during general stochastic-gradient training, not only when learning a new task.
- The experiment tracks one 100-example mini-batch over 30 epochs, recording its loss every 50 iterations during unshuffled training.
- The less-forgetting graph is smoother than traditional learning, but its training loss is initially higher because freezing the boundary obstructs learning new data.
- The modified algorithm periodically switches source parameters from target parameters and repeatedly unfreezes the network boundary.
- A larger Ns reduces adaptation to new data, while Nf has a similar role to Ns in controlling the less-forgetting behavior.
IV. EXPERIMENTS
The experiments evaluate the proposed algorithms through unforgetting and generalization tests. The unforgetting test uses manually constructed source and target domains, while the generalization test uses CIFAR-10.
- The study establishes separate unforgetting and generalization recognition experiments for Algorithms 1 and 2, respectively.
- The unforgetting experiment constructs source and target domains from CIFAR-10, MNIST, and SVHN, whereas the generalization experiment uses CIFAR-10.
- The experiments compare the proposed methods with dropout-based approaches and evaluate recognition performance.
A. Unforgetting Test
The unforgetting test evaluates whether target-domain learning preserves source-domain recognition without source examples. It uses split and transformed CIFAR-10 domains, plus an MNIST-to-SVHN digit-domain test.
- The test measures source-information retention while the network also performs on target data, with source examples inaccessible during target learning.
- CIFAR-10 is split into 40,000 source images and 10,000 target images, with target pixels converted before training and testing.
- The digit unforgetting test uses MNIST as the source domain and SVHN as the target domain.
- The proposed method predicts true labels more accurately than traditional transfer learning, LWTA, and Maxout in the reported experiments.
- In CIFAR10, the highest target recognition rate of less forgetting exceeds those of transfer learning, LWTA, and Maxout.
B. Generalization Test
The generalization test compares methods as training progresses and examines the target–source accuracy trade-off through λe. Less-forgetting learning improves performance relative to methods without it, with its CIFAR-10 target rate exceeding several alternatives.
- The test reports accuracy for original, batch normalization, batch normalization plus less forgetting, and less-forgetting learning as iterations increase.
- Figure 4 plots source accuracy against target accuracy while varying λe in Equation 1.
- The left plot covers CIFAR-10 object recognition, while the right covers MNIST source and SVHN target digit recognition.
- Less-forgetting learning improves generalization performance over methods without it across the reported cases.
V. CONCLUSION
The paper proposes less-forgetting learning to alleviate catastrophic forgetting and reports that forgetting also occurs during ordinary stochastic-gradient training. The method mitigates this problem and improves deep-network generalization.
- The paper proposes a less-forgetting learning method to alleviate catastrophic forgetting in deep neural networks.
- Forgetting occurs during general learning with stochastic gradient descent, and the proposed method mitigates this phenomenon.
- Less-forgetting learning is useful for improving the generalization ability of deep neural networks.