PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning without Forgetting

Zhizhong Li, Derek Hoiem

arXiv:1606.09282v3cs.CVcs.LGstat.ML

TL;DR

Adding new capabilities to a CNN without access to training data for existing tasks risks degrading prior performance. Learning without Forgetting uses new-task data to learn the new capability while preserving old-task outputs, performing similarly to joint training and often outperforming fine-tuning on the new task.

  • Problem

    Vision systems need to learn new capabilities while preserving existing performance, despite unavailable, proprietary, or cumbersome legacy training data.

  • Method

    Learning without Forgetting combines fine-tuning with knowledge distillation, using only new-task images and labels while preserving the original network’s outputs for existing tasks.

  • Results

    The method performs similarly to joint training, often outperforms fine-tuning on the new task, and preserves old-task performance without requiring previous-task data.

  • Takeaways & Limitations

    Learning without Forgetting offers a practical way to expand an existing vision system when previous-task training data are unavailable.

  • Takeaways & Limitations

    The method has been demonstrated mainly for image classification, with only one tracking experiment, leaving segmentation, detection, and non-vision applications for future work.

Abstract

from arXiv · show

When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

1 INTRODUCTION

The paper addresses adding visual capabilities to CNN systems without retaining prior-task training data while preserving existing performance. Learning without Forgetting uses only new-task examples to optimize new-task accuracy and preserve responses from the original network.

  • Problem: Practical vision systems must learn new visual capabilities while maintaining performance on existing ones.Examples include adding site-specific object models or detecting improper footwear to an existing safety system.
  • Limitations of Existing Strategies: Feature extraction underperforms on new tasks, fine-tuning harms previous tasks, and duplicated specialized networks increase test time linearly with added tasks.These drawbacks arise from insufficient shared representations, drift in shared parameters, and maintaining separate networks.
  • Learning without Forgetting: Learning without Forgetting optimizes new-task accuracy and preservation of original-task responses using only examples from the new task.It is similar to joint training but does not require old-task images or labels.
  • Learning without Forgetting: Preserving original-network outputs on new-task examples remains effective despite poor sampling of the old-task domain and can regularize new-task learning.The paper reports that this strategy preserves old-task performance while improving new-task performance.
  • Advantages: Learning without Forgetting outperforms feature extraction and fine-tuning on the new task, greatly outperforms fine-tuned shared parameters on the old task, and improves efficiency and deployment simplicity.Training is faster than joint training and only slightly slower than fine-tuning; testing is faster than using multiple fine-tuned networks, and prior training data need not be retained.

2 RELATED WORK

Learning without Forgetting combines distillation and fine-tuning to learn new tasks while preserving old-task outputs using only new-task data. It differs from feature extraction, multitask learning, architectural expansion, and related continual-learning methods in their parameter updates, data requirements, or preservation mechanisms.

  • Learning without Forgetting: Learning without Forgetting combines distillation with fine-tuning, optimizing shared and task-specific parameters using new-task data to train new tasks and guide old-task outputs.The same data supervises new tasks and provides unsupervised output guidance for old tasks.
  • Feature Extraction and Fine-tuning: Feature extraction uses fixed pre-trained CNN activations for new-task classifiers, whereas fine-tuning modifies existing parameters, extends the output layer, and may freeze convolutional layers.With suitable hyperparameters, fine-tuning often outperforms feature extraction.
  • Multitask Learning: Multitask learning jointly optimizes tasks using data from all tasks, typically sharing bottom layers and keeping top layers task-specific.Each task’s data regularizes parameters shared or constrained across tasks.
  • Parameter-Preserving Expansion: Adding nodes preserves original parameters while learning new features, but substantially expands the network and can underperform fine-tuning and feature extraction with insufficient data.Examples include Deep Block-Modular Neural Networks and Progressive Neural Networks.
  • Continual and Related Methods: Unlike related lifelong-learning and continual-task methods, Learning without Forgetting preserves existing-task performance without original data and jointly optimizes representation and final layers.Compared with A-LTM, it reports maintaining good performance; compared with Less Forgetting Learning, it outperforms on the new task.

3 LEARNING WITHOUT FORGETTING

Learning without Forgetting adds new task capabilities using only new-task data while preserving old-task outputs through recorded responses and output regularization. It jointly optimizes shared and task-specific parameters, avoiding stored old datasets while retaining advantages over simpler adaptation methods.

  • Learning without Forgetting: Learning without Forgetting adds new-task parameters and learns shared and task-specific parameters using only new-task images and labels, without existing-task data.The method substitutes recorded old-task responses on new images for unavailable old-task training data.
  • Learning without Forgetting: The method records old-task outputs on each new-task image, adds randomly initialized output nodes for new classes, and then trains the expanded network.New parameters equal new classes times nodes in the last shared layer and are typically a very small percentage of total parameters.
  • Optimization: The procedure first freezes shared and old-task parameters to train new parameters, then jointly optimizes all parameters until convergence.The warm-up step greatly improves fine-tuning’s old-task performance but is less crucial for Learning without Forgetting and Less Forgetting.
  • Loss functions: Training uses a new-task multinomial logistic loss plus knowledge-distillation losses that keep old-task output probabilities close to the original network.For multiple tasks or labels, the corresponding losses are summed; knowledge distillation increases the weight of smaller probabilities.
  • Loss functions: Constraining original-task outputs is important, while the specific similarity measure is not crucial; knowledge distillation performs slightly better but similarly to other reasonable losses.The old-new task tradeoff can be adjusted through λo, which was set to 1 in most experiments.
  • Efficiency comparison: LwF is slightly slower than fine-tuning but faster than joint training, while all methods take approximately the same time to evaluate a test image.Duplicating and fine-tuning the network for each task takes m times as long to evaluate, where m is the total number of tasks.

4 EXPERIMENTS

The experiments evaluate whether Learning without Forgetting preserves old-task performance while learning new tasks without original-task training data. Across single-task, cumulative-task, dataset-size, and ablation studies, LwF generally outperforms adaptation baselines while retaining old capabilities.

  • Experimental setup: Experiments compare LwF with feature extraction, fine-tuning, fine-tuning FC, and Less Forgetting Learning without requiring original-task training data.The study uses image-classification tasks with varying inter-task similarity, primarily on AlexNet, with additional verification on 16-layer VGGnet.
  • Single new task scenario: On new tasks, LwF consistently outperforms fine-tuning, LFL, fine-tuning FC, and feature extraction except for ImageNet→MNIST and Places365→CUB under fine-tuning.The improvement over fine-tuning suggests that preserving old-task outputs acts as an effective regularizer, motivating LwF as a standard adaptation approach.
  • Multiple new task scenario: During cumulative task addition, LwF degrades more slowly over time than fine-tuning and generally preserves performance across older and newly added tasks.Older-task responses are recomputed after each addition to emulate the absence of training data for all original tasks.
  • Influence of dataset size: With CUB subsampled to 30%, 10%, and 3%, LwF maintains the same qualitative advantage over fine-tuning on both tasks.The experiment trains on reduced CUB subsets and evaluates on the entire validation set.
  • Ablation studies: Network Expansion + LwF performs similarly to LwF with additional computational cost and complexity, while expansion alone trails LwF on the new task.Simply lowering the shared-parameter learning rate fails to preserve old-task performance and also reduces new-task performance.
  • Ablation studies: LwF's knowledge-distillation loss slightly outperforms the compared response-preserving losses, although the advantage is not large.It also outperforms the evaluated baseline shown in Figure 7, whose result lies between feature extraction and another adaptation approach.

5 DISCUSSION

The discussion presents Learning without Forgetting (LwF) as a way to add new capabilities without original-task data while preserving existing performance. It also highlights benefits over fine-tuning, tracking improvements, and directions for broader evaluation and online learning.

  • Method and implications: LwF adapts CNNs to new tasks without original-task training data by combining knowledge distillation with fine-tuning.It learns parameters discriminative for the new task while preserving original-task outputs on new-task training data.
  • Tracking application: In tracking, replacing online fine-tuning with LwF produced some improvements, although the difference was not fully reported in the passage.The experiment builds on MD-Net, which treats tracking as template classification.
  • Method and implications: For expanding predictions, LwF performs similarly to joint training while training faster and avoiding previous-task data.For new-task performance alone, it often outperforms standard fine-tuning.
  • Limitations and future work: The authors demonstrated LwF for image classification and one tracking experiment, and propose further evaluation on semantic segmentation, detection, and tasks outside computer vision.They also identify representative unlabeled images, theoretical performance bounds, and online learning across tasks as future directions.

APPENDIX A TRACKING WITH MD-NET USING LWF

The appendix tests Learning without Forgetting beyond classification by replacing fine-tuning in MD-Net for causal general object tracking. LwF slightly improves tracking performance over MD-Net, though the supplied passage truncates the qualification of this difference.

  • Tracking setup: The experiment evaluates LwF for general object tracking, where each frame requires predicting the tracked object’s bounding box from the first frame’s known groundtruth box.The tracker is expected to be causal, so frame t’s result should not depend on later frames t + 1 onward.
  • MD-Net adaptation: MD-Net treats tracking as template classification and fine-tunes a general network jointly trained on multiple videos into a classifier for each test video.The paper identifies fine-tuning as potentially causing undue drift from the original parameters.
  • LwF result: Replacing MD-Net’s fine-tuning with LwF slightly improves tracking performance, but the supplied passage truncates the statement qualifying the difference.The experiment tests whether LwF can preserve or improve tracking while avoiding potential drift from the original parameters.

A.1 MD-Net

MD-Net tracks objects by classifying nearby bounding boxes and refining the highest-scoring candidate with bounding-box regression. At test time, it adapts fully connected layers to the first frame and later samples while freezing convolutional layers, and MD-Net + LwF appears to improve MD-Net on VOT 2015 without a statistically significant difference.

  • A.1 MD-Net: MD-Net samples bounding boxes near the previous frame’s box, classifies them as foreground or background, then regresses the highest-scoring box.The reported regression result becomes the object location.
  • A.1 MD-Net: At test time, MD-Net discards final layers, inserts a randomly initialized layer, freezes convolutional layers, and trains the remaining network on first-frame samples.A bounding-box regression layer is trained on first-frame data and kept unchanged.
  • A.1 MD-Net: MD-Net later updates its fully connected layers using samples from previous frames while tracking consequent frames.
  • A.1 MD-Net: VOT 2015 evaluates MD-Net using expected average overlap, combining tracking accuracy and robustness over 15 runs.
  • A.1 MD-Net: MD-Net + LwF appears to improve upon MD-Net on VOT 2015, but the difference is not statistically significant.

A.2 MD-Net + LwF

MD-Net + LwF applies Learning without Forgetting during online training to preserve old-task parameters while adapting to confined new-task data. It slightly improves MD-Net, although the improvement is not statistically significant and single-run overlap varies greatly.

  • Motivation: LwF replaces the usual fine-tune FC online-training step because it generally outperforms fine-tune FC on the new task.The additional regularization is intended to benefit updates from crops drawn from a single video.
  • Method: The method keeps task-specific final layers as old-task parameters and retains a copy of the original pretrained network to compute old-task responses during online training.The original network is needed because online new-task updates change the working network.
  • Implementation: The procedure pretrains on OTB-100 excluding VOT 2015 sequences, then tests the tracking algorithm on VOT 2015 for 15 runs.Tracking and testing otherwise remain unchanged from MD-Net.
  • Results: Our method slightly improves MD-Net, which reports 0.386, but the improvement is not statistically significant.Both methods start from the same pretrained network, while single-run expected average-overlap scores vary greatly.

APPENDIX B SPLIT OF VOC AND SCENE

Appendix B describes how the VOC and Scene new tasks were divided into three category groups each for the multiple-new-task experiment.

  • VOC: VOC was split into Transport, Animals, and Objects categories.Transport includes vehicles; Animals includes animal and person classes; Objects includes household and furniture-related classes.
  • Scene: Scene was split into Large rooms, Medium rooms, and Small rooms.The appendix lists scene classes assigned to each room-size group, and notes that this split is also used in.
Loading 1606.09282v3…