Source-linked AI summary
Compacting, Picking and Growing for Unforgetting Continual Learning
Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, Chu-Song Chen
TL;DR
Continual learning must learn sequential tasks without access to previous-task data while avoiding catastrophic forgetting. The paper proposes CPG, an iterative loop of compaction, critical-weight picking, and optional growth that preserves old knowledge and limits model expansion. Experiments report unforgetting, compact models, and improved learning of new tasks using accumulated knowledge.
Problem
Continual learning lacks previous-task training data, while fine-tuning new tasks can cause catastrophic forgetting.
Method
CPG iteratively prunes redundant newly added weights, selectively reuses critical old-task weights with masks, and expands the architecture when necessary.
Results
CPG preserves previously learned function mappings exactly, maintains compactness with slight or no architecture growth, and improves learning of subsequent tasks through accumulated knowledge.
Takeaways & Limitations
The approach provides a simple continual-learning method for multiple sequential tasks with limited additional space and no forgetting.
Takeaways & Limitations
Regularization approaches remain unsuitable for unlimited sequential tasks when previous data are unavailable and network capacity is fixed and limited.
Abstract
from arXiv · showhide
Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.
1 Introduction
The paper targets continual learning without previous-task data, combining compaction, critical-weight picking, and optional growth to preserve prior skills while keeping models compact. CPG also uses accumulated knowledge to improve learning of subsequent tasks.
- Problem: Continual learning must handle unknown sequential tasks without previous-task training data, because fine-tuning new tasks causes catastrophic forgetting.
- Method: CPG repeatedly combines model pruning, critical-weight selection, and progressive architecture expansion for sequential tasks.
- Picking and Growing: A differentiable binary mask selects critical old-task weights for new-task training, while released weights are reused; expansion follows only if the performance goal remains unmet.
- Compacting: Gradual pruning removes newly added redundant weights while retraining remaining weights until a predefined accuracy goal is met; preserved old-task weights remain unchanged.
- Results: The approach preserves previous function mappings exactly when new tasks are added, avoiding forgetting.
- Results: CPG permits architecture expansion while retaining compactness, and experiments show multiple tasks can be condensed with slight or no architecture growth.
- Results: The accumulated condensed model serves as a knowledge base for picking weights and enhances learning of subsequent tasks compared with independent task training.
2 Related Work
Prior continual-learning methods use regularization, replay, or dynamic architectures, but each has limitations involving forgetting, memory, or model growth. CPG combines compression with selective reuse to preserve old-task performance while limiting expansion.
- Categories: Continual lifelong learning methods are grouped into network regularization, memory or data replay, and dynamic architecture.
- Network regularization: Regularization methods restrict updates to learned weights, but missing old-task data and fixed capacity can cause gradual forgetting, especially for earlier tasks.
- Memory replay: Memory-replay methods use additional models, such as GANs, to retain or regenerate previous-task data information for training new tasks.
- Dynamic architecture: Dynamic-architecture methods reduce forgetting by expanding models for new tasks, but their structures grow monotonically and can become redundant.
- CPG: Unlike DEN, CPG selects critical old-task weights without modifying them, avoids performance restoration, and experimentally outperforms DEN and PAE.
3 The CPG approach for Continual Lifelong Learning
The CPG approach alternates pruning, weight selection, and selective architecture growth across sequential tasks. Preserved old-task weights remain fixed for recall, while released and newly added weights support subsequent tasks.
- Task 1: CPG begins with gradual pruning that removes redundant weights while restoring performance, dividing the model into preserved task weights and released weights.Pruning is performed iteratively rather than once, with retraining after each removal until the pruning criteria are met.
- Task k to k+1: For each new task, a learnable binary mask selects critical weights from the compact model, which are combined with released weights for training.The mask is learned through a real-valued surrogate and thresholded for the forward pass.
- Task k to k+1: If the new task fails to meet its accuracy goal, CPG expands the architecture with additional filters or nodes and resumes training.Previously released weights are used before further expansion is introduced.
- Task k to k+1: During new-task training, selected old weights remain fixed while the mask and additional weights adapt, allowing old tasks to be exactly recalled.The fixed weights are picked from the preserved old-task model rather than updated by back-propagation.
- Compaction of task k+1: After training, gradual pruning compresses the new-task model, producing preserved weights for all learned tasks and released redundant weights for later tasks.This compaction and picking/growing loop repeats from task to task.
4 Experiments and Results
Across CIFAR-100, fine-grained classification, and facial-informatic tasks, CPG incrementally learns without forgetting while maintaining comparatively compact models. Its results indicate advantages over fine-tuning and alternative continual-learning methods, especially for later tasks and model size.
- CIFAR-100 twenty tasks: CPG outperforms fine-tuning on tasks 5, 10, and 15, suggesting accumulated knowledge improves performance on subsequent tasks.The methods have roughly the same initial task-1 accuracy because both are trained from scratch.
- CIFAR-100 twenty tasks: 1.5× expansion with 0.41× redundant weights makes CPG more compact than PAE's 2× expansion while achieving better accuracy through critical-weight picking.PackNet and PAE ensure unforgetting, but PAE uses all old weights and grows to twice the original weight count.
- CIFAR-100 twenty tasks: Including overheads, CPG model sizes are 2.16×, 2.40×, and 2.41× the original VGG16-BN, versus 20× for independent models preserving old-task accuracy.Higher accuracy goals generally improve performance but consume more weights; the lowest setting retains 0.41× redundant weights for future use.
- Fine-grained image classification tasks: On six fine-grained image-classification tasks, CPG outperforms competing methods on almost all later tasks and has model size comparable to Piggyback.CPG and PackNet perform slightly worse on the first ImageNet task because both compress ResNet50 through pruning.
5 Conclusion and Future Work
CPG combines compacting, picking, and growing to support continual learning without forgetting while limiting model expansion. The authors report favorable accuracy with limited additional space and identify task boundaries and pruning scope as future-work considerations.
- CPG uses weight pruning to control model complexity as the number of sequential tasks increases.
- Binary masks select learned weights for reuse alongside newly added weights, while preserving old-task weights to prevent forgetting.
- Growing the model for new tasks supports learning unlimited and unknown or unrelated tasks.
- Experiments show similar or better accuracy with limited additional space.
- Future work includes channel pruning, learning without clear task boundaries, and selectively forgetting previous tasks through recorded masks.
Supplementary Material of "Compacting, Picking and Growing for Unforgetting Continual Learning," NeurIPS 2019
The supplementary experiments evaluate CPG on CIFAR-100 divided into twenty five-class tasks, including several random task orders. Average accuracy remains roughly unchanged across reordered sequences, supporting applicability when task order is arbitrary.
- CIFAR-100 is divided into 20 tasks, with 5 classes assigned to each task.
- The supplementary evaluation tests CPG under three reshuffled task sequences in addition to the original ordering.
- Average accuracy across the 20 tasks is roughly the same across random-order settings.
- These results indicate potential applicability when the order of continual-learning tasks is arbitrary.