Source-linked AI summary
Continual Learning via Neural Pruning
Siavash Golkar, Michael Kagan, Kyunghyun Cho
TL;DR
Continual learning in fixed-capacity models is hindered by catastrophic forgetting and lacks performance guarantees under standard training. CLNP uses neural pruning to reserve inactive capacity for new tasks, with controlled graceful forgetting, and empirically outperforms prior weight-elasticity approaches.
Problem
Fixed-capacity continual-learning models can forget previously learned tasks, while existing approaches lack guarantees under standard SGD training.
Method
CLNP sparsifies networks through activity-based pruning, trains subsequent tasks in inactive neurons and filters, and permits controlled graceful forgetting to regain capacity.
Results
CLNP noticeably improves results over previous approaches based on weight elasticity across multiple benchmarks.
Takeaways & Limitations
CLNP provides diagnostics for free capacity and neuron reuse while exploiting previously learned features through transfer across the network.
Takeaways & Limitations
Performance and the number of tasks supported depend crucially on the effectiveness of the sparsification method used.
Abstract
from arXiv · showhide
We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.
1 Introduction
Continual learning seeks to add new task capabilities while preserving earlier skills, but fixed-capacity models face catastrophic forgetting. CLNP addresses this by training later tasks in pruned inactive capacity, using controlled forgetting to balance capacity and accuracy.
- Motivation: Continual learning aims to improve subsequent-task performance and resource use while enabling models to build on prior experiences.
- Problem: Catastrophic forgetting occurs when training on new problems causes machines to forget previously learned tasks.
- Approach: CLNP trains subsequent tasks in inactive neurons and filters of a sparsified fixed-capacity network using standard gradient descent.
- Approach: CLNP prevents interference with prior task pathways while allowing new tasks to use previously learned features.
- Approach: Graceful forgetting trades a small, controlled accuracy loss for recovered network capacity and protection against uncontrolled future performance loss.
- Results: CLNP significantly outperforms weight-elasticity approaches across multiple benchmarks.
Related work
Prior continual-learning methods either use specialized or expanding modular architectures or constrain important weights, while CLNP adapts the latter category with activation-based sparsification and graceful forgetting. The paper positions this approach as a fixed-capacity alternative that retains zero forgetting during subsequent-task training.
- Modular approaches: Modular approaches prevent forgetting through routing or added modules, but may require specialized structures or expanding rather than fixed-capacity models.
- Weight-constrained approaches: Weight-constrained approaches use simple network structures but can still suffer catastrophic forgetting, whereas CLNP retains zero forgetting during subsequent-task training.
- CLNP positioning: CLNP adapts path-based continual learning with activation-based sparsification and graceful forgetting while remaining directly comparable to weight-constrained methods.
- Network superposition: Related superposition work includes architecture-search methods and a Fourier-space approach, whereas CLNP targets forgetting prevention through sparsification.
- Sparsification: Neural sparsification is used as a tool in CLNP rather than treated as the paper’s primary research focus.
- Sparsification: Unlike parameter-based sparsity, CLNP uses activation-based sparsity in a fixed-capacity model and targets zero catastrophic forgetting.
2 Methodology
CLNP uses activation-based neuronal sparsity to reserve inactive network capacity for new tasks while protecting prior outputs. Its workflow combines severed interference connections, task-specific output structures, activity-based pruning, and graceful forgetting to balance sparsity with accuracy.
- Network sparsity: Activation-based sparsity treats a sparse network as a narrower network embedded within the original architecture, with active and inactive neurons.The method concerns neuron or filter usage rather than merely zero-valued weights.
- Network sparsity: Weights are partitioned into active, free, and interference groups according to whether they connect active or inactive neurons.Free weights connect to inactive nodes, while interference weights connect inactive nodes to active nodes.
- Continual learning mechanism: Setting interference weights to zero lets CLNP change free weights for new tasks without changing the previous network output.Free connections can support transfer through previously learned features or form new pathways among inactive neurons.
- Output architecture: CLNP supports multi-head expansion with new output layers and single-head masking when task outputs are structurally compatible.The multi-head option is not fully fixed-capacity because each task adds output-layer weights, whereas single-head masking partitions the final hidden layer within fixed capacity.
- Sparsification: The sparsification scheme applies L1 regularization during training, prunes neurons using average activity, and omits fine-tuning unless specified.Pruning thresholds and regularization coefficients can vary by layer; active neurons exceed the post-training activity threshold.
- Sparsification: Graceful forgetting controls the sparsity–performance compromise by accepting a small controlled loss before later tasks use the recovered capacity.After each task, the model is sparsified to an acceptable performance level, after which future-task training causes no further deterioration of prior tasks.
3 Experiments
CLNP is evaluated on permuted MNIST and split CIFAR tasks, showing strong retention, controlled capacity use, and transfer across layers. Graceful forgetting helps balance sparsity and future-task performance, while wider architectures accommodate more continual learning.
- Permuted MNIST: With test error within 0.05% of single-task SGD, CLNP virtually eliminates catastrophic forgetting on permuted MNIST and outperforms previous methods.The experiment uses a fixed-capacity single-head architecture and reports mean and standard deviation over five runs.
- Permuted MNIST: First-layer neuron usage grows little after task 1, whereas second-layer usage grows linearly because final-layer neurons are not reused in the single-head design.The usage diagnostic reveals substantial reuse of earlier-layer features while preventing interference at the output-connected layer.
- Permuted MNIST: After 10 permuted MNIST tasks, the two hidden layers are 18% and 40% utilized, leaving capacity for about 25 tasks at 98.4% test accuracy.The final hidden layer becomes fully depleted only after approximately 25 random-permutation tasks.
- Split CIFAR: On split CIFAR, the m = 2% forgetting scheme maintains high performance throughout all tasks and runs out of capacity only when the final task is trained.The more restrictive m = 1% scheme depletes capacity after task 4 and performance then plummets.
- Split CIFAR: In the wide single-head network, first-layer channels stop growing after task 2, while deeper layers continue learning channels through later tasks.This pattern supports the paper’s observation that lower-layer features are more transferable and higher layers require greater width.
- Split CIFAR: Compared with a reported roughly 10% forgetting under superposition, CLNP causes less than a 2% drop through graceful forgetting and remains unchanged across nine subsequent tasks.The comparison concerns the first task in a single-head CIFAR experiment.
4 Conclusion
CLNP uses inactive neurons and filters to learn new tasks without catastrophic forgetting, while graceful forgetting can trade a small controlled accuracy loss for reclaimed capacity. Diagnostics reveal layerwise capacity bottlenecks and support the transferability of earlier-layer features.
- CLNP trains new tasks in inactive neurons and filters, avoiding catastrophic forgetting on previously learned tasks.
- Graceful forgetting sacrifices a small amount of accuracy after each task to regain network capacity for later tasks.
- Capacity-usage diagnostics report free neurons and identify layers that exhaust capacity prematurely, which can be widened for subsequent tasks.
- CLNP empirically verifies that earlier-layer features are more transferable than higher-layer features.
- The method depends critically on the sparsification scheme, with fine-tuning after pruning producing better results than single-iteration pruning.
A Multi-task training on CIFAR-10/100
The multi-task CIFAR-10/100 comparison balances unequal dataset sizes through a fixed minibatch mixture and adapts the model with separate task heads. Its training setup uses a 5:1 sample ratio and a scheduled Adam optimizer, while acknowledging that other ratios may perform better.
- The mixed CIFAR-10/100 problem has a first-task dataset ten times larger than the other task datasets.
- Comparison training uses minibatches containing 50 samples from task 1 and 10 samples from each other task, for 140 samples total.
- The 5:1 sampling ratio starts a new epoch when the smaller datasets are exhausted, after 50% of the large dataset has been seen.
- The chosen sample ratio biases relative task performance, and other ratios may produce better overall results.
- The multi-task model partitions the final hidden layer into 10 equal task-specific heads and trains on 10 of 11 tasks for comparison fairness.
- Training uses Adam for 120 epochs with learning rate 0.001 and schedule milestones at epochs 50 and 90.