Source-linked AI summary
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Arun Mallya, Svetlana Lazebnik
TL;DR
Continual learning seeks to add tasks without catastrophic forgetting or storing separate large models. PackNet iteratively prunes and retrains a network, freezing retained weights and reusing freed parameters for later tasks. It adds multiple tasks with performance close to individually trained networks and low storage overhead, while requiring task-specific inference masks.
Problem
Continual learning must acquire new tasks while maintaining earlier-task performance without storing large specialized models, but naive fine-tuning can cause catastrophic forgetting.
Method
PackNet iteratively prunes redundant weights, retrains for each task, freezes retained parameters, and uses task-specific masks to select each task's network state.
Results
The method adds three fine-grained classification tasks to one ImageNet-trained VGG-16 with accuracies very close to separately trained networks and outperforms prior work in robustness to catastrophic forgetting.
Takeaways & Limitations
PackNet packs multiple tasks into a single network with minimal prior-task performance loss across VGG-16, ResNet, and DenseNet architectures.
Takeaways & Limitations
The pruning-based approach cannot perform simultaneous inference on all tasks because nonlinearities make sparsity-dependent filter responses nonseparable.
Abstract
from arXiv · showhide
This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks. By performing iterative pruning and network re-training, we are able to sequentially "pack" multiple tasks into a single network while ensuring minimal drop in performance and minimal storage overhead. Unlike prior work that uses proxy losses to maintain accuracy on older tasks, we always optimize for the task at hand. We perform extensive experiments on a variety of network architectures and large-scale datasets, and observe much better robustness against catastrophic forgetting than prior work. In particular, we are able to add three fine-grained classification tasks to a single ImageNet-trained VGG-16 network and achieve accuracies close to those of separately trained networks for each task. Code available at https://github.com/arunmallya/packnet
1. Introduction
Continual learning requires acquiring new tasks without degrading prior-task performance or storing large specialized models. PackNet uses pruning to reuse redundant parameters for new tasks while preserving earlier task performance.
- Fine-tuning can alter weights important to prior tasks, causing catastrophic forgetting.
- Proxy-loss methods preserve prior-task knowledge, whereas PackNet frees redundant parameters through pruning and keeps surviving parameters fixed.
- Three fine-grained classification tasks were added to one ImageNet-trained VGG-16 with accuracies close to separately trained networks.
- Figure 1 illustrates sequential pruning, retraining, weight sharing, and task-specific masking across Tasks I–III.
2. Related Work
Related work addresses continual learning through proxy losses, model compression, or growing architectures. PackNet instead applies iterative pruning to add entire diverse datasets and reports experiments across full-scale image tasks and multiple networks.
- LwF uses distillation on current-task data, while EWC penalizes modifying parameters important to earlier tasks.
- PackNet uses weight-magnitude pruning to free parameters for new tasks and repeatedly prunes and retrains for diverse tasks.
- Progressive neural networks preserve old weights but replicate the architecture for each new dataset, increasing overall model size.
- The paper evaluates combining up to four entire image-classification tasks in one network, contrasting with prior evaluations on smaller or synthetic settings.
- Unlike incremental learning methods that add a few classes at a time, PackNet adds entire image-classification tasks or datasets.
3. Approach
PackNet iteratively prunes eligible weights, retrains each task, freezes retained weights, and reuses freed parameters for subsequent tasks. Task-specific masks select the appropriate network state at inference, with low storage overhead but no simultaneous inference across tasks.
- The method prunes weights of an initial network, retrains to recover performance, and uses freed parameters for a new task without adding network capacity.
- Each new task reactivates pruned weights, reuses earlier-task weights, then prunes and retrains while previously assigned weights remain fixed.
- Each pruning round removes the lowest-magnitude eligible weights from every convolutional and fully connected layer.
- Biases and batch-normalization parameters remain fixed after the first pruning and retraining round, reducing per-task overhead.
- At most log2(N) bits encode a per-parameter task mask, with one- and three-task additions costing approximately 17 MB and 34 MB on a 537 MB VGG-16.
- Inference applies a task-specific mask to reproduce the selected task's trained network state, with no additional runtime computation.
- The method cannot perform simultaneous inference for all tasks because nonlinearities make sparsity-dependent filter responses nonseparable.
4. Experiments and Results
Experiments evaluate PackNet across fine-grained, large-scale, and multiple network architectures, comparing pruning with classifier-only, individual-network, joint-training, and LwF baselines. The method preserves prior-task performance while achieving accuracy close to separately trained models with substantially lower storage.
- Multiple fine-grained classification tasks: PackNet adds CUBS, Stanford Cars, and Flowers to an ImageNet-trained VGG-16, with errors only 2.38%, 1.78%, and 1.10% above the Individual Networks best case.Results average six task orderings with three runs per ordering.
- Multiple fine-grained classification tasks: 11.04%, 30.41%, and 10.41% error reductions versus Classifier Only are obtained for the three fine-grained tasks.Increasing initial pruning from 50% to 75% improves Cars top-1 error from 18.08% to 15.75%.
- Storage: 595 MB versus 2,173 MB is required by the pruning-based model compared with separate networks, while it is only 33 MB larger than Classifier Only.The comparison is reported for the final row of Table 2.
- Adding another large-scale dataset task: 0.64% top-1 error and 0.10% top-5 error separate PackNet from an individually trained network when adding Places365 to a 75% pruned ImageNet network.Places365 contains 1.8 M images versus ImageNet's 1.3 M images.
- Adding another large-scale dataset task: 33.49% versus 28.42% top-1 error shows that joint training performs much worse than an individual ImageNet network, whereas iterative pruning supports task-specific hyperparameters.The pruned Places365 model was trained for 10 epochs, compared with 60–90 epochs for joint and individual networks.
- Extension to other networks: 0.45% and 0.04% top-1 accuracy losses on ImageNet are observed after pruning ResNet-50 and DenseNet-121, respectively.Top-5 error increases by 0.05% for ResNet and decreases by 0.13% for DenseNet; batch-normalization parameters remain frozen.
5. Detailed Analysis
The analysis examines task-order effects, pruning and retraining, layer selection, bias sharing, and filter-based pruning. It finds that retraining and access to convolutional layers are important, while aggressive filter pruning is constrained by accuracy loss.
- Effect of training order: Task addition leaves fewer parameters available for later tasks, so accuracy decreases with task order.For the 0.50, 0.75, 0.75 sequence, Task II uses 16.75 M parameters and Task III uses around 13 M, reducing the free-parameter pool.
- Effect of pruning ratios: Retraining reduces the error increase caused by pruning and can lower error below the original unpruned value.The error rises immediately after pruning because network connectivity changes, then decreases after retraining; retraining is especially important at large pruning ratios.
- Effect of pruning ratios: A 90% single-shot pruning followed by retraining produces small top-1 error increases for Stanford Cars, CUBS, and Flowers.Errors change from 15.75% to 17.84% for Stanford Cars, 24.13% to 24.72% for CUBS, and 8.96% to 9.48% for Flowers.
- Effect of finetuning various layers: Task-specific biases do not noticeably improve performance, so the reported method shares biases to reduce storage overhead.The comparison averages six task orderings with three runs per ordering.
- Effect of finetuning various layers: Fine-tuning convolutional layers provides the largest accuracy boost, making them necessary for good performance on new tasks.Fine-tuning fully connected layers improves over the classifier-only baseline, but convolutional-layer fine-tuning gives the biggest improvement.
6. Conclusion
The conclusion presents PackNet as a way to pack multiple tasks into one network with minimal performance loss, including in compact architectures. It identifies jointly learned task-specific sparsity masks as a direction for future work.
- Conclusion: PackNet packs multiple tasks into one network with minimal loss of performance on prior tasks.The method modifies all network layers and influences many filters and features to approach individually trained task performance.
- Conclusion: The method works on both roomy VGG-16 and compact ResNet and DenseNet architectures.The conclusion explicitly extends the reported scope beyond VGG-16 to parameter-efficient networks.
- Future work: Future work will explore jointly training network weights and binary sparsity masks for individual tasks.The current masks result from pruning, while future masks might be learned using techniques related to binary-weight networks.