Source-linked AI summary
Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
Arun Mallya, Dillon Davis, Svetlana Lazebnik
TL;DR
Adapting a network to sequentially arriving tasks normally risks catastrophic forgetting or requires storing additional task-specific models. Piggyback keeps a backbone fixed and learns a differentiable binary mask for each task, achieving comparable performance across diverse tasks and architectures with low overhead. Its scope is limited because new tasks cannot benefit from one another's learned features.
Problem
Fine-tuning established networks for new tasks can cause catastrophic forgetting, motivating adaptation methods that preserve old-task performance while adding few parameters.
Method
Piggyback learns task-specific binary masks by thresholding real-valued mask weights and applying them elementwise to fixed backbone weights.
Results
Piggyback achieves performance similar to separate task-specific networks across diverse datasets and architectures, and is competitive with the best Visual Decathlon methods using the least additional parameters.
Takeaways & Limitations
The fixed backbone and independent per-task masks avoid catastrophic forgetting and make performance agnostic to task ordering.
Takeaways & Limitations
Added tasks cannot benefit from one another because only features from the initial task are reused and adapted.
Abstract
from arXiv · showhide
This work presents a method for adapting a single, fixed deep neural network to multiple tasks without affecting performance on already learned tasks. By building upon ideas from network quantization and pruning, we learn binary masks that piggyback on an existing network, or are applied to unmodified weights of that network to provide good performance on a new task. These masks are learned in an end-to-end differentiable fashion, and incur a low overhead of 1 bit per network parameter, per task. Even though the underlying network is fixed, the ability to mask individual weights allows for the learning of a large number of filters. We show performance comparable to dedicated fine-tuned networks for a variety of classification tasks, including those with large domain shifts from the initial task (ImageNet), and a variety of network architectures. Unlike prior work, we do not suffer from catastrophic forgetting or competition between tasks, and our performance is agnostic to task ordering. Code available at https://github.com/arunmallya/piggyback.
1 Introduction
Piggyback adapts a fixed backbone to new tasks by learning differentiable binary masks over unchanged weights, avoiding catastrophic forgetting. It achieves performance similar to separate task-specific networks across diverse datasets and architectures, including substantial domain shifts.
- 1 Introduction: Fine-tuning can cause catastrophic forgetting, degrading performance on old tasks and requiring specialized models for each task or dataset.The introduction frames avoiding this degradation while adding few parameters as a continual-learning requirement.
- 1 Introduction: Piggyback learns differentiable binary masks over fixed backbone weights, producing a separate mask for each new task.The masks are learned end-to-end and applied elementwise, while the underlying network weights remain unchanged.
- 1 Introduction: Performance is similar to using a separate network per task across datasets ranging from CUBS birds, Stanford cars, and Oxford flowers to WikiArt paintings and human sketches.The latter datasets represent substantial departures from the natural-image domain of ImageNet.
- 1 Introduction: The method applies to VGG-16, ResNets, and DenseNets, and remains competitive with the best Visual Decathlon methods while using the least additional parameters.It is also demonstrated for training a fully convolutional semantic-segmentation network from a classification backbone.
2 Related Work
Related work addresses continual learning through regularization, pruning, architectural expansion, or selective routing. Piggyback instead learns task-specific binary masks over fixed weights, achieving per-task adaptation with a 1-bit-per-parameter overhead.
- 2 Related Work: Sequential-task methods such as LwF and EWC preserve older-task performance with proxy losses or regularization while modifying network weights.PackNet instead prunes and fine-tunes weights iteratively, producing binary parameter-usage masks as a by-product.
- 2 Related Work: Piggyback leaves the initial backbone weights unchanged and learns a different mask per task, making task addition agnostic to ordering and preventing one task from affecting another.Separate masks also decouple task learning from choices such as batch-mixing, pruning, and cost-weighting hyperparameters.
- 2 Related Work: Progressive Neural Networks, Residual Adapters, and DAN add task-specific parameters or filters, whereas Piggyback constrains new filters to masked versions of existing filters.Its binary masks require 1 bit per network parameter and do not require task-specific biases or batch-normalization parameters.
- 2 Related Work: Piggyback adapts binary-network training by updating real-valued mask weights through backpropagation while using thresholded masks during the forward pass.This provides differentiable optimization for task-specific masking rather than evolutionary routing or pruning-based mask generation.
3 Approach
Piggyback adapts a fixed backbone to a new task by learning binary masks over its weights while keeping the backbone unchanged. The method trains continuous mask variables with thresholding and backpropagation, then stores only compact binary masks per task.
- Mask-based adaptation: Binary masks selectively activate or disable fixed backbone weights, allowing task-specific adaptation without modifying the pretrained network.The masks are learned elementwise over convolutional and fully connected weights, combining ideas from network binarization and pruning.
- Mask-based adaptation: The modified layer computes y = (W ⊙ m)x, with the pretrained weights W fixed and only real-valued mask weights mr trained.A hard threshold converts mr into binary mask values m, where each value switches a corresponding weight on or off.
- Optimization: Real-valued mask weights are optimized end-to-end using gradients propagated through the thresholded masks.The method uses gradients of the thresholded mask as a noisy estimator for the real-valued mask gradients despite hard thresholding being non-differentiable.
- Storage: After training, only thresholded binary masks are retained, adding approximately 1/32, or 3.12%, of the backbone size per task.The real-valued mask weights are discarded after training, while each binary mask requires one extra bit per network parameter.
- Optimization: The second initialization strategy with Adam improved accuracy by approximately 2% over alternative optimization choices.Experiments initialized real-valued mask weights to 1e-2 and used a threshold of 5e-3; balanced random binary initialization performed poorly.
- Mask design: Ternary masks {−1, 0, 1} did not perform significantly differently from binary masks, so subsequent experiments focus on binary masks.The ternary variant used two cutoff thresholds and was not retained as the paper’s main approach.
4 Experiments and Results
Experiments evaluate Piggyback across diverse datasets, baselines, task orders, and network architectures. The method achieves near-individual-network performance while remaining order-agnostic and applicable to large-scale and deeper-network settings.
- Experimental setup: The evaluation spans large-scale ImageNet and Places365, fine-grained CUBS, Cars, and Flowers, and domain-shifted WikiArt and sketch datasets.Dataset statistics are summarized in Table 1, and the experiments include multiple network architectures.
- Experimental setup: Table 2 compares Classifier Only, Individual Networks, PackNet, and Piggyback using top-1 errors, with parenthesized values indicating top-5 errors.The table starts from an ImageNet-trained VGG-16 network and evaluates new fine-grained classification tasks.
- Fine-grained classification: Piggyback achieves errors slightly lower than individual networks on fine-grained tasks while remaining agnostic to task ordering.PackNet errors increase by 4–7% when task addition is delayed, whereas independent Piggyback masks avoid this ordering sensitivity.
- Large-scale dataset: Places365 top-1 errors are within 0.36% of the individual-network result, with performance comparable to PackNet despite shorter training.Both Piggyback and PackNet were trained for 20 epochs, while the individual-network baselines were trained for 60–90 epochs.
- Network architectures: Piggyback applies without architectural changes to VGG-16 with batch normalization, ResNet-50, and DenseNet-121, but its gap versus individual networks grows with depth.The gap is about 2% for ResNet and DenseNet, and 4–5% on WikiArt under the reported setting; task-specific batch-normalization parameters help close the latter gap.
5 Analysis
The analysis shows that ImageNet initialization matters, sparsity increases with network depth, and domain shift exposes batch-normalization limitations.
- 5.1 Does Initialization Matter?: ImageNet initialization outperforms Places365 and random initialization on all evaluated tasks, while matching scratch-trained Places365 accuracy on Places365.The authors attribute this to ImageNet's more diverse learned features.
- 5.1 Does Initialization Matter?: Even a randomly initialized backbone achieves non-trivial accuracy across datasets, indicating that learned masks can exploit fixed filters without pretrained features.
- 5.2 Learned sparsity and its distribution across network layers: Places365 initialization requires more backbone changes than ImageNet initialization, reinforcing ImageNet features as the more diverse starting point.
- 5.2 Learned sparsity and its distribution across network layers: Sparsity generally increases with layer depth; CUBS and Flowers mostly reuse low-level features, whereas WikiArt also changes them under larger domain shift.All tasks require changes to mid-level features, according to the analysis.
- 5.3 Handling large input domain shifts: Allowing task-specific batch-normalization parameters reduces WikiArt top-1 error from 28.67% to 25.92% on ResNet-50 and from 29.56% to 25.90% on DenseNet-121.The fixed-batch-normalization gap is larger for WikiArt's domain shift, especially in deeper networks.
6 Results on Visual Decathlon & Semantic Segmentation
Piggyback achieves competitive Visual Decathlon performance with minimal parameter overhead and extends to semantic segmentation with mixed training.
- 6.1 Results on Visual Decathlon: The method is competitive with state-of-the-art Visual Decathlon performance while using the least additional parameters over a single network.The evaluation covers 10 diverse classification tasks using an ImageNet-trained backbone and masks for the remaining nine datasets.
- 6.1 Results on Visual Decathlon: Nine binary masks add 9n bits to a 32n-bit backbone, yielding approximately 41n bits overall and a 1.28 parameter ratio.
- 6.2 Semantic segmentation: On PASCAL segmentation, piggyback reaches 61.41 mean IOU versus 61.081 for a VGG-16 fine-tuned network.It uses 17 MB for backbone masking and 7.5 MB for newly added layers instead of replicating the approximately 500 MB network.
- 6.2 Semantic segmentation: The segmentation experiment shows that piggyback supports mixed training with newly added layers and remains a competitive alternative to full-network fine-tuning.
7 Conclusions
The conclusion presents piggyback as a reusable fixed-backbone approach for new tasks, while identifying limits in task-to-task transfer and specialized architectures.
- 7 Conclusions: Piggyback reuses fixed network weights through learned masks and works across multiple datasets and network architectures.
- 7 Conclusions: The method can support deployed-device learning without modifying existing weights or downloading a new large network.Reusing the backbone and masks is intended to simplify scaling across users and devices.
- 7 Conclusions: Added tasks cannot benefit from one another because only features from the initial task are reused and adapted.
- 7 Conclusions: Future work includes object detection with specialized layers and expanding existing layers when task accuracy requires more capacity.