Adding New Tasks to a Single Network with Weight Transformations using Binary Masks

1 Introduction

• Fine-tuning can degrade old-task performance, while replicating a network for each task preserves it at the cost of parameter growth.These are the forgetting and scalability challenges motivating incremental adaptation.Source: §1 · p. 1
• Prior binary-mask adaptation assigns a learned mask to each task and uses it to select useful network weights, adding one bit per parameter per task.The proposed method builds on this strategy.Source: §1 · p. 2
• The proposed transformation uses a learned binary mask both to filter pretrained weights and as a scaled, shifted additive component.The affine transformation changes the mask's range and minimum value; the pretrained kernel and original mask are also scaled before the terms are summed.Source: §1 · p. 2, Figure 1 · p. 2
• Slightly more than 1 bit per parameter per task supports higher task-specific performance through greater flexibility in perturbing the pretrained network.The approach aims to limit additional parameters while improving each task-specific network.Source: §1 · p. 3
• The method achieves a new state of the art on the Visual Decathlon Challenge while performing comparably to separate fine-tuned networks across benchmarks.It retains a very small per-task parameter overhead.Source: §1 · p. 3

2 Related works

• Knowledge distillation can preserve previous-task activations during training on novel tasks, while other constrained updates account for parameter importance to earlier tasks.These are examples of approaches that constrain optimization around the initial network weights.Source: §2 · p. 3
• Weight-preservation approaches keep the original network parameter count but limit forgetting at the expense of lower performance on old and new tasks.Adding task-specific parameters can avoid forgetting if the original parameters remain untouched, but requires additional parameters.Source: §2 · p. 3
• Disjoint task-specific parameter subsets limit what remains available to each task as task count grows, whereas learned binary masks offer a compact, more flexible alternative.The paper reports that this formulation improves performance and reduces the gap to individually end-to-end trained architectures while maintaining low per-task overhead.Source: §2 · pp. 3–4
• Binarization and quantization methods compress whole architectures or activations, while this work compresses only the parameters added for each task.The paper also uses optimization techniques from these works as building blocks.Source: §2 · p. 4

3.1 Overview

• Task-specific networks share the baseline structure and parameters, while each task has its own parameters and potentially a differently sized classifier.The baseline parameters are divided into shareable parameters and task-specific parameters.Source: §3.1 · pp. 4–5
• The method stores a binary mask for each task and convolutional kernel, requiring one bit per parameter in each convolutional layer.The transformed parameters are used in the task-specific network’s convolutional layer.Source: §3.1 · p. 5
• Unlike Piggyback’s binary mask filter, the full method applies a task-specific affine transformation to each shared convolutional kernel using a binary mask and additional parameters.The transformation combines the baseline kernel, a tensor of ones, the mask, and the element-wise product of the kernel and mask.Source: §3.1 · p. 5
• The simplified version sets k3 to 0, removing the Piggyback multiplicative component while retaining the affine transformation of the mask.In the illustrated version, k2 scales and k1 translates the binary mask before it is added to the pretrained kernel.Source: §3.1 · p. 6, Figure 3 · p. 6
• Task-specific batch-normalization parameters are learned, and k0 is fixed to 1 for convolutional layers followed by batch normalization.The stated reason is that batch-normalization output is invariant to convolutional-weight scale.Source: §3.1 · p. 6

3.2 Learning Binary Masks

• Binary masks are learned indirectly through thresholded real-valued matrices, with a strictly increasing surrogate used for gradient computation.The paper states that this surrogate gradient points in the downhill direction for the error surface; identity and sigmoid choices are discussed.Source: §3.2 · p. 7

4 Experiments

• The benchmarks cover six varied recognition datasets and the 10-task Visual Decathlon, using pretrained ResNet-50 and DenseNet-121 for ImageNet-to-Sketch and Wide ResNet-28 for the challenge.The ImageNet-to-Sketch protocol averages results across runs; the Visual Decathlon uses a shared schedule for the nine non-ImageNet tasks without task-specific hyperparameter strategies.Source: §4 · pp. 7–8, §4.1 · p. 11

4.1 Results

• Both model versions nearly close the gap between classifier-only and individually fine-tuned networks, and consistently outperform basic Piggyback and PackNet except for comparable DenseNet results on Sketch.With task-specific batch normalization for Piggyback, the remaining gap is reduced in some settings but remains large for Flowers, Stanford Cars, and WikiArt; the models’ total parameter counts are 1.17 times baseline for ResNet-50 and 1.21 times for DenseNet-121.Source: §4.1 · pp. 9–10, Table 1 · p. 10, Table 2 · p. 10
• More than 400 points for the simple model and more than 600 for the full model over the previous state-of-the-art model [45] establish new Visual Decathlon records.The full model achieves top-two or comparable results in 7 of 9 non-ImageNet tasks; Aircraft is the main exception, with nearly 13 points lower accuracy than the best model [33].Source: Table 3 · p. 11, §4.1 · p. 11, Table 4 · p. 12

4.2 Ablation Study

• Adding k1 significantly boosts the simple model, while adding k2 or making k1 channel-wise provides no further gain; extra parameters help somewhat on larger WikiArt.For the surrogate function, σ(x) shows no particular advantage over the straight-through estimator, whose noise may regularize parameters [2] [37].Source: §4.2 · p. 12, Table 5 · p. 13

4.3 Parameter Analysis

• The learned masks remain nearly half active, while Piggyback masks are denser and tend to thin with depth; the affine transformation grows more prominent in deeper layers.The authors suggest the affine transformation’s flexibility may reduce the need to keep more weights active, and report that k2 and k3 tend to have larger magnitudes than k1.Source: §4.3 · pp. 12–13

• Affine transformations acting on learned binary masks provide richer modifications of a fixed, pre-trained network for adapting to new tasks.The authors report that experiments on two public benchmarks confirm the approach's performance.Source: §5 · p. 13
• The method exceeds the previously reported best Visual Decathlon Challenge result by more than 600 points.Source: §5 · p. 14
• Future work may test the approach on incremental-class-learning and open-world-recognition scenarios, and explore cross-task affine transformations to reuse previously collected knowledge.Source: §5 · p. 14
• Fig. 4 reports mask percentages by layer depth and learned k1, k2, and k3 values across three datasets and two architectures.Source: Figure 4 · p. 14