Source-linked AI summary
Conditional Channel Gated Networks for Task-Aware Continual Learning
Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, Babak Ehteshami Bejnordi
TL;DR
The paper tackles catastrophic forgetting in convolutional networks trained on sequential tasks, including settings where task identity is unavailable at test time. It uses task-specific conditional gating to protect important filters and preserve capacity, together with a task classifier that selects the appropriate head. Across four continual-learning datasets, the model outperforms existing methods with and without a task oracle, while requiring substantial GPU memory for larger backbones and depending partly on task semantic separation.
Problem
Continual learning must prevent neural networks from overwriting knowledge from earlier tasks, while many applications cannot provide task labels during inference.
Method
The framework uses task-specific input-dependent gates with sparsity to freeze relevant kernels, reinitialize unused ones, and a task classifier to select the classification head.
Results
The model consistently outperforms existing methods in task-incremental and class-incremental settings across four continual-learning datasets, with a slight but consistent margin over HAT in repeated task-incremental experiments.
Takeaways & Limitations
Conditional gating can protect previously learned knowledge while dynamically reusing features and reserving capacity for new tasks, with task classification extending the approach beyond task-oracle inference.
Takeaways & Limitations
Training can require substantial GPU memory for larger backbones, and task-classifier performance may depend on semantic separation among tasks.
Abstract
from arXiv · showhide
Convolutional Neural Networks experience catastrophic forgetting when optimized on a sequence of learning problems: as they meet the objective of the current training examples, their performance on previous tasks drops drastically. In this work, we introduce a novel framework to tackle this problem with conditional computation. We equip each convolutional layer with task-specific gating modules, selecting which filters to apply on the given input. This way, we achieve two appealing properties. Firstly, the execution patterns of the gates allow to identify and protect important filters, ensuring no loss in the performance of the model for previously learned tasks. Secondly, by using a sparsity objective, we can promote the selection of a limited set of kernels, allowing to retain sufficient model capacity to digest new tasks.Existing solutions require, at test time, awareness of the task to which each example belongs to. This knowledge, however, may not be available in many practical scenarios. Therefore, we additionally introduce a task classifier that predicts the task label of each example, to deal with settings in which a task oracle is not available. We validate our proposal on four continual learning datasets. Results show that our model consistently outperforms existing methods both in the presence and the absence of a task oracle. Notably, on Split SVHN and Imagenet-50 datasets, our model yields up to 23.98% and 17.42% improvement in accuracy w.r.t. competing methods.
1. Introduction
Continual learning addresses catastrophic forgetting when models encounter sequential tasks, especially when task identity may be unavailable at inference. The proposed conditional-computation framework protects prior knowledge, preserves capacity for new tasks, and adds task prediction without an oracle.
- Online task streams can cause catastrophic forgetting because learning new tasks overwrites parameters critical to previous tasks.
- Task-incremental methods typically assume that each input’s task label is available during inference, an assumption that may not hold in real-world applications.
- Task-specific gates select kernels dynamically, freeze relevant filters for prior tasks, and reinitialize unused filters for future tasks.The framework also uses sparsity to encourage selecting fewer units and conserve capacity.
- A task classifier predicts which classification head to query, enabling inference without a task oracle and using task-level rehearsal to mitigate forgetting.
2. Related work
Prior continual-learning methods address forgetting through architectural expansion, replay, distillation, or parameter masks, but mask-based approaches generally require task labels at inference. This work combines input-dependent channel gating with a task classifier for class-incremental settings.
- Progressive Neural Networks avoid forgetting by adding parameters for each task, but this comes at the expense of limited scalability.
- Knowledge distillation and replay are established strategies for alleviating catastrophic forgetting in continual learning.
- Existing mask-based methods can protect task-relevant parameters, but they operate only when an oracle supplies the task label.
- The proposed gates are dynamically generated from network inputs and select specific kernels rather than entire computational blocks.
- The framework applies data-dependent channel gating to continual learning and introduces a task classifier for class-incremental learning without task-oracle access.
3. Model
The model uses task-specific conditional gates to select convolutional kernels, protect filters important to earlier tasks, and preserve capacity for new tasks. A task classifier extends this design to task-agnostic inference by predicting which task-specific stream to use.
- 3.1. Problem setting and objective: Each task is modeled as a classification problem in a sequential stream, where task identity may be known during training and testing in task-incremental learning.Without task conditioning, classes from different tasks share a single classifier and compete during prediction.
- 3.1. Problem setting and objective: The objective jointly predicts the class conditioned on the task and the task itself, shifting single-head prediction from classes to task labels.The task classifier is trained incrementally, while class prediction remains task-conditioned.
- 3.2. Multi-head learning of class labels: For each convolutional layer, a task-specific gating module uses the input feature map to select active channels and forward a sparse feature map.A new lightweight gating module is instantiated for each newly observed task.
- 3.2. Multi-head learning of class labels: The sparsity objective selects a minimal set of kernels, conserving filters for future tasks and adapting allocated capacity to task difficulty and individual observations.The gates use a hard threshold in the forward pass and a sigmoid-based backward pass to optimize binary decisions.
- 3.2. Multi-head learning of class labels: Thresholded firing probabilities identify relevant kernels to freeze for a task and non-relevant kernels to reinitialize for later tasks.The experiments use threshold 0, preventing forgetting while reducing capacity available to future tasks.
- 3.3. Single-head learning of task labels: In the task-agnostic setting, all task-specific gating modules generate parallel streams, and a task classifier receives their concatenated feature maps to predict the task.The shared backbone can remain computationally cheaper than the ungated backbone despite the parallel streams; prior-task examples rehearse the task classifier.
4. Experiments
Experiments evaluate the model across four continual-learning datasets, task-incremental and class-incremental settings, memory strategies, gate behavior, inference cost, and practical limitations.
- Datasets and backbone architectures: Experiments use Split MNIST, Split SVHN, Split CIFAR-10, and Imagenet-50 with SimpleCNN or ResNet-18 backbones.MNIST and SVHN use three-layer CNNs, while CIFAR-10 and Imagenet-50 use ResNet-18.
- Task-incremental setting: Task-incremental evaluation uses ground-truth task labels to select active gating modules and classification heads.The comparison includes Joint, Ewc-On, LwF, and HAT.
- Task-incremental setting: The model performs on-par with or better than HAT across datasets, while gating-based methods outperform consolidation strategies as task difficulty increases.LwF becomes less flexible as the number of tasks grows, whereas EWC eventually incurs forgetting by penalizing rather than preventing relevant-parameter updates.
- Class-incremental with episodic memory: With episodic memory, the model consistently outperforms competing approaches on class-incremental Split-MNIST and Split-SVHN across buffer sizes.It uses stored examples mainly to rehearse coarse-grained task prediction while retaining fine-grained class prediction.
- Class-incremental and memory analysis: Generative replay uses multiple task-specific generators, while the model’s buffer-based version beats DGMw on Split-SVHN with 3.6 times less memory.On CIFAR-10, generative replay matches an episodic memory of ≈ 1.5 MB, over 20 times smaller than its generators.
- Model analysis: Gate analysis indicates feature reuse across tasks, task-related filter sharing, and unused gates that preserve backbone capacity for additional learning.Figure 6 examines gate firing probabilities in selected layers and shows correlated classes activating the same filters.
- Limitations and future works: Training can require substantial GPU memory for larger backbones, and task-classifier performance may depend on semantic separation among tasks.Experiments randomly assign classes to tasks, so coherent task groupings could potentially yield better performance.
5. Conclusions
The framework uses conditional computation and task-specific gating to address catastrophic forgetting in convolutional neural networks. It supports testing with or without task labels and achieves state-of-the-art results across four continual learning datasets.
- The framework applies conditional computation to convolutional neural networks to tackle catastrophic forgetting.
- Task-specific lightweight gating modules help prevent forgetting while dynamically reusing previously learned knowledge.
- The method supports both task-incremental and class-incremental settings, with a task classifier replacing a task oracle when task labels are unavailable.
- Experiments on four continual learning datasets show state-of-the-art results against existing methods.
1. Training details and hyperparameters
Training used fixed optimization choices for fairness and separate model-specific hyperparameters governing sparsity behavior. The reported details include optimization and scheduling procedures.
- Training used stochastic gradient descent with momentum, gradient clipping, and learning-rate division by 10 at scheduled epochs.
- Optimization choices were kept fixed across the proposed model and competing methods to ensure fairness.
- Model-specific hyperparameters included the sparsity-loss weight and the patience before sparsity was introduced.
2. WGAN details
The generative-replay experiments use WGANs with gradient penalty and address a task-classifier failure caused by mixing generated prior-task examples with real current-task examples. The proposed remedy also generates examples for the current task.
- The generative models were implemented with Wasserstein GANs with Gradient Penalty.
- WGAN training used 2×10^5 iterations, five discriminator updates, one generator update, Adam, learning rate 10^-4, batch size 64, and gradient-penalty weight 10.
- Using generated examples for prior tasks and real examples for the current task produced poor task-classifier test performance despite good training discrimination.
- Figure 7 illustrates the task classifier’s degenerate behavior under mixed real and generated rehearsal and the proposed solution.
- The proposed solution also uses fake examples for the current task by shifting WGAN training to the beginning of that task.
3. Quantitative results for figures
The supplementary material reports numerical values corresponding to main-paper figures and describes experiments on episodic-memory accuracy and memory consumption. It also compares generative memory with a class-conditional generator baseline.
- Numerical results corresponding to Figures 4 and 5 of the main paper are reported in supplementary tables.
- The memory-consumption experiment reports numerical values for the comparison represented in Figure 5.
- Generative memory is evaluated against a baseline using one class-conditional generator for each task.
4. Comparison w.r.t. conditional generators
The model with generative memory is compared against a class-conditional generator baseline, showing that task-level generative rehearsal benefits Split SVHN and Split CIFAR-10.
- C-Gen trains one class-conditional WGAN-GP per task, then trains the backbone on independently generated labeled examples.
- Task-level generative rehearsal performs better than class-level generative rehearsal on Split SVHN and Split CIFAR-10.
- When task labels are predicted correctly, avoiding generated-example updates to classification heads preserves accuracy comparable to the task-incremental setup.
5. Confidence of task-incremental results
The experiment was repeated across five random seeds, showing that the proposal’s performance margin over HAT is slight but consistent.
- Five runs with different random seeds were used to assess the confidence of the performance gap between the proposal and HAT.The experiment corresponds to the comparison reported in Table 1 of the main paper.
- The proposal’s margin over HAT is slight yet consistent across the repeated experiments.
- Table 8 reports task-incremental learning results averaged across five runs.
- Table 9 describes the WGAN architecture used for the generative experiment, including kernel sizes, strides, and negative-ReLU slopes.