Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Introduction

• Human learning of new skills typically does not harm previously learned tasks, whereas sequential-task DNNs usually largely forget earlier tasks after training on a new one.Learning highly related tasks can also advance related human skills.Source: §1 · p. 1
• Parameter-constraint methods such as EWC (Kirkpatrick et al., 2017) and synaptic intelligence (Zenke et al., 2017) still have non-negligible forgetting as task counts rise; memory-based methods store task information.Other approaches add small parameter amounts while freezing previously estimated parameters, but rely on a strong base network and have limited transfer mainly between consecutive tasks (Mallya & Lazebnik, 2018; Rebuffi et al., 2017a; 2018; Mancini et al., 2018).Source: §1 · p. 1
• Task-specific structures can grow distinct components while retaining shared feature layers; structure refers to fine-grained choices such as layer count and operations.The introduction argues that different tasks may require different structures, rather than the same structure for all tasks.Source: §1 · pp. 1–2
• The learn-to-grow framework searches for a structure for each sequential task, considering parameter sharing or adding parameters, then estimates parameters for the chosen structure.Its stated aim is to decouple explicit structure learning from parameter estimation.Source: §1 · p. 2
• Separating structure and parameter learning is reported to improve parameter use and performance, produce sensible task structures, and reduce forgetting versus similarly complex baselines.The authors attribute these benefits to explicit continual structure learning and the separation of structure learning from parameter estimation.Source: §1 · p. 2

2.1. Problem Definition of Continual Learning

• Continual learning trains on tasks sequentially without access to earlier task test data, seeking strong performance on the current task while minimizing forgetting on earlier tasks.The model is evaluated on each task’s test set, including tasks observed before the current one.Source: §2.1 · p. 2

2.2. Our Proposed Learn-to-Grow Framework

• The framework grows each task’s model on top of the previously trained model, flexibly reusing prior parameters and optionally keeping reused parameters fixed or allowing penalized changes.This combines structural growth with an elastic-parameter strategy, aiming to avoid forgetting without sacrificing new-task efficiency or accuracy.Source: §2.2 · pp. 2–3
• A penalized loss jointly accounts for network structure and model parameters, preventing task-specific models from growing linearly without bound as tasks accumulate.The structure can select a task-specific network from a shared super-network or define a new model per task; the latter can grow linearly, while a parameter-count regularizer can bound total size.Source: §2.2 · p. 3

3. Our Implementation

• The framework has two components: structure optimization selects or creates building blocks for the current task, while parameter learning estimates new parameters and may fine-tune existing ones.Structure optimization builds on the DNN trained on previous tasks.Source: §3 · p. 3

3.1. Structure Optimization

• The method assumes a global network topology and maintains a super network for task-specific and new shareable layers, while the search space can grow exponentially with tasks.The authors report that limiting choices with a priority queue was unnecessary in all their experiments.Source: §3.1 · p. 3
• For each shareable layer, the search selects reuse, adaptation, or new; adaptation adds parameters, while new creates a layer with the same parameter size.Source: §3.1 · p. 3
• NAS updates architecture weights using validation loss and parameters using training loss, alternating updates with a first-order approximation.The training data is split into validation and training subsets; the authors found this simple approach sufficient.Source: §3.1 · p. 4

3.2. Parameter Optimization

• After search, retraining either keeps reused layers fixed or fine-tunes them with regularization, then updates the super network with created or tuned layers, adapters, and classifiers.Keeping reused layers fixed avoids forgetting but forgoes potential positive backward transfer; the paper suggests regularized fine-tuning may also benefit previous tasks.Source: §3.2 · p. 4

4.1. Are Sensible Structures Sought in Learn-to-Grow?

• Structure search chose “new” at the first layer and “reuse” thereafter across permuted-MNIST tasks, while VDD shared layers for related natural images and mostly added layers for ImageNet–Omniglot.The learned MNIST wiring was better than the tested alternatives that made another single layer new.Source: §4.1 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

4.2. Are Forgetting Addressed in Learn-to-Grow?

• On permuted MNIST, SGD average accuracy fell from 97.9% to 63.0%, while EWC (Kirkpatrick et al., 2017) reached 96.8%; fixing or lightly tuning reused layers preserved individual-task-level accuracy.Fine-tuning reused layers with simple ℓ2 regularization or a smaller learning rate without regularization behaved similarly to fixing them.Source: Figure 3 · p. 5, §4.2 · p. 6
• In VDD, learn-to-grow significantly outperformed the all-backbone-SGD baseline, and its fine-tuned parameters stayed closer to first-task values than baseline parameters.The reported ℓ2 distances compare the first and last layers; the paper notes that learn-to-grow distances were typically very small across layers, while baseline movement generally increased in higher layers.Source: §4.2 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
• On VDD, learn-to-grow achieved the best overall results and led on five of nine tasks, with total model size similar to the adapter approach (Rebuffi et al., 2017a).The comparison reports top-1 validation classification accuracy and total parameter size after training ten tasks.Source: §4.2 · p. 6, Table 1 · p. 7

4.3. Comparisons with State-of-the-Art Methods

• On permuted MNIST, learn-to-grow was competitive with or better than recent continual-learning methods; on split CIFAR100, it showed similar performance and selected new final-convolution parameters while reusing the rest.For split CIFAR100, the paper attributes this structure to similar natural-image inputs but disjoint output classes; it reports that fully connected layers were reused, possibly because their capacity sufficed.Source: §4.3 · p. 7, Figure 7 · p. 7

Related Work

• Replay methods train on stored examples or generated samples from earlier tasks alongside current-task data to mitigate forgetting.Generative replay learns a model of previous-task data distributions and trains with generated and real current-task samples.Source: §5 · p. 8
• Regularization methods constrain parameter updates, including EWC’s quadratic penalty on important weights and synapse-level importance penalties.Schwarz et al. (2018) use progress and compression phases, distilling the newly learned model into the old model with EWC during compression.Source: §5 · p. 8
• Other methods preserve prior-task performance with attention or avoid forgetting by preventing changes to weights used by previous tasks.Examples of the latter include Rusu et al. (2016), Mallya & Lazebnik (2018), and Fernando et al. (2017).Source: §5 · p. 8
• Expandable approaches include dynamically expanding or duplicating layers and progressive networks, while PathNet tunes only selected unused modules; learn-to-grow permits tuning parameters from previous tasks.The paper distinguishes learn-to-grow as more flexible because NAS optimizes its structure.Source: §5 · p. 8
• Learn-to-grow uses neural architecture search for structure optimization, employing DARTS’s continuous relaxation for search efficiency.The paper situates its search methods alongside work by Stanley & Miikkulainen (2002), Zoph & Le (2016), Baker et al. (2017), and Liu et al. (2018).Source: §5 · p. 8

6. Conclusion

• The framework offers three choices for each layer when learning a new task: reuse, adapt, or add a new layer.It explicitly optimizes neural structure through differentiable neural architecture search.Source: §6 · p. 8
• Structure learning produces sensible models, and forgetting can be completely avoided without fine-tuning reuse layers or significantly alleviated with fine-tuning.Source: §6 · p. 8
• Across permuted MNIST, Visual Domain Decathlon, and split CIFAR-100, the method achieves highly comparable or better performance than state-of-the-art methods.Source: §6 · p. 8

1. Additional Experimental Details for permuted MNIST

• The proposed method used three 300-unit hidden layers and one shared output layer, while the other methods were configured to have at least as many parameters.Except for PathNet and progressive networks, other methods used 3,000 units in the first layer and 300 in the remaining layers; PathNet and progressive networks used their specified modular, task-specific configurations.Source: §1 · p. 11
• For a fair comparison, VCL was run with a coreset size of zero.Source: §1 · p. 11
• Beyond five tasks, the deep generative replay implementation showed significant drops on previous tasks; performance averaged around 96% when task count was at most five.The passage says reasonable results were obtained only within five tasks and does not provide a complete value for the subsequent “Reaching 60%” statement.Source: §1 · p. 11
• All methods used the same permutations and the same subset of training data for the permuted-MNIST experiments.Source: §1 · p. 11

• The AlexNet-like network has three convolutional and max-pooling layers followed by two fully connected layers.Source: §2 · p. 11
• Its convolutional layers use kernel sizes (4,4), (3,3), and (2,2), with 64, 128, and 256 filters, respectively.Each convolution is followed by a (2,2) max-pooling layer and rectified linear activations.Source: §2 · p. 11
• Each of the two fully connected layers has 2048 hidden units.Source: §2 · p. 11

• Tasks were trained in a random sequence with ImageNet first, and each task had its own Batch Normalization layers.When the search selected “reuse,” retraining weights were fixed so previous-task results were unaffected; forgetting evaluation on VDD was left for future work.Source: §3 · p. 11
• Baselines included individually trained, randomly initialized models and classifier-only transfer models with the first 25 ImageNet-pretrained layers fixed.The Adapter baseline adds a 1×1 convolution alongside each 3×3 convolution, adding approximately one ninth of the whole model size per task.Source: §3 · pp. 11–12
• The framework outperformed other baselines, achieved the best results on five of nine tasks, and had a total model size similar to Adapter.Its advantage was especially large on the small-data VGG-Flowers and Aircraft tasks.Source: §3 · p. 12, Table 1 · p. 12
• β = 0.1 gave the best average accuracy while keeping total model size at approximately 2.3× the original model size.The parameter-loss term penalizes choices according to additional parameter size and its corresponding weight α_l; β controls the penalty’s scaling.Source: §3 · p. 12