Source-linked AI summary
Lifelong Learning with Dynamically Expandable Networks
Jaehong Yoon, Eunho Yang, Jeongtae Lee, Sung Ju Hwang
TL;DR
Lifelong learning must reuse knowledge across sequential tasks without sacrificing performance on earlier tasks or incurring unnecessary growth. DEN selectively retrains and dynamically expands a shared network, using splitting, duplication, and timestamps to control semantic drift. Across multiple public datasets, it matches or exceeds separate-task and batch-model performance with substantially fewer parameters, while fine-tuning can further improve results.
Problem
Sequential deep-network learning must transfer knowledge across tasks while addressing retraining costs, unnecessary expansion, and catastrophic forgetting.
Method
DEN selectively retrains relevant old-network components, adds or splits neurons when needed, regularizes added parameters, and timestamps units for task-specific inference.
Results
11.9%p −60.3%p of separate-task model parameters achieve similar or better performance across multiple public datasets, while fine-tuning outperforms batch models by up to 0.05%p −4.8%p.
Takeaways & Limitations
DEN can estimate an effective network structure for lifelong learning and for batch settings where all tasks are available.
Takeaways & Limitations
When many tasks are semantically disparate, finding a solution that performs well on both previous and later tasks may become difficult, and previous datasets are unavailable at later steps.
Abstract
from arXiv · showhide
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.
1 INTRODUCTION
Lifelong learning trains models on sequentially arriving tasks while transferring earlier knowledge, but retraining can cause forgetting and task interference. DEN addresses these challenges through selective retraining and dynamic capacity expansion, achieving near-separate-task performance with far fewer parameters.
- Lifelong learning uses knowledge from earlier tasks to improve performance or training speed on later tasks.
- Sequential retraining can degrade performance on both new and old tasks when new tasks differ substantially from earlier ones.
- DEN selectively retrains relevant portions of the previous network, shares subnetworks across tasks, and expands capacity when necessary.
- The approach must reduce growing retraining costs, add only necessary neurons, and prevent semantic drift as tasks accumulate.
- 11.9%p −60.3%p of separate-task model parameters yield similar or better performance across multiple public datasets.
2 RELATED WORK
Related work frames lifelong learning as sequential knowledge transfer with efficient online training. Prior deep-learning approaches address catastrophic forgetting through parameter regularization or by preventing updates to old task parameters, while other work explores dynamic network expansion.
- Lifelong learning transfers knowledge from earlier tasks to later tasks while emphasizing efficient training and knowledge transfer.
- Catastrophic forgetting occurs when retraining for new tasks causes a deep network to forget knowledge learned for previous tasks.
- Elastic Weight Consolidation uses Fisher-information-weighted regularization to balance learning new tasks with preserving earlier knowledge.
- Other approaches prevent forgetting by blocking changes to old task parameters or by dynamically adding neurons during training.
3 INCREMENTAL LEARNING OF A DYNAMICALLY EXPANDABLE NETWORK
DEN incrementally learns a sequence of tasks by selectively retraining relevant subnetworks, expanding capacity only when needed, and splitting or duplicating drifting units. Timestamped inference preserves access to earlier task knowledge while allowing later learning to reuse shared structure.
- Problem setting: DEN models an unbounded sequence of tasks arriving with previously unavailable training datasets, retaining access to prior task parameters.Each task may be a single task or a set of subtasks, with the current task represented by training data D_t.
- Selective retraining: Selective retraining identifies the subnetwork connected to the new task and updates only its weights, reducing overhead and limiting negative transfer.DEN fits a sparse model at the top layer, traces affected units breadth-first, and leaves unconnected network components unchanged.
- Dynamic network expansion: If existing features cannot accurately represent a task, DEN expands the relevant layer and uses group sparsity to select the necessary number of new neurons without repeated unit-by-unit retraining.Expansion introduces incoming and outgoing parameter matrices for added units and regularizes groups of incoming weights.
- Network split/duplication: DEN addresses semantic drift by identifying units that have moved too far from their original values and duplicating them before continued training.The network split procedure uses a drift measure and copies affected units with new edges.
- Timestamped inference: Timestamped inference restricts each task to units introduced no later than that task's training stage, while permitting later learning through units that are further trained but not split.New units receive timestamp t when added during expansion or splitting.
4 EXPERIMENT
Experiments evaluate DEN across multiple datasets and compare prediction performance, efficiency, selective retraining, network expansion, and semantic-drift prevention. DEN matches batch-model performance with substantially fewer parameters, while its components improve training efficiency and preserve earlier-task performance.
- Quantitative evaluation: DEN performs almost as well as batch models across datasets and outperforms them on MNIST-Variation.Progressive networks underperform DEN, while L2 and EWC perform worse because they cannot dynamically increase capacity.
- Network capacity: DEN matches STL performance using 18.0%, 60.3%, and 11.9% of STL capacity on MNIST-Variation, CIFAR-100, and AWA, respectively.DEN-Finetune achieves the best performance on all datasets and supports network-capacity estimation when all tasks are available.
- Selective retraining: Selective retraining takes significantly less GPU computation time than full retraining and improves accuracy over the base network by 2%p.It selects fewer upper-level task-specific units and more lower-level generic units.
- Network expansion: DNN-Dynamic achieves the best mean AU-ROC while increasing network size substantially less than DNN-Constant with k=20.A capacity-matched DNN-Constant with k=13 improves accuracy but still underperforms DEN.
- Semantic drift: DEN with timestamped inference shows no noticeable performance degeneration across training stages and significantly outperforms DNN-Progressive.DNN-L2 preserves early-task performance but worsens on later tasks, while DEN without timestamping shows slight degeneration over time.
5 CONCLUSION
The paper proposes Dynamically Expandable Networks for lifelong learning, combining partial retraining, capacity expansion, and semantic-drift prevention. Across multiple classification datasets, DEN approaches batch performance with 11.9%p–60.3%p of the batch model’s capacity, and fine-tuning can outperform batch models.
- 5 CONCLUSION: DEN partially retrains old-task networks, expands capacity when new tasks require it, and uses splitting, duplication, and timestamping to prevent semantic drift.The method is implemented for both feedforward and convolutional networks.
- 5 CONCLUSION: DEN significantly outperforms existing lifelong-learning methods while achieving nearly batch-level performance with 11.9%p–60.3%p of batch-model capacity.The evaluation covers multiple classification datasets under lifelong-learning scenarios.
- 5 CONCLUSION: Fine-tuning DEN on all tasks produces models that outperform batch models and supports network-structure estimation.This conclusion applies when all tasks are available for fine-tuning.
A.1 RESULTS ON PERMUTED MNIST
On Permuted MNIST, DEN outperforms lifelong-learning baselines with modest network expansion. DEN-Finetune achieves the highest AUROC among the evaluated models.
- A.1 RESULTS ON PERMUTED MNIST: DEN outperforms all lifelong-learning baselines while using only 1.39 times the base-network capacity.Permuted MNIST contains ten tasks, each defined by a different random permutation of input pixels.
- A.1 RESULTS ON PERMUTED MNIST: DEN-Finetune achieves the best AUROC among all models, including DNN-STL and DNN-MTL.The figure reports average per-task AUROC and network capacity relative to MTL.