Source-linked AI summary
PathNet: Evolution Channels Gradient Descent in Super Neural Networks
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, Daan Wierstra
TL;DR
PathNet addresses how a shared giant neural network could learn many users’ tasks while reusing parameters without catastrophic forgetting. It evolves pathways that direct gradient-based learning to selected modules, fixes learned paths between tasks, and demonstrates positive transfer across supervised and reinforcement-learning domains. The authors present the method as a scalable direction, while noting that its demonstrated network is fairly small and that further investigation is needed.
Problem
Training one giant neural network across many tasks requires parameter reuse while avoiding catastrophic forgetting and interference between users’ updates.
Method
PathNet evolves populations of pathways that select limited parameter subsets for gradient-based or A3C learning, then fixes the best path and evolves new paths for subsequent tasks.
Results
Positive transfer was demonstrated across supervised and reinforcement-learning tasks, including Atari and Labyrinth, compared with learning from scratch and fine-tuning controls.
Takeaways & Limitations
PathNet provides a learned pathway mechanism for reusing parts of a neural network across tasks while preserving previously fixed functionality.
Takeaways & Limitations
The method was demonstrated in a fairly small network, and further investigation is required to establish how much it outperforms fixed paths.
Abstract
from arXiv · showhide
For artificial general intelligence (AGI) it would be efficient if multiple users trained the same giant neural network, permitting parameter reuse, without catastrophic forgetting. PathNet is a first step in this direction. It is a neural network algorithm that uses agents embedded in the neural network whose task is to discover which parts of the network to re-use for new tasks. Agents are pathways (views) through the network which determine the subset of parameters that are used and updated by the forwards and backwards passes of the backpropogation algorithm. During learning, a tournament selection genetic algorithm is used to select pathways through the neural network for replication and mutation. Pathway fitness is the performance of that pathway measured according to a cost function. We demonstrate successful transfer learning; fixing the parameters along a path learned on task A and re-evolving a new population of paths for task B, allows task B to be learned faster than it could be learned from scratch or after fine-tuning. Paths evolved on task B re-use parts of the optimal path evolved on task A. Positive transfer was demonstrated for binary MNIST, CIFAR, and SVHN supervised learning classification tasks, and a set of Atari and Labyrinth reinforcement learning tasks, suggesting PathNets have general applicability for neural network training. Finally, PathNet also significantly improves the robustness to hyperparameter choices of a parallel asynchronous reinforcement learning algorithm (A3C).
1. INTRODUCTION
PathNet addresses the challenge of training one large neural network across many tasks by evolving pathways that reuse parameters while limiting interference. It combines genetic pathway selection with gradient-based learning and demonstrates transfer across supervised and reinforcement-learning settings.
- Motivation: PathNet targets multi-user training of a shared giant neural network so existing knowledge can be reused across tasks.The framework is motivated by transfer, continual, and multitask learning, with parameter sharing when transfer is possible and disjoint updates when interference is significant.
- Algorithm: PathNet evolves pathways that select which subset of network parameters gradient descent uses and updates.Evolution guides where gradient descent is applied, while the pathway determines the active parameters.
- Algorithm: A tournament-selection genetic algorithm evaluates pathways through brief gradient-based training, then replicates and mutates fitter pathways.Evolution and learning occur simultaneously, with evolution determining where learning is applied.
- Transfer: After task A, the best pathway is fixed for task B, preserving its learned parameters while a new population of pathways is evolved.The fixed path prevents the functionality learned on task A from being changed during subsequent training.
- Results: Positive transfer was demonstrated in supervised and reinforcement-learning experiments, including Atari and Labyrinth tasks.The experiments compare transfer against learning from scratch and fine-tuning controls.
2. METHODS
PathNet uses a modular network in which evolved genotypes select limited pathways, while gradient descent or A3C trains only the selected parameters. Across supervised and reinforcement-learning setups, task-specific paths are fixed between tasks and transfer is evaluated against de novo and fine-tuning controls.
- 2.1 PathNet Architecture: A PathNet contains L layers of M neural-network modules, with active modules selected by a pathway genotype.Modules are arranged in layers, and their outputs are summed before reaching active modules in the next layer.
- 2.1 PathNet Architecture: Each pathway includes at most N modules per layer, while the final layer has a separate task-specific readout.Supervised tasks use separate linear readouts; A3C tasks use value-function and policy readouts.
- 2.2 Pathway Evolution: Serial and Parallel: In parallel A3C training, 64 workers evaluate pathways simultaneously, restricting each worker’s parameter updates to its pathway’s subset.Workers use accumulated return over T episodes as pathway fitness and asynchronously replace lower-fitness genotypes through tournament selection.
- Transfer protocol: After task A, the best pathway is fixed and all parameters outside it are reinitialized before evolving pathways for task B.Without reinitialization, transfer performance did not exceed fine-tuning.
- Evaluation: Transfer is compared with independent target-task training from scratch and fine-tuning using the first task’s path.These controls are used in both supervised and reinforcement-learning settings.
- 2.4 Binary MNIST classification tasks: Binary MNIST uses noisy two-class classification, with paths evolved on a first task and then re-evolved on a second task after fixing and resetting parameters.The reported outcome is the training time required to reach perfect training-set classification.
- 2.5 CIFAR and SVHN classification tasks: CIFAR and cropped SVHN use a larger modular network and evaluate accuracy after a fixed 500-generation training period.The experiment treats higher second-task accuracy than scratch learning as evidence of positive transfer.
- 2.6 Atari games: Atari experiments test whether learning Pong, RiverRaid, or Seaquest first speeds learning on a second game using 64-worker A3C.Source games are trained for 80M or 40M total timesteps, followed by 40M timesteps for the target game.
3. RESULTS
Across supervised, Atari, and Labyrinth tasks, PathNet generally accelerated transfer or learning relative to fixed-path controls, although transfer gains varied by task pair. The experiments also show layer-specific exploration and performance advantages under broad hyperparameter sweeps.
- Binary MNIST Classification: 167 generations was PathNet’s mean time to solution on target MNIST, versus 229 for fine-tuning and 195 for learning from scratch.The reported total speedup ratio versus independent controls was 1.18.
- Binary MNIST Classification: PathNet’s layer exploration was uneven: early layers converged fastest, while the final layer received more exploration and training.The passage reports that many final-layer modules contributed to high fitness, whereas only a few first-layer modules did so; path overlap did not correlate with speedup.
- CIFAR and SVHN: 35.7% and 39.8% accuracy were achieved when cSVHN and CIFAR were learned second, versus 25.5% and 35.3% when learned first.The comparison used a fixed 500-generation evaluation, and the authors state that both datasets were learned faster when learned second.
- Atari Games: PathNet was superior to independent and fine-tuning controls on four Atari target games after hyperparameter sweeps.The strongest reported settings used tournament size B = 10, T = 10 game episodes per evaluation, and mutation rates of 0.01–0.001.
- Atari Games: 1.33 times speedup was achieved by PathNet on average across the investigated Atari games, compared with 1.16 for fine-tuning and 1.0 for the control.The speedup comparison used a control value of 1.0.
- Labyrinth Games: PathNet learned the second task faster than fine-tuning in several Labyrinth transfers and outperformed fine-tuning from scratch on stairway to melon and seekavoid arena.The cited transfers include transfer to lt chasm and transfer from lt chasm to seekavoid arena.
- Labyrinth Games: 1.26 was PathNet’s average performance ratio across Labyrinth game transfers, compared with 1.00 for fine-tuning.PathNet performed worse than de novo learning when transferring to lt chasm, while the other two games showed positive transfer.
- Labyrinth Games: PathNet had significantly higher mean performance than controls on seekavoid arena when learning from scratch and when relearning the same task.The comparison used the same sweep of 243 hyperparameters as the Atari experiments.
4. CONCLUSION
PathNet combines gradient-descent learning with evolution to select trainable parameter subsets, supporting transfer while preventing learned functionality from being lost. The paper demonstrates this approach across supervised and reinforcement-learning settings, while identifying scalability and multitask learning as open areas.
- Conclusion: PathNet extends deep learning by using evolution to select which subset of network parameters gradient descent trains.Weights and biases are learned by gradient descent, while pathway evolution determines the trained subset.
- Conclusion: PathNet sustained transfer learning across at least four tasks in both supervised and reinforcement-learning settings.The conclusion reports capability in both learning settings without enumerating the individual tasks here.
- Conclusion: Pathways are evolved as thinned subnetworks, making dropout frequency an emergent consequence of population convergence across layers.The paper relates this mechanism to evolutionary dropout and notes faster convergence in early layers than later layers.
- Limitations and future work: The demonstrated network was fairly small, so scaling PathNet to much larger networks depends on more efficient pathway-gating implementations.The authors also propose testing additional reinforcement-learning tasks, including continuous robotic control.
- Limitations and future work: Multitask learning had not yet been addressed in this paper, and the benefits of module duplication remained under investigation.Both topics are identified as directions for further work.
- Limitations and future work: The paper suggests replacing tournament selection and mutation with learned reinforcement-learning variation operators, potentially using softer gating.These are proposed extensions rather than evaluated components of the presented method.