Source-linked AI summary
Evolutionary Architecture Search For Deep Multitask Networks
Jason Liang, Elliot Meyerson, Risto Miikkulainen
TL;DR
Deep multitask architectures are difficult to design because there are many ways to share across tasks and the search space is complex. This paper evolves modules and routings from soft ordering; on Omniglot, the combined approach reaches 88% accuracy and is reported as the best-performing method.
Problem
Deep multitask architecture design is difficult because many task-sharing choices matter and complex architectures are hard for people to optimize.
Source: §1 · p. 1Method
Starting from soft ordering, the paper evolves module architectures and task-specific routings, including a combined approach with evolved shared modules.
Source: §1 · p. 1Results
On Omniglot, the combined capstone approach performs best, reaching 88% accuracy; soft ordering reaches 67% and single-task training 61%.
Source: §1 · p. 1, §4.3 · p. 7Takeaways & Limitations
The authors conclude that evolutionary architecture search can make a large difference in deep-learning performance and is particularly powerful when combined with multitask learning.
Source: §1 · p. 1Soft ordering's fixed grid-like topology biases and restricts the types of sharing it can learn.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Multitask learning, i.e. learning several tasks at once with the same neural network, can improve performance in each of the tasks. Designing deep neural network architectures for multitask learning is a challenge: There are many ways to tie the tasks together, and the design choices matter. The size and complexity of this problem exceeds human design ability, making it a compelling domain for evolutionary optimization. Using the existing state of the art soft ordering architecture as the starting point, methods for evolving the modules of this architecture and for evolving the overall topology or routing between modules are evaluated in this paper. A synergetic approach of evolving custom routings with evolved, shared modules for each task is found to be very powerful, significantly improving the state of the art in the Omniglot multitask, multialphabet character recognition domain. This result demonstrates how evolution can be instrumental in advancing deep neural network and complex system design in general.
1 INTRODUCTION
Deep multitask architecture design is difficult because tasks can be tied together in many ways, and complex architectures are hard for people to optimize. This paper evolves multitask network modules and routing, finding the combined approach performs best on Omniglot.
1 INTRODUCTION
- Deep multitask architecture design is challenging because tasks can be connected in many ways, and large, complex networks are difficult for human designers to optimize.The introduction also frames evolutionary search and multitask learning together as a particularly powerful approach.Source: §1 · p. 1
- 88% accuracy: the capstone approach performs best on Omniglot, while soft ordering reaches 67% and single-task training 61%.The paper reports that evolution of modules and topologies improves on soft ordering, and their coevolution improves further.Source: §1 · p. 1
- The paper evolves multitask network hyperparameters, modules, and routing topologies, extending soft ordering with task-specific routing and evolved modules.Its four evaluated approaches culminate in evolving task-specific routings together with general modules.Source: §1 · p. 1
2 BACKGROUND AND RELATED WORK
The background reviews deep multitask learning and architecture search, focusing on how tasks share parameters and how evolutionary methods can search network structures. Soft ordering permits sharing across depths but retains a fixed grid-like topology, motivating broader module and routing search.
2.1 Deep Multitask Learning
- Multitask network design centers on how parameters are shared; choosing architectures and sharing patterns across tasks creates a combinatorial search space.Prior approaches commonly align task feature hierarchies, while another learns task hierarchies but still assumes that alignment.Source: §2.1 · p. 2
- Soft ordering avoids fixed hierarchy alignment by learning task-specific scalar weights that let shared layers contribute at different depths through a soft merge.The weights are learned jointly with layer weights through backpropagation, with softmax enforcing that the merge weights sum to 1.Source: §2.1 · p. 2
- Soft ordering retains a fixed grid-like topology, restricting learnable sharing; this paper generalizes the modules and evolves both modules and their assembly topologies.The authors describe this as the first evolutionary approach to deep multitask learning.Source: §2.1 · p. 2
2.2 Architecture and Hyperparameter Search
- Evolutionary architecture search includes topology-search methods such as NEAT and CoDeepNEAT, which combines neural topology search with hierarchical evolution.This paper extends CoDeepNEAT to coevolve modules and blueprints for deep multitask networks and combines it with task-specific architecture evolution.Source: §2.2 · p. 2
- The tested methods extend soft ordering with CoDeepNEAT and task-specific routing, with their combination identified as the best-performing approach.The figure presents the five methods and their relationships.Source: Figure 3 · p. 3
- CoDeepNEAT evolves module and blueprint populations separately, assembles their members into candidate networks, and assigns fitness from supervised-task performance.Blueprints specify modules at network locations, while modules encode layers and their hyperparameters; fitness is attributed back to the blueprints and modules used.Source: §2.2 · pp. 2–3, Figure 2 · p. 2
3 ALGORITHMS FOR DEEP MTL EVOLUTION
The paper builds from soft ordering to methods that evolve module architectures, shared routing, task-specific routing, or combinations of these components. The methods differ in which components they evolve and how they train them.
3.1 Coevolution of Modules
- CM uses CoDeepNEAT to evolve module architectures within the fixed soft-ordering grid, leaving routing fixed and omitting blueprint evolution.Module hyperparameters are evolved, while the blueprint population that determines module connections is not used.Source: §3.1 · p. 3
3.2 Coevolution of Modules/Shared Routing
- CMSR extends CM by coevolving module architectures and a shared routing represented as an arbitrary DAG rather than a fixed grid.Blueprint nodes point to module species, and nodes can evolve independently in whether their weights are shared.Source: §3.2 · pp. 3–4
3.3 Coevolution of Task Routing
- CTR keeps one shared module set but evolves a distinct routing for each task, allowing modules to be assembled at different locations and adapted through soft merges.CTR incurs no additional backpropagation iterations over training a single fixed-topology multitask model.Source: §3.3 · p. 4, §3.3.1 · p. 4
3.3.2 Algorithm Description.
- In CTR, each task's DAG routes shared modules through soft merges; champions and challengers train jointly, then validation accuracy selects replacements and best mean-accuracy checkpoints.The final model comes from the best checkpoint and is evaluated on a held-out test set.Source: §3.3.2 · pp. 4–5
3.4 Coevolution of Modules and Task Routing
- CMTR combines module evolution with task-specific routing evolution, whose improvements over soft ordering are described as largely orthogonal and more powerful together.CoDeepNEAT evolves modules in an outer loop while CTR evolves and assembles task-specific routings in an inner loop; CMTR keeps weights between modules shared.Source: §3.4 · p. 5
4 EXPERIMENTS
The experiments compare evolutionary variants of soft ordering on a fixed subset of Omniglot alphabet-recognition tasks. CMTR, which combines routing and module evolution, performs best, while component analyses examine training, topology, and weight-sharing choices.
4.1 Omniglot Character Recognition
- The experiments use a fixed 50%/20%/30% training, validation, and test split and select 20 of Omniglot’s 50 alphabet tasks in a fixed random order.Each alphabet induces its own character-recognition task.Source: §4.1 · pp. 5–6
4.2 Experimental Setup
- CoDeepNEAT networks are evaluated after 3,000 iterations, while CTR receives 120 meta-iterations (30,000 iterations); fitness is mean validation accuracy across 20 tasks.Each iteration uses one randomly selected image and label from each task.Source: §4.2 · p. 6
- The top 50 CoDeepNEAT networks are retrained for 30,000 iterations, and test accuracy is measured after fresh 30,000-iteration training using the highest-validation snapshot.Learning-rate decay boosts CM and CMSR moderately but does not similarly boost CTR or CMTR.Source: §4.2 · p. 6
4.3 Results
- Around 78% validation accuracy is reached by all three CoDeepNEAT variants; CMTR converges fastest, followed by CM and CMSR, but CTR training gives CMTR better final performance.The search spaces grow from CMTR to CM to CMSR.Source: §4.3 · p. 6, Figure 6 · p. 6
- CTR improves faster during the first 10 meta-iterations, but CMTR with evolved modules soon overtakes it during training.The comparison averages fitness across 10 runs.Source: §4.3 · p. 6, Figure 7 · p. 6
- CMTR, combining CoDeepNEAT and routing evolution, performs best; evolving both modules and topologies beats baselines and either alone, with pairwise differences significant (p < 0.05).Table 1 reports average validation and test accuracy over 20 tasks for each algorithm.Source: Table 1 · p. 7, §4.3 · p. 7
- In CM, no weight sharing outperforms forced sharing, but evolving sharing performs best; sharing near output decoders is a useful strategy.The paper notes that disabling sharing is not optimal for CTR’s more complex routing.Source: §4.3 · pp. 6–7, Figure 8 · p. 6
- Approximately three million parameters characterize the best networks, whose simple and complex modules combine with routing topologies that vary across alphabets.Similar alphabet-specific topologies emerge for similar alphabets and across repeated evolutionary runs.Source: §4.3 · p. 7
5 DISCUSSION AND FUTURE WORK
The experiments show that multitask performance depends on architecture, with differentiated task routings and shared modules particularly effective. The authors identify vision, language, and multimodal tasks as directions for future application.
5 DISCUSSION AND FUTURE WORK
- The experiments show that multitask learning can significantly improve performance across tasks, while the architecture used for it matters substantially.The authors also conclude that combining multitask learning with evolutionary architecture search is particularly powerful.Source: §5 · p. 7
- Evolution discovers systematically diverse modules, from simple to complex, and CMTR routing topologies range from simple to complex, with similar alphabets showing similar topologies.The figure describes two CMTR module species, one simple and one complex.Source: §5 · p. 7, Figure 9 · p. 7
- CMSR's shared but evolvable routing gives modules greater flexibility for reuse and extends soft ordering's approach.The authors note that CM and CMSR often move away from sharing module weights despite reusing module architectures.Source: §5 · pp. 7–8
- CTR and CMTR, which evolve different topologies for tasks while sharing modules, are more powerful than CM and CMSR, which use shared topology and differentiated modules.Sharing components, including weights, is crucial to CMTR's performance.Source: §5 · p. 8
- Future work could apply the algorithms to vision, language, and multimodal tasks, including pairing object recognition with caption generation.The authors identify discovering ways to tie multimodal tasks together as a direction for evolutionary optimization.Source: §5 · p. 8
6 CONCLUSION
The paper presents evolutionary algorithms for optimizing deep multitask network architectures and reports state-of-the-art performance on a widely used multitask benchmark.
6 CONCLUSION
- The proposed evolutionary algorithms achieve state-of-the-art performance on a widely used multitask learning dataset and benchmark.The algorithms optimize deep multitask network architectures.Source: §6 · p. 8