Source-linked AI summary
Evolving Deep Neural Networks
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, Babak Hodjat
TL;DR
Deep neural networks have become too complex to configure reliably by hand as their topologies and hyperparameters grow. CoDeepNEAT evolves components, topologies, and hyperparameters, producing designs comparable to hand-designed architectures and supporting a real-world image-captioning application.
Problem
Deep neural networks have complex topologies and hundreds of hyperparameters, making architecture configuration difficult for human engineers.
Method
CoDeepNEAT extends NEAT to coevolve network components, topologies, and hyperparameters, evaluating networks after partial gradient-descent training.
Results
The evolutionary approach discovers designs comparable to hand-designed architectures in benchmark tasks and supports practical image-captioning applications.
Takeaways & Limitations
Evolutionary optimization can construct complex deep-learning architectures and may benefit from increasing computational resources.
Takeaways & Limitations
The approach is computationally demanding, and the application still needs improved caption evaluation and filtering for incorrect impressions.
Abstract
from arXiv · showhide
The success of deep learning depends on finding an architecture to fit the task. As deep learning has scaled up to more challenging tasks, the architectures have become difficult to design by hand. This paper proposes an automated method, CoDeepNEAT, for optimizing deep learning architectures through evolution. By extending existing neuroevolution methods to topology, components, and hyperparameters, this method achieves results comparable to best human designs in standard benchmarks in object recognition and language modeling. It also supports building a real-world application of automated image captioning on a magazine website. Given the anticipated increases in available computing power, evolution of deep networks is promising approach to constructing deep learning applications in the future.
1 INTRODUCTION
As deep neural networks have grown more complex, configuring their topologies, components, and hyperparameters has become difficult to do by hand. The paper presents an evolutionary approach that automatically optimizes these aspects and discovers designs comparable to the state of the art.
- Motivation: Complex DNN topologies and hundreds of hyperparameters make architecture design difficult for human engineers.The relevant design choices can determine success on a given problem.
- Motivation: DNN architecture optimization involves designing components, assembling a full network topology, and setting component and global hyperparameters for each task.The paper identifies these as three separate aspects requiring optimization.
- Approach: CoDeepNEAT extends NEAT to coevolve DNN components, topologies, and hyperparameters, evaluating networks by how well gradient descent trains them for the task.The approach is demonstrated on object recognition, language modeling, and image captioning.
- Results: The approach discovers designs comparable to the state of the art automatically and without much development effort.The paper presents this as a result enabled by the evolutionary design process.
2 BACKGROUND AND RELATED WORK
Prior neuroevolution work optimized neural-network weights or topologies and, more recently, selected specific DNN design elements under computational constraints. This paper develops a broader evolutionary approach that combines topology, components, and hyperparameters, including cooperative assembly of modules and blueprints.
- Prior neuroevolution: Neuroevolution has long optimized neural-network weights and topologies, especially for sequential decision tasks where gradients are unavailable.Recurrent connections can help disambiguate hidden states in partially observable problems.
- Prior neuroevolution: Evolution has also been combined with gradient-based learning, while deep neuroevolution applies evolution to supervised DNN design when gradients are available.The paper characterizes this as a higher-level optimization of neural-network design.
- Bilevel optimization: Bilevel evolution previously improved helicopter-control performance by expanding the hand-optimized neuroevolution parameter set from eight to 15 parameters.The example motivates evolutionary search over interacting system-design parameters.
- DNN evolution: Recent DNN-evolution studies targeted specific design elements, including hyperparameters, autoencoder weights, or topology and hyperparameters, because computational resources were limited.These approaches did not optimize the full design space described by the paper.
- CoDeepNEAT: CoDeepNEAT evolves separate populations of modules and blueprints, then assembles them by replacing blueprint nodes with modules during fitness evaluation.This representation supports repetitive and deep network structures.
- Paper scope: The paper applies systematic DNN evolution to convolutional networks, LSTM language models, and a real-world image-captioning application.The reported benchmark includes CIFAR-10 image classification and language modeling.
3 EVOLUTION OF DEEP LEARNING ARCHITECTURES
DeepNEAT extends NEAT to evolve deep-network topologies and hyperparameters, while CoDeepNEAT coevolves reusable modules and blueprints. On CIFAR-10, the resulting networks achieved state-of-the-art-comparable error with a repetitive architecture and faster training convergence.
- DeepNEAT: DeepNEAT evolves DNN layer topologies and hyperparameters by representing layers as graph nodes and mutating their properties.Node hyperparameters specify layer types and properties, while global hyperparameters include learning rate, training algorithm, and preprocessing.
- CoDeepNEAT: CoDeepNEAT separately evolves module and blueprint populations, then combines them into larger assembled networks for evaluation.Blueprint nodes point to module species, allowing the same module to be reused at multiple blueprint locations.
- CoDeepNEAT: CoDeepNEAT efficiently evolves repetitive modular structures and explores more diverse, deeper architectures than DeepNEAT through mutations in modules and blueprints.Small mutations can produce large changes in the assembled network structure.
- CIFAR-10 benchmark: 12 epochs were needed to reach 20% test error, versus over 30 epochs for Snoek et al.; convergence took around 120 versus over 200 epochs.The evolved network’s repetitive structure reused the same modules multiple times.
4 EVOLUTION OF LSTM ARCHITECTURES
CoDeepNEAT was extended to search jointly over LSTM unit variants and layer connectivity, including skip connections. On the PTB language-modeling benchmark, the best evolved network improved performance over vanilla LSTM by 5%.
- LSTM evolution: CoDeepNEAT searches simultaneously for new LSTM memory-unit architectures and connectivity patterns across multiple layers.This addresses both directions of recent LSTM architecture research: modifying individual units and stitching layers together.
- LSTM evolution: Two LSTM-specific mutations enable or disable inter-layer connections and add or remove skip connections between LSTM nodes.Multiple LSTM layers are flattened into a graph that neuroevolution modifies.
- Language-modeling benchmark: The PTB language-modeling benchmark contains 929k training words, 73k validation words, 82k test words, and a 10k-word vocabulary.The task is predicting the next word in a large text corpus.
- Language-modeling benchmark: 5% improvement over vanilla LSTM was achieved after 25 generations, with a test-perplexity score of 78.The best variant used a feedback skip connection between the memory cells of two LSTM layers.
- Implications: The initial results show that CoDeepNEAT can automatically discover improved LSTM variants using only two LSTM-specific mutations.The paper suggests that expanding the search space with additional mutation and layer or connection types could lead to further improvements.
5 APPLICATION CASE STUDY: IMAGE CAPTIONING FOR THE BLIND
CoDeepNEAT evolved image-captioning architectures that combine visual and textual representations, then deployed the resulting model to caption magazine images for screen readers. The system outperformed a hand-tuned MSCOCO baseline and received manual evaluations on the magazine dataset.
- 5.1 Evolving DNNs for Image Captioning: CoDeepNEAT combines image embeddings and one-hot text inputs to evolve captioning architectures containing dense, LSTM, sum, concatenation, and associated hyperparameter choices.The search space includes the hand-designed Show and Tell architecture as a baseline.
- 5.1 Evolving DNNs for Image Captioning: Evolution evaluates networks on a 100,000-image MSCOCO subset for six epochs, favoring architectures that converge quickly; the best architectures train to near convergence 6 times faster than Show and Tell.After evolution, the optimized learning rate is scaled by one-fifth to compensate for subsampling.
- 5.1 Evolving DNNs for Image Captioning: The case study supplements MSCOCO with 17,000 magazine image-caption pairs targeting iconic images, which are underrepresented in MSCOCO.MSCOCO emphasizes common objects in context, whereas the magazine includes portraits, infographics, cartoons, abstract designs, and out-of-context objects.
- 5.2 Building the Application: The trained model is served through a browser-injected JavaScript tool that writes generated captions into image alt fields for screen readers.This supports blind users accessing captions while browsing the magazine website.
- 5.3 Image Captioning Results: 29.1 BLEU-4, 88.0 CIDEr, and 23.8 METEOR were achieved by the evolved model on MSCOCO, compared with 27.7, 85.5, and 23.7 for the baseline.The table reports the evolved network's improvement over the hand-designed baseline when trained on MSCOCO alone.
- 5.3 Image Captioning Results: Because suitable automatic metrics were unavailable for the magazine captions, all 3100 holdout images were manually rated from 1 to 4.On iconic images, about one half were rated correct; across all images, about one fifth were rated correct.
6 DISCUSSION AND FUTURE WORK
The authors conclude that evolving deep neural networks is feasible for benchmark performance and real-world applications, but computational cost currently limits the approach. Partial training also biases evolution toward fast learners rather than only top performers.
- Discussion: Evolutionary optimization produced architectures comparable to hand-designed systems in benchmark tasks and supported a real-world application.The authors frame this feasibility result as evidence that the approach can be used beyond benchmarks.
- Discussion: Thousands of networks must be trained during evolution, so results are limited by available computational power.A single deep network can take a couple of days to train on a state-of-the-art GPU.
- Discussion: Partial network training biases evolution toward fast learners rather than solely toward top-performing networks.The authors identify training time, execution time, and memory requirements as alternative evolutionary objectives.
- Future Work: Anticipated growth in cloud and distributed GPU resources is presented as an opportunity to scale evolutionary architecture search.The paper cites cloud GPU services and distributed computing efforts as emerging sources of computational capacity.
7 CONCLUSION
The paper concludes that evolutionary optimization can construct deep architectures more complex than hand-designed systems by jointly optimizing topology, components, and hyperparameters. Such automated design is currently comparable to leading human designs and may enable applications across multiple domains.
- Conclusion: Evolutionary optimization jointly adapts network topology, components, and hyperparameters to task requirements.The paper presents this simultaneous optimization as enabling architectures more complex than humans can design by hand.
- Conclusion: The resulting automated designs are currently comparable with the best human-designed architectures.The conclusion connects this comparison to potential applications in vision, speech, language, and other areas.