PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Neural AutoML for Deep Learning

Jason Liang, Elliot Meyerson, Babak Hodjat, Dan Fink, Karl Mutch, Risto Miikkulainen

arXiv:1902.06827v3cs.NE

TL;DR

Configuring DNNs requires choices about architecture and hyperparameters, and practical applications may also impose size constraints. LEAF applies evolutionary search to these choices and can optimize network complexity alongside performance. The paper reports that LEAF can outperform existing AutoML systems and hand-designed solutions while supporting smaller architectures.

  • Problem

    DNN architecture and hyperparameter choices are difficult to configure, and many existing approaches optimize only part of the configuration despite the need to control network size.

    Source: §1 · p. 1

  • Method

    LEAF uses CoDeepNEAT to evolve DNN architectures and hyperparameters, with multiobjective search for network complexity and distributed training and evaluation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    LEAF can outperform existing state-of-the-art AutoML systems and the best hand-designed solutions while optimizing architecture, hyperparameters, and network size.

    Source: §6 · p. 8

  • Takeaways & Limitations

    LEAF can produce useful solutions from a naive starting point with little domain knowledge, while allowing network size to be optimized for practical requirements.

    Source: §6 · p. 8

    The discussed RL architecture-search approach is limited to linear or tree-like cores, excluding arbitrary graph topologies from its search space.

    Source: §2.2 · p. 2

Abstract

from arXiv · show

Deep neural networks (DNNs) have produced state-of-the-art results in many benchmarks and problem domains. However, the success of DNNs depends on the proper configuration of its architecture and hyperparameters. Such a configuration is difficult and as a result, DNNs are often not used to their full potential. In addition, DNNs in commercial applications often need to satisfy real-world design constraints such as size or number of parameters. To make configuration easier, automatic machine learning (AutoML) systems for deep learning have been developed, focusing mostly on optimization of hyperparameters. This paper takes AutoML a step further. It introduces an evolutionary AutoML framework called LEAF that not only optimizes hyperparameters but also network architectures and the size of the network. LEAF makes use of both state-of-the-art evolutionary algorithms (EAs) and distributed computing frameworks. Experimental results on medical image classification and natural language analysis show that the framework can be used to achieve state-of-the-art performance. In particular, LEAF demonstrates that architecture optimization provides a significant boost over hyperparameter optimization, and that networks can be minimized at the same time with little drop in performance. LEAF therefore forms a foundation for democratizing and improving AI, as well as making AI practical in future applications.

1 INTRODUCTION

Configuring DNN architectures and hyperparameters is difficult, while practical deployments can also require smaller networks. LEAF addresses these needs by evolving hyperparameters and network structure, with multiobjective optimization for minimizing network size.

1 INTRODUCTION

  • Finding effective DNN architectures and hyperparameters is a black-box optimization problem that manual testing makes tedious and time-consuming.Without guiding principles, choices may rely on history and convenience rather than theoretical or empirical principles.Source: §1 · p. 1
  • Common partial optimization rarely tunes architecture and hyperparameters together, while grid search becomes intractable beyond four or five hyperparameters.The search space grows combinatorially, and grid search does not determine the optimal architecture.Source: §1 · p. 1
  • Modern DNNs can have hundreds of millions of weights, creating a size constraint for smartphone applications with only a few gigabytes of RAM.The paper motivates optimizing network complexity while maximizing performance.Source: §1 · p. 1
  • Existing AutoML systems are described as limited in problem scope or feedback, and neither cited system minimizes network size or complexity.Google AutoML hides architecture and training; MOE tunes hyperparameters but not architecture.Source: §1 · p. 2
  • LEAF combines evolutionary hyperparameter and architecture search, and adapts CoDeepNEAT for multiobjective search to find minimal architectures.The paper evaluates the framework on Wikipedia comment toxicity classification and multitask Chest X-rays classification.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The section reviews approaches to hyperparameter tuning and architecture search, including their scaling and search-space constraints. It also introduces LEAF’s layered separation of evolutionary algorithms from system-level training and evaluation.

2.1 Hyperparameter Tuning for DNNs

  • Bayesian optimization can work with multimodal, non-separable, and noisy objectives using relatively few evaluations, but scales cubically in evaluated points and performs poorly above 10–15 hyperparameters.DNGO replaces Gaussian processes with linearly scaling Bayesian neural networks to address the computational cost.Source: §2.1 · p. 2

2.2 Architecture Search for DNNs

  • The discussed recurrent-controller architecture search achieved results within 1–2 percentage points of state of the art on CIFAR-10 and ImageNet, and state-of-the-art language modeling performance at the time.The controller generates layer sequences and is trained with REINFORCE.Source: §2.2 · p. 2
  • That architecture-search approach restricts optimized networks to linear or tree-like cores, excluding arbitrary graph topologies and leaving users to define a search space beforehand.The number of layer hyperparameters is limited, and generating the final network requires training many thousands of candidates.Source: §2.2 · p. 2

2 BACKGROUND AND RELATED WORK

  • LEAF separates the algorithm layer, which evolves hyperparameters or networks, from the system layer that trains and evaluates them; the problem-domain layer uses both to optimize DNNs.This decoupling supports varying problem types, including multiobjective optimization and different neural-network layers.Source: Figure 1 · p. 3

3 LEAF OVERVIEW

LEAF combines evolutionary search with distributed training and evaluation. Its problem-domain layer applies this framework to hyperparameter tuning, architecture search, and network-complexity minimization.

3 LEAF OVERVIEW

  • LEAF’s algorithm layer evolves DNN architectures and hyperparameters, while its system layer parallelizes training on cloud infrastructure and returns fitness information.The algorithm layer sends network architectures in Keras JSON format; the problem-domain layer uses both layers to solve optimization tasks.Source: §3 · p. 3

3.1 Algorithm Layer

  • CoDeepNEAT evolves blueprint and module populations separately, then assembles networks by replacing each blueprint node with a module from its pointed-to species.Nodes encode DNN layers and their hyperparameters, while edges specify layer connections; a shared module species can reuse the same module across blueprint nodes.Source: §3.1 · pp. 3–4
  • Fitness from validation performance is attributed to blueprints and modules as their average across assembled networks, reducing evaluation noise.After evolution, the best network is trained to convergence and evaluated on a holdout test set.Source: §3.1 · p. 4

3.2 System Layer

  • LEAF’s system layer distributes candidate-network training through a queue of worker machines, making it possible to train thousands of candidates in days.Workers train networks and return their fitness information to the server.Source: §3.2 · p. 4

3.3 Problem-Domain Layer

  • For multitask learning, LEAF coevolves module architectures and the blueprint routing between modules, extending a soft-ordering multitask architecture.Different tasks use the modules to different degrees.Source: §3.3 · p. 4
  • Multiobjective CoDeepNEAT ranks candidates through successive Pareto fronts to optimize performance and network complexity simultaneously.The experiments use parameter count as the secondary objective; complexity can also be characterized by FLOPS or training/inference time.Source: §3.3 · pp. 4–5

4 EXPERIMENTAL RESULTS

The experiments test LEAF on Wikipedia toxicity classification and chest X-ray multitask classification against existing AutoML and hand-designed approaches. LEAF achieves strong benchmark performance and discovers useful trade-offs between network complexity and fitness.

4 EXPERIMENTAL RESULTS

  • Wikidetox uses 160K comments with LSTM and FastText search; Chest X-rays uses 112,120 images with convolutional search and eight-epoch AUROC fitness evaluations.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.1 Wikipedia Comment Toxicity Classification Domain

  • After about 9000 CPU hours, LEAF architectures surpassed the Kaggle competition winner on Wikidetox, after earlier runs exceeded Google AutoML, Microsoft TLC, and MOE.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 Chest X-rays Multitask Image Classification

  • LEAF matched CheXNet and exceeded Google AutoML by nearly 4 AUROC points on the Chest X-rays testing set.Source: Table 1 · p. 6, §4.2 · p. 7
  • Multiobjective LEAF converged faster to the same final fitness and produced Pareto fronts that dominated single-objective LEAF across complexity and fitness.Source: §4.2 · p. 7, Figure 4 · p. 7
  • Multiobjective LEAF initially improved fitness at similar complexity, then by generation 40 had lower average complexity as the fitness gap narrowed.It shifted attention to complexity as fitness began to converge.Source: §4.2 · p. 7
  • After eight epochs, multiobjective LEAF found 125K-parameter networks at 77% AUROC and 56K-parameter networks at 74% AUROC.The smaller network used the complex module twice, while the larger one used it four times.Source: §4.2 · p. 7

5 DISCUSSION

In Wikidetox, LEAF’s results suggest that evolutionary optimization can improve on naive starting points and existing systems, with compute spending offering a flexibility-quality trade-off. Multiobjective LEAF also finds networks that balance fitness and complexity, while the paper points to related datasets as a further opportunity.

5 DISCUSSION

  • LEAF can improve over a naive Wikidetox starting point and beat existing AutoML systems and hand-designed approaches with more effort.The authors note that training thousands of networks requires substantial compute, but results can improve by running LEAF longer and spending more.Source: §5 · p. 7
  • Multiobjective LEAF found a 56K network whose performance was nearly as good as a 125K network, using two rather than four instances of the same module.The authors describe these networks as a trade-off between conflicting objectives.Source: Figure 5 · p. 8
  • The authors expect cheaper, more available computing power to yield better results and say evolutionary AutoML can make use of that power.They frame this as a prospective benefit, not a measured result.Source: §5 · p. 8
  • Multiobjective LEAF networks dominate single-objective networks on complexity and fitness in almost every generation, while maintaining higher average fitness.The authors suggest complexity minimization may regularize networks; they propose module reuse as a possible explanation.Source: §5 · p. 8
  • The authors identify using multiple related datasets as a fourth goal, noting prior work that other tasks can help when a task has little training data.They present evolutionary AutoML as a framework for using DNNs in domains that might otherwise be impractical because of data scarcity.Source: §5 · p. 8

6 CONCLUSION

LEAF jointly optimizes DNN hyperparameters and architecture, including network size, and the paper reports that it can outperform existing AutoML systems and hand-designed solutions. The authors present automated design as a route to new applications, while suggesting that it may eventually replace hand-design in some roles.

6 CONCLUSION

  • The paper reports that LEAF can outperform existing state-of-the-art AutoML systems and the best hand-designed solutions.It says this performance can result from simultaneously optimizing hyperparameters, components, and architecture topology for a task.Source: §6 · p. 8
  • LEAF can optimize network size alongside other architecture requirements, and the paper says it can work with little domain knowledge and a naive starting point.The authors frame these capabilities as helping democratize AI and make discovered solutions more useful in practice.Source: §6 · p. 8
  • The authors suggest automated design frameworks could make new applications of deep learning possible and may eventually replace some hand-design of algorithms and DNNs.They describe this as a long-term possibility for systems supporting scientific research and AI product design.Source: §6 · p. 8

APPENDIX

The appendix includes tables describing evolution configuration and the search space explored for Wikidetox and Chest X-rays.

APPENDIX

  • Table 2 is identified as covering evolution configuration for Wikidetox and Chest X-rays.The caption does not provide specific configuration values.Source: Table 2 · p. 8
  • Table 3 is identified as covering the search space explored for Wikidetox and Chest X-rays.The caption does not specify the search-space contents.Source: Table 3 · p. 8

Related papers

Loading 1902.06827v3…