PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter

arXiv:1804.09081v4stat.MLcs.LG

TL;DR

Neural architecture search must balance predictive performance with resource consumption while avoiding the vast computation required by many existing methods. LEMONADE combines multi-objective evolutionary search with Lamarckian inheritance through approximate network morphisms, and finds competitive models in substantially less search time than resource-intensive alternatives.

  • Problem

    Existing architecture-search methods often optimize predictive performance without resource penalties, while requiring enormous computational resources.

  • Method

    LEMONADE uses multi-objective evolutionary search and Lamarckian inheritance through approximate network morphisms to warmstart and evaluate candidate architectures efficiently.

  • Results

    Within 5 days on 16 GPUs, LEMONADE discovers architectures competitive with hand-designed networks and methods using 40x greater resources.

  • Takeaways & Limitations

    LEMONADE returns architectures spanning 10 000 to 10 000 000 parameters while remaining competitive in predictive performance and resource consumption.

  • Takeaways & Limitations

    The authors suggest that more sophisticated multi-objective algorithms and additional network operators could further improve LEMONADE.

Abstract

from arXiv · show

Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption, (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.

1 INTRODUCTION

LEMONADE addresses two limitations of neural architecture search: excessive resource consumption and enormous search costs. It combines multi-objective evolutionary search with Lamarckian inheritance to find resource-efficient, competitive architectures more efficiently.

  • Motivation: Prior architecture-search methods often optimize predictive performance alone, despite practical requirements for low inference time, model size, or energy consumption.Predictive performance and resource consumption typically involve an implicit trade-off.
  • Lamarckian inheritance: Network morphisms preserve a network’s function while generating children, reducing training from scratch and thereby lowering training time per architecture.Approximate morphisms also permit shrinking networks, which is essential for multi-objective search.
  • LEMONADE: LEMONADE jointly optimizes objectives such as predictive performance, inference time, and parameter count while maintaining a population approximating their Pareto front.It handles cheap and expensive objectives differently to reduce unnecessary training and evaluation.
  • LEMONADE: LEMONADE selects architectures likely to fill gaps in cheap objectives before training and evaluating only that subset.This staged strategy further reduces computational requirements during architecture search.
  • Evaluation: 10 000 to 10 000 000 parameters are covered by LEMONADE’s returned CNN population across up to five objectives and two image-classification search spaces.The evaluated spaces include non-modular architectures and repeatable cells transferable to other datasets.
  • Evaluation: Within only 5 days on 16 GPUs, LEMONADE finds models competitive with hand-designed networks and architectures designed using 40x greater resources.The comparison includes MobileNet V2 and automatically designed architectures such as those from Zoph et al. (2018).

2 BACKGROUND AND RELATED WORK

The background frames architecture search as a multi-objective problem and reviews reinforcement-learning, neuro-evolutionary, performance-prediction, one-shot, and network-morphism approaches. LEMONADE extends this landscape by combining multi-objective optimization with Lamarckian inheritance.

  • Multi-objective optimization: Multi-objective optimization seeks Pareto-optimal solutions when multiple objectives cannot generally be minimized simultaneously.A Pareto-optimal solution cannot improve one objective without worsening at least one other.
  • Neural architecture search: Reinforcement-learning NAS methods generate architectures and use validation performance of trained models as the reward signal.Examples include recurrent controllers that generate architecture descriptions or cells repeated within fixed macro-architectures.
  • Neural architecture search: Neuro-evolutionary NAS uses genetic algorithms, while LEMONADE additionally supports multi-objective optimization and passes learned parameters to offspring.This learned-parameter inheritance is the distinguishing Lamarckian element described in the comparison.
  • Efficiency: Many NAS approaches require vast computation because they train and validate thousands of architectures, motivating performance prediction and weight-sharing alternatives.One-shot methods train a single supergraph and evaluate architectures as its subgraphs.
  • Network morphisms: Network morphisms initialize new architectures from similar trained models to reduce the cost of training every architecture from scratch.LEMONADE extends this idea with approximate morphisms that can also shrink networks.
  • Multi-objective NAS: Multi-objective NAS methods consider resource consumption alongside accuracy, although some approaches restrict the search space through fixed-length architecture descriptions.The related work includes methods targeting accuracy and resource consumption jointly.

3 NETWORK OPERATORS

LEMONADE uses network operators as evolutionary mutations, combining function-preserving network morphisms with approximate morphisms that can reduce network capacity.

  • A network operator maps a parameterized neural network to another neural network, potentially with a different number of parameters.
  • 3.1 NETWORK MORPHISMS: Network morphisms preserve the represented function for every input by appropriately initializing the child network’s parameters.
  • 3.1 NETWORK MORPHISMS: LEMONADE uses operators that insert blocks, increase convolutional filters, or add skip connections as network-morphism mutations.
  • 3.2 APPROXIMATE NETWORK MORPHISMS: Because ordinary network morphisms only increase capacity, they cannot directly reverse architectural decisions or reduce resource requirements.
  • 3.2 APPROXIMATE NETWORK MORPHISMS: Approximate network morphisms allow capacity-reducing operators by making the child approximately represent the parent’s function, using knowledge distillation for initialization.
  • 3.2 APPROXIMATE NETWORK MORPHISMS: LEMONADE’s Lamarckian mutations warmstart children: network morphisms retain the parent’s initial error, while approximate morphisms may cause a small increase.

4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH

LEMONADE searches multiple architecture objectives by maintaining a Pareto-front population, using cheap objectives to focus expensive evaluations, and iteratively retaining non-dominated children.

  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: LEMONADE minimizes expensive objectives such as validation error alongside cheap objectives such as model size.
  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: The population consists of non-dominated parent networks representing the current approximation of the multi-objective Pareto front.
  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: The algorithm is implemented through an iterative procedure that initializes a population, computes parent and child distributions, generates children, and returns the updated population.
  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: Parent selection and child generation use distributions based on cheap-objective density, favoring less dense regions of the search space.
  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: LEMONADE generates more proposed children than it accepts, then evaluates only accepted children on expensive objectives.
  • 4 LEMONADE: MULTI-OBJECTIVE NEURAL ARCHITECTURE SEARCH: After evaluation, the algorithm combines the current generation with generated children, recomputes the Pareto front, and repeats for a preset number of generations.

5 EXPERIMENTS

The experiments evaluate LEMONADE across multi-objective architecture and cell searches, comparing predictive performance with resource consumption and transferability. LEMONADE finds competitive models and cells across several benchmarks and resource regimes.

  • The search covers entire architectures and cells, with discovered cells transferred from CIFAR-10 to ImageNet64x64 and ImageNet.
  • The Pareto front improves over time while covering models ranging from 10 000 to 10 000 000 parameters.
  • LEMONADE evaluates five objectives spanning CIFAR-10 and CIFAR-100 performance, parameter count, multiply-add operations, and inference time.
  • LEMONADE matches NASNets and MobileNets V2 for resource-intensive models and outperforms them among very efficient models, including models below 100,000 parameters.
  • Using only 80 GPU days, LEMONADE finds models with inference times superior to the baselines, compared with 2000 GPU days for Zoph et al. (2018).
  • LEMONADE is on par with or outperforms DPP-Net across parameter regimes while remaining competitive with methods using substantially more computation.
  • On ImageNet64x64, the combined discovered cells and Cell 2 dominate NASNets, WRNs, and MobileNets V2 across the entire parameter range.
  • On regular ImageNet’s mobile setting, Cell 2 achieves a top-1 error of 28.3% and top-5 error of 9.6%, slightly worse than published NASNet results.

6 CONCLUSION

The conclusion presents LEMONADE as a multi-objective evolutionary architecture-search algorithm using Lamarckian inheritance and computationally differentiated objective evaluation. Experiments show competitive accuracy and resource efficiency, while future operator and evolutionary-method improvements remain possible.

  • LEMONADE combines multi-objective evolutionary search with Lamarckian inheritance based on approximate network morphism operators.
  • The algorithm exploits that evaluating predictive performance is orders of magnitude more expensive than evaluating objectives such as parameter count.
  • Experiments on CIFAR-10 and ImageNet64x64 find competitive models and cells in both accuracy and resource efficiency.
  • The authors suggest that more sophisticated multi-objective evolutionary concepts and additional operators could improve LEMONADE in the future.

A.1 DETAILS ON NETWORK OPERATORS

The appendix introduces the network operators used in LEMONADE and points to subsequent subsections for detailed descriptions of network morphisms and approximate network morphisms.

  • The appendix provides detailed information on the network morphisms and approximate network morphisms employed in LEMONADE.

A.1.1 DETAILS ON NETWORK MORPHISMS

Network morphisms modify neural architectures while preserving the represented function through suitable parameter initialization. The described operators support adding, widening, composing, and bypassing network components.

  • A network morphism replaces part of a network while preserving its represented function through properly initialized new parameters.
  • Network morphism Type I: Type I morphisms add fully connected or convolutional layers, including normalization transformations with compensating parameters.
  • Network morphism Type II: Type II morphisms can widen layers or add concatenation-based skip connections while retaining the original computation through initialization.
  • Network morphism Type III: Type III morphisms replace an idempotent function with a composition initialized to reproduce the original function.
  • Network morphism Type IV: Type IV morphisms blend an existing layer with an arbitrary function using a learnable parameter initialized to one.
  • Combinations of network morphisms remain network morphisms, enabling compound blocks such as Conv-BatchNorm-ReLU to be inserted.

A.1.2 DETAILS ON APPROXIMATE NETWORK MORPHISMS

Approximate network morphisms generate altered architectures while retaining inherited behavior, with practical training performed on observed data and affected layers rather than retraining everything from scratch.

  • Definition: An ε-approximate network morphism transforms a network while minimizing a distance between the original and transformed network outputs.Exact network morphisms satisfy the approximate condition for every ε.
  • Practical approximation: The optimal transformed weights can be approximated with the same optimization procedure used for training, such as SGD.The approach is described as akin to knowledge distillation.
  • Practical approximation: Because the true data distribution is unknown, the morphism objective is replaced by an empirical counterpart evaluated on training data.The approximation uses given training data Xtrain and categorical crossentropy as the distance measure.
  • Efficient implementation: To reduce cost further, LEMONADE inherits unaffected weights, freezes them, and trains only affected layers for a few epochs.This procedure applies when operators affect only selected layers and their immediate neighbors.
  • Operators: The experiments use layer or skip-connection removal, filter pruning, and substitution of convolutions with depthwise separable convolutions as approximate morphisms.Affected layers are trained for 5 epochs to minimize the morphism distance.

A.2 ADDITIONAL EXPERIMENTS

The additional CIFAR-10 experiment compares LEMONADE with validation error and model size as joint objectives under identical training conditions.

  • Objectives: The experiment minimizes validation error as the primary objective and log(#params(N)) as a proxy for memory consumption.Only two objectives are optimized for simplicity and computational resource constraints.
  • Experimental design: The comparison uses model architecture as the intended source of performance differences by keeping the training setup identical.This controls training conditions across evaluated models.
  • Experimental design: The study is conducted on CIFAR-10 and compares architectures along predictive performance and parameter count.Parameter count serves as the resource-consumption axis.

A.2.1 ADDITIONAL BASELINES

Additional experiments compare LEMONADE with random search, NAS-generated models, and hand-crafted MobileNet architectures, while examining component ablations and transferred cells.

  • Additional baselines: The baseline comparison includes LEMONADE, random search, NASNets, and differently sized MobileNet V1 and V2 architectures on CIFAR-10.NASNets are described as reinforcement-learning-based search results, while MobileNets are manually designed for resource efficiency.
  • Additional baselines: Random search uses the same number and parameter range of networks and the same training time as LEMONADE for a fair comparison.Its resulting Pareto front is computed with respect to validation data.
  • CIFAR-10 comparisons: Figure 5 reports CIFAR-10 test performance for models trained under identical conditions, with Figure 7 zooming into errors below 0.06%.Table 2 provides detailed comparisons for different model sizes.
  • Method components: LEMONADE’s three components are approximate morphisms for shrinking architectures, Lamarckian inheritance, and two-staged sampling.The method can search entire architectures without limiting discovered size, subject to the operators defining the search space.
  • Ablation study: The ablation study reports that deactivating approximate morphisms, Lamarckism, or KDE-based sampling reduces LEMONADE’s performance.Without approximate morphisms, networks can only grow; without Lamarckism, networks are initialized from scratch; without KDE, sampling is uniform.
Loading 1804.09081v4…