PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularized Evolution for Image Classifier Architecture Search

Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le

arXiv:1802.01548v7cs.NEcs.AIcs.CVcs.DC

TL;DR

Evolutionary architecture search had not matched human-designed image classifiers, limiting evidence for its effectiveness relative to other search methods. This paper introduces aging evolution and evolves AmoebaNet-A, which surpasses hand-designed models and reaches 83.9% top-1 / 96.6% top-5 ImageNet accuracy at larger scale.

  • Problem

    Evolutionary algorithms had not reached the accuracy of human-designed image classifier architectures, leaving their effectiveness in architecture search unresolved.

  • Method

    The paper introduces aging evolution, a tournament-selection variant that favors younger genotypes, and compares evolution with reinforcement learning and random search.

  • Results

    83.9% top-1 / 96.6% top-5 ImageNet accuracy: scaled AmoebaNet-A sets a new state-of-the-art and surpasses hand-designed models.

  • Takeaways & Limitations

    Evolution is a simple method that can discover competitive image classifiers and search somewhat faster than reinforcement learning when resources are scarce or searches stop early.

  • Takeaways & Limitations

    Some findings may be restricted to the search spaces and datasets used, so broader evaluation is needed to verify generality.

Abstract

from arXiv · show

The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted ones. Here, we evolve an image classifier---AmoebaNet-A---that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new state-of-the-art 83.9% / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. This is relevant when fewer compute resources are available. Evolution is, thus, a simple method to effectively discover high-quality architectures.

Introduction · Related Work

The paper situates image-classifier architecture search as an automated alternative to expert-designed architectures and introduces regularized evolution for controlled comparison with reinforcement learning. Its approach produces competitive AmoebaNet-A models, including a scaled version with state-of-the-art ImageNet accuracy, while related work spans diverse search strategies and efficiency techniques.

  • Introduction: Human experts designed most state-of-the-art image classifiers, motivating automated architecture-search algorithms and scaled neuro-evolution approaches.Improved hardware enabled evolution to produce higher-quality image classifiers.
  • Introduction: Regularized evolution modifies tournament selection by associating each genotype with an age and biasing selection toward younger genotypes.The paper presents this age-based change as one of two additions to the standard evolutionary process.
  • Introduction: The method uses simple mutations in the NASNet search space, whose architectures are represented as small directed graphs.This enables a controlled comparison between evolution and the reinforcement-learning method for which the search space was designed.
  • Introduction: 82.8% top-1 / 96.1% top-5 ImageNet accuracy makes AmoebaNet-A competitive with similarly sized models discovered by other algorithms.The paper names AmoebaNet-A as the high-quality model evolved by its approach.
  • Introduction: 83.9% top-1 / 96.6% top-5 ImageNet accuracy is achieved by AmoebaNet-A when scaled up, establishing a new state-of-the-art accuracy.The paper also reports that evolution performs well in its benchmark against reinforcement learning.
  • Related Work: Architecture-search research includes reinforcement learning, evolution, cascade-correlation, boosting, hill-climbing, MCTS, SMBO, random search, and grid search.The related-work discussion also notes methods that do not use independent architectures and excludes non-image-classification work from scope.
  • Related Work: Many architecture-search methods use substantial resources, while progressive-complexity stages, hypernets, accuracy prediction, warm-starting, ensembling, parallelization, reward shaping, early stopping, and Net2Net can reduce compute cost.The paper states that most of these techniques could in principle also be applied to evolution, but does not pursue that extension.
  • Related Work: Generational evolution can leave faster-training machines idle, whereas real-time algorithms such as rtNEAT and tournament selection address distributed-training inefficiency.An existing age-based method instead assigns age to genes and organizes individuals into age-layers to restrict competition.

Methods

The study searches the NASNet image-classifier space using aging evolution, which maintains trained populations and favors newer models through age-based removal. It compares evolution with reinforcement learning and random search under controlled CIFAR-10 conditions, then enlarges discovered architectures for baseline-matched evaluation.

  • NASNet search space: All experiments use the NASNet search space, a feed-forward stack of Inception-like normal and reduction cells with shared architectures within each cell type.Each cell receives inputs from preceding cells, and the normal-cell architecture is independent of the reduction-cell architecture.
  • NASNet search space: An architecture is specified by five pairwise combinations in each cell type, while manually chosen N and F control model size and accuracy.N is the number of normal cells per stack, and F is the number of output filters in convolution operations.
  • Aging evolution: Aging evolution initializes P randomly architected trained models, selects the highest-accuracy parent from S sampled population members, mutates it, and removes the oldest model each cycle.The search continues for C cycles and returns the highest-accuracy model in the full history.
  • Aging evolution: Removing the oldest rather than the worst sampled model favors newer models and promotes broader search-space exploration instead of premature focus on good models.The paper contrasts this aging approach with non-aging tournament selection, which discards the worst model in the random sample.
  • Aging evolution: Mutations randomly modify either a hidden state or an operation in the normal or reduction cell, with identity also available; only one mutation is applied per cycle.Hidden-state mutations replace a selected hidden state subject to an acyclicity constraint, while op mutations replace an operation from a fixed list.
  • Experimental comparison: Evolution, RL, and random search were compared under identical construction, training, and evaluation code while searching CIFAR-10, with RL using an LSTM controller and trained-model rewards.Search first evaluated 20k small models, then enlarged discovered architectures by increasing N and F to match baseline model sizes.

Methods Details

The experiments used a defined operation set and specified evolutionary population and selection sizes across CIFAR-10 and ImageNet. Identity mutation was fixed at 0.05, while reinforcement-learning optimization selected a 0.0008 learning rate.

  • Experimental setup: P=100 and S=25 were used for evolution on CIFAR-10 and ImageNet experiments.CIFAR-10 withheld 5k examples for validation; ImageNet used 1.2M 331x331 images, 1k classes, and 50k withheld validation examples.
  • Search space: The search operation set included identity, separable convolutions, pooling, dilated separable convolution, and factorized 1x7-then-7x1 convolution.Separable convolutions used 3x3, 5x5, or 7x7 kernels; pooling used 3x3 average or max pooling.
  • Mutation and optimization: 0.05 was the fixed probability for identity mutation, while other mutation probabilities were uniform.The identity-mutation probability was arbitrary and not tuned.
  • Mutation and optimization: 0.0008 was the best learning rate among eight configurations used to further optimize reinforcement learning.The tested learning rates ranged from 0.00003 to 0.0032, and optimization runs were excluded from plots to avoid selection bias.

Results

On CIFAR-10, evolution produced more accurate models than reinforcement learning early in the search and comparable accuracy after 20k models, while augmented evolved architectures offered favorable accuracy–compute tradeoffs. Transferred to ImageNet, AmoebaNet-A reached state-of-the-art performance at similar model sizes and set a new state of the art when enlarged.

  • CIFAR-10 search comparison: Evolution yielded more accurate models than reinforcement learning at earlier search stages, benefiting resource-constrained experiments that may stop early.The comparison used five repeated large-scale architecture-search experiments on CIFAR-10.
  • CIFAR-10 search comparison: After 20k evaluated models, evolution produced models with similar accuracy to reinforcement learning.The experiments were stopped at 20k models, matching the baseline study.
  • Augmented-model comparison: Evolved architectures achieved higher accuracy with similar FLOPs than random search and similar accuracy with lower FLOPs than reinforcement learning.Parameter counts showed similar behavior to FLOPs, and lower FLOPs indicate lower forward-pass operation cost.
  • AmoebaNet-A comparison: AmoebaNet-A was selected as the evolved architecture with the highest validation accuracy and compared against NASNet-A from the baseline study.The comparison was not entirely controlled because the network training code could not be ensured to match.
  • ImageNet results: 83.9%/96.6% top-1/5 ImageNet accuracy was achieved by enlarged AmoebaNet-A with 469M parameters, establishing a new state of the art.AmoebaNet-A was evolved on CIFAR-10 and then transferred to ImageNet, preventing ImageNet overfitting during architecture evolution.

Discussion

The discussion identifies limits to the generality of the findings and proposes broader comparisons and larger search spaces. It also reports faster search, lower-FLOP evolved models, benefits of aging evolution, and future analyses of discovered design patterns.

  • Scope of results: The findings may be restricted to the selected search spaces and datasets, motivating controlled comparisons across more spaces, datasets, tasks, and algorithms.The authors also suggest exploring even larger spaces because speed may matter when available compute cannot reach the optimum.
  • Algorithm speed: Evolution reached half-maximum accuracy in roughly half the time of reinforcement learning, although the authors avoid further quantification because speed depends on its measurement.The comparison suggested both algorithms approach a common accuracy asymptote.
  • Model speed: Evolved models were faster in terms of lower FLOPs, possibly because asynchronous evolution indirectly favors models that reproduce quickly.The authors identify verifying this explanation as future work.
  • Benefits of aging evolution: Aging evolution appeared advantageous in small-compute experiments, including CPU searches on grayscale CIFAR-10, with results tending to hold across datasets and search spaces.The comparison used reduced compute requirements, and the supplement varied the dataset or search space.
  • Understanding aging evolution and regularization: Aging may improve exploration under noisy training by giving models short lifespans, frequently renewing the population and preventing lucky models from dominating.In non-aging evolution, lucky models may remain for the whole experiment and produce many children.
  • Interpreting architecture search: Future work should analyze architecture-search experiments for design patterns; architectures with high output-vertex fan-in were favored across the experiments.The final evolved populations had a mean fan-in value three standard deviations above the relevant comparison, but the supplied passage is truncated before completing that comparison.

Conclusion

The paper introduced aging evolution for image-classifier architecture search, finding it more effective than standard tournament selection and efficient at scale. Evolution searched faster in scarce-resource settings, matched reinforcement learning in final model quality, and produced AmoebaNet-A, the first evolved model to surpass hand-designed classifiers.

  • Aging evolution: Aging evolution favors younger genotypes, improves upon standard tournament selection, and supports efficient asynchronous population updating.The authors also open-sourced the implementation.
  • Algorithm comparison: Evolution showed somewhat faster search speed than reinforcement learning and stood out under scarce resources or early stopping.The comparison also included random search.
  • Algorithm comparison: Evolution matched reinforcement learning in final model quality while using a simpler method.This result came from a controlled comparison of evolution, RL, and random search.
  • AmoebaNet-A: AmoebaNet-A was the first evolved image classifier to surpass hand-designed models, matched the accuracy of similarly sized search-discovered classifiers, and achieved state-of-the-art accuracy at large size.The authors open-sourced the code and checkpoint.

Supplement A: Evolution and Reinforcement Learning · Motivation

The supplement extends the comparison between evolution and reinforcement learning in architecture search. It emphasizes that direct algorithmic comparisons are difficult because studies often use different search spaces, which can obscure the source of performance differences.

  • Motivation: Evolutionary algorithms and reinforcement learning have both recently been applied to architecture search.
  • Motivation: The supplement extends the evolution–reinforcement learning comparison reported in the Results Section.
  • Motivation: Direct attribution of observed results to the search algorithm is difficult when studies use novel search spaces.
  • Motivation: A small search space can make an approach appear fast even when the algorithm itself is not responsible for the speed.
  • Motivation: Different search spaces prevent straightforward comparisons between evolutionary algorithms and reinforcement learning.
  • Motivation: The supplement presents the comparison as further complicated by studies using differing experimental setups.

Setup · Findings · Outcome

The study compares evolution with reinforcement learning under matched conditions across search spaces, datasets, and compute scales. Evolution was at least as good as RL in the tested contexts, with statistical significance.

  • Setup: All experiments used the NASNet search-space design, with three spaces—SP-I, SP-II, and SP-III—ordered by increasing size.The spaces differ in the number of pairwise combinations and allowed operations; SP-I is the main-text and baseline variant.
  • Setup: The comparison used the main-text evolutionary algorithm and the baseline study’s RL algorithm, selected for its prior CIFAR-10 accuracy.The baseline was considered the most accurate CIFAR-10 result available when the study began.
  • Setup: Evolution and RL were evaluated at different compute scales under identical experimental conditions.Both approaches used the same code for network construction, training, and evaluation.
  • Setup: The supplementary experiments reduced resource usage by using grayscale datasets, CPUs instead of GPUs, and a smaller compute scale than the main text.Examples included “G-Imagenet” instead of ImageNet.
  • Findings: Evolution was better than or equal to RL, with statistical significance, across five contexts that varied the dataset or search space.The study first optimized meta-parameters for both algorithms, then compared them under the five contexts.
  • Findings: The most favorable contexts for evolution and RL were examined in greater detail using repeated runs.Figures A-1c and A-1d show progress for five repeats of each algorithm in their respective best contexts.
  • Outcome: The outcome section references the main text’s algorithm comparison in the context of the SP- search space.The supplied passage truncates the specific search-space designation after “SP-”.

I search space … Findings

The supplement extends the aging-evolution comparison across datasets and search spaces, finding higher final accuracy for aging evolution in most tested contexts. Additional scaled experiments verify that observations from smaller CPU experiments generalize to the large-compute regime.

  • I search space: The supplement varies datasets and search spaces through many small experiments to confirm the main-text conclusions.These experiments focus on CIFAR-10 at scale and extend the results across conditions.
  • Motivation: The comparison distinguishes aging evolution, which removes the oldest model, from non-aging evolution, which removes the worst tournament participant.Aging evolution keeps population size constant by removing the oldest model whenever a new model is added.
  • Setup: The experiments use the same search spaces and datasets as Supplement A.This establishes the setup for the subsequent comparisons across contexts.
  • Findings: Five contexts compare non-aging and aging evolution across G-CIFAR, MNIST, and G-ImageNet search-space combinations.The contexts are G-CIFAR/SP-I, G-CIFAR/SP-II, G-CIFAR/SP-III, MNIST/SP-I, and G-ImageNet/SP-I.
  • Findings: 4 of 5 contexts showed statistically significant higher final accuracy for aging evolution on average.The exception was G-ImageNet, where experiments were extremely short because training required substantial CPU computation.
  • Findings: Three scaled experiments compared aging and non-aging evolution under the same conditions as the main text.These experiments tested whether findings from smaller CPU experiments generalized to the large-compute regime.
  • Findings: The scaled comparison used the main-text dataset, search space, resources, and duration across aging and non-aging evolution experiments.It included aging evolution with favorable meta-parameters and two non-aging evolution configurations.

Outcome · Supplement C: Aging Evolution in Toy Search Space · Setup

The supplement extends the finding that aging evolution is robust across contexts and uses a noisy toy optimization problem to test why aging may help. The toy setup represents candidate architectures as binary cube vertices with noisy evaluations, while explicitly serving only as a testbed rather than a neural architecture-search simulation.

  • Outcome: The supplement tests whether aging evolution’s advantage holds across varied contexts, beyond one fixed search space–dataset setting.This robustness matters because computationally demanding architecture-search experiments may not permit many runs for meta-parameter optimization.
  • Supplement C: Aging Evolution in Toy Search Space: Aging evolution performs better than non-aging evolution in a toy search space designed to isolate noisy evaluation as the only difficulty.The authors present this as a sanity check motivated by the suspicion that aging helps navigate noisy evolutionary evaluations, leaving formal verification for future work.
  • Setup: The toy problem is a single-optimum, D-dimensional noisy optimization task with a signal-to-noise ratio matching the neuroevolution experiments.It contains no neural networks and is intended to evolve solutions to a deliberately simple problem.
  • Setup: Candidate solutions are vertices of a D-dimensional unit cube, represented as bit-strings of 0s and 1s analogous to neural network architectures.Each vertex receives a noisy simulated accuracy, paralleling the noisy accuracy obtained by training and evaluating an architecture.
  • Setup: The cube optimization can be viewed as a bounded-support, multidimensional integer-valued paraboloid whose coordinates are restricted to {0, 1}.Under this restriction, the paraboloid’s value at a cube corner equals the number of nonzero coordinates, with mutations treating coordinate values categorically.
  • Setup: The simulations are not intended to faithfully mimic neural-network architecture search experiments.They are used only as a testing ground for methods that evolve solutions under noisy evaluations.

Findings … Setup

The findings compare optimized NAE and AE across toy problems of varying dimensionality, while supplementary experiments describe additional aging-evolution AmoebaNets and their distinct selection and hardware-aware setups.

  • Findings: Optimized NAE and AE perform similarly in low-dimensional, easier problems, while AE becomes relatively better as dimensionality D increases.The comparison uses simulated accuracy across problem dimensionality.
  • Findings: The toy-search-space analysis averages 100 evolutionary simulations for each meta-parameter combination and reports optima with ± 2 SEM error bars.Simulated accuracy is plotted against dimensionality D, which measures problem difficulty.
  • Outcome: The findings provide circumstantial evidence that aging may help evolutionary search navigate noise, motivating more general verification as future work.The paper identifies this as a suspicion rather than a definitive conclusion.
  • Supplement D: Additional AmoebaNets: AmoebaNet-B, AmoebaNet-C, and AmoebaNet-D all used the aging evolution algorithm but differed in platform awareness, selection criterion, and search procedure.B used platform-aware search, C used a Pareto-optimal criterion, and D used multi-stage search with manual extrapolation.
  • Setup: AmoebaNet-B was evolved directly on Google TPUv2 hardware, allowing search to optimize hardware-dependent accuracy factors such as compiler optimizations.This contrasts with the main-text discovery on GPUs and TPU evaluation of the largest model.
  • Setup: The top AmoebaNet-B candidate was selected from K=100 models by parameter-count bins, validation-accuracy ranking, augmentation to N=6 and F=32, and final validation accuracy.Each bin contributed its top K/B models before augmentation and final selection.
  • Setup: AmoebaNet-C was chosen during the ongoing main-text experiments for expediency because it appeared promising in a Pareto-optimal sense.It was not selected by the main-text procedure of choosing the highest validation accuracy after experiments ended.
  • Setup: AmoebaNet-D was manually derived from AmoebaNet-B by extrapolating evolution, including specified convolution and pooling substitutions plus reduced precision from 32-bit to 16-bit floats.The modifications targeted mutations that continued improving fitness late in the experiment.

Findings · Outcome

The supplement reports additional evolutionary experiments that produced three models, while directing readers to other results and the main text for benchmarking and empirical rigor. Among the reported outcomes, AmoebaNet-D won DAWNBench for ImageNet training cost, while AmoebaNet-B and AmoebaNet-C had earlier state-of-the-art results that were later superseded or surpassed.

  • Findings: The three model architectures are presented without benchmarking in this section, with results deferred to elsewhere.The section explicitly refers readers to results presented elsewhere in the Outcome section.
  • Outcome: Additional evolutionary experiments produced three new models, but the supplement omitted ablations and baselines because the methods were resource-intensive.The supplement identifies the main text’s AmoebaNet-A process as the more empirically rigorous reference.
  • Outcome: 2.13% test error: AmoebaNet-B had set a CIFAR-10 state of the art after training with cutout, but was later superseded.This result appeared in a previous preprint of the paper.
  • Outcome: AmoebaNet-C had set the previous state-of-the-art top-1 accuracy on ImageNet after training with advanced data augmentation.The reported result is attributed to reference [11].
  • Outcome: AmoebaNet-D won Stanford DAWNBench for the lowest monetary training cost on ImageNet to reach 93% top-5 accuracy.The competition category minimized the cost of training a model to 93% top-5 accuracy.
  • Outcome: $49.30: AmoebaNet-D’s ImageNet training cost was 16% better than the second-best ResNet trained on the same hardware.The results were published in [9].
Loading 1802.01548v7…