Regularized Evolution for Image Classifier Architecture Search

Introduction

• Human experts had designed most state-of-the-art image classifiers, while evolutionary and genetic-programming architectures had not yet matched their accuracy; this work reports surpassing hand-designs [20] [23] [24].The paper identifies architecture search as an effort to automate classifier design and accelerate the process.Source: Introduction · p. 1
• Aging evolution biases tournament selection toward younger genotypes, while simple mutations reconnect edge origins or relabel edges across the NASNet search space [19, 54].The NASNet space represents architectures as directed graphs whose vertices are hidden states and whose labeled edges denote operations.Source: Introduction · p. 1
• Evolution matched reinforcement learning with a simpler method and searched faster than RL and random search in all experiments, especially during early stages with limited compute [54].The paper presents this as a controlled comparison in the NASNet search space and the first comparative case study of architecture-search algorithms for image classification.Source: Introduction · p. 1
• AmoebaNet-A reaches 82.8% top-1 / 96.1% top-5 ImageNet accuracy at similar sizes and 83.9% top-1 / 96.6% when scaled, setting a new state of the art.The paper describes the similar-size result as competitive with the best image classifiers obtained by other algorithms.Source: Introduction · p. 1

Related Work

• Related work includes reinforcement-learning and evolutionary searches, as well as cascade-correlation, boosting [10], hill-climbing [14], MCTS [34], SMBO [29], random search, and grid search.The review also notes methods that do not use independent architectures [38] and excludes architecture search beyond image classification from its scope.Source: Related Work · pp. 1–2
• Generational evolution such as NEAT can leave faster-training machines idle while waiting for a generation, whereas real-time methods such as rtNEAT and tournament selection avoid that wait.The paper says these real-time methods may leave models alive for long periods; it presents finite lifetimes in aging evolution as a way to improve results over direct tournament selection while retaining efficiency.Source: Related Work · p. 2
• Unlike age-layer evolution, which assigns ages to genes and requires two meta-parameters, this method tracks individual ages and removes the oldest each cycle, keeping population size constant without extra meta-parameters.The paper describes this as a simple approach in which young individuals are less likely to die than very old ones.Source: Related Work · p. 2

Search Space

• Architectures use NASNet’s fixed feed-forward stack of normal and reduction cells, whose five pairwise combinations are searched independently; N and F are manually set.Normal cells preserve image size, while reduction cells halve it; both cell types use five pairwise combinations to define their architectures.Source: Search Space · pp. 2–3

Evolutionary Algorithm

• Aging evolution initializes P random models, selects the highest-validation-fitness model from S sampled candidates, mutates it, then removes the population’s oldest model.Removing the oldest model favors newer candidates and is intended to explore the search space rather than focus on good models too early.Source: Evolutionary Algorithm · p. 3
• Each cycle randomly chooses hidden-state, op, or identity mutation; hidden-state changes preserve feed-forward structure, while op mutation replaces an op from a fixed list.A hidden-state mutation replaces one selected state with another state in the cell, subject to a no-loops constraint.Source: Evolutionary Algorithm · pp. 3–4

Baseline Algorithms

• The RL baseline follows the controller algorithm and code in, while random search independently samples equally likely architectures rather than mutating existing models.The RL controller is an LSTM that constructs pairwise combinations and receives a reward after each architecture is trained and evaluated.Source: Baseline Algorithms · p. 4

Experimental Setup

• All methods share code and search CIFAR-10; after 20k small-model evaluations, discovered architectures are enlarged to baseline-matched N/F and trained longer on CIFAR-10 or ImageNet.The enlargement follows the model augmentation procedure; ImageNet models add an input stem to reduce image size.Source: Experimental Setup · p. 4

• The search space included identity, separable 3x3/5x5/7x7 and dilated 3x3 convolutions, pooling, and 1x7-then-7x1 convolutions; evolution used P=100 and S=25.The operations also included 3x3 average and max pooling.Source: Methods Details · p. 4
• CIFAR-10 used 5k withheld validation examples; ImageNet used 1.2M 331x331 images, 1k classes, 50k withheld validation examples, and its standard validation set for testing.Search-phase models trained for 25 epochs each, and each experiment evaluated 20k models on 450 K40 GPUs in approximately 7 days.Source: Methods Details · p. 4
• Evolution tested P/S configurations 100/2, 100/50, 20/20, 100/25, and 64/16; 100/25 performed best, while identity mutation was fixed at 0.05 and not tuned.Other mutation probabilities were uniform, as described in the Methods.Source: Methods Details · p. 4
• RL learning rates tested were 0.00003, 0.00006, 0.00012, 0.0002, 0.0004, 0.0008, 0.0016, and 0.0032; 0.0008 was best.Optimization runs were excluded from plots to avoid selection bias, as decided a priori.Source: Methods Details · p. 4
• The best 20 models per experiment were augmented to N=6/F=32 and trained for 600 epochs with batch 128, SGD momentum 0.9, L2 weight decay 5 × 10−4, and cosine-decayed initial lr 0.024.Training also used Scheduled-DropPath to 0.7 probability and an auxiliary softmax weighted at half the main softmax; Table 1 used N/F of 6/32 and 6/36.Source: Methods Details · p. 4
• ImageNet models used N/F=6/190 and 6/448, distributed synchronous SGD on 100 P100 GPUs, RMSProp, and the specified standard training settings.The largest model used N=6/F=448; F counts first-stack convolution filters and doubles after each reduction cell. Scheduled-DropPath to 0.7 probability contributed 0.3% top-1 ImageNet accuracy, and applicable conditions matched the baseline study.Source: Methods Details · p. 4

Comparison With RL and RS Baselines

• Evolution yielded more accurate CIFAR-10 models early, while at 20,000 models it achieved accuracy similar to RL; both outperformed random search.Figure 3 reports accuracy before augmentation, not model compute cost; its caption notes RL models had higher compute cost.Source: Comparison With RL and RS Baselines · p. 4, Figure 3 · p. 5
• On CIFAR-10, evolved architectures had higher accuracy at similar FLOPs than random search, and lower FLOPs at similar accuracy than RL.Parameter counts showed similar behavior to FLOPs; lower FLOPs indicate lower forward-pass operation counts.Source: Comparison With RL and RS Baselines · p. 5, Figure 4 · p. 5
• AmoebaNet-A, selected for highest validation accuracy, was slightly more accurate than NASNet-A at matched size or considerably smaller at matched accuracy on CIFAR-10.The authors note this comparison was not fully controlled because training code and the number of experiments could differ.Source: Comparison With RL and RS Baselines · p. 5, Table 1 · p. 5

ImageNet Results

• AmoebaNet-A was evolved on CIFAR-10 and transferred to ImageNet, where it performed comparably to the baseline at the same parameter count.This comparison used the F=190 model.Source: ImageNet Results · p. 6, Table 2 · p. 6
• 83.9%/96.6% top-1/5 ImageNet accuracy with 469M parameters set a new state of the art for enlarged AmoebaNet-A.The model is the F=448 version.Source: ImageNet Results · p. 6, Table 2 · p. 6

Scope of results

• Evolution matched or exceeded RL accuracy in preliminary small-compute searches across three search spaces and three grayscale datasets, though the authors caution that findings may be limited to the tested settings.The discussion calls for comparisons across more spaces, datasets, tasks, and algorithms.Source: Discussion · p. 6, Figure 6 · p. 6

Algorithm speed

• Evolution reached half-maximum accuracy in roughly half the time indicated for RL, but the paper declines to quantify speed further because the measure depends on the accuracy threshold and definition of speed.The discussion notes that speed may matter more in larger search spaces and points to SP-III as an example where evolution stood out.Source: Discussion · pp. 6–7

Model speed

• Evolved models had lower FLOPs, and the authors speculate that asynchronous evolution may indirectly favor faster models because they reproduce sooner.They considered only asynchronous algorithms for resource utilization and suggest comparing asynchronous and generational methods for accuracy.Source: Discussion · p. 7

Benefits of aging evolution

• In small-compute experiments on grayscale CIFAR-10, aging evolution tended to outperform non-aging evolution across population and sample settings.The experiments used CPU, and the supplement reports that the trend also held when varying the dataset or search space.Source: Discussion · p. 7, Figure 7 · p. 7

Understanding aging evolution and regularization

• The authors hypothesize that aging reduces overfitting to training noise by renewing the population and favoring architectures that retrain well; a toy noisy-evaluation search found results consistent with this conjecture.They leave further verification, potentially including theoretical work, to future research.Source: Discussion · p. 7

Simplicity of aging evolution

• The authors characterize evolution as random search plus selection and note that it requires few meta-parameters, while RL has additional implementation and tuning complexity [37].They acknowledge that careful RL tuning might yield better models, but could be costly; very long random search might also match results at high cost [21] [36].Source: Discussion · pp. 7–8

Interpreting architecture search

• Final evolved populations had mean output fan-in 3 standard deviations above random-model expectations, and training models with different fan-in values confirmed that accuracy increases with fan-in.The pattern was also found in ResNeXt; broader pattern discovery may require search spaces designed for that purpose.Source: Discussion · p. 8

Additional AmoebaNets

• AmoebaNet-B came from platform-aware search, AmoebaNet-C from an early promising model, and AmoebaNet-D from manual extrapolation optimized for training speed.AmoebaNet-C was referenced in other work [11], and AmoebaNet-D won Stanford DAWNBench for lowest ImageNet training cost [9].Source: Discussion · p. 8

Conclusion

• Aging evolution makes genotypes die according to age, favoring younger ones; it improved on standard tournament selection while asynchronous population updates retained efficiency at scale.Simple mutations also enabled applying evolution to the popular NASNet search space, and the code was open-sourced.Source: Conclusion · p. 8
• In a controlled comparison with reinforcement learning and random search, evolution searched somewhat faster, excelled with scarce resources or early stopping, and matched reinforcement learning’s final model quality with a simpler method.Source: Conclusion · p. 8
• AmoebaNet-A was the first evolved model to surpass hand-designed ImageNet models, matched the accuracy of same-size search-discovered classifiers, and set a new state of the art at large size.The paper also reports open-sourcing its code and checkpoint.Source: Conclusion · p. 8

Motivation

• The supplement compares both approaches across image-classification contexts using repeated experiments without sampling bias.This design aims to support a statistically confident comparison.Source: Motivation · p. 11

Setup

• The experiments used three NASNet search spaces of increasing size: SP-I, SP-II, and SP-III; SP-I matched the main-text variant and the baseline study.SP-II expanded the allowed operations from 8 to 19, while SP-III used larger cell trees (C=15) with the same 19 operations.Source: Setup · p. 11
• The evolutionary algorithm matched the main-text method, while RL used the baseline study’s algorithm, selected because it had then achieved the most accurate CIFAR-10 results [54].The cited baseline was described as a popular dataset and architecture-search study when the experiments began.Source: Setup · p. 11
• Both algorithms used identical network construction, training, and evaluation code and were compared under matched conditions at different compute scales.Supplementary runs used grayscale datasets, CPU, small models (F=8), and four epochs; unless stated otherwise, they used SP-I and G-CIFAR.Source: Setup · p. 11

Findings

• Across five contexts, evolution was statistically significantly better than or equal to RL, and its accuracy rose significantly faster during the initial stage.Higher RL learning rates did not accelerate that initial stage; the initial speed advantage was especially noticeable in SP-III.Source: Findings · p. 11
• Evolution and RL were each repeatedly run to optimize their meta-parameters before comparison across contexts that varied the dataset or search space.The comparison contexts included different datasets or search spaces.Source: Findings · p. 11

Outcome

• The Outcome passage identifies a main-text comparison of image-classifier architecture-search algorithms in an SP- context, but the supplied passage ends before specifying the search space.The excerpt does not provide additional outcome results.Source: Outcome · p. 11

I search space

• The supplement extends the main-text results with small experiments varying datasets and search spaces to confirm the conclusions.Figure A-1 spans G-CIFAR, MNIST, and G-ImageNet contexts and describes evolution performing better than RL at 5k models in all five contexts.Source: I search space · p. 11, Figure A-1 · p. 12

Motivation

• AE keeps population size constant by removing the oldest model, whereas NAE removes the worst model in each tournament [36].The supplement compares the two algorithms across multiple contexts.Source: Motivation · p. 13

Setup

• The AE–NAE experiments used the same search spaces and datasets as Supplement A.Figure B-1 compares G-CIFAR/SP-I, G-CIFAR/SP-II, G-CIFAR/SP-III, MNIST/SP-I, and G-ImageNet/SP-I.Source: Setup · p. 13, Figure B-1 · p. 13

Findings

• AE achieved statistically significantly higher final accuracy on average in 4 of 5 contexts; Figure B-1 shows the comparisons.The exception was G-ImageNet, where experiments were extremely short because CPU training on the larger dataset was compute-demanding.Source: Figure B-1 · p. 13, Findings · p. 13
• In the larger SP-II and SP-III search spaces, every AE run outperformed every NAE run.These were two of the five dataset–search-space contexts compared.Source: Findings · p. 13
• Three at-scale AE–NAE experiments under the main-text conditions provided verification that the smaller CPU-experiment observations generalize to large compute.Figure B-2 presents these comparisons, but the supplied passages do not report their specific outcomes or values.Source: Findings · p. 13, Figure B-2 · p. 13

Outcome

• AE tends to outperform NAE across varied parameters and search space–dataset contexts, a robustness benefit when computational costs limit meta-parameter optimization runs.The supplement extends the Discussion’s observation beyond one fixed context.Source: Outcome · p. 13

Motivation

• In the toy search space, aging evolution outperforms non-aging evolution, although whether aging helps navigate noisy evaluations remains unverified and is left to future work.The authors present the experiment as a sanity check motivated by that suspicion.Source: Motivation · p. 14

Setup

• The toy problem contains no neural networks; it is a single-optimum, D-dimensional noisy optimization task whose signal-to-noise ratio matches the neuroevolution experiments.Source: Setup · p. 14
• Each D-dimensional cube vertex is a bit-string simulated architecture, scored by its zero-coordinate fraction plus Gaussian noise (µ = 0, σ = 0.01); the optimum is the origin.The noise level matches that observed for neural networks.Source: Setup · p. 14
• The search uses 10k sample complexity and runs in milliseconds; the unit cube restricts a bounded-support integer-valued paraboloid problem to {0, 1} coordinates for faster computation.At each cube corner, the paraboloid value equals the number of nonzero coordinates.Source: Setup · p. 14
• The simulations are not intended to mimic neural-network architecture search, but only to test techniques for evolving solutions under noisy evaluations.Source: Setup · p. 14

Findings

• AE was never worse than NAE and was significantly better at larger problem dimensionalities in the toy search space.The graph summarizes thousands of simulations, with 100 simulations per meta-parameter combination and ±2 SEM error bars.Source: Figure C-1 · p. 14, Findings · p. 14

Outcome

• The findings provide circumstantial evidence that aging may help navigate noise, while verifying this more generally remains future work.Source: Outcome · p. 14

Motivation

• AmoebaNet-B, C, and D all used aging evolution, but differed in search platform, selection criterion, or manual extrapolation.Source: Motivation · p. 15

Supplement D: Additional AmoebaNets

• AmoebaNet-B was searched and evaluated on TPUv2, letting evolution optimize hardware-dependent factors such as compiler optimizations.The search used the larger SP-II space and trained F=32 models for 50 epochs.Source: Setup · p. 15

Setup

• For AmoebaNet-B, researchers sampled 100 models across parameter-count bins, augmented them to N=6 and F=32, and chose the highest-validation-accuracy model.Source: Setup · p. 15

Supplement D: Additional AmoebaNets

• AmoebaNet-C was chosen during the ongoing search as a high-accuracy, relatively small model, using CIFAR-10 test accuracy for selection.It was selected for expediency and intended only for ImageNet benchmarking, rather than by the main text’s final-validation-accuracy procedure.Source: Setup · p. 15

Setup

• AmoebaNet-D was manually derived from AmoebaNet-B by extrapolating later-stage improvements, changing cell operations, reducing precision to 16-bit, and training for 35 epochs.The changes included factorized 1x7 and 7x1 convolutions and average pooling; the DAWNBench submission used N=2 and F=256.Source: Setup · pp. 15–16

Findings

• Figure D-1 shows the three model architectures, but the paper does not benchmark them there and directs readers to results presented elsewhere.Source: Findings · p. 16

Outcome

• The supplement describes additional evolutionary experiments that produced three new models, primarily in search of better models.Source: Outcome · p. 16
• Because these methods were resource-intensive, the supplement omits ablations and baselines and points to the main-text AmoebaNet-A process for a more empirically rigorous approach.Source: Outcome · p. 16
• 2.13% test error on CIFAR-10 was AmoebaNet-B’s result in a previous preprint after training with cutout, but it has since been superseded.The previous preprint’s result is noted as historical, not current state of the art.Source: Outcome · p. 16
• AmoebaNet-C had set the previous state-of-the-art top-1 accuracy on ImageNet after training with advanced data augmentation [11].Source: Outcome · p. 16
• $49.30 was AmoebaNet-D’s ImageNet training cost to reach 93% top-5 accuracy in the Stanford DAWNBench lowest-cost competition [9].The category aimed to minimize the monetary cost of training to that accuracy.Source: Outcome · p. 16
• 16% better than ResNet on the same hardware was AmoebaNet-D’s training-cost result in the Stanford DAWNBench competition [9].The passage identifies ResNet as the second-best model.Source: Outcome · p. 16