Source-linked AI summary
Large-Scale Evolution of Image Classifiers
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, Alex Kurakin
TL;DR
Neural-network architecture design remains difficult, motivating automated discovery with less human participation. The paper evolves architectures from trivial initial conditions using simple evolutionary methods and large-scale computation, producing fully trained models. It reports competitive results on CIFAR-10 and CIFAR-100, while acknowledging that computation costs were not reduced.
Problem
Designing neural-network architectures remains challenging, including for image classification, so the paper seeks to minimize human participation in architecture discovery.
Method
The paper evolves populations of trained architectures using fitness-based selection, mutations, and inherited weights, starting from trivial initial conditions in large search spaces.
Results
The method reaches 94.6% test accuracy on CIFAR-10, 95.6% with an ensemble, and 77.0% on CIFAR-100.
Takeaways & Limitations
Once started, neuro-evolution requires no experimenter participation and produces fully trained models on two challenging image-classification benchmarks.
Takeaways & Limitations
The study did not focus on reducing computation costs and did not attempt datasets beyond CIFAR-10 and CIFAR-100.
Abstract
from arXiv · showhide
Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%, respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in the outcomes and the computational requirements.
1. Introduction
The paper addresses laborious neural-network architecture design by scaling simple evolutionary techniques to discover competitive image classifiers with minimal human participation. It emphasizes fully trained outputs, broad search spaces, result variability, and computational cost.
- Architecture discovery remains laborious even for image classification, despite neural networks’ success on difficult data-rich tasks.
- The study scales slightly modified evolutionary algorithms and introduces mutation operators to search large architecture spaces from trivial initial conditions.The search allows variable depth, arbitrary skip connections, and broadly mutable numerical parameters.
- 94.6% test accuracy was achieved by the top CIFAR-10 model, while an ensemble reached 95.6%.The reported single-model result used 4×10^20 FLOPs, and the ensemble required no additional training cost.
- 77.0% test accuracy was achieved on CIFAR-100, and the study reports comparisons with hand-designed architectures using C10+ and C100+ test accuracies.
- The method is one-shot, producing a fully trained neural network without post-processing and with few impactful meta-parameters.
- The paper reports result variability, researcher degrees of freedom, and computation requirements to support more transparent evaluation of automated discovery.It motivates evaluating model performance relative to computational investment and opportunity cost.
2. Related Work
Related work spans neuro-evolution, reinforcement learning, Bayesian optimization, and other automated architecture-search methods with differing encodings, search spaces, and initial conditions. This paper distinguishes its approach through large unrestricted searches from basic initial conditions and mutations that can add or remove whole layers.
- Early neuro-evolution evolved weights in fixed architectures, while NEAT extended evolution to architectures through weight, connection, node, recombination, and diversity mutations.
- Earlier studies generally emphasized encoding or evolutionary efficiency rather than the scale required for realistic image-classification datasets such as CIFAR.
- Prior automated discovery methods differ in architecture depth, parameter discreteness, skip-connection support, and whether they require retraining or manual post-processing.The comparison table summarizes these differences for automatically discovered architectures.
- Reinforcement-learning approaches construct convolutional networks sequentially, with some methods allowing the discoverer to determine the number of layers.
- Comparisons across automated discovery methods are difficult because their search spaces and initial conditions differ substantially.
- The proposed method uses a simplified graph encoding, mutates whole layers, permits layer removal, leaves layer parameters broadly mutable, and inherits back-propagated weights across mutations.
3. Methods
The method evolves trained neural-network architectures from simple initial models through asynchronous tournament selection, mutations, validation-based fitness, and inherited weights. It uses a graph encoding, massively parallel computation, and reporting procedures designed to limit overfitting and quantify computational cost.
- Evolutionary algorithm: Each worker compares two individuals, removes the lower-fitness model, mutates a copy of the better parent, trains the child, and returns it to the population.Fitness is validation accuracy, making the process repeated pairwise tournament selection.
- Parallel infrastructure: The infrastructure scales evolution through asynchronous, lock-free workers that share a filesystem rather than communicating directly.Population operations are represented through filesystem directories and atomic renames.
- Encoding and mutations: Architectures use a directed graph whose edges encode identity connections or convolutions and whose vertices apply nonlinearities to activation tensors.The DNA stores mutable convolutional parameters and the learning rate, while inconsistent incoming shapes are resolved before activation.
- Encoding and mutations: Mutations alter learning rates, weights, convolutions, strides, and related architectural choices, with numerical parameters sampled uniformly around existing values.The resulting search space is dense and does not constrain parameters to values already known to work well.
- Training and evaluation: Experiments begin with single-layer, non-convolutional models and evolve them using validation accuracy, while training uses 25,600 steps and standard SGD settings.The test set is withheld from both evolution and neural-network training; only reproduced experiments are included.
- Computation cost: Experiment cost is estimated by summing training and validation FLOPs across individuals, but the estimate excludes input/output, preprocessing, graph-building, memory-copying, and other overheads.The authors expect the estimate to be more useful for large architectures.
- Weight inheritance: Children inherit parent weights whenever layer shapes match, preserving compatible weights across mutations to support evolution within limited training budgets.Some mutations preserve all weights, some preserve none, and most preserve only a subset.
4. Experiments and Results
The experiments test whether simple one-shot evolution can produce competitive models, how outcomes and costs vary, and whether the unchanged algorithm transfers from CIFAR-10 to CIFAR-100. Across repeated runs and controls, evolution reaches high CIFAR-10 accuracy, while weight inheritance and selection improve results relative to their controls.
- Evaluation questions: The experiments are designed to assess variability, parallelizability, computation cost, and unchanged transfer from CIFAR-10 to CIFAR-100.These questions explicitly define the study’s evaluation scope.
- Controls: 87.3% accuracy was achieved by random search under the same runtime and hardware, compared with the selected evolutionary process.The random-search control disabled selection and used 2×10^17 FLOPs.
- Controls: 92.2% accuracy was achieved when weight inheritance was disabled, lower than the corresponding evolutionary result under the same amount of time.This control used 9×10^19 FLOPs.
- Transfer to CIFAR-100: 77.0% accuracy was achieved on CIFAR-100 using the CIFAR-10 algorithm without changes or altered meta-parameters.The single CIFAR-100 experiment used 2×10^20 FLOPs, and no other datasets were attempted.
5. Analysis
The analysis links evolutionary performance to population size, training duration, and mechanisms for escaping local optima, while recombination did not improve recombination-free results.
- Effect of population size: Larger populations explore models more thoroughly and reach better optima, whereas populations of size 2 can become trapped at very low fitness.A super-fit individual can repeatedly defeat its one-mutation-away child, preventing multi-mutation escape sequences.
- Effect of number of training steps: Accuracy increases with the number T of training steps per individual.Larger T reduces the number of identity mutations needed to reach a given training level.
- Escaping local optima: Increased mutation rates and population-wide weight resets can free populations trapped at local optima.These interventions work well but are costly.
- Recombination: None of the three explored recombination strategies improved the recombination-free results.The strategies recombined mutation probabilities, trained weights, or architectural structures.
6. Conclusion
The conclusion presents neuro-evolution as able to construct fully trained image-classification networks from trivial starting conditions without experimenter participation. It also emphasizes repeatability and predictable meta-parameter effects, while acknowledging substantial computational costs.
- Conclusion: Neuro-evolution constructs large, accurate networks for two challenging image-classification benchmarks from trivial initial conditions and a large search space.The process requires no experimenter participation after starting and yields fully trained models.
- Repeatability: High accuracies obtained in five large-scale evolution experiments are repeatable, as indicated by the narrow ±2σ region around the mean curve.Figure 2 also compares weight-inheritance-disabled evolution and random search under the same hardware conditions.
- Meta-parameters: Evolution has fewer tunable meta-parameters, with effects on result variance that are fairly predictable and can be made small.This conclusion is presented as a property of the evolutionary framework.
- Computational requirements: Reducing computation costs was not a focus of the paper, and more economical implementation is left to future algorithmic and hardware improvements.The authors identify computation as a condition for broader practical appeal.
S1. Methods Details
The method evolves serialized graph-based DNA representations of neural architectures through selection, reproduction, training, and mutation. Fitness is validation accuracy, while mutations alter learning rates and graph structure, including convolutional columns and skip connections.
- The population loop selects two random individuals, usually reproduces the fitter individual, and usually removes the less-fit individual while maintaining a population-size set-point.
- Workers handle small populations, population-size errors, and concurrency conflicts by conditionally reproducing, creating individuals, or abandoning the current task.
- The DNA is a serializable directed acyclic graph whose edges represent convolutions and vertices represent nonlinearities, while excluding trained weights.
- Mutations modify DNA attributes and structure, including learning rate changes, vertex-edge insertion or excision, and skip-connection edges.
- Fitness is held-out validation accuracy, and training and evaluation follow a standard image-model procedure.
S2. FLOPs estimation
The paper estimates evolution-wide computational cost by summing training and validation FLOPs across every constructed individual. Per-step and per-validation-batch FLOPs are derived from the executed TensorFlow graph and operation-specific statistics.
- Total evolution cost sums the FLOPs of every constructed individual, including training and validation computation.For an individual, the estimate is FtNt + FvNv.
- Ft and Fv are obtained analytically by summing FLOPs for operations executed during one training step and one validation evaluation.
- The implementation identifies executed graph nodes using runtime metadata and associates them with analytically logged operation statistics.
- FLOP rules are declared for operation types including unary math, binary element-wise operations, reductions, convolutions, and pooling.
S3. Escaping Local Optima Details
The authors examine whether increasing mutation rates helps populations escape local optima. Some populations escape, but the strategy is not frequent or fast enough to be proposed as efficient.
- Increasing mutation rates was introduced to encourage exploration after populations converged at either high or poor fitnesses.
- Identity mutations can let poorly structured but more-trained architectures temporarily outperform better architectures that still require training.