PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Esteban Real, Chen Liang, David R. So, Quoc V. Le

arXiv:2003.03384v2cs.LGcs.NEstat.ML

TL;DR

AutoML research has mostly used restrictive, human-designed search spaces, especially for neural-network architectures. AutoML-Zero searches for complete algorithms from basic mathematical operations with evolutionary methods. It discovers neural networks and task-adapted techniques, while the authors note that the current search space still imposes important limitations.

  • Problem

    AutoML has made progress mainly through restrictive search spaces built from expert-designed layers or isolated algorithmic components, limiting searches for complete algorithms.

  • Method

    AutoML-Zero represents whole algorithms as programs with Setup, Predict, and Learn functions and searches their instructions using basic operations and evolutionary search.

  • Results

    Evolution discovers gradient-descent neural networks and, on task-specific datasets, algorithms using multiplicative interactions, normalized gradients, weight averaging, and dropout-like operations.

  • Takeaways & Limitations

    The results provide preliminary evidence that evolutionary search can discover complete machine-learning algorithms from scratch while adapting them to task types.

  • Takeaways & Limitations

    The current search space processes one example at a time, so techniques requiring batches or higher-order tensors would require additional search-space mechanisms.

Abstract

from arXiv · show

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

1. Introduction

AutoML-Zero extends AutoML from constrained architecture search toward discovering complete machine-learning algorithms from basic operations. Evolutionary search makes this generic, sparse space tractable enough to recover neural networks and task-adapted techniques.

  • Existing AutoML studies largely constrain search to expert-designed layers or isolated algorithmic components such as learning rules and data augmentation.
  • AutoML-Zero searches jointly over model structure, optimization, initialization, and other algorithm components using simple mathematical operations.
  • The framework represents algorithms as programs with Setup, Predict, and Learn functions, making the search space generic but sparse.
  • Evolution can discover neural networks trained by gradient descent from empty programs and teacher-generated data.
  • On CIFAR-10-derived tasks, evolved algorithms surpass gradient-descent neural networks by discovering multiplicative interactions, normalized gradients, and weight averaging.
  • The paper presents AutoML-Zero, a minimally designed search framework, together with results showing nuanced algorithms can emerge through evolutionary search.

3. Methods

AutoML-Zero encodes learning algorithms as small programs and evaluates them across tasks, then uses regularized evolutionary search to mutate and select candidates. The implementation adds throughput mechanisms while retaining a broad operation-level search space.

  • Search seeks a high-quality, generalizable algorithm in a large space by evaluating candidates on search tasks and selecting among them using held-out performance.
  • Search Space: Algorithms operate on small virtual memories with scalar, vector, and matrix variables, while instructions select basic operations and memory addresses.
  • Search Space: Setup initializes state, Predict produces outputs, and Learn updates the algorithm during example-by-example training before validation evaluates Predict.
  • Search Method: Mutations can insert or remove instructions, randomize a component function, or modify an instruction argument.
  • Search Method: Regularized evolution maintains a population, removes its oldest member, selects the best algorithm from a random tournament, then copies and mutates it.
  • Search Method: Functional-equivalence checking reuses duplicate evaluation scores, while hurdles and distributed workers increase algorithm-evaluation throughput.

4. Results

AutoML-Zero searches a highly generic, sparse space of complete machine-learning algorithms using evolutionary methods, which outperform random search as tasks become harder. The experiments show that evolution can rediscover backpropagation, exceed hand-designed baselines on image tasks, and adapt algorithms to data, training-speed, and class-count conditions.

  • The experiments ask how difficult AutoML-Zero search is, whether minimal human input can discover reasonable algorithms, and whether task variation produces different algorithms.
  • Search difficulty: Evolution greatly outperforms random search as task difficulty increases because acceptable algorithms become increasingly sparse.Task difficulty is estimated from random-search success rates relative to hand-designed references.
  • Rediscovering neural networks: With multiple search tasks, evolution discovers both a neural-network forward pass and backpropagation code for learning task-specific weights.Using one task can instead produce a hard-coded prediction function and weights.
  • Image-task evaluation: Evolved algorithms perform better than the hand-designed neural-network reference in 13 out of 20 held-out experiments.The final evaluation uses unseen binary-classification tasks from the original CIFAR-10 data, while the discovered algorithm includes noise injection, bilinear interactions, and normalized gradients.
  • Image-task evaluation: 84.06 ± 0.10% accuracy on CIFAR-10 significantly exceeds logistic regression at 77.65 ± 0.22% and a 2-layer fully connected neural network at 82.22 ± 0.17%.The best evolved algorithm also outperforms the baselines on binary tasks from SVHN, downsampled ImageNet, and Fashion MNIST.
  • Algorithm adaptations: Task conditions elicit distinct adaptations: noisy ReLU with few examples, learning-rate decay for fast training, and transformed weight means as learning rates for multiple classes.These patterns recur across controlled experiments: decay appears in 30/30 fast-training trials versus 3/30 controls, while the multi-class adaptation appears in 24/30 versus 0/30 controls.

5. Conclusion and Discussion

The paper demonstrates that AutoML-Zero can evolve whole machine-learning algorithms from basic operations with minimal restrictions, while identifying practical limitations and future search-space improvements.

  • AutoML-Zero represents algorithms as Setup, Predict, and Learn programs and evolves neural networks, gradient descent, multiplicative interactions, weight averaging, and normalized gradients.
  • Four search-method upgrades—migration, FEC, increased diversity, and hurdles—were beneficial across three tested compute scales.
  • Evaluating evolved algorithms requires hyperparameter tuning, but program variables may have unknown meanings and interacting effects.
  • Hyperparameter coupling can produce values that work on specific tasks but fail to generalize, requiring time-consuming manual decoupling.
  • Interpreting evolved code requires simplification, recurring-motif analysis, and ablation-based verification because raw programs are complex.
  • The current search space processes one example at a time and lacks loops or function calls, limiting discovery of batch-based techniques and deeper networks.

S1. Additional Related Work

Related work has automated individual algorithm components such as model structure and optimization, whereas AutoML-Zero searches more broadly across the complete algorithm.

  • Architecture search typically targets model structure using expert-designed layers and constrained neural-network building blocks.
  • Numerically discovered optimizers search weight-update rules, producing coefficients or learned networks that can be effective but difficult to interpret.
  • Symbolic optimizer work evolves mathematical expressions, while later approaches add hand-tuned operations to optimize neural-network accuracy.
  • Unlike prior optimizer searches, AutoML-Zero does not assume an existing neural network or gradient; these must be discovered with the rest of the algorithm.
  • The paper also relates to program synthesis but focuses on learning algorithms for common machine-learning tasks rather than self-reflexive programs.

S2. Search Space Additional Details

The search space uses basic mathematical operations without derivatives, while evolutionary search is optimized to explore sparse programs efficiently and safely. Additional mechanisms reduce redundant evaluation, adapt computation across workers and tasks, and stop degenerate or underperforming algorithms early.

  • Operation vocabulary: Gradient computation must be evolved because the operation vocabulary contains no derivatives.
  • Evolutionary mutations: Mutations add, remove, randomize, or modify instructions, with removal twice as likely as addition and real constants perturbed multiplicatively.Categorical choices are uniform; constants are scaled by a random factor in [0.5, 2.0] and have a 10% sign-flip probability.
  • Evaluation efficiency: Functional equivalence checking caches fingerprints from 10 training steps and 10 validation examples to avoid repeatedly evaluating identically functioning algorithms.The cache stores 100k fingerprint–accuracy pairs.
  • Parallelism: Parallel workers evolve separate populations and exchange 50 algorithms through a central migration server every 100–10000 evaluations.
  • Resource control: Hurdles early-stop low-performing models using a rolling 75th-percentile accuracy threshold, consistently saving 75% of compute.Degenerate algorithms are also terminated for NaN, Inf, excessive error, or runtime exceeding 4x a plain gradient-descent neural network.

S4. Task Generation Details

Binary classification tasks are generated by pairing classes, projecting their features with a Gaussian random matrix, and using the resulting examples for search and selection.

  • Each binary task randomly selects two CIFAR-10 classes as positive and negative examples, then projects their features using a Gaussian random matrix.

S5. Detailed Search Experiment Setups

The supplementary experiments progressively expand the operation space, program capacity, task diversity, and evolutionary search setup across Sections 4.1–4.3.

  • Section 4.1: Section 4.1 first restricts operations and instruction counts to discover known linear and affine algorithms, using P=1000, T=10, and U=0.9.
  • Section 4.1: Section 4.1 then searches a two-layer-neural-network operation subset with 21/3/9 Setup/Predict/Learn instructions and P=1000, T=10, U=0.9, and W=1k.
  • Sections 4.2–4.3: Sections 4.2 and 4.3 allow broad operation vocabularies and larger programs, with Section 4.2 combining projected CIFAR-10 and MNIST tasks across multiple data, epoch, dimension, and feature settings.Section 4.2 uses 8/14/3 scalar/vector/matrix addresses and maximum instruction counts of 21/21/45; Section 4.3 uses 10/16/4 addresses and the same maxima.

S6. Evolved Algorithms

The supplementary material exposes raw evolved programs and contrasts them with simplified versions produced by static analysis and ablation-based pruning. Figure S2 provides raw code for the algorithms shown in Figure 7.

  • Figure S1: Supplementary Figure S1 shows the best Section 4.2 algorithm before and after redundant instructions are removed through static analysis.
  • Program simplification: Static analysis removes instructions such as v3 = gaussian(0.7,0.4), while further ablations remove instructions whose deletion does not significantly alter accuracy.The simplified main-text version also reorders instructions and renames variables consistently.
  • Figure S2: Supplementary Figure S2 contains raw code for the algorithms displayed as selected snippets in Figure 7.

S7. Algorithm Selection and Evaluation

The evaluation pipeline searches algorithms on low-dimensional projected tasks, selects candidates using separate tasks, tunes constants for the target dimensionality, and finally evaluates on full-size CIFAR-10 tasks. Hyperparameter coupling can cause performance to drop when moving between datasets, requiring manual decoupling to recover performance.

  • Algorithm Selection and Evaluation: Candidates are first ranked by median accuracy on Tsearch tasks, selected using Tselect tasks, and finally evaluated on full-dimensionality CIFAR-10 tasks.The final evaluation uses binary classification tasks with the original 3072-dimensional CIFAR-10 inputs.
  • Algorithm Selection and Evaluation: Constants are jointly tuned by random search on the full-size dimensionality before final evaluation because evolution starts on 16-dimensional tasks.Each constant is scaled by a random log-scale factor between 0.001 and 1000.
  • Algorithm Selection and Evaluation: Hyperparameter coupling hinders algorithm selection and tuning when evolved hyperparameters are computed from other variables rather than represented as independently tunable constants.For example, setting s2 = norm(v1) can make the learning rate inappropriate on higher-dimensional datasets.
  • Algorithm Selection and Evaluation: Manual inspection and decoupling of coupled parameters can recover performance on a new dataset, while automating this process remains future work.In the example, s2 is replaced with a tunable constant before random-search tuning.

S8. Interpreting Algorithms

Because raw evolved programs are difficult to interpret, the paper simplifies code, identifies recurring sections, and tests hypotheses through ablations and knock-ins. Ablations found four code sections whose removal caused large accuracy losses, while the remaining sections were removed for readability.

  • S8. Interpreting Algorithms: Raw evolved code is interpreted by automatically removing redundant instructions, tracking code sections that recur across independent searches, and testing suspected contributions experimentally.The procedures focus interpretation on computations affecting final predictions and repeatedly discovered sections.
  • S8. Interpreting Algorithms: Four of six ablations caused large validation-accuracy drops: input noise (−0.16%), bilinear modeling (−1.46%), normalized gradients (−1.20%), and weight averaging (−4.11%).The remaining two sections showed no significant loss after ablation and were removed for readability.
  • S8. Interpreting Algorithms: Knock-ins insert a previously helpful code section into simpler algorithms to test whether it improves their performance there as well.This procedure confirmed the usefulness of the four sections identified through ablation.
  • S8. Interpreting Algorithms: Ablation Table S2 summarizes 30 search runs per experimental setting using best accuracy and success fraction on unseen projected binary CIFAR-10 tasks.The table reports uncertainty using ±2 SEM for best accuracy and ±2σ for success fraction.

S9. More Search Method Ablations

Additional ablations test method upgrades under simplified, lower-compute settings, while baseline experiments provide comparisons across compute scales. The four upgrades remain beneficial across three tested compute scales, although the lowest-scale setting shows no defined successes.

  • S9. More Search Method Ablations: Upgrade ablations restrict operations to neural-network construction and gradient-descent learning, reduce projected dimensionality from 256 to 16, and use 1k processes for 5 days.All tasks use T = 8000 and E = 10; later experiments found that more epochs did not change the conclusions.
  • S9. More Search Method Ablations: All four upgrades are beneficial across the three compute scales tested, according to Supplementary Tables S2–S4.Figure 8 presents a subset of these results in plot form.
  • S9. More Search Method Ablations: The paper provides baselines at three compute scales to facilitate comparisons with other search algorithms on the same search space.The study states that its focus was not the search method and identifies room for future work there.
  • S9. More Search Method Ablations: Baseline experiments use a severely restricted search space and end after each process runs 100B training steps.Training steps are used instead of algorithm counts or time to avoid variation from early stopping and hardware differences.
  • S9. More Search Method Ablations: Table S5 summarizes 100 repeats of baseline experiments at three compute scales, with each process running on one commodity CPU core.The table compares restricted baselines with the full and basic search settings.
  • S9. More Search Method Ablations: At 100 processes, results remain similar to higher-compute experiments and support the same conclusions; at 10 processes, no defined successes are observed.The lower-compute table reports consistency but no successes under its stated success definition.
Loading 2003.03384v2…