PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Neural Architecture Search

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy

arXiv:1712.00559v3cs.CVcs.LGstat.ML

TL;DR

Directly navigating the exponentially large architecture space is difficult, particularly early in search when there is little knowledge of what makes a good model. The method progressively searches CNN structures from simple to complex, using a learned predictor to rank candidate structures for evaluation. The approach is 5 times more efficient than the RL method of in number of models evaluated, and 8 times faster in total compute.

  • Problem

    Directly navigating the exponentially large architecture space is difficult, particularly early in search when there is little knowledge of what makes a good model.

    Source: §4.1 · p. 6

  • Method

    The method progressively searches CNN structures from simple to complex, using a learned predictor to rank candidate structures for evaluation.

    Source: §1 · p. 2

  • Results

    The approach is 5 times more efficient than the RL method of in number of models evaluated, and 8 times faster in total compute.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting models match previous work's performance while requiring a fraction of its computational cost.

    Source: §6 · p. 14

    The search greedily selects the top K candidates rather than using Bayesian-optimization acquisition functions, which the authors identify as a future direction.

    Source: §6 · p. 14

Abstract

from arXiv · show

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

1 Introduction

Progressive Neural Architecture Search searches convolutional-cell structures from simple to complex while learning a surrogate to guide the search. It reports state-of-the-art classification accuracy with fewer model evaluations and less compute than the RL comparison.

1 Introduction

  • Earlier evolutionary and reinforcement-learning methods could outperform manually designed architectures but required substantial compute; one RL method evaluated 20,000 networks on 500 P100 GPUs over 4 days.Evolutionary algorithms mutate and recombine encoded structures, while RL controllers specify structures through action sequences and use validation performance as reward.Source: §1 · p. 1
  • The search space encodes a CNN as a cell of B blocks, each combining two transformed inputs; cells are stacked to suit dataset size and desired runtime, enabling transfer across datasets.The structured search space is attributed to.Source: §1 · p. 2
  • At each iteration, the search expands K candidate cells of b blocks into K′ children, ranks them with a learned surrogate, evaluates the top K, and repeats until the maximum block count B.The search begins with shallow structures and prunes unpromising candidates as complexity increases.Source: §1 · p. 2
  • 5 times more efficient than the RL method of in models evaluated and 8 times faster in total compute; discovered structures achieve state-of-the-art classification accuracies on CIFAR-10 and ImageNet.The introduction also describes the method as matching previous state of the art while requiring 5 times fewer model evaluations during architecture search.Source: §1 · pp. 1–2
  • Progressive search uses faster-to-train simple structures for early surrogate training, predicts quality only for nearby larger structures, and factorizes the search space into smaller spaces.The nearby-structure prediction is compared to trust-region methods.Source: §1 · p. 2

2 Related Work

The paper positions progressive, predictor-guided structure search against RL and evolutionary methods that search fully specified networks. It distinguishes its hierarchical cell-based CNN search from earlier SMBO work using a flat CNN space and from surrogate methods limited to fixed-size structures.

2 Related Work

  • NAS used REINFORCE [34] to train an RNN policy rewarded by validation accuracy, while introduced cell-based search and PPO.The resulting method outperformed almost all previous methods in accuracy versus speed on CIFAR-10, ImageNet, and COCO object detection.Source: §2 · pp. 2–3
  • [39] and use Q-learning; applies policy gradients to widen or deepen layers, uses network morphisms for hill climbing, and accelerates search through parameter sharing.Source: §2 · p. 3
  • Evolutionary algorithms learn both network structure and parameters in early work such as, while more recent methods evolve structures and use SGD for parameters.Source: §2 · p. 3
  • Unlike RL and EA local search over fully specified graphs, the paper adopts progressive heuristic search, with SMBO using a predictive model to guide node expansion.Prior CNN SMBO used a flat rather than hierarchical cell-based search space; MCTS used random branch selection and was inefficient.Source: §2 · p. 3
  • Most surrogate-function methods predict performance zero-shot or extrapolate learning curves after limited training, but target fixed-size structures and do not suit progressive search.Source: §2 · p. 3

3 Architecture Search Space

The method searches cell structures and stacks copies of a learned cell to build the final CNN, following the hierarchical approach in.

3.1 Cell Topologies

  • A cell maps an H × W × F tensor to an H′ × W′ × F′ tensor; stride 2 halves spatial dimensions and doubles filters, while stride 1 preserves them.Source: §3.1 · p. 4
  • Each cell is a DAG of B blocks, each taking two inputs from earlier blocks or the previous cell and combining two operation outputs by addition.This makes each block a four-part specification once addition is fixed as the combiner.Source: §3.1 · p. 4
  • The eight operators include depthwise-separable convolutions, a 1x7-then-7x1 convolution, identity, average and max pooling, and dilated convolution.Source: §3.1 · p. 4
  • For cells of up to five blocks, the raw search space is 5.6 × 10^14 structures, pruned to ∼10^12 unique cells versus ’s 10^28.The authors note that the resulting space remains extremely large and requires efficient optimization.Source: §3.1 · p. 4

3.2 From Cell to CNN

  • To evaluate a cell, the method stacks untied-weight copies with stride 1 or 2, then applies global average pooling and a softmax classifier.CIFAR-10 uses 32 × 32 images; ImageNet uses 331 × 331 or 224 × 224 images, with the smaller setting faster but less accurate and an initial stride-2 convolution.Source: §3.2 · p. 5, Figure 1 · p. 5

4 Method

PNAS searches CNN cell structures from simple to complex, using measured performance to train a surrogate that selects promising deeper candidates. Its predictor is designed to handle variable-length cells and rank candidates when training data are scarce.

4.1 Progressive Neural Architecture Search

  • Rather than navigate the exponentially large full-structure space directly, PNAS evaluates one-block cells, expands candidates, and uses a learned predictor to select top-K cells at each depth.Expanding the 256 one-block structures by 576 possible second blocks yields 147,456 candidate cells, too many to train and evaluate exhaustively.Source: §4.1 · p. 6

4.2 Performance Prediction with Surrogate Model

  • The surrogate must handle variable-length cells, predict performance for b+1 blocks from data through b blocks, rank candidates roughly by true performance, and learn from scarce evaluations.Source: §4.2 · pp. 6–7
  • The authors compare an LSTM over block-token sequences with an MLP that averages embedded block vectors; both regress validation accuracy and use L1 loss.Source: §4.2 · p. 7
  • At each search step, an ensemble of five predictors is fit from scratch on four-fifths of the available data, which empirically reduces prediction variance.Source: §4.2 · p. 7
  • The authors plan to investigate Gaussian processes with string kernels as potentially more sample-efficient predictors that provide uncertainty estimates.Source: §4.2 · p. 7

5 Experiments and Results

PNAS uses surrogate-guided progressive search to reduce search cost, while its discovered models match or surpass comparison methods on CIFAR-10 and ImageNet. Predictor accuracy is weaker when extrapolating to larger cells, but improves with more and larger training cells.

5.1 Experimental Details

  • 5000 CIFAR-10 training images formed the validation set, and search evaluated up to K = 256 networks per stage, with each child trained for 20 epochs.The experiments also used whitened images, random horizontal flips, and 32 × 32 crops from images upsampled to 40 × 40; cells had at most B = 5 blocks.Source: §5.1 · p. 8

5.2 Performance of the Surrogate Predictors

  • RNNs predicted seen-cell performance better, while MLPs did slightly better on unseen larger cells, and ensembling helped extrapolation; prediction of larger cells improved with more and larger training cells.Predictors performed well on training-set models but less well on larger models.Source: Table 1 · p. 9, §5.2 · pp. 9–10

5.3 Search Efficiency

  • 8 times faster in total cost and up to 5 times fewer evaluated models made PNAS more efficient than NAS under the same search space.The total-cost comparison includes NAS's reranking stage, whereas PNAS selects its top model directly from the search.Source: Table 2 · p. 11, §5.3 · pp. 11–12

5.4 Results on CIFAR-10 Image Classification

  • 21 times less compute let PNAS find a CIFAR-10 model with the same accuracy as NAS; it also used 36 times less compute than Hierarchical EA and 63 times fewer resources than AmoebaNets.The cross-method comparisons use different search spaces, so the paper cautions that they are not directly comparable.Source: Table 3 · p. 12, §5.4 · p. 13

5.5 Results on ImageNet Image Classification

  • 82.9% top-1 and 96.2% top-5 accuracy in the Large ImageNet setting exceeded prior methods; in Mobile, PNASNet-5 reached 74.2% top-1 versus NASNet-A’s 74.0%.Both methods also surpassed MobileNet (70.6%) and ShuffleNet (70.9%) in the Mobile setting.Source: §5.5 · p. 13, Table 5 · p. 14, Table 4 · p. 14

6 Discussion and Future Work

Progressive search through increasingly complex CNN graphs, guided by a learned prediction function, finds models matching previous work’s performance at a fraction of the computational cost.

6 Discussion and Future Work

  • The resulting models match previous work’s performance at a fraction of the computational cost.The method combines progressive search through increasingly complex graphs with a learned prediction function to identify promising models.Source: §6 · p. 14
  • Future work could improve prediction and training, change candidate selection, or explore speed-accuracy tradeoffs.Suggestions include Gaussian processes with string kernels, model-based early stopping, warm-starting larger models from smaller parents, Bayesian optimization using expected improvement or upper confidence bound, adaptively varying K, and speed-accuracy tradeoffs.Source: §6 · p. 14

A Search Efficiency of PNAS with RNN-ensemble

With an RNN-ensemble surrogate, PNAS is more efficient than NAS under the same search space. Its advantage is larger by examples than by models trained and evaluated because NAS adds a reranking stage.

A Search Efficiency of PNAS with RNN-ensemble

B Searching Cells with More Blocks

PNAS found good-performing models for cells with more than five blocks, while progressively exploring search spaces that grow to about 10^33 configurations at ten blocks.

B Searching Cells with More Blocks

  • The comparison NAS curve is for B = 5; the passage suggests its performance would likely decline if searching cells with more blocks.This is a prediction attributed to the growth in search space, not a reported result for NAS at B > 5.Source: B Searching Cells with More Blocks · p. 18
  • PNAS found good-performing models for B > 5, reaching an unconstrained search space of about 10^33 configurations at B = 10.The search space grows by about four orders of magnitude at each block level.Source: Figure 2 · p. 18, B Searching Cells with More Blocks · p. 18

C Intermediate Level PNASNet Models

Progressive Neural Architecture Search grows cells from simple to complex, and PNASNet models from intermediate levels remain competitive on CIFAR-10 as test error decreases with each level.

C Intermediate Level PNASNet Models

D Transferring from CIFAR-10 to ImageNet

CIFAR-10 accuracy is strongly correlated with ImageNet accuracy, supporting CIFAR-10 as a fast proxy for selecting architectures. The relationship is weaker among architectures with higher CIFAR-10 scores.

D Transferring from CIFAR-10 to ImageNet

  • CIFAR-10 accuracy, even after only 20 training epochs, strongly correlates with ImageNet accuracy and is a reasonable fast proxy for searching models.Source: D Transferring from CIFAR-10 to ImageNet · p. 20
  • 0.727 top-1 rank correlation links CIFAR-10 and ImageNet performance across architectures, suggesting that the best CIFAR-10 architecture transfers to other datasets.The reported rank correlation is for the full set of points in Figure 4.Source: Figure 4 · p. 20
  • 0.505 top-1 and 0.460 top-5 rank correlations apply to the higher-value points with CIFAR-10 scores above 0.89.Source: Figure 4 · p. 20

Related papers

Loading 1712.00559v3…