Source-linked AI summary
Progressive Neural Architecture Search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy
TL;DR
Neural architecture search needs efficient ways to explore large CNN structure spaces. PNAS progressively searches increasingly complex cell structures while learning a surrogate predictor, achieving state-of-the-art classification accuracy with fewer evaluations and less compute.
Problem
Directly navigating exponentially large spaces of fully specified architectures is difficult, especially before the search has learned what makes a good model.
Method
PNAS uses sequential model-based optimization to expand cell structures from simple to complex and uses a learned predictor to select promising candidates for evaluation.
Results
The discovered structures achieve state-of-the-art classification accuracies on CIFAR-10 and ImageNet while using 5 times fewer model evaluations and 8 times less total compute than the compared RL method.
Takeaways & Limitations
Progressive search with learned prediction matches prior performance while reducing the computational cost of CNN architecture search.
Takeaways & Limitations
The paper identifies future improvements including better surrogate predictors, early stopping, warm starts, Bayesian optimization, and adaptive evaluation budgets.
Abstract
from arXiv · showhide
We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.
1 Introduction
PNAS searches CNN cell structures progressively from simple to complex while learning a surrogate model to prune unpromising candidates. It matches prior state-of-the-art accuracy with substantially fewer evaluations and lower compute.
- 1 Introduction: 5 times fewer model evaluations are required during architecture search while matching previous state-of-the-art accuracy.This comparison is reported for the CNN learned by the proposed method.
- 1 Introduction: The search uses a structured cell space rather than directly searching complete CNNs.Cells are stacked to construct CNNs, enabling architecture transfer across datasets.
- 1 Introduction: PNAS searches convolutional cells from simple structures toward increasingly complex ones, using a learned predictor to guide candidate selection.The method expands candidate cells by one block, ranks children with the surrogate, and evaluates only the top candidates.
- 1 Introduction: 8 times faster total compute is achieved than the RL method of under the reported comparison.The paper attributes the efficiency to progressive search through increasingly complex structures and learned prediction.
2 Related Work
Prior neural architecture search largely uses evolutionary algorithms or reinforcement learning to explore fully specified structures. Related work also studies progressive search and surrogate prediction, but these approaches have important efficiency or applicability limitations.
- 2 Related Work: Reinforcement-learning controllers generate action sequences specifying CNN structures and receive validation accuracy as the reward.NAS later used PPO and a structured cell-based search space.
- 2 Related Work: Evolutionary algorithms mutate and recombine encoded network structures, training and evaluating models whose top performers generate children.Recent evolutionary methods search structures while using SGD to estimate parameters.
- 2 Related Work: RL and EA are described as local searches through fully specified graph structures, whereas PNAS searches progressively from simple to complex.This distinction motivates the paper’s heuristic search strategy.
- 2 Related Work: MCTS-based progressive search used random branch selection at each node, which the paper characterizes as inefficient.PNAS instead uses a learned predictor to focus expansion.
- 2 Related Work: Prior surrogate methods generally target fixed-sized structures, limiting their direct use with progressive search.Other predictors operate without training or extrapolate learning curves after limited training.
3 Architecture Search Space
The search space represents CNNs through reusable convolutional cells composed of DAG blocks. Each block selects two inputs, applies operators, and combines the results, while cells are stacked into dataset-specific CNNs.
- Cell Topologies: A cell is a DAG of B blocks, with each block mapping two input tensors to one output tensor.The block is specified by input choices, two operators, and a combination operator.
- Cell Topologies: Each block selects two inputs from earlier cell blocks or the previous cell, applies two operators, and combines them by addition.The operator set includes convolutions, pooling, identity, and dilated convolution.
- Cell Topologies: The operator space contains 8 functions, including depthwise-separable convolutions, pooling, identity, and dilated convolution.This is smaller than the 13-operator space used in.
- Search-space size: 5.6 × 10^14 possible cell structures arise for cells with up to 5 blocks before symmetry pruning.The number of unique cells is approximately 10^12 after pruning, still making exhaustive search impractical.
- CNN construction: PNAS learns one cell type and emulates reduction cells using the same cell with stride 2.The resulting CNN construction otherwise follows the strategy in.
- CNN construction: Cells are stacked with stride 1 or stride 2 to form CNNs, with repeats adjusted for the dataset and desired model setting.The construction uses global average pooling and a softmax classification layer.
4 Method
PNAS progressively expands cell structures, trains and evaluates a beam of candidates, and uses a learned predictor to rank larger children before evaluation. The predictor must support variable-sized cells, preserve performance ordering, and learn from scarce data.
- 4.1 Progressive Neural Architecture Search: Directly navigating exponentially large architecture spaces is difficult when the search begins without knowledge of good models.PNAS addresses this by ordering the search from simplest models first.
- 4.1 Progressive Neural Architecture Search: PNAS trains and evaluates all one-block cells, expands them into 147,456 two-block candidates, then uses a predictor to select the most promising children.The process repeats until cells reach the target maximum depth.
- 4.2 Performance Prediction with Surrogate Model: The predictor must handle variable-length cell descriptions, including cells with b + 1 blocks after training on cells up to b blocks.This requirement motivates sequence-based prediction with an LSTM.
- 4.1 Progressive Neural Architecture Search: At each depth, the beam retains K predicted top candidates for training and evaluation before updating the predictor.The beam size K is the number of models evaluated per search-tree level.
- 4.2 Performance Prediction with Surrogate Model: The predictor is intended to rank models in roughly true-performance order while using scarce training data efficiently.Low mean squared error is not required if ranking quality is adequate.
- 4.2 Performance Prediction with Surrogate Model: An ensemble of 5 predictors, each trained from scratch on 4/5 of the available data, empirically reduces prediction variance.The ensemble is used as new search data arrives.
- 4.2 Performance Prediction with Surrogate Model: Future work includes more sample-efficient predictors, model-based early stopping, warm starts, Bayesian optimization, and adaptive beam sizes.These directions identify extensions to the current search and prediction procedure.
5 Experiments and Results
Experiments evaluate surrogate predictors, search efficiency, computational cost, and transfer to ImageNet. PNAS progressively identifies competitive CNNs while substantially reducing search effort relative to NAS and evolutionary methods.
- Experimental setting: CIFAR-10 experiments use 50,000 training images, a 5,000-image validation split, and 20-epoch CNN evaluations.The search uses K = 256 networks per stage, maximum cell depth B = 5, F = 24 filters, and N = 2 cell unrolls.
- Surrogate predictors: The surrogate predicts training-set performance well but performs less well on larger unseen models; extrapolation improves with more and larger training cells.The comparison evaluates predictions on both previously seen cell sizes and cells one block larger than the training data.
- Surrogate predictors: The MLP ensemble is slightly better than the RNN ensemble for predicting unseen larger models, while the RNN performs better on the training set.The authors attribute the difference as likely due to overfitting and report that ensembling helps extrapolation.
- Search efficiency: 5 times fewer model evaluations are required by PNAS than NAS to reach equal mean performance under the same search space.For B = 5, PNAS reaches the final cell size after evaluating 1,160 models.
- Computational cost: 8 times faster total compute is achieved by PNAS than NAS, while PNAS matches NAS accuracy and uses 21 times less compute in the broader comparison.The broader comparisons also report 36 times less compute than Hierarchical EA and 63 times fewer resources than AmoebaNets, but the authors caution that search spaces differ.
- ImageNet transfer: 74.2% top-1 accuracy is achieved by PNASNet-5 in the ImageNet Mobile setting, versus 74.0% for NASNet-A.Under the Large setting, PNASNet-5 reaches 82.9% top-1 and 96.2% top-5 accuracy.
6 Discussion and Future Work
The paper accelerates CNN architecture search by progressively exploring increasingly complex graphs with a learned predictor. It reports comparable performance to prior work at a fraction of the computational cost and identifies several directions for further improvement.
- Future work includes better surrogate predictors, model-based early stopping, warm starting, Bayesian optimization, adaptive evaluation budgets, and automated speed-accuracy tradeoffs.
(Supplementary Material)
The supplementary material identifies the authors and their affiliations: Johns Hopkins University, Google AI, and Stanford University.
- The paper lists Chenxi Liu, Alan Yuille, and Kevin Murphy among its authors.
- The authors are affiliated with Johns Hopkins University, Google AI, and Stanford University.
A Search Efficiency of PNAS with RNN-ensemble
Supplementary experiments compare PNAS with an RNN-ensemble predictor against NAS and random search under the same search space. PNAS is reported as substantially more efficient, especially when measured by the number of training examples.
- The supplementary analysis focuses on the performance of an RNN-ensemble as the surrogate model.
- PNAS with an RNN-ensemble is compared with NAS and random search under the same search space.
- 5 times as efficient by the number of examples, PNAS with an RNN-ensemble outperforms NAS in the supplementary comparison.The passage also reports roughly twice the efficiency by number of models trained and evaluated.
B Searching Cells with More Blocks
The supplementary experiment extends progressive search to cells with up to 10 blocks. PNAS continues to find good-performing models in much larger search spaces, reaching about 10^33 configurations at 10 blocks.
- PNAS finds good-performing models while searching cells with more than 5 blocks, including levels up to B = 10.
- ∼10^33 possible model configurations are reached at B = 10 as the unconstrained search space expands by about 4 orders of magnitude per block level.
- Figure 2 visualizes progressive search with an MLP-ensemble from cells containing 1 block through cells containing 10 blocks.
C Intermediate Level PNASNet Models
The progressive search reaches PNASNet-5 at five blocks, while intermediate PNASNet models provide competitive performance with fewer blocks. Their cell structures are visualized and evaluated on CIFAR-10.
- B = 5 defines the final search level, whose best model is PNASNet-5.
- Figure 3 visualizes the cell structures used in PNASNet-{1, 2, 3, 4}.
- CIFAR-10 test performance is reported using top-1 error, parameter count, and search cost.
- Test error decreases from b = 1 to b = 5, while smaller PNASNets remain competitive.
D Transferring from CIFAR-10 to ImageNet
CIFAR-10 accuracy is strongly associated with ImageNet accuracy across architectures, supporting CIFAR-10 as a proxy for architecture search. The association is weaker among the highest-CIFAR-scoring models.
- CIFAR-10 accuracy remains strongly correlated with ImageNet accuracy even for models trained for only 20 epochs.
- 0.727 rank correlation links CIFAR-10 and ImageNet top-1 performance across architectures.
- 0.505 top-1 and 0.460 top-5 rank correlations apply to models with CIFAR-10 scores above 0.89.