Efficient Neural Architecture Search via Parameter Sharing

Hieu Pham; Melody Y. Guan; Barret Zoph; Quoc V. Le; Jeff Dean

Source-linked AI summary

Efficient Neural Architecture Search via Parameter Sharing

Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean

arXiv:1802.03268v2Feb 9, 2018cs.LGcs.CLcs.CVcs.NEstat.ML

TL;DR

NAS has demonstrated strong architecture-design performance but is computationally expensive because child models are trained separately. ENAS searches subgraphs in a shared computational graph and shares child-model parameters, achieving strong CIFAR-10 and Penn Treebank results with more than 1000x fewer GPU-hours than NAS.

Problem
Conventional NAS is computationally expensive because each candidate child model is trained to convergence solely to measure its performance, after which its weights are discarded.
Method
ENAS represents architectures as subgraphs of a larger directed acyclic graph and shares parameters among child models while training shared weights and a controller.
Results
ENAS achieves 2.89% CIFAR-10 test error and 55.8 Penn Treebank test perplexity, while reducing NAS search cost by more than 1000x in GPU-hours.
Takeaways & Limitations
ENAS works across CIFAR-10 and Penn Treebank, showing that parameter sharing can support strong architecture-search performance at substantially lower computational cost.
Takeaways & Limitations
Penn Treebank experiments limit the ENAS cell to 24M parameters and do not use post-training techniques, extensive hyperparameter tuning, or selection among multiple trained architectures.

Abstract

from arXiv · show

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Thanks to parameter sharing between child models, ENAS is fast: it delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new state-of-the-art among all methods without post-training processing. On the CIFAR-10 dataset, ENAS designs novel architectures that achieve a test error of 2.89%, which is on par with NASNet (Zoph et al., 2018), whose test error is 2.65%.

1. Introduction

Neural architecture search can produce strong model designs, but conventional NAS is costly because it trains each sampled child model to convergence and discards its weights. ENAS addresses this bottleneck by sharing weights across child models while retaining strong results.

Motivation: Conventional NAS repeatedly samples child models, trains each to convergence, and uses validation performance to guide subsequent architecture choices.This iterative process makes architecture evaluation expensive because trained weights are discarded after measuring accuracy.
Motivation: NAS can require 32,400-43,200 GPU hours, while using fewer resources tends to produce less compelling results.Zoph et al. (2018) used 450 GPUs for 3-4 days.
Contribution: ENAS improves NAS efficiency by forcing all child models to share weights instead of training each architecture from scratch to convergence.The approach is motivated by evidence that learned parameters can transfer across models and tasks with little to no modification.
Results: ENAS achieves a 2.89% CIFAR-10 test error versus NAS’s 2.65% and a Penn Treebank test perplexity of 55.8 versus NAS’s 62.4.The Penn Treebank result is reported as a new state-of-the-art among approaches without post-training processing.
Results: Architecture search takes less than 16 hours on a single Nvidia GTX 1080Ti GPU, reducing GPU-hours by more than 1000x compared with NAS.The method is named Efficient Neural Architecture Search because of this efficiency.

2. Methods

ENAS represents neural architectures as subgraphs of a shared DAG, allowing child models to reuse parameters while a controller searches recurrent and convolutional spaces. Training alternates between optimizing shared model parameters with cross-entropy and optimizing controller decisions using validation rewards.

Core design: ENAS represents the NAS search space as a superposition DAG whose activated subgraphs define individual child architectures and share local computation parameters.Nodes represent local computations, while edges represent information flow; parameters are used when their computations are activated.
Recurrent cells: The controller RNN selects recurrent-cell edges, node operations, and topology, making ENAS more flexible than a fixed binary-tree search space.The recurrent-cell search space includes four activation functions and approximately 10^15 configurations when N = 12.
Recurrent cells: For recurrent cells, the controller samples node decisions and the cell output is the average of loose-end nodes that are never selected as inputs elsewhere.In the four-node example, nodes 3 and 4 are loose ends, so the output is (h3 + h4)/2.
Training: ENAS trains shared child-model parameters by minimizing expected sampled-model cross-entropy, using a Monte Carlo gradient estimate that can work with one sampled model.The controller policy is fixed during this phase, and the shared parameters are updated with stochastic gradient descent.
Training: The controller parameters are optimized separately with policy gradient to maximize expected validation reward, using REINFORCE with Adam and a moving-average baseline.The reward is validation perplexity-derived for language modeling and validation accuracy for image classification, encouraging selection based on validation performance.

3. Experiments

ENAS was evaluated on Penn Treebank and CIFAR-10, combining architecture-search experiments with ablations of its controller and search spaces. It achieved strong task performance while substantially reducing search cost relative to NAS.

Penn Treebank: ENAS found a Penn Treebank recurrent cell in about 10 hours on one Nvidia GTX 1080Ti GPU, achieving 55.8 test perplexity.This result was obtained without post-training processing and was comparable to the 56.0 perplexity of Mixture of Softmaxes.
Penn Treebank: More than 1000x faster in GPU hours than NAS, ENAS achieved 55.8 test perplexity versus NAS’s 62.4.The comparison is reported for the Penn Treebank search process.
CIFAR-10: 4.23% test error was achieved by ENAS’s macro-search network, improving on NAS’s 4.47% model.Increasing the highest-layer filters to 512 reduced the error to 3.87%, compared with 3.65% for NAS’s best model.
CIFAR-10: More than 50,000x fewer GPU-hours were required for ENAS to find the macro-search architecture than for NAS.The macro-search architecture took about 7 hours to discover.
CIFAR-10: 3.54% test error was achieved with ENAS-discovered convolution and reduction cells, compared with 3.41% for NASNet-A.With CutOut, ENAS reached 2.89% error versus 2.65% for NASNet-A.
Ablations: 1.7% higher test error resulted when ENAS’s separable convolutions were replaced with normal convolutions at matched parameter counts.Randomly changing several cell connections in the micro search space increased test error by 2.1%.
Ablations: 8.92% error resulted when the ENAS controller was not updated, while an ensemble of 250 Monte Carlo configurations reached only 5.49% test error.The ablation supports the importance of appropriately training the ENAS controller.

4. Related Work and Discussions

ENAS improves NAS efficiency through shared weights and applies flexibly across search spaces and domains. Its design differs from related approaches that use hypernetworks or other efficiency strategies.

Efficiency: ENAS is significantly more efficient than other NAS-efficiency methods in GPU hours.The comparison includes performance prediction, iterative search, and hierarchical architecture representations.
Design connections: ENAS shares weights between architectures, extending the idea of weight inheritance to a DAG-based search space.The DAG representation enables architectures to be selected as subgraphs while reusing parameters across child models.
Comparison with related methods: SMASH generates child-model weights with a hypernetwork, which inherently restricts those weights to a low-rank space.The authors suspect this restriction explains ENAS’s superior empirical performance to SMASH.
Scope: ENAS applies flexibly to RNN cells for text, entire-network macro search, and convolutional-cell micro search for images.These examples span multiple search spaces and disparate domains.

5. Conclusion

ENAS addresses NAS’s computational expense by sharing parameters across child models while searching subgraphs within a larger graph. The method works well on both CIFAR-10 and Penn Treebank.

Conclusion: ENAS speeds up NAS by more than 1000x in terms of GPU hours.The paper identifies parameter sharing across child models as the key contribution enabling this efficiency.
Conclusion: ENAS searches for an architecture as a subgraph within a larger graph that incorporates the architectures in the search space.This implements parameter sharing across child models during architecture search.
Conclusion: ENAS works well on both CIFAR-10 and Penn Treebank datasets.

A. Details on Penn Treebank Experiments

The Penn Treebank recurrent-cell search space represents cells as DAGs whose controller selects connections and activation functions. Shared parameters and stabilization choices support the resulting model training.

Recurrent-cell structure: Penn Treebank recurrent cells use N computational nodes represented as a DAG.Nodes represent local computations, while edges represent information flow between nodes.
Recurrent-cell structure: The first node receives the current RNN signal x(t) and the previous cell output h(t−1).
Controller decisions: For later nodes, the controller selects each node’s incoming connection and activation function.Each node hℓ receives input from a preceding node jℓ and applies a controller-chosen activation fℓ.
Parameter sharing: Shared recurrent-cell parameters include transformation matrices, word embeddings, and untied softmax weights.These parameters are reused across different recurrent child models.
Stabilization: Batch normalization is added after averaging cell layers during architecture search to stabilize updates, but can be removed for a fixed sampled cell without performance loss.

B. Details on CIFAR-10 Experiments

The CIFAR-10 experiments use architectural and training techniques designed to stabilize ENAS’s shared-parameter updates. These include convolutional ordering, repeated separable convolutions, skip-connection handling, and global average pooling.

Convolutional design: CIFAR-10 convolutions use relu-conv-batchnorm ordering, while depthwise separable convolutions are applied twice in the micro search space.
Skip connections: Skip connections from multiple layers are concatenated and projected with a 1 × 1 convolution, batch normalization, and ReLU.This keeps the number of output channels consistent across architectures.
Pooling: Global average pooling reduces parameters in the dense connection to the Softmax layer to avoid overfitting.The activations of each channel are averaged after the final convolutional layer.
Training stability: Without the final two stabilization tricks, ENAS training is very unstable because shared-parameter gradient updates have high variance.The cited tricks are global average pooling and stabilization of stochastic skip connections.