PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Architecture Search with Reinforcement Learning

Barret Zoph, Quoc V. Le

arXiv:1611.01578v2cs.LGcs.AIcs.NE

TL;DR

Designing neural-network architectures still requires substantial expert knowledge and time, so this paper introduces Neural Architecture Search, using a recurrent controller to generate architectures. The method finds strong models from scratch across image and language tasks, including a CIFAR-10 model with 3.65% test error and a recurrent cell that outperforms prior RNN and LSTM architectures.

  • Problem

    Designing neural-network architectures remains time-consuming and dependent on substantial expert knowledge despite advances in deep learning.

  • Method

    Neural Architecture Search uses a recurrent controller to generate neural-network architectural hyperparameters and search variable-length architecture spaces.

  • Results

    The method designs strong models from scratch across CIFAR-10 and Penn Treebank, including a CIFAR-10 model achieving 3.65% test error.

  • Takeaways & Limitations

    Neural Architecture Search establishes a research direction for automatically finding effective neural-network architectures.

  • Takeaways & Limitations

    The described search space is restrictive because it excludes learning-rate prediction and initially assumes architectures contain only convolutional layers.

Abstract

from arXiv · show

Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.

1 INTRODUCTION

The paper introduces Neural Architecture Search, which uses a recurrent controller to generate neural-network architectures and reinforcement learning to improve them from validation accuracy. Experiments show it can design models from scratch, including a competitive CIFAR-10 ConvNet.

  • Motivation: Deep learning success has shifted emphasis from feature designing toward architecture designing.The introduction contrasts hand-designed features such as SIFT and HOG with architectures such as AlexNet and VGG-style models.
  • Method: Neural Architecture Search uses a recurrent controller to generate variable-length strings specifying neural-network structure and connectivity.The generated architecture defines a child network that is trained on real data.
  • Method: Validation accuracy serves as the reward signal for policy-gradient updates that make the controller more likely to generate better architectures.The controller is updated after training the child network and measuring its validation-set accuracy.
  • Results: Neural Architecture Search can design good models from scratch, including a novel ConvNet that is better than most human-invented architectures on CIFAR-10.The introduction presents this capability as an achievement considered not possible with other methods.
  • Results: 3.65 test set error is achieved by the CIFAR-10 model, which is also 1.05x faster than the current best model.Both figures refer to the reported CIFAR-10 model comparison.

2 RELATED WORK

The paper situates Neural Architecture Search among hyperparameter optimization, neuro-evolution, program synthesis, sequence-to-sequence learning, and meta-learning. It distinguishes these approaches by search-space flexibility, scalability, autoregressive prediction, non-differentiable optimization, and learning across tasks.

  • Hyperparameter optimization: Hyperparameter optimization methods search fixed-length model spaces, limiting their ability to generate variable-length configurations specifying network structure and connectivity.The passage identifies hyperparameter optimization as important and widely used in machine learning.
  • Neuro-evolution: Modern neuro-evolution algorithms compose novel models more flexibly but are less practical at large scale because search-based methods are slow or require many heuristics.
  • Program synthesis: Neural Architecture Search parallels program synthesis and inductive programming, which search for programs from examples, including probabilistic program induction for several learning tasks.Examples include simple question answering, sorting numbers, and learning with very few examples.
  • Sequence-to-sequence learning: The controller predicts hyperparameters autoregressively, conditioned on previous predictions, borrowing this structure from sequence-to-sequence decoders while optimizing the child network’s non-differentiable accuracy.The passage relates this optimization to BLEU optimization in neural machine translation.
  • Meta-learning: The work is also related to meta-learning, including neural networks learning gradient-descent updates or reinforcement-learning policies for another network.Meta-learning uses information learned in one task to improve a future task.

3 METHODS

The method uses a recurrent controller trained with policy-gradient reinforcement learning to generate neural architectures, whose validation accuracy provides the reward. It expands the search with parallel training, skip connections, and recurrent-cell generation.

  • Controller and reinforcement learning: A recurrent controller generates architectural hyperparameters as token sequences, with each prediction produced by a softmax and fed into the next time step.The generated architecture is trained to convergence, and its held-out validation accuracy is recorded as the reward.
  • Controller and reinforcement learning: Policy gradients optimize the controller because validation accuracy is non-differentiable, using REINFORCE with an exponential-moving-average baseline to reduce variance.The baseline remains unbiased when it does not depend on the current action.
  • Accelerate Training with Parallelism and Asynchronous Updates: Distributed training uses S parameter-server shards and K controller replicas, each sampling m architectures and training child models in parallel before returning gradients.Asynchronous parameter updates accelerate controller learning because training each child network to convergence can take hours.
  • Skip connections: Set-selection attention lets the controller propose skip connections and branching layers by sampling content-based sigmoid decisions over previous layers.Because these connections are probability distributions, REINFORCE applies without significant modifications.
  • Generating recurrent cells: For recurrent-cell search, the controller constructs a computation tree that combines x_t and h_t−1 through selected operations and activation functions to produce h_t.The controller also predicts connections involving c_t−1 and c_t memory-state variables.

4 EXPERIMENTS AND RESULTS

Neural Architecture Search was evaluated on CIFAR-10 for convolutional architectures and Penn Treebank for recurrent cells, using validation-based rewards to select models. The search produced competitive CIFAR-10 networks, improved Penn Treebank perplexity, and transferred to character language modeling and GNMT settings.

  • Experimental setup: Experiments targeted CIFAR-10 convolutional architectures and Penn Treebank recurrent cells, with separate held-out validation sets providing reward signals.Test performance was computed only once for the network achieving the best validation performance.
  • CIFAR-10 results: 5.50% error rate was achieved by a 15-layer CIFAR-10 architecture when the controller predicted neither strides nor pooling.The architecture was described as shallow and relatively inexpensive among top-performing networks.
  • CIFAR-10 results: 6.01% error rate was achieved by a 20-layer CIFAR-10 architecture when the controller also predicted strides.This larger search space produced performance not much worse than the fixed-stride experiment.
  • CIFAR-10 results: 4.47% was achieved by a 39-layer CIFAR-10 network with two pooling layers, approaching the 3.74% achieved by the best human-invented architecture.The search space was constrained by predicting 13 fully connected blocks, each representing 3 layers.
  • Penn Treebank results: Neural Architecture Search models outperformed other state-of-the-art Penn Treebank models, with one model gaining almost 3.6 perplexity.The experiments trained two-layer child models while adjusting hidden units to approximately match medium-baseline parameter counts.
  • Transfer learning results: 0.5 test set BLEU improvement was obtained over the default LSTM cell when the discovered cell replaced it at matched computational complexity.The new cell was used without tuning the existing GNMT framework settings beyond hyperparameter adjustment for equal complexity.

5 CONCLUSION

The paper introduces Neural Architecture Search, using a recurrent neural network controller to compose architectures across variable-length search spaces. The method shows strong empirical performance on challenging benchmarks and opens a research direction for automatically finding effective neural network architectures.

  • 5 CONCLUSION: Neural Architecture Search uses a recurrent neural network as a controller to compose neural network architectures.The controller makes the method flexible for architecture generation.
  • 5 CONCLUSION: The recurrent controller enables search over variable-length architecture spaces.
  • 5 CONCLUSION: The method achieves strong empirical performance on challenging benchmarks and motivates automatic discovery of effective neural network architectures.The paper presents this approach as a new research direction.

A APPENDIX

The appendix illustrates a convolutional architecture discovered without strides or pooling and compares the original LSTM cell with two cells found by the model under different search spaces.

  • Convolutional architecture: The discovered convolutional architecture uses concatenation across input layers rather than residual connections.Figure 7 defines FH as filter height, FW as filter width, and N as the number of filters.
  • Recurrent cells: The model found two strong recurrent cells, one without max and sin and another with max and sin available, though the latter omitted sin.Figure 8 contrasts both discovered cells with the original LSTM cell.
Loading 1611.01578v2…