PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Architecture Optimization

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, Tie-Yan Liu

arXiv:1808.07233v5cs.LGstat.ML

TL;DR

Existing architecture-search methods typically search a discrete space inefficiently. NAO embeds architectures continuously, predicts performance, optimizes embeddings with gradients, and decodes them into architectures, achieving competitive CIFAR-10 and PTB results with reduced computational resources.

  • Problem

    Reinforcement-learning and evolutionary architecture-search methods search discrete spaces, which is inefficient as the number of architectural choices grows.

  • Method

    NAO uses an encoder, performance predictor, and decoder to optimize neural-architecture embeddings in continuous space and decode improved embeddings into architectures.

  • Results

    NAO achieves competitive results across CIFAR-10 and PTB, including 1.93% CIFAR-10 test error and 56.0 PTB perplexity, while weight sharing enables searches under 10 GPU hours.

  • Takeaways & Limitations

    Continuous embedding optimization provides an effective and computationally efficient route to discovering architectures for image classification and language modeling.

Abstract

from arXiv · show

Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method to automatic neural architecture design based on continuous optimization. We call this new approach neural architecture optimization (NAO). There are three key components in our proposed approach: (1) An encoder embeds/maps neural network architectures into a continuous space. (2) A predictor takes the continuous representation of a network as input and predicts its accuracy. (3) A decoder maps a continuous representation of a network back to its architecture. The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy. Such a better embedding is then decoded to a network by the decoder. Experiments show that the architecture discovered by our method is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources. Specifically we obtain 1.93% test set error rate for CIFAR-10 image classification task and 56.0 test set perplexity of PTB language modeling task. Furthermore, combined with the recent proposed weight sharing mechanism, we discover powerful architecture on CIFAR-10 (with error rate 2.93%) and on PTB (with test set perplexity 56.6), with very limited computational resources (less than 10 GPU hours) for both tasks.

1 Introduction

NAO addresses the inefficiency of discrete architecture search by optimizing neural-network embeddings in a continuous space. Its encoder, predictor, and decoder support gradient-based discovery and competitive results across vision and language tasks.

  • Existing reinforcement-learning and evolutionary architecture-search methods operate in a discrete space that becomes inefficient as architectural choices grow.
  • NAO maps architectures into a continuous vector space and uses gradient-based optimization to seek embeddings of potentially better networks.
  • The method combines an encoder, a performance predictor, and a decoder to optimize architecture embeddings and reconstruct new neural-network architectures.
  • 1.93% test set error rate was achieved on CIFAR-10, while PTB language modeling reached 56.0 perplexity.

2 Related Work

Prior architecture-search approaches mainly use reinforcement learning, evolutionary algorithms, or Bayesian optimization. NAO instead optimizes architecture embeddings, while DARTS optimizes mixture weights and recovers decisions through argmax.

  • Reinforcement-learning methods treat architecture-component choices as actions, whereas evolutionary methods guide mutation and recombination.
  • DARTS and NAO both apply gradient-based continuous optimization, but DARTS optimizes mixture weights while NAO optimizes neural-architecture embeddings.
  • NAO’s framework maps an architecture to an embedding, improves that embedding through predictor-guided optimization, and decodes it into a new architecture.
  • DARTS selects architecture decisions by mixture-weight argmax, whereas NAO uses a decoder to recover the discrete architecture.
  • Bayesian optimization models architecture performance with Gaussian processes whose effectiveness depends on covariance functions representing architectural similarity.

3 Approach

NAO represents CNN and RNN architectures as token sequences, encodes them continuously, predicts their performance, and decodes optimized embeddings into architectures. Joint training and iterative seed-based optimization connect representation learning with architecture search.

  • 3.1 Architecture Space: CNN cells use B = 5 nodes, with each node choosing two previous inputs and operations from an 11-operator set.
  • 3.1 Architecture Space: RNN cells choose a previous node and activation function for each node, using B = 12 nodes in the experiments.
  • 3.1 Architecture Space: Architectures are serialized as fixed-length token sequences encoding node indices, operation types, and operation sizes.
  • 3.2 Components of Neural Architecture Optimization: The encoder maps architecture sequences to continuous representations, while the performance predictor maps those representations to predicted dev-set accuracy.
  • 3.2 Components of Neural Architecture Optimization: The decoder autoregressively reconstructs architecture tokens from an embedding and restricts token choices to valid architectures.
  • 3.3 Training and Inference: The encoder, predictor, and decoder are jointly trained with performance-prediction and structure-reconstruction losses, with λ controlling their trade-off.
  • 3.3 Training and Inference: Gradient ascent moves seed architectures toward embeddings with higher predicted performance, which are decoded into new architectures and iterated across rounds.

4 Experiments

Experiments evaluate NAO on image classification and language modeling, including predictor/decoder behavior, search efficiency, and transfer to additional datasets.

  • CIFAR-10 classification: NAO achieves 1.93% test error on CIFAR-10, while NAO-WS reaches 2.93% using 2.5M parameters within 7 hours.The comparison includes 1000 evaluated architectures for NAO and a smaller search cost than prior methods.
  • CIFAR-10 classification: NAO reduces CIFAR-10 error from 3.34 to 3.18 versus AmoebaNet while evaluating 1000 rather than 20000 architectures.The paper reports this comparison within a smaller search space of #op = 11.
  • NAO analysis: The predictor exceeds 78% pairwise accuracy with roughly 500 evaluated architectures, while the decoder’s average Hamming distance is below 0.5 tokens.The decoder comparison uses 60-token architecture descriptions.
  • NAO analysis: Architectures generated through NAO have gradually increasing performance across optimization iterations, and predicted and real performances remain closely aligned.Figure 2 compares mean real and predicted performance for candidate architectures over iterations l = 1, 2, 3.
  • Transfer evaluation: The CIFAR-10-discovered NAONet transfers to CIFAR-100 with 14.75 test error versus 15.20 for the previous cutout-based state of the art.The same architecture and training setting are used for CIFAR-100.
  • Transfer evaluation: NAONet obtains 25.7 top-1 test error on ImageNet, outperforming most previous works under the related-work training setting.The architecture transferred to ImageNet is the best architecture discovered on CIFAR-10.
  • PTB language modeling: On PTB, NAO achieves 56.0 perplexity, while NAO-WS achieves 56.6 perplexity through a 10-hour search.The results are compared with expert-designed and previous NAS methods.
  • WikiText-2 transfer: The PTB-discovered NAONet is on par with or surpasses ENAS and DARTS on the larger WikiText-2 dataset.The transferred architecture is evaluated on another language-modeling task.

5 Conclusion

NAO performs gradient-based neural architecture optimization in continuous space rather than searching discrete decisions, using an encoder, performance predictor, and decoder. The method achieves competitive results across image classification and language modeling.

  • NAO optimizes neural architectures in continuous space with gradient-based methods instead of searching discrete decisions.
  • Its encoder, performance predictor, and decoder together improve the effectiveness and efficiency of discovering architectures.
  • NAO achieves competitive results on both image classification and language modeling tasks.

7 Appendix

The appendix specifies the datasets, operator and activation search spaces, and the CNN and recurrent cells discovered by NAO. It also identifies the corresponding NAONet and PTB cell visualizations.

  • Datasets and training: CIFAR-10 uses 50k training images and 10k test images, with 5000 training images held out as a development set.
  • Datasets and training: PTB experiments follow ENAS setups with variational dropout, L2 regularization, tied embeddings and softmax weights, and scheduled learning-rate decay.
  • Search spaces: Without weight sharing, CNN searches use 11 operators, while weight-sharing searches use five operators including separable convolutions and pooling.
  • Search spaces: Recurrent-cell searches use tanh, ReLU, identity, and sigmoid activation functions.
  • Discovered architectures: NAO visualizes the best discovered CNN cells as NAONet normal and reduction cells, and the best discovered recurrent cell for PTB.
Loading 1808.07233v5…