PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Transferable Architectures for Scalable Image Recognition

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le

arXiv:1707.07012v4cs.CVcs.LGstat.ML

TL;DR

Neural architecture search is costly on large datasets, so the paper learns transferable convolutional cells on CIFAR-10 and scales them to ImageNet and other tasks. NASNet reaches state-of-the-art performance across classification and object detection while using fewer computational resources than human-designed architectures.

  • Problem

    Searching directly for architectures on large image datasets is computationally expensive, motivating methods that learn transferable architectures from smaller proxy datasets.

  • Method

    The paper designs a search space that decouples architectural complexity from network depth, enabling convolutional cells learned on CIFAR-10 to transfer across datasets and computational scales.

  • Results

    NASNet matches or exceeds state-of-the-art across CIFAR-10, ImageNet, and COCO object detection, including 82.7% top-1 ImageNet accuracy and 43.1% COCO mAP.

  • Takeaways & Limitations

    Learned convolutional cells can be scaled across computational budgets and transferred to multiple vision tasks while outperforming human-designed architectures.

  • Takeaways & Limitations

    Object-detection gains may also reflect a novel loss and ensembling across model instances and image crops, rather than NASNet features alone.

Abstract

from arXiv · show

Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet, NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.

1. Introduction

The paper introduces a scalable paradigm for optimizing convolutional architectures on a target dataset by searching for transferable architectures on smaller datasets. NASNet achieves state-of-the-art results on CIFAR-10, ImageNet, and COCO while offering strong accuracy across computational budgets.

  • Transferable architecture search: NASNet transfers the best architecture found on CIFAR-10 to ImageNet with little modification, achieving 82.7% top-1 and 96.2% top-5 accuracy.The transferred architecture achieves state-of-the-art accuracy among published works on ImageNet.
  • CIFAR-10 results: 2.4% error rate on CIFAR-10 makes NASNet state-of-the-art on the dataset used for architecture search.
  • ImageNet efficiency: 1.2% improvement in top-1 accuracy over the best human-invented architectures requires 9 billion fewer FLOPS.
  • Accuracy across computational budgets: 74.0% top-1 accuracy on ImageNet is achieved by the smallest NASNet, 3.1% better than equivalently sized state-of-the-art mobile models.Varying the number of convolutional cells and filters produces models with different computational demands.
  • Transfer to object detection: 43.1% mAP on COCO is achieved by the largest NASNet with Faster-RCNN features, 4% better than previous state-of-the-art.The learned ImageNet features transfer to other computer vision problems, including object detection.

2. Related Work

The method connects neural architecture search to hyperparameter optimization, evolutionary algorithms, and meta-learning, while drawing its search-space and cell design from prior recurrent and convolutional architectures. Earlier approaches often struggled to scale to ImageNet, motivating the paper’s focus on scalable transfer.

  • Architecture search: The method relates to hyperparameter optimization and architecture-design approaches including Neural Fabrics, DiffRNN, MetaQNN, and DeepArchitect.Evolutionary algorithms offer a more flexible architecture-design class but had not achieved comparable success at large scale.
  • Meta-learning: Most meta-learning approaches had not scaled to large problems like ImageNet, although learned optimizers for ImageNet classification had achieved notable improvements.The passage frames scalability as a key limitation in related learning-to-learn methods.
  • Search-space and cell design: The search space drew inspiration from LSTMs and Neural Architecture Search Cell, while the modular convolutional cell relates to VGG, Inception, ResNet/ResNext, and Xception/MobileNet.These precedents span recurrent search-space ideas and modular convolutional architectures used on ImageNet.

3. Method

The method uses NAS reinforcement learning to search convolutional cells, with a transferable NASNet search space designed to scale architectures from CIFAR-10 to larger image datasets. Architectures are built from repeated Normal and Reduction Cells whose structures are recursively predicted by a controller RNN.

  • Neural Architecture Search: NAS trains child networks sampled by a controller RNN, then uses held-out validation accuracy to update the controller toward better architectures.The controller predicts an architecture from a search space, and the trained child network’s accuracy provides the reward signal.
  • NASNet Search Space: The NASNet search space is designed so architectures found on CIFAR-10 can scale to larger, higher-resolution datasets across computational settings.The paper names this transferable space NASNet because it produces the best architecture found in the experiments.
  • Cell-Based Architecture: Scalable networks repeat convolutional cells with shared architecture but different weights, using Normal Cells for unchanged resolution and Reduction Cells for halving spatial dimensions.Reduction Cells use stride two initially; ImageNet uses more Reduction Cells than CIFAR-10 because its input images are larger.
  • Cell Search Procedure: For each cell, the controller RNN recursively selects two hidden-state inputs, two operations, and a combination method across B blocks.Each newly created hidden state becomes available to subsequent blocks, and experiments use B = 5.
  • Cell Search Procedure: The controller makes 2 × 5B predictions: the first 5B construct the Normal Cell and the second 5B construct the Reduction Cell.The two cell types therefore receive separate searched architectures rather than sharing one architecture.
  • Search Optimization: The search uses NAS reinforcement learning, while uniform random search is an alternative that performs slightly worse on CIFAR-10.Cell outputs combine selected hidden states by element-wise addition or filter-dimension concatenation, with unused states concatenated into the final output.

4. Experiments and Results

Experiments show that convolutional cells learned on CIFAR-10 transfer across ImageNet model scales, achieving strong classification and detection results. ScheduledDropPath and reinforcement-learning search further support NASNet performance and architecture quality.

  • Search and architecture construction: 500 GPUs searched convolutional cells on CIFAR-10, producing NASNet-A, NASNet-B, and NASNet-C from the three best searches.The controller RNN used PPO and a global workqueue to generate child networks.
  • Regularization: ScheduledDropPath, which increases path-drop probability during training, was effective regularization for NASNet.It modifies DropPath by scheduling the probability rather than keeping it fixed.
  • CIFAR-10 classification: 2.40% error rate was achieved by large NASNet-A with cutout on CIFAR-10, improving the previous best record of 2.56%.The best single run achieved 2.19% error rate.
  • ImageNet classification: 82.7% single-model ImageNet accuracy established a new state of the art, surpassing the previous best published result by ∼1.2%.Cells learned on CIFAR-10 exceeded the predictive performance of corresponding hand-designed models.
  • ImageNet classification: 74.0% accuracy was achieved at ∼550M multiply-add operations, while learned cells delivered state-of-the-art performance across almost 2 orders of magnitude in computational budget.The result used 224x224 images and comparable computational demand to previous models.
  • Object detection: 43.1% mAP was achieved by the best single NASNet model for object detection after increasing input resolution, exceeding the previous best by over 4.0%.At 800 × 800 resolution, the system achieved mAP = 40.7%, while the mobile-optimized network achieved a mAP of 29.6%.

5. Conclusion … A.1. Dataset for Architecture Search

The paper learns transferable convolutional cells in a search space that separates architectural complexity from network depth, enabling scalable models across datasets and computational budgets. NASNet architectures approach or exceed state-of-the-art performance with lower computational demand, while architecture search uses an augmented CIFAR-10 dataset and a 5,000-image validation subset.

  • 5. Conclusion: The learned convolutional cells transfer across multiple image-classification tasks and can scale in computational cost and parameters.The resulting models exceed human-designed models across mobile and computationally intensive settings.
  • 5. Conclusion: The NASNet search space decouples architecture complexity from network depth, allowing architectures found on CIFAR-10 to transfer across data and computational scales.This design identifies candidate architectures on a small dataset before applying them to broader image-classification settings.
  • 5. Conclusion: NASNet architectures approach or exceed state-of-the-art performance on CIFAR-10 and ImageNet while requiring less computation than human-designed architectures.The ImageNet results matter because many vision systems derive image features or architectures from ImageNet classification models.
  • A.1. Dataset for Architecture Search: A random subset of 5,000 training images is held out as a validation set for the controller RNN.The remaining training images support the architecture-search dataset setup described for CIFAR-10.
  • A.1. Dataset for Architecture Search: All CIFAR-10 images are whitened before augmentation with random crops from upsampled 40x40 images and random horizontal flips.The crops are 32x32 patches, and the augmentation procedure is common among related work.

A.2. Controller architecture · A.3. Training of the Controller · A.4. Details of architecture search space

The architecture search uses an LSTM controller trained with PPO to sample convolutional cells, distributed child-network evaluation, and a predefined selection pipeline. The search space combines flexible convolutional-cell choices with specific nonlinearities, shape matching, ordering, and separable-convolution rules.

  • A.2. Controller architecture: The controller is a one-layer LSTM with 100 hidden units, producing 2 × 5B softmax predictions for two convolutional cells per architecture decision.Each of the 10B predictions has a probability, and the child network’s joint probability is their product for controller-gradient computation.
  • A.2. Controller architecture: PPO replaces REINFORCE with learning rate 0.00035, alongside a 0.00001 entropy penalty, a 0.95 exponential-moving-average reward baseline, and uniform initialization from -0.1 to 0.1.The stated motivation for PPO is faster and more stable controller training, while the entropy penalty encourages exploration.
  • A.3. Training of the Controller: A global workqueue distributes controller-generated samples to 450 child workers, which train sampled networks and report held-out validation accuracy back to the controller.Workers request new work when free, and the process continues until the predetermined architecture-sampling limit is reached.
  • A.3. Training of the Controller: 20,000 child models define the search limit; the controller updates in minibatches of 20 architectures, after which the top 250 are trained to convergence on CIFAR-10.The final CIFAR-10 training stage determines the very best architecture among the selected candidates.
  • A.4. Details of architecture search space: The search space was developed through preliminary small-scale experiments intended to identify flexible, expressive architectures and procedures for large-scale search.These experiments focused on determining how to run the subsequent architecture search effectively.
  • A.4. Details of architecture search space: All convolutions use ReLU, while 1x1 convolutions are inserted as necessary to ensure matching shapes within convolutional cells.Experiments with ELU showed minimal benefit.
  • A.4. Details of architecture search space: Depthwise separable convolutions omit Batch Normalization and ReLU between depthwise and pointwise operations, unlike the cited prior design.This rule applies throughout the architecture search space.
  • A.4. Details of architecture search space: Convolutions follow ReLU, convolution operation, and Batch Normalization; selected separable convolutions are applied twice to hidden states to improve overall performance.The twice-applied separable-convolution rule was found empirically to improve performance.

A.5. Training with ScheduledDropPath

Naive dropout across convolutional filters degraded NASNet performance, whereas ScheduledDropPath, a modified DropPath technique, effectively regularized NASNets. DropPath stochastically removes individual paths within a cell at a fixed probability.

  • Regularization method: ScheduledDropPath is a modified version of DropPath that works well for regularizing NASNets.The authors discovered this technique while comparing several stochastic regularization methods.
  • Regularization method: Naive dropout applied across convolutional filters degraded performance.
  • Regularization mechanism: DropPath stochastically drops each path, or edge, in a cell with a fixed probability.The paper describes ScheduledDropPath as a modified form of this procedure.

A.6. Training of CIFAR models

CIFAR models use cosine learning-rate decay, momentum optimization, and L2 weight decay. During architecture search, each model is trained for 20 fixed epochs, with cosine decay helping differentiate architectures.

  • 20 fixed epochs are used to train each CIFAR-10 model during architecture search.
  • Cosine learning-rate decay is applied throughout CIFAR training and helps further differentiate good architectures.
  • Momentum optimization uses a 0.9 momentum rate, and all models apply L2 weight decay.

A.7. Training of ImageNet models

ImageNet models are trained on the 1.2M-image, 1,000-class ILSVRC 2012 dataset using distributed synchronous SGD, established augmentation and decay procedures, and regularization components including an auxiliary classifier and dropout.

  • Training setup: ImageNet models use 299x299 or 331x331 inputs, established data augmentation, and distributed synchronous SGD with 50 workers.The dataset contains ∼1.2M images labeled across 1,000 classes.
  • Regularization and optimization: An auxiliary classifier placed two-thirds up the network contributes loss weighted by 0.4.The network is empirically insensitive to the auxiliary classifier’s parameter count and loss weight.
  • Regularization and optimization: All models use L2 regularization, exponential learning-rate decay, and dropout with probability 0.5 on the final softmax matrix.The decay scheme follows the procedure in [60].

B. Additional Experiments

The paper presents two additional cells, NASNet-B and NASNet-C, that performed well on CIFAR and ImageNet. Their search spaces modify how hidden states are passed between layers and which normalization operations follow addition.

  • NASNet-B: NASNet-B uses B = 4 and feeds all hidden states created within the convolutional cell into the next layer, rather than concatenating only unused states.The architecture takes four hidden states as input to the cell, and each output hidden state is reused in future layers.
  • Additional cells: Both NASNet-B and NASNet-C are additional cells that performed well on CIFAR and ImageNet.Their search spaces are slightly different from the search space used for NASNet-A.
  • NASNet-C: NASNet-C concatenates unused hidden states as in NASNet-A while allowing addition followed by layer normalization or instance normalization as in NASNet-B.This combines the hidden-state handling of NASNet-A with the normalization options introduced for NASNet-B.

C. Example object detection results

NASNet-A featurization works well with Faster-RCNN on the COCO dataset, producing accurate object localization. Example detections show improvements over a previous state-of-the-art Inception-ResNet-v2-based model.

  • NASNet-A featurization works well with Faster-RCNN on COCO and gives accurate localization of objects.
  • Example detections show improvements over the previous state-of-the-art Faster-RCNN model using Inception-ResNet-v2 featurization.
  • The best-performing NASNet-A featurization is illustrated with Faster-RCNN trained on the COCO dataset.
Loading 1707.07012v4…