Source-linked AI summary
Searching for A Robust Neural Architecture in Four GPU Hours
Xuanyi Dong, Yi Yang
TL;DR
NAS traditionally relies on reinforcement learning or evolutionary strategies that require extensive computational resources. GDAS instead uses differentiable, learnable sampling over a DAG-based neural-cell search space and optimizes it by gradient descent. It completes searches in several GPU hours and reports competitive CNN and RNN performance, while direct ImageNet search remains unsuccessful without further tuning.
Problem
RL- and EA-based NAS require validation accuracy from numerous architecture candidates, making architecture search computationally expensive.
Method
GDAS represents neural-cell architectures as DAG sub-graphs and uses differentiable, learnable sampling to optimize the architecture distribution by gradient descent.
Results
2.82% CIFAR-10 test error and 57.5 PTB test perplexity were achieved, with searches completed in several GPU hours and discovered networks transferred to ImageNet and WT2.
Takeaways & Limitations
GDAS reduces standard NAS search cost by about 10^4 times while producing CNN and RNN models with competitive performance.
Takeaways & Limitations
Direct ImageNet search failed to obtain good performance with the small-dataset hyper-parameters and requires further tuning.
Abstract
from arXiv · showhide
Conventional neural architecture search (NAS) approaches are based on reinforcement learning or evolutionary strategy, which take more than 3000 GPU hours to find a good model on CIFAR-10. We propose an efficient NAS approach learning to search by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG). This DAG contains billions of sub-graphs, each of which indicates a kind of neural architecture. To avoid traversing all the possibilities of the sub-graphs, we develop a differentiable sampler over the DAG. This sampler is learnable and optimized by the validation loss after training the sampled architecture. In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS). In experiments, we can finish one searching procedure in four GPU hours on CIFAR-10, and the discovered model obtains a test error of 2.82\% with only 2.5M parameters, which is on par with the state-of-the-art. Code is publicly available on GitHub: https://github.com/D-X-Y/NAS-Projects.
1. Introduction
GDAS addresses the computational cost of NAS by using differentiable, learnable sampling over a DAG-based cell search space. It searches efficiently by gradient descent and reports strong CIFAR-10 and PTB results.
- Motivation: NAS reduces human effort but remains difficult because the space of possible network architectures is effectively unbounded.Manual architecture design requires substantial effort and experimentation.
- Motivation: RL- and EA-based NAS methods require validation accuracy from numerous architecture candidates, making search computationally expensive.Their feedback is obtained through prolonged evaluation procedures.
- Approach: GDAS represents a neural cell search space as a DAG and samples one intermediate feature between each pair of nodes to form a sub-graph.Each node aggregates transformed intermediate features from previous nodes.
- Approach: GDAS makes architecture sampling differentiable and learnable, enabling end-to-end optimization by gradient descent with feedback at every iteration.The method samples a sub-graph rather than optimizing all DAG operations simultaneously.
- Results: 2.82% test error on CIFAR-10 and 57.5 test perplexity on PTB were achieved while completing a search in several GPU hours.The discovered CIFAR and PTB networks were also successfully transferred to ImageNet and WT2.
2. Related Work
Related NAS work includes macro search over whole networks and micro search over reusable cells. GDAS follows the micro-search paradigm while targeting substantially lower search cost.
- Search paradigms: NAS methods are commonly divided into macro search, which discovers entire networks, and micro search, which discovers reusable neural cells.Micro-search methods construct networks by stacking copies of discovered cells.
- Macro search: Macro-search methods often limit CNN depth because their network search spaces are too large to explore effectively.The cited approaches typically constrain models to fewer than 12 layers.
- Micro search: Micro-search methods such as NASNet commonly required more than 100 GPU days, while some reduced-cost methods still required more than one GPU day.GDAS is positioned as a micro-search method focused on reducing search cost.
- Efficiency: GDAS searches for a robust network within fewer GPU hours, reported as 1000× less than the standard NAS approach.The method is presented as a cost-reduction approach within the micro-search setting.
- Efficiency: Related efficiency techniques include progressive search stages, accuracy prediction, HyperNet, Net2Net transformation, and parameter sharing.These methods aim to reduce the computational resources required by NAS.
- Concurrent work: Xie et al. concurrently used a similar relaxation of discrete sampling, but focused on attention-based NAS loss inconsistency rather than search acceleration.The distinction concerns the primary objective of the sampling relaxation.
3. Methodology
GDAS searches neural cells within a DAG by sampling differentiable candidate operations and optimizing their distribution with alternating training and validation updates. The learned cells are then assembled into networks, with a fixed reduction cell used to simplify the search and improve results.
- Cell-based architecture search: GDAS searches a neural cell and stacks copies of the discovered cell to compose the complete network.For CNNs, cells transform outputs of previous cells; normal and reduction cells use stride 1 and 2, respectively.
- Search space as a DAG: Each computational node combines transformations of two earlier feature tensors in a DAG.The cell represents each node as the sum of two functions applied to outputs of previous nodes.
- Search space as a DAG: The candidate function set contains eight operations, including identity, zeroization, depth-wise separable convolutions, dilated convolutions, and pooling.The same candidate set is used for the cell search, with operation strides differing between normal and reduction cells.
- Differentiable model sampling: GDAS samples one transformation per edge from a learnable discrete distribution encoded by architecture parameters A.The distribution is parameterized by learnable vectors A_i,j, while each sampled operation uses its associated weights.
- Differentiable model sampling: The Gumbel-Max trick and softmax relaxation make discrete architecture sampling differentiable during backpropagation.GDAS uses arg max in the forward pass and softmax in the backward pass to permit gradient propagation.
- Training and architecture derivation: Architecture parameters are updated from validation loss while operation weights are updated from training loss through alternating optimization.This strategy avoids directly solving the nested NAS objective with high-order derivatives and is intended to improve generalization.
- Training and architecture derivation: A fixed reduction cell reduces search difficulty, and GDAS(FRC) completes CIFAR-10 searches in less than four hours.The final architecture retains the most important incoming connections according to learned operation probabilities.
4. Experimental Study
GDAS is evaluated across image-classification and language-modeling benchmarks, with experiments emphasizing search efficiency, transferability, and the performance of discovered architectures. The study also examines cell choices and notes that direct ImageNet search remains future work.
- Experimental setup: CIFAR-10 experiments split the 50K training images into separate 25K training and validation sets for cell search.The candidate function set contains eight functions, and the experiments use standard CNN search settings.
- CIFAR-10 results: 2.82% CIFAR-10 error is achieved with 2.5M parameters using GDAS (FRC), while GDAS reaches 2.93% error with 3.3M parameters.GDAS (FRC) fixes the reduction cell and searches only the normal cell.
- Search efficiency: Less than four hours on one V100 GPU is sufficient for GDAS (FRC), but reported search costs are not normalized across GPU devices.On a Titan 1080Ti, the reported GDAS (FRC) cost increases to about seven GPU hours.
- ImageNet transfer: ImageNet results show that the CIFAR-10-discovered cell transfers successfully and achieves competitive performance under a 600M multiply-add limit.GDAS (FRC) uses about 20% fewer multiply-add operations than DARTS while obtaining the same top-5 error; some methods achieve slightly lower test error at much higher cost.
- Language modeling: 57.5 test perplexity is obtained on PTB, while WT2 experiments report competitive results among automatically searched approaches.The PTB validation perplexity is 59.8, and the PTB result is described as on par with state-of-the-art models.
- Analysis and scope: Replacing the automatically discovered reduction cell with a designed one produces similar CIFAR-10 accuracy, while direct ImageNet search remains a stated future direction.The authors report that the reduction cell may have a negligible effect on network performance and identify ImageNet search as computationally challenging.
5. Conclusion
GDAS reduces NAS search cost by about 10^4 while producing CNN and RNN models with competitive performance against state-of-the-art models.
- 10^4 times lower search cost than standard NAS is achieved by GDAS.
- GDAS discovers CNN and RNN models with competitive performance compared to state-of-the-art models.