Source-linked AI summary
One-Shot Neural Architecture Search via Self-Evaluated Template Network
Xuanyi Dong, Yi Yang
TL;DR
NAS seeks efficient architecture discovery, but one-shot methods still sample candidates randomly despite sharing parameters. SETN learns candidate-selection probabilities alongside a parameter-sharing template network, and reports state-of-the-art results on three benchmarks within two GPU days, with limitations for very large search spaces and direct ground-truth evaluation.
Problem
One-shot NAS reduces training cost through parameter sharing, but randomly sampled candidates are often poor and unlikely to include the best architecture.
Method
SETN combines a parameter-sharing template network with an evaluator that estimates candidates likely to have lower validation loss for selective sampling.
Results
SETN completes search within two GPU days and achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet.
Takeaways & Limitations
SETN improves the quality of candidates used for one-shot evaluation and can find high-performing CNNs from a search space exceeding 10^16 possibilities.
Takeaways & Limitations
For search spaces as large as 10^30 candidates, a small number of evaluations T may not find a good model, requiring higher evaluation cost.
Abstract
from arXiv · showhide
Neural architecture search (NAS) aims to automate the search procedure of architecture instead of manual design. Even if recent NAS approaches finish the search within days, lengthy training is still required for a specific architecture candidate to get the parameters for its accurate evaluation. Recently one-shot NAS methods are proposed to largely squeeze the tedious training process by sharing parameters across candidates. In this way, the parameters for each candidate can be directly extracted from the shared parameters instead of training them from scratch. However, they have no sense of which candidate will perform better until evaluation so that the candidates to evaluate are randomly sampled and the top-1 candidate is considered the best. In this paper, we propose a Self-Evaluated Template Network (SETN) to improve the quality of the architecture candidates for evaluation so that it is more likely to cover competitive candidates. SETN consists of two components: (1) an evaluator, which learns to indicate the probability of each individual architecture being likely to have a lower validation loss. The candidates for evaluation can thus be selectively sampled according to this evaluator. (2) a template network, which shares parameters among all candidates to amortize the training cost of generated candidates. In experiments, the architecture found by SETN achieves state-of-the-art performance on CIFAR and ImageNet benchmarks within comparable computation costs. Code is publicly available on GitHub: https://github.com/D-X-Y/AutoDL-Projects.
1. Introduction
Neural architecture search reduces reliance on manual network design, but conventional and one-shot approaches still face substantial search or candidate-selection costs. SETN combines parameter sharing with learned candidate selection and achieves strong benchmark results within comparable computation.
- Motivation: Neural architecture search aims to discover networks that maximize validation accuracy while reducing the expert knowledge and computational resources required for architecture design.Typical reinforcement-learning and evolutionary approaches can require 500 GPUs over four days.
- Motivation: One-shot NAS shares parameters across candidates, but randomly sampled candidates are often poor and cover only a small portion of the search space.This makes inclusion of the best architecture unlikely during evaluation.
- SETN: SETN uses a template network that shares parameters across candidate CNNs and trains candidates stochastically so their parameters can be reused without additional training.At each iteration, one candidate is sampled and only its partial template parameters are optimized.
- SETN: Its evaluator estimates which individual candidates are likely to have lower validation loss and selectively samples them for one-shot evaluation.The evaluator is trained on validation data with assistance from the template network.
- Results: SETN discovers a superior CNN on CIFAR-10 within two GPU days and achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet.The reported result covers three benchmarks.
2. Related Work
Prior NAS methods trade computational cost, evaluation strategy, and parameter sharing in different ways. SETN is positioned as a one-shot method that retains rapid evaluation while replacing random candidate selection with selective sampling.
- Comparison dimensions: Table 1 compares NAS algorithms by search type, efficiency, parameter sharing, rapid evaluation, and candidate-generation strategy.Efficiency is defined as discovering a CNN within five GPU days.
- Positioning: SETN focuses on CNN search and is closely related to one-shot NAS while differing from DARTS in how candidate evaluation and selection are handled.The paper contrasts DARTS's direct architecture selection with one-shot methods' random candidate sampling.
- Search strategies: Early reinforcement-learning and evolutionary NAS methods train many candidates or mutate networks toward higher accuracy, but their computational costs are high.These approaches use validation accuracy as supervision after training candidate networks for tens of epochs.
- One-shot NAS: One-shot approaches estimate network performance without additional training through generated or shared weights, but randomly select candidates with uneven quality.SETN instead selectively samples candidates expected to have lower validation loss.
3. Background
The paper searches neural cells rather than complete CNN structures, representing each cell as a sequence of operations over node inputs and assembling cells into CNNs for CIFAR and ImageNet. The resulting search space contains 9.1 × 10^16 candidates.
- Cell-based search: Recent NAS methods search for an effective neural cell instead of an entire CNN structure.The paper follows this cell-based search approach.
- Cell-based search: A cell maps an input tensor to an output tensor, with stride 1 defining a normal cell and stride 2 defining a reduction cell.The cited formulation distinguishes the two cell types by their spatial and channel transformations.
- Cell structure: Each cell node is specified by two inputs and two transformation functions, producing an intermediate tensor by summing the transformed inputs.The cell output concatenates intermediate outputs from its nodes.
- Cell structure: The candidate operation set includes pooling, skip connections, separable convolutions, and asymmetric convolutions.The listed operations include 3x3 max pooling, 3x3 average pooling, skip connection, 3x3 and 5x5 separable convolutions, and 1x3 & 3x1 convolution.
- CNN construction: 9.1 × 10^16 candidates form the search space when each cell has B = 4 nodes, after which normal and reduction cells are assembled into CNNs.The paper gives separate overall CNN layouts for CIFAR and ImageNet.
4. Methodology
SETN combines a parameter-sharing template network with a learned evaluator to train and selectively sample architecture candidates. It alternates training and validation updates, then evaluates 1,000 evaluator-selected candidates using shared parameters.
- Template Network: The template network contains all candidate CNNs and shares its parameters across them, amortizing candidate training.Each candidate uses the corresponding template parameters rather than being trained from scratch.
- Template Network: At each iteration, SETN uniformly samples one candidate and optimizes only that candidate’s template parameters, giving candidates equal training opportunity.This stochastic strategy is intended to avoid the Matthew effect and improve candidate training compared with joint optimization.
- Architecture Encoding: Candidate architectures are encoded by categorical choices of inputs and operations, with ordered operation sampling removing redundant combinations.The encoding uses vectors for input choices and operation pairs; the ordering constraint enforces r2 ≤ r1.
- Evaluator: The evaluator is optimized on validation data through a continuous relaxation of discrete architecture choices so it can estimate candidates’ validation loss.The evaluator parameters α receive gradients from validation loss, while template parameters ω are trained on the training set.
- Searching Algorithm: SETN samples T=1000 low-validation-loss candidates according to the evaluator, evaluates them with parameters extracted from the template network, and selects the lowest-loss candidate.The search procedure avoids traversing the full candidate space and replaces random candidate selection with evaluator-guided sampling.
- Connections with Other NAS Approaches: SETN is presented as combining one-shot NAS’s shared-parameter evaluation with higher-quality candidate selection while generalizing the DARTS and one-shot NAS streams.The paper contrasts DARTS’s direct selection and variable evaluation with one-shot NAS’s random candidate sampling.
5. Experiments
SETN is evaluated across CIFAR and ImageNet, including efficiency, scalability, candidate quality, ranking reliability, and discovered-cell analyses. It achieves competitive benchmark performance with substantially lower search costs, while large search spaces and direct evaluation of the evaluator remain limitations.
- Comparison with the state-of-the-art: 1.8 GPU days completes SETN's CIFAR-10 and CIFAR-100 search, with the lowest error among efficient approaches using similar or fewer parameters.On CIFAR-100, SETN achieves 17.25% error; on CIFAR-10, NASNet-A is slightly better but requires more than 1000× the computational cost.
- Scalability: A search space with 9.1×10^18 candidates requires 50 GPU hours for SETN training and under three hours to evaluate 1K candidates, only about 18% more GPU days overall.The expanded space is about 100× larger than the small space, and the discovered network has similar performance.
- Comparison with the state-of-the-art: 74.3% top-1 accuracy is achieved by SETN (N=2 & C=58) on ImageNet, with competitive accuracy among efficient NAS approaches.The ImageNet experiment transfers the CIFAR-10-discovered CNN structure without using ImageNet images during search.
- Comparison with the state-of-the-art: 1750× lower search cost than AmoebaNet accompanies similar accuracy, while SETN completes its search within acceptable computational costs.AmoebaNet costs 3150 GPU days, compared with SETN's substantially lower cost.
- The quality of estimated candidates: SETN generates more accurate candidates than random sampling and SETN-NON, and more candidates in the 85%–90% accuracy range than SETN-LR.SETN's overall candidate performance is similar to SETN-LR, but its histogram contains more highly accurate candidates.
- The quality of estimated candidates: More than 80% of sampled network pairs receive the same relative ranking from template-parameter accuracy and retrained ground-truth accuracy.The analysis evaluates 2,000 networks arranged into 1,000 pairs.
6. Conclusion
SETN searches for higher-accuracy CNNs by improving the quality of candidates selected for one-shot evaluation. It finds a good CNN from more than 10^16 possibilities and achieves state-of-the-art performance on three benchmarks.
- 6. Conclusion: SETN improves one-shot NAS by generating higher-quality architecture candidates for evaluation.Its sampled candidates can cover better architectures and support finding higher-performance models.
- 6. Conclusion: SETN completes the search procedure within two GPU days.
- 6. Conclusion: SETN finds a good CNN from more than 10^16 network possibilities and achieves state-of-the-art performance on three benchmarks.