Source-linked AI summary
Single Path One-Shot Neural Architecture Search with Uniform Sampling
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, Jian Sun
TL;DR
NAS must reduce the cost of exploring architectures while avoiding the training difficulties caused by weight coupling in one-shot methods. The paper proposes Single Path One-Shot with single-path architectures and uniform sampling, and reports flexible, efficient search with strong ImageNet performance.
Problem
Existing one-shot NAS methods remain hard to train because supernet weights are coupled, and they have not shown competitive results on large datasets.
Method
Single Path One-Shot uses a simplified single-path supernet with uniform stochastic sampling, decoupling supernet training from architecture search.
Results
The approach achieves better results across several search spaces, with 74.7% accuracy under approximately 300M FLOPs in joint building-block and channel search.
Takeaways & Limitations
The method provides an efficient and flexible NAS approach that supports complex search spaces and practical constraints such as low latency.
Takeaways & Limitations
Path-dropout-based training is sensitive to its dropout rate, while latency-aware soft losses make hard latency constraints difficult to guarantee.
Abstract
from arXiv · showhide
We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method, however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex search spaces (e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency). It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.
1 Introduction
NAS automates architecture design, but early approaches are computationally expensive and existing one-shot methods remain difficult to train. The proposed single path one-shot approach uses uniform sampling to simplify training and supports flexible, efficient search.
- Early NAS samples and trains many architectures from scratch, making computation unaffordable on large datasets.
- Weight-sharing methods reduce computation by training a supernet once and inheriting its weights for candidate architectures.Only fine-tuning is needed afterward.
- Continuous-relaxation approaches jointly optimize architecture distributions and supernet weights, creating coupled weights whose effectiveness for individual architectures is unclear.
- Existing one-shot methods decouple architecture search from supernet training, but still involve coupled weights, sensitive hyperparameters, and weak large-dataset results.
- Single Path One-Shot uses single-path architectures and uniform sampling so architectures are optimized simultaneously with reduced weight coupling.The authors describe the training as hyperparameter-free and easy to converge.
- The approach supports rich search spaces and real-world constraints, while experiments on ImageNet verify state-of-the-art accuracy, memory consumption, training time, search efficiency, and flexibility.Evolutionary search supports constraints such as low latency.
2 Review of NAS Approaches
NAS combines weight optimization with architecture optimization, but training architectures from scratch is expensive. Weight sharing accelerates search, while continuous relaxation and joint optimization introduce coupling, bias, and difficulty satisfying hard architecture constraints.
- NAS optimizes network weights on training loss and selects architectures using validation accuracy after training.
- Nested NAS trains numerous sampled architectures from scratch, restricting practical use to small datasets and small search spaces.
- Weight-sharing NAS trains one supernet whose architectures inherit shared weights, avoiding training from scratch and enabling fast search on large datasets such as ImageNet.
- Continuous relaxation expands the discrete space to A(θ), enabling gradient methods but allowing sampled architectures that may be invalid in the original space A.
- Supernet node weights become deeply coupled, so inherited weights for a specific architecture may not remain effective.
- Joint optimization biases training toward some architecture parameters and nodes, making architectures with differently mature weights non-comparable and potentially misleading sampling.
- Architecture constraints concern properties such as memory, FLOPs, latency, and energy because they depend on architecture rather than weights.
- Soft latency losses make it difficult to guarantee hard constraints, so satisfying training objectives and preset architecture constraints simultaneously is challenging.
3 Our Single Path One-Shot Approach
The approach trains a stochastic single-path supernet by uniformly sampling architectures, then searches efficiently with inherited weights. Its simplified design supports flexible constrained searches and complex architecture variables while reducing weight coupling.
- Architecture Search: Architecture search evaluates inherited weights by inference, allowing repeated searches with different constraints such as 100ms and 200ms latency.The search algorithm can exactly satisfy architecture constraints and reuse a trained supernet.
- Uniform Sampling: Uniform sampling optimizes sampled architectures’ active weights across the search space while using efficient memory through stochastic supernet training.Only the weights of the sampled architecture are activated and updated at each optimization step.
- Single Path Supernet: Single-path architectures reduce co-adaptation between node weights without requiring a dropout-rate hyperparameter.Each sampled architecture activates one path, and the strategy corresponds to drop rate 1 in the comparison.
- Search Space Design: Choice-block designs extend the search space to channel number and other architecture variables while preserving the approach’s simple structure.Channel search preallocates maximum-width weights, samples a channel count, and slices the corresponding subtensor.
- Architecture Search: Evolutionary search is more effective than random search for improving validation accuracy over generations.The comparison is reported in Fig. 2.
- Empirical Summary: Comprehensive comparisons report that the approach is easiest to train, uses the smallest memory, satisfies latency constraints well, and supports large datasets.The summary also characterizes it as efficient, flexible, and state-of-the-art in the reported experiments.
4 Experiment Results
Experiments on ImageNet evaluate the method across building-block, channel, mixed-precision, and constrained search spaces. Results support competitive accuracy, reduced search cost, repeated searches from one supernet, and partial ranking correlation.
- Building Blocks: 74.3 accuracy is achieved by evolutionary search for building blocks, exceeding the reported baselines under FLOPs ≤330M.Random search with the single-path supernet reaches 73.8, while random selection reaches 73.7.
- Channels: 73.9 accuracy is obtained by channel search under FLOPs ≤330M, higher than the baselines.The search varies midchannels from 0.2x to 1.6x of the default channels.
- Channels: 74.7% accuracy is achieved by jointly searching building blocks and channels at approximately 300M FLOPs, surpassing listed manually designed and automatically searched models.Searching channels after building-block selection is slightly better than searching the combined space directly.
- Constraint-Based Search: The method achieves comparable or higher accuracy than baseline methods under matched FLOPs or latency constraints.Different constrained architectures are searched from the same supernet.
- Constraint-Based Search: One supernet supports searches under different complexity constraints, whereas previous methods train multiple supernets for those constraints.The paper reports that searching is much cheaper than supernet training and repeated search is supported.
- Mixed-Precision Quantization: Mixed-precision search achieves superior accuracy in most tested cases against uniform- and mixed-precision baselines.ResNet-18 and ResNet-34 results are searched on the same supernet.
- Search Cost: The single-path supernet uses less memory and the overall method is more efficient, with the extra search step costing less than 1 GPU day.The cost analysis trains compared supernets for 150000 iterations using batch size 256 and 8 GPUs.
- Correlation Analysis: Kendall Tau analysis finds positive but imperfect correlation between supernet and stand-alone model performance across search spaces and datasets.The method can identify superior architectures near the best, but cannot guarantee the true best architecture.
5 Conclusion
The paper revisits weight coupling in one-shot NAS and proposes a simple single-path approach to alleviate it. Experiments report better results across several search spaces, greater efficiency for repeated searches, and significant but imperfect correlation between predicted and real rankings.
- Conclusion: The proposed single-path one-shot approach addresses weight-coupling drawbacks while remaining simple and effective across several search spaces.The conclusion presents the method as a revision of the one-shot NAS paradigm.
- Conclusion: The method is more efficient when multiple searches are needed.The conclusion links this advantage to the method's search-cost analysis.
- Conclusion: Supernet rankings are significantly correlated with real rankings, but the correlation is partial rather than perfect and depends on search-space simplicity.The method can find superior architectures around the best without guaranteeing the real best architecture.