Source-linked AI summary
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu
TL;DR
NAS must design accurate ConvNets under mobile runtime constraints, but combinatorially large search spaces make existing approaches expensive. Single-Path NAS encodes candidate operations as shared-weight subsets of one superkernel, achieving strong ImageNet accuracy with substantially lower search cost.
Problem
NAS design spaces are combinatorially large, making hardware-efficient ConvNet search costly, with prior approaches requiring at least 200 GPU-hours.
Method
Single-Path NAS uses one over-parameterized superkernel with shared convolutional weights and learns which kernel-weight subsets define each layer's operation.
Results
74.96% top-1 accuracy on ImageNet at 79ms Pixel 1 latency was achieved with a search cost of 8 epochs, or 30 TPU-hours.
Takeaways & Limitations
Single-Path NAS reduces hardware-efficient NAS search to less than 4 hours and its design-space encoding can be incorporated into other NAS methodologies.
Abstract
from arXiv · showhide
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours. Our contributions are as follows: 1. Single-path search space: Compared to previous differentiable NAS methods, Single-Path NAS uses one single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional kernel parameters, hence drastically decreasing the number of trainable parameters and the search cost down to few epochs. 2. Hardware-efficient ImageNet classification: Single-Path NAS achieves 74.96% top-1 accuracy on ImageNet with 79ms latency on a Pixel 1 phone, which is state-of-the-art accuracy compared to NAS methods with similar constraints (<80ms). 3. NAS efficiency: Single-Path NAS search cost is only 8 epochs (30 TPU-hours), which is up to 5,000x faster compared to prior work. 4. Reproducibility: Unlike all recent mobile-efficient NAS methods which only release pretrained models, we open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas.
1 Introduction
NAS automates hardware-efficient ConvNet design, but its enormous architecture spaces and multi-path supernets make searching computationally expensive. Single-Path NAS shares kernel weights in one superkernel to reduce trainable parameters and search cost while maintaining strong mobile ImageNet performance.
- Motivation: NAS seeks hardware-efficient ConvNets under mobile-device constraints, but its search remains costly because architecture choices create combinatorially large design spaces.For a 22-layer mobile ConvNet with five candidate operations per layer, the passage estimates approximately 10^15 architectures.
- Motivation: At least 200 GPU-hours remain necessary for prior differentiable NAS approaches despite proxy datasets and partial path updates.These workarounds address memory and data demands but do not eliminate the substantial computational cost.
- Single-Path NAS: Single-Path NAS represents candidate convolution operations as subsets of shared weights in one over-parameterized superkernel rather than separate trainable paths.The method formulates architecture search as selecting which kernel-weight subsets to use in each layer.
- Single-Path NAS: This single-path encoding reduces trainable parameters and gradients, enabling batch sizes of 1024, four times prior search efficiency.The reduction in maintained parameters is the central mechanism behind the reported search-efficiency improvement.
- Results: 74.96% top-1 accuracy on ImageNet with 79ms Pixel 1 latency represents a +0.31% improvement over the best hardware-aware NAS under 80ms.The reported search cost is 8 epochs, or 3.75 hours on TPUs and 30 TPU-hours.
- Results: Single-Path NAS reports up to 5,000× faster search than prior work while providing state-of-the-art accuracy under the same mobile latency setting.The authors also open-source and fully document the method for reproducibility.
2 Related Work
Hardware-efficient NAS targets ConvNet deployment constraints that accuracy-focused or FLOP-focused design does not fully capture. The field has progressed from human-designed operations and costly controller-based search toward hardware-aware, one-shot differentiable methods, whose multi-path supernets remain inefficient.
- Hardware-efficient ConvNets: Hardware-efficient ConvNet research addresses the deployment challenge created by higher computational complexity in increasingly capable ConvNet designs.Earlier approaches relied on human expertise to introduce efficient operations, while pruning and quantization pursued similar efficiency goals.
- Neural Architecture Search: NAS automates ConvNet design through reinforcement learning, evolutionary algorithms, and gradient-based methods, but controller training across sampled architectures is costly.Earlier cell-based approaches repeated a searched cell across layers, limiting the search focus to cell architecture.
- Hardware-aware NAS: Low FLOP counts do not necessarily translate to hardware efficiency, motivating NAS objectives that incorporate runtime or power.Recent hardware-aware methods also move beyond cell-based assumptions toward generalized MobileNetV2-based search spaces.
- Hardware-aware Differentiable NAS: One-shot differentiable NAS uses over-parameterized supernets with separate trainable paths for every candidate operation in each layer.This formulation relaxes architecture search into an operation- or path-selection problem.
- Hardware-aware Differentiable NAS: Multi-path search spaces scale trainable parameters with candidate operations, forcing workarounds such as proxy datasets and memory-wise partial path updates.These methods remain constrained because the full supermodel must still be maintained during search, limiting batch sizes.
3 Proposed Method: Single-Path NAS
Single-Path NAS encodes candidate MBConv architectures within one over-parameterized superkernel rather than separate paths, reducing the NAS problem to selecting subsets of shared kernel weights. Differentiable indicators and trainable thresholds represent kernel size, expansion ratio, and skip decisions while supporting hardware-aware optimization.
- 3.1 Mobile ConvNets Search Space: A Novel View: MBConv layers vary by depthwise kernel size and expansion ratio, with candidate values {3, 5} and {3, 6}, plus a skip operation that drops the layer.The MBConv micro-architecture uses point-wise, depthwise, and linear convolutions.
- 3.1 Mobile ConvNets Search Space: A Novel View: Single-Path NAS represents all candidate architectures with one superkernel per layer instead of separate trainable paths.The design searches over shared convolutional kernel parameters in a compact one-shot network.
- 3.2 Proposed Methodology: Single-Path NAS formulation: A 5 × 5 kernel combines the 3 × 3 inner weights with an additional outer-shell subset, allowing kernel-size choices to share parameters.The relation is w5×5 = w3×3 + w5×5\3×3.
- 3.2 Proposed Methodology: Single-Path NAS formulation: Trainable thresholds and group-Lasso-based importance signals determine whether kernel subsets are active, with sigmoid relaxation enabling gradient-based optimization.The indicator condition depends on subset weights, and thresholds are learned rather than manually selected.
- 3.2 Proposed Methodology: Single-Path NAS formulation: The superkernel encodes expansion-ratio and skip decisions by zeroing selected channel or filter subsets after setting intermediate channels to the maximum candidate expansion ratio.The same kernel-based formulation is applied recursively to expansion-ratio choices, while zeroing the relevant filters represents the skip operation.
- 3.3 Single-Path vs. Existing Multi-Path Assumptions: Unlike multi-path NAS, Single-Path NAS directly optimizes one compact network’s kernel weights instead of alternating updates in a bi-level architecture-and-weight optimization.The multi-path formulation maintains separate candidate paths and requires additional gradient steps when alternating architecture and weight updates.
4 Experiments
Single-Path NAS is evaluated for ImageNet classification on Pixel 1 under an 80ms latency target, with comparisons covering accuracy, search cost, runtime prediction, and kernel-based architectural behavior.
- ImageNet classification: 74.96% top-1 accuracy is achieved on ImageNet under the hardware-efficient NAS setting, establishing the reported state-of-the-art result.The evaluation targets Pixel 1 latency of ≤80ms and compares against prior mobile-efficient NAS methods.
- ImageNet classification: +0.31% top-1 accuracy over ProxylessNAS is reported while maintaining target latency of ≤80ms on the same mobile platform.The method also reports gains over MnasNet, FBNet-B, and MobileNetV2 in this latency range.
- NAS search cost: 8 epochs reduce search cost by 5,000× versus MnasNet, 25× versus ProxylessNAS, and 11× versus FBNet.The search takes 3.75 hours on a TPUv2, using approximately 10k steps and batch size 1024.
- Kernel-based analysis: Kernel-based visualizations show that subsets of shared superkernel weights capture architectural decisions across layers and reflect accuracy-runtime trade-offs.The visualization uses log-scale standard deviations across channel dimensions, with lighter colors indicating smaller values.
- Search comparison: Randomly sampled designs have accuracy comparable to MobileNetV2, while the authors report that random search does not outperform NAS methods.Samples were retained when their predicted runtime fell between 75ms and 80ms.
- Runtime modeling: 1.76% average prediction error validates the runtime model used to estimate ConvNet runtimes during search.Across 100 randomly designed ConvNets, the model has 1.32ms RMSE.
5 Conclusion
Single-Path NAS reduces hardware-efficient ConvNet search to less than four hours through a single-path supernet that selects kernel-weight subsets. It achieves strong ImageNet accuracy under mobile latency constraints while substantially reducing search cost.
- Single-Path NAS reduces hardware-efficient ConvNet search to less than 4 hours by finding which subset of kernel weights to use per layer.Its single-path view revisits one-shot supernet design for lower-cost architecture search.
- 74.96% top-1 accuracy on ImageNet is achieved with 79ms latency on a Pixel 1 phone.The reported latency is on-par with previous NAS methods at ≤80ms.
- 8 epochs (30 TPU-hours) reduces hardware-efficient NAS search cost by up to 5,000× compared to prior work.The paper reports this as a major efficiency improvement over previous methods.