Source-linked AI summary
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, Kurt Keutzer
TL;DR
Mobile ConvNet design must navigate a combinatorial architecture space whose best choices vary with input resolution and target hardware, while FLOPs do not reliably represent latency. The paper introduces differentiable neural architecture search using gradient-based optimization and reports FBNets that outperform prior efficient models with substantially lower search cost. A key scope assumption is that operator runtimes are independent and sequential on the target processor.
Problem
Designing accurate and efficient mobile ConvNets is difficult because the architecture space is combinatorial, search is expensive, and optimal designs depend on resolution and hardware.
Method
DNAS optimizes a layer-wise stochastic super net with Gumbel Softmax and gradient-based methods, using target-device latency in the loss.
Results
FBNet-B achieves 74.1% top-1 accuracy with 295M FLOPs and 23.1 ms latency, while its estimated search cost is 420x lower than MnasNet's.
Takeaways & Limitations
DNAS enables case-by-case ConvNet redesigns across resolutions, channel sizes, and target devices, with FBNets achieving higher accuracy than MobileNetV2 across these settings.
Takeaways & Limitations
The latency estimate assumes operators run sequentially and that each operator's runtime is independent of the others on the target processor.
Abstract
from arXiv · showhide
Designing accurate and efficient ConvNets for mobile devices is challenging because the design space is combinatorially large. Due to this, previous neural architecture search (NAS) methods are computationally expensive. ConvNet architecture optimality depends on factors such as input resolution and target devices. However, existing approaches are too expensive for case-by-case redesigns. Also, previous work focuses primarily on reducing FLOPs, but FLOP count does not always reflect actual latency. To address these, we propose a differentiable neural architecture search (DNAS) framework that uses gradient-based methods to optimize ConvNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets, a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically. FBNet-B achieves 74.1% top-1 accuracy on ImageNet with 295M FLOPs and 23.1 ms latency on a Samsung S8 phone, 2.4x smaller and 1.5x faster than MobileNetV2-1.3 with similar accuracy. Despite higher accuracy and lower latency than MnasNet, we estimate FBNet-B's search cost is 420x smaller than MnasNet's, at only 216 GPU-hours. Searched for different resolutions and channel sizes, FBNets achieve 1.5% to 6.4% higher accuracy than MobileNetV2. The smallest FBNet achieves 50.2% accuracy and 2.9 ms latency (345 frames per second) on a Samsung S8. Over a Samsung-optimized FBNet, the iPhone-X-optimized model achieves a 1.4x speedup on an iPhone X.
1. Introduction
Efficient mobile ConvNet design is difficult because the architecture space is enormous, training-based search is costly, and optimal designs depend on resolution and hardware. DNAS addresses these challenges with differentiable, hardware-aware search, producing FBNets with strong accuracy-latency results and much lower search cost.
- ConvNet design must balance accuracy with efficiency because mobile devices have limited computing capacity.
- (3 × 5)^16 ≈ 6 × 10^18 possible architectures arise even under simplified per-layer kernel and filter choices.The combinatorial space makes exhaustive design impractical.
- Days or weeks of ConvNet training restrict manual design to only a few iterations, while reinforcement-learning NAS trains sampled architectures on proxy data or for fewer epochs.
- Optimal architectures vary with input resolution and target device because receptive-field needs and operator latency can change across settings.
- FLOPs are hardware-agnostic, whereas latency depends on hardware and software configurations, motivating direct latency optimization.
- DNAS uses a stochastic super net, Gumbel Softmax, and gradient-based optimization to search layer-wise architectures without separately training each candidate.
- 74.1% top-1 accuracy, 295M FLOPs, and 23.1 ms latency make FBNet-B 2.4x smaller and 1.5x faster than MobileNetV2-1.3, while its search costs 216 GPU-hours.
2. Related work
Prior efficient-ConvNet work reduced parameters or FLOPs, while NAS methods increasingly automated architecture search but often remained computationally expensive. Later approaches introduced weight sharing, gradient-based optimization, and target-device latency objectives, though some searched only stage-level structures.
- SqueezeNet, SqueezeNext, and ShiftNet primarily reduce ConvNet parameter size, while later work shifts attention toward FLOPs.
- Early NAS methods use reinforcement learning to search architectures for competitive accuracy with low FLOPs, but they are computationally expensive.
- Weight sharing and gradient-based optimization reduce the computational cost of neural architecture search.
- Latency-focused NAS optimizes target-device latency, while related work also adapts existing models to improve efficiency.
3. Method
FBNet uses a layer-wise differentiable search space and a latency-aware loss to optimize ConvNet architectures without brute-force training across the combinatorial design space.
- Problem formulation: DNAS formulates architecture design around a search space, a latency-aware loss, and an efficient search algorithm.The objective seeks an architecture minimizing loss after its weights are trained, while accounting for target-hardware latency.
- Search space: Each searchable layer can select a different block within a fixed macro-architecture, rather than repeating one cell throughout the network.The macro-architecture fixes layer counts and dimensions, while searchable layers choose block types.
- Search space: Candidate blocks vary in expansion rate, kernel size, and grouped-convolution configuration, with skip blocks allowing reduced network depth.The search space uses 9 candidate blocks across 22 layers, yielding 9^22 ≈ 10^21 possible architectures.
- Latency-aware loss: The loss combines cross-entropy with a latency penalty measured on target hardware, controlled by coefficients α and β.CE(a, w_a) is the cross-entropy term, while LAT(a) denotes architecture latency in microseconds.
- Latency-aware loss: A latency lookup table estimates network runtime from individual operator runtimes, avoiding measurements of all 10^21 architectures.The approach assumes operator runtimes are independent on the target processor and benchmarks only a few hundred operators.
- Search algorithm: Gumbel Softmax relaxes discrete block-selection masks into differentiable variables, enabling SGD to optimize weights and architecture-distribution parameters.After super-net training, architectures are sampled from the learned distribution P_θ.
- Search algorithm: The proposed DNAS algorithm is reported to be orders of magnitude faster than previous reinforcement-learning NAS while generating better architectures.The stochastic super net represents the search space with parallel candidate blocks and samples one block per layer during inference.
4. Experiments
Experiments evaluate FBNets on ImageNet across accuracy, latency, search cost, input resolutions, channel scalings, and target devices. The results show strong efficiency, rapid search, and benefits from device- and resolution-specific redesign.
- 4.1. ImageNet Classification: 73.0% accuracy: FBNet-A outperforms several efficient baselines while achieving lower latency and FLOP count than key comparison models.FBNet-A improves over 1.0-MobileNetV2, 1.5-ShuffleNet V2, and CondenseNet, while matching DARTS and MnasNet-65 in accuracy.
- 4.2. Different Resolution and Channel Size Scaling: 1.5% to 6.4% better accuracy: FBNets outperform MobileNetV2 at matched input and channel scaling with similar latency.FBNet-96-0.35-1 achieves 50.2% accuracy and 2.9 ms latency, equivalent to 345 frames per second, on a Samsung Galaxy S8.
- 4.2. Different Resolution and Channel Size Scaling: 2.9 ms latency: FBNet-96-0.35-1 uses a much shallower architecture with many skipped layers compared with the 224-input FBNet-A, B, and C models.The paper conjectures that smaller input sizes require smaller receptive fields and therefore fewer layers.
- 4.3. Different Target Devices: 39% higher latency: deploying FBNet-S8 on iPhone X is slower than deploying the iPhone-X-optimized model on its target device, despite similar accuracy.The target-device models achieve 73.20% and 73.27% accuracy, while their cross-device latencies are 27.53 ms and 23.33 ms.
- 4.3. Different Target Devices: Different operators are selected for each device because their runtimes differ across iPhone X and Samsung S8.The searched architectures use device-specific operator choices rather than relying on one architecture for both platforms.
5. Conclusion
DNAS discovers hardware-aware ConvNets through differentiable search, producing FBNets that improve accuracy-efficiency trade-offs across models, resolutions, and devices.
- DNAS uses a differentiable neural architecture search framework to discover hardware-aware efficient ConvNets.
- 74.1% top-1 accuracy with 295M FLOPs and 23.1 ms latency makes FBNet-B 2.4x smaller and 1.5x faster than MobileNetV2-1.3 at the same accuracy.
- FBNet-B achieves better accuracy and lower latency than MnasNet, while DNAS search cost is estimated to be 420x smaller.
- 1.5% to 6.4% accuracy gains over MobileNetV2 are achieved by searching across different input resolutions and channel scaling.
- 50.2% accuracy with 2.9 ms latency, or 345 frames/sec, is achieved by the smallest FBNet on a Samsung S8.
- A model optimized for iPhone X delivers a 1.4x speedup over the Samsung-optimized FBNet on an iPhone X.
A. Experiment details
The architecture search uses a two-stage pipeline: train a stochastic super net to obtain an architecture distribution, then sample and train architectures from scratch.
- The search has two stages: training the stochastic super net, followed by sampling architectures and training them from scratch.
- 100 ImageNet classes are used as a smaller proxy dataset to accelerate stochastic super-net training.
- FBNet-{A, B, C} models are trained for 360 epochs on 8 GPUs with batch size 256 and learning-rate drops at epochs 90, 180, and 270.