Source-linked AI summary
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, Hartwig Adam
TL;DR
DNNs can be too computationally intensive for mobile platforms, while optimizing MACs or weights may not reliably reduce latency or energy. NetAdapt adapts pretrained networks using empirically measured direct metrics and progressively simplifies them under a resource budget. It achieves up to 1.7× faster measured inference latency with equal or higher accuracy than state-of-the-art simplification algorithms.
Problem
Indirect metrics such as MACs and weights may not reliably approximate direct mobile resource metrics such as latency and energy consumption.
Method
NetAdapt iteratively adapts a pretrained network using empirical measurements of direct metrics on the target platform while maximizing accuracy under a resource budget.
Results
Up to 1.7× faster measured inference latency is achieved with equal or higher accuracy than state-of-the-art network simplification algorithms.
Takeaways & Limitations
NetAdapt can adapt pretrained networks to mobile platforms without detailed knowledge of the underlying platform implementation, provided the desired metrics can be measured.
Takeaways & Limitations
The experiments assume a single latency budget and demonstrate latency reduction using filter removal from convolutional or fully-connected layers.
Abstract
from arXiv · showhide
This work proposes an algorithm, called NetAdapt, that automatically adapts a pre-trained deep neural network to a mobile platform given a resource budget. While many existing algorithms simplify networks based on the number of MACs or weights, optimizing those indirect metrics may not necessarily reduce the direct metrics, such as latency and energy consumption. To solve this problem, NetAdapt incorporates direct metrics into its adaptation algorithm. These direct metrics are evaluated using empirical measurements, so that detailed knowledge of the platform and toolchain is not required. NetAdapt automatically and progressively simplifies a pre-trained network until the resource budget is met while maximizing the accuracy. Experiment results show that NetAdapt achieves better accuracy versus latency trade-offs on both mobile CPU and mobile GPU, compared with the state-of-the-art automated network simplification algorithms. For image classification on the ImageNet dataset, NetAdapt achieves up to a 1.7$\times$ speedup in measured inference latency with equal or higher accuracy on MobileNets (V1&V2).
1 Introduction
NetAdapt addresses the difficulty of deploying accurate DNNs on resource-constrained mobile platforms by optimizing direct resource metrics through empirical measurements. It progressively adapts pretrained networks to meet budgets while maximizing accuracy, achieving improved measured-latency trade-offs.
- Mobile deployment is constrained because accurate DNNs are often too computationally intensive for resource-limited phones.
- 19% fewer MACs can nevertheless incur 29% longer practical latency, showing that indirect metrics may poorly approximate mobile performance.
- NetAdapt incorporates direct metrics into optimization and evaluates them with empirical measurements from the target platform, without requiring detailed platform-specific knowledge.
- The algorithm iteratively reduces a pretrained network's resource consumption while maximizing accuracy until the resource budget is met.
- NetAdapt generates a family of simplified networks with different trade-offs, supporting dynamic network selection and further study.
- Up to 1.7× lower measured inference latency is achieved with equal or higher accuracy than state-of-the-art automatic simplification algorithms.
2 Related Work
Related work simplifies DNNs through pruning, architecture design, decomposition, or quantization, but many approaches are not guided by direct platform metrics. NetAdapt instead uses empirical measurements and can complement these methods.
- Pruning methods remove individual weights or entire filters, while some methods automatically or learnedly determine layerwise compression rates.
- Most platforms cannot fully exploit unstructured sparse filters, and filter-pruning methods may require manually choosing each layer's compression rate.
- Unlike the aforementioned simplification methods, NetAdapt is guided by direct metrics rather than only indirect network measures.
- Energy-aware pruning uses estimated energy models but requires detailed knowledge of target-platform hardware and toolchain mappings, whereas NetAdapt directly uses empirical measurements.
- NetAdapt is complementary to efficient architectures, decomposition, and quantization, including adapting MobileNets to further improve efficiency.
3 Methodology: NetAdapt
NetAdapt progressively simplifies a pretrained network under tightening resource constraints, using empirical direct-metric estimates and accuracy-guided layer proposals until the target budget is met.
- Core approach: NetAdapt automatically simplifies a pretrained network to meet a resource budget while maximizing accuracy.The algorithm can target latency, energy, memory footprint, or combinations of these resources.
- Iterative adaptation: The resource-reduction schedule progressively tightens constraints until every resource is within its budget.The schedule specifies the per-iteration reduction, and the algorithm terminates when the tightened constraints reach the target budgets.
- Iterative adaptation: Each iteration reduces one layer’s filters, evaluates candidate simplifications, and carries forward the highest-accuracy proposal.Candidates are generated for each CONV or FC layer individually, then the best proposal is selected.
- Fine-tuning: Short-term end-to-end fine-tuning restores accuracy during adaptation, followed by long-term fine-tuning until convergence for the final network.Short-term fine-tuning is particularly important during large reductions in small networks, where accuracy could otherwise collapse.
- Layer simplification: Filter counts are chosen from empirical measurements, while filters retained in a layer are those with the largest ℓ2-norm magnitudes.Associated channels in following layers are also removed when filters are removed.
- Fast resource estimation: Layer-wise look-up tables estimate network resource consumption by summing pre-measured layer costs, avoiding prohibitively expensive repeated device measurements.Estimated and real latency were highly correlated on a single large Google Pixel 1 CPU core during MobileNetV1 adaptation.
4 Experiment Results
The experiments evaluate NetAdapt on MobileNetV1 across network sizes and platforms, comparing it with automated simplification baselines and multipliers. They also include a MobileNetV2 adaptation to extend the efficiency evaluation.
- Experimental scope: NetAdapt is evaluated on MobileNets using ImageNet image classification experiments.The study focuses primarily on MobileNetV1 and also adapts large MobileNetV2.
- Baselines: Multipliers simplify networks by scaling width, resolution, or both across layers.The notation “50% MobileNetV1 (128)” denotes a 50% width multiplier with input resolution 128.
- Baselines: MorphNet uses sparsifying regularization for automatic network simplification.
- Baselines: ADC is an automatic network simplification algorithm based on reinforcement learning.
- Experimental scope: The experiments target both real-time-class small MobileNetV1 and larger MobileNetV1 networks across platforms.The small model is described as more difficult to simplify than larger networks.
4.1 Detailed Settings for MobileNetV1 Experiments
The MobileNetV1 experiments adapt depthwise-separable networks layerwise and use controlled fine-tuning and mobile inference measurements. Settings cover training, latency measurement, and the small-model comparison setup.
- NetAdapt configuration: MobileNetV1 depthwise layers are adapted together with their corresponding pointwise layers, with filters selected using the pointwise layer.For the small MobileNetV1, the latency-reduction schedule starts at 0.5 and decays by 0.96 per iteration.
- Network training: Short-term fine-tuning uses a reduced training set, while a holdout set selects the highest-accuracy proposal and the full set supports final fine-tuning.
- Network training: The same training configuration is applied across networks for fair comparison, using MorphNet’s configuration with a batch size of 128.The long-term and short-term learning rates are 0.045 and 0.0045, respectively.
- Mobile inference: Inference uses TensorFlow Lite on mobile CPUs and SNPE on mobile GPUs.CPU latency is measured on a single large Google Pixel 1 core; GPU latency is measured on a Samsung Galaxy S8.
- Comparison setup: Figure 5 compares small-MobileNetV1 NetAdapt with multipliers and MorphNet on a Google Pixel 1 mobile CPU.
- Mobile inference: Each reported latency is the median of 11 measurements.
4.2 Comparison with Benchmark Algorithms
NetAdapt improves accuracy–latency trade-offs over benchmark algorithms on small and large MobileNetV1 models across mobile CPU and GPU settings. Better training configurations can further increase adapted-network accuracy.
- Small MobileNetV1 on mobile CPU: 1.7× faster than multipliers, with the same or higher accuracy, NetAdapt adapts small MobileNetV1 on a Google Pixel 1 CPU.Against MorphNet, NetAdapt is 1.6× faster with 0.3% higher accuracy.
- Large MobileNetV1 on mobile CPU: 1.4× and 1.2× speedups over multipliers and ADC, respectively, are achieved by NetAdapt with higher accuracy on large MobileNetV1 CPU experiments.The large model’s latency is approximately 8× that of the small MobileNetV1, so the initial latency reduction is scaled accordingly.
- Training configuration: 1.3% additional accuracy is obtained after adding dropout and label smoothing to the large-MobileNetV1 training configuration.The improvement is reported for the adapted networks in the large-MobileNetV1 comparison.
- Large MobileNetV1 on mobile GPU: 1.2× speedup with higher accuracy is reported for NetAdapt versus benchmark algorithms on large MobileNetV1 running on a Samsung Galaxy S8 mobile GPU.The same better training configuration can further increase accuracy by 1.3%.
Adapting Large MobileNetV1 on a Mobile GPU
The ablation studies examine indirect MAC-based optimization and fine-tuning effects for MobileNetV1 on a Google Pixel 1 CPU. They show why measured latency matters and quantify the benefit of long-term fine-tuning.
- Fine-tuning: Long-term fine-tuning adds 3.4% accuracy on average, increasing gains from 1.8% to 4.5%.The comparison is made after adapting small MobileNetV1 on a Google Pixel 1 CPU.
- Fine-tuning: Short-term fine-tuning preserves accuracy well, while long-term fine-tuning provides additional accuracy after the target network is selected.
- Direct versus indirect metrics: Lower MAC counts do not necessarily produce faster networks when NetAdapt is guided by the indirect metric.The experiment compares MAC-based guidance with direct latency-based optimization.
Impact of Direct Metrics
Short-term fine-tuning stabilizes NetAdapt’s network-proposal decisions, while long-term fine-tuning recovers additional accuracy after adaptation.
- With zero short-term fine-tuning iterations, accuracy drops to nearly zero, causing proposal selection to rely solely on noise.
- After 10,000 short-term fine-tuning iterations, accuracy remains above 20%, enabling better network-proposal decisions.
- Long-term fine-tuning increases accuracy by up to 4.5% or 3.4% on average after short-term fine-tuning.
Impact of Long-Term Fine-Tuning
Resource-reduction schedules trade adaptation speed against accuracy, while NetAdapt’s layerwise choices preserve filters where feature-map resolution or classification demands may be higher.
- Resource Reduction Schedules: Larger resource reductions per iteration accelerate adaptation but reduce accuracy; with equal total iterations, smaller initial reductions with slower decay are preferable.
- Architecture Comparison: NetAdapt and multiplier-based networks achieve similar latency to 25% MobileNetV1 (128), enabling architectural comparison at a matched latency target.
- Architecture Comparison: NetAdapt removes more filters in layers 7–10 but fewer in layer 6, where reduced feature-map resolution may otherwise create an information bottleneck.
- Architecture Comparison: NetAdapt retains more filters in layer 13, possibly because ImageNet’s 1,000 classes require more feature maps for final classification.
4.5 Adapting Large MobileNetV2 on a Mobile CPU
On a mobile CPU, NetAdapt also adapts the large MobileNetV2 and outperforms multipliers in accuracy or speed.
- For MobileNetV2, NetAdapt adapts individual expansion layers or locksteps bottleneck layers sharing the same resolution because of residual connections.
- Each network proposal is short-term fine-tuned for 10,000 iterations, with a 1 ms initial latency reduction and 0.995 latency-reduction decay.
5 Conclusion
NetAdapt automatically adapts pretrained networks to mobile platforms under real resource budgets by optimizing direct metrics through empirical measurement. It achieves better accuracy–latency trade-offs than state-of-the-art simplification methods.
- NetAdapt automatically adapts a pretrained network to a mobile platform given a real resource budget.
- The algorithm incorporates direct metrics such as latency and energy, using empirical measurements without requiring knowledge of the platform’s implementation.
- Up to 1.7× faster with equal or higher accuracy, NetAdapt achieves better accuracy–latency trade-offs than other state-of-the-art network simplification algorithms.
- The conclusion emphasizes using direct metrics in efficient-network optimization as a direction for further performance improvements.