Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Han Cai; Ji Lin; Yujun Lin; Zhijian Liu; Haotian Tang; Hanrui Wang; Ligeng Zhu; Song Han

Source-linked AI summary

Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Han Cai, Ji Lin, Yujun Lin, Zhijian Liu, Haotian Tang, Hanrui Wang, Ligeng Zhu, Song Han

arXiv:2204.11786v1Apr 25, 2022cs.LGcs.CLcs.CV

TL;DR

DNNs deliver strong AI performance but are too computationally intensive for many resource-constrained devices. This survey reviews compression, AutoML, on-device training, task-specific acceleration, and supporting software/hardware systems. It presents a broad view spanning manual and automated methods, inference and training, and general and application-specific optimization.

Problem
DNN computational complexity limits deployment on mobile and IoT devices despite their strong performance across AI tasks.
Method
The paper systematically surveys model compression, AutoML, on-device training, task-specific acceleration, and efficient deep learning software and hardware.
Results
The survey covers efficient deep learning across manual-to-automated methods, training-to-inference, algorithm-to-system design, and general-to-application-specific optimization.
Takeaways & Limitations
Efficient deep learning requires coordinated advances in algorithms, on-device training, task-specific acceleration, and software/hardware systems.
Takeaways & Limitations
Current AutoML methods mainly search restricted spaces built from human-designed architectures, limiting optimization headroom and novel operation discovery.

Abstract

from arXiv · show

Deep neural networks (DNNs) have achieved unprecedented success in the field of artificial intelligence (AI), including computer vision, natural language processing and speech recognition. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that are able to lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand in order to enable numerous edge AI applications. This paper provides an overview of efficient deep learning methods, systems and applications. We start from introducing popular model compression methods, including pruning, factorization, quantization as well as compact model design. To reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization. We then cover efficient on-device training to enable user customization based on the local data on mobile devices. Apart from general acceleration techniques, we also showcase several task-specific accelerations for point cloud, video and natural language processing by exploiting their spatial sparsity and temporal/token redundancy. Finally, to support all these algorithmic advancements, we introduce the efficient deep learning system design from both software and hardware perspectives.

1 Introduction

Efficient deep learning is needed to bring high-performing DNNs to resource-constrained devices where computation, latency, privacy, and sustainability impose practical barriers. The survey organizes solutions across compression, automation, on-device training, task-specific acceleration, and software/hardware systems.

Motivation: More than 10G MACs are required to process a 30-word translation sentence, while a LiDAR perception model needs more than 2000G MACs per second.These workloads exceed the capabilities of many mobile, vehicle, and IoT devices.
Motivation: Mobile and IoT devices have constrained hardware resources, while cloud delegation can violate latency or privacy requirements.Autonomous driving is latency-sensitive, and healthcare data can be privacy-sensitive.
Efficient inference: Inference efficiency uses pruning, low-rank factorization, quantization, and compact neural-network designs, but jointly tuning their many choices requires substantial human effort.Examples include MobileNets, ShuffleNets, and SqueezeNets.
On-device training: Efficient on-device training combines memory-efficient algorithms with transfer learning and federated learning to support local customization while aggregating models without compromising privacy.The cited approaches include gradient checkpointing, activation pruning, low-bit quantization, and federated learning.
System support: Algorithmic efficiency does not automatically produce measured hardware acceleration because general-purpose libraries and hardware may lack native support for sparse and low-bit computation.Specialized software and hardware systems help bridge this algorithm-to-implementation gap.
Scope and organization: The survey covers efficient deep learning from manual compression to AutoML, training to inference, algorithms to systems, and general-purpose to application-specific optimization.Its scope includes pruning, factorization, quantization, compact models, NAS, automated pruning and quantization, on-device training, and specialized acceleration.

2 Model Compression

Model compression reduces deep neural network size and computation through pruning, factorization, quantization, and knowledge distillation, while compact architectures redesign networks for efficiency.

2.1 Parameter Pruning: Pruning removes redundant network elements, reducing model size and computation through choices of granularity, importance criteria, and recovery training.Fine-grained, pattern-based, structured, and dynamic schemes trade compression, accuracy, and hardware friendliness differently.
2.2 Low-Rank Factorization: Low-rank factorization decomposes convolutional or fully connected weights, replacing expensive operations with lower-complexity approximations.Truncated SVD reduces fully connected multiplication from O(mk) to O(mt + tk) when t is much smaller than m and k; tensor decompositions target convolutional kernels.
2.3 Quantization: Quantization lowers weight and activation bit precision, reducing model size and potentially accelerating inference when hardware supports the resulting arithmetic.Post-training INT8 quantization usually causes minor or no accuracy loss, while quantization-aware training emulates deployment-time quantization to reduce accuracy loss.
2.4 Knowledge Distillation: Knowledge distillation transfers information from a large teacher to a smaller student through output logits, softened probabilities, or intermediate activations.Feature-map and attention-transfer methods require matching spatial resolutions between teacher and student models.

3 Automated Compression and Neural Architecture Search

This section surveys automated methods for compressing and designing neural networks, reducing reliance on costly hand-crafted exploration while accounting for accuracy and deployment efficiency. It covers NAS search spaces and algorithms, automated compression, joint optimization, empirical comparisons, and remaining limitations.

Automated Compression and Neural Architecture Search: Automated methods address the time-consuming, sub-optimal human tuning of model size, latency, energy, and accuracy trade-offs.The surveyed approaches automate compression and architecture decisions across large design spaces.
Automated Neural Architecture Design: NAS searches neural network architectures through a generator–evaluator loop, using design spaces that range from whole networks to reusable cells and hierarchical structures.Cell-based spaces reduce complexity independently of network depth and allow learned cells to transfer across datasets.
Automated Neural Architecture Design: NAS search efficiency is improved by restricting the search space and using sample-efficient algorithms, but conventional searches may still require 12,800 network evaluations and 10^4 GPU hours.Search methods include random search, reinforcement learning, evolutionary algorithms, and Bayesian optimization.
Automated Neural Architecture Design: Auto-designed CNNs save engineer labor and outperform human-designed models on ImageNet, object detection, and semantic segmentation, while MAC reduction alone may not predict hardware speed.NASNet-A and AmoebaNet-A use fewer MACs than MobileNetV2-1.4 but run slower on a Google Pixel 1.
Joint Compression and Neural Architecture Search: APQ jointly optimizes architecture and quantization, attaining 74.1% versus 71.8% accuracy under the same latency/energy constraint.It uses a once-for-all network and predictor transfer to reduce the cost of evaluating quantized models.
Limitations and Future Directions: Current AutoML methods mainly search restricted, human-designed spaces, limiting novel primitives and requiring hardware-specific spaces; broader spaces and domain-aware methods remain future directions.Specialized CNN design also becomes costly across many deployment scenarios because each case requires renewed design and training.

4 Efficient On-Device Learning

Efficient on-device learning addresses memory, energy, bandwidth, latency, and privacy constraints while enabling models to adapt to local data. The section surveys activation-reduction methods, transfer learning, and federated learning techniques for these settings.

On-device learning must overcome severe memory and energy constraints: MobileNetV2 training approaches 1GB, while DRAM access consumes two orders of magnitude more energy than on-chip SRAM.A Raspberry Pi 1 Model A has only 256MB of memory, and training activations often cannot fit in SRAM.
Training activations are the main memory bottleneck because they grow linearly with batch size and can reach 13.9× the parameter size in ResNet50 at batch size 16.Inference does not require storing all intermediate activations for back-propagation, whereas training does.
Gradient checkpointing reduces activation storage from O(n) to O(n/k) + O(k) by discarding intermediate activations and recomputing them during back-propagation.Choosing k = √n yields O(2√n) activation storage.
Other general training accelerations prune activations, quantize weights, activations, and gradients, or compress gradients to reduce computation, memory, and communication costs.Gradient compression is especially relevant when edge networks have lower bandwidth and higher latency than high-end distributed-training infrastructure.
With limited local data, transfer learning is preferable to training from scratch, but updating only the final layer can have limited capacity and poor accuracy under distribution shift.TinyTL instead freezes pretrained feature-extractor weights and updates biases to reduce training memory while addressing capacity loss.
TinyTL reduces transfer-learning memory from more than 250MB to 16MB, while federated learning supports collaborative training without explicitly sharing local data.Future efficient systems must jointly address SRAM-resident training, unstable Wi-Fi connections, latency, and data ownership.

5 Domain-Specific Optimization

The survey reviews task-specific acceleration for point clouds, videos, and NLP by exploiting spatial sparsity, temporal redundancy, and token redundancy. It reports efficiency results for 3D segmentation and describes neural architecture search and quantization approaches for video and language models.

3D Point Cloud Processing: Point-cloud convolution methods range from voxel-based and direct point-based processing to efficient designs using sparse or hierarchical representations.The general operation indexes neighboring points, applies a kernel to neighboring features, and produces an output for each center point.
3D Point Cloud Processing: KPConv and PVCNN achieve state-of-the-art accuracy with up to 2.7× speedup and 1.9× parameter reduction over PointCNN on ShapeNet part segmentation.MinkowskiNet is reported as strongest overall for indoor semantic segmentation, while KPConv is strongest among point-based methods.
3D Point Cloud Processing: SPVNAS achieves 3× measured speedup and 8× computation reduction over MinkowskiNet for outdoor 3D scene segmentation.Sparse-convolution methods have the best reported accuracy and efficiency in the summarized outdoor results.
Video Understanding: Video acceleration research applies NAS to temporal architectures, modality fusion, temporal resolutions, and spatial-temporal search spaces.The surveyed methods include EvaNet, AssembleNet, Tiny Video Networks, X3D, and PV-NAS.
Natural Language Processing: NLP efficiency methods quantize attention-based models to integer, ternary, binary, or adaptive floating-point formats, but lower precision can reduce BERT accuracy.The cited results report 3%–7% loss at 4-bit quantization and an additional 4%–13% loss at 2 bits across tasks.
Discussions and Future Directions: The section identifies future directions including hardware-aware point-cloud design, new video primitives and NAS, data-efficient video understanding, and deployable super-transformers for edge devices.It also highlights system-algorithm co-design for 3D modules and reduced training costs for deploying NLP models across devices.

6 Efficient System Design

The survey presents software and hardware system techniques for converting algorithmic efficiency into measured speed and energy gains. It covers specialized compilation, memory and dataflow optimization, sparse and low-precision hardware, heterogeneous placement, and joint neural-hardware design.

System Motivation: Algorithmic reductions in FLOPs and model size do not necessarily produce measured speedup or energy efficiency, motivating specialized software and hardware systems.The survey frames system design as the bridge between theoretical benefits and deployment performance.
Software Systems: Specialized software systems optimize deep-learning programs for hardware characteristics and include compilers, model-serving libraries, and domain-specific point-cloud implementations.Examples include TVM, TinyEngine, Kaolin, and PyTorch3D.
Hardware Systems: Domain-specific accelerators improve efficiency through data specialization, parallelism, optimized local memory, and reduced overhead.These techniques address the memory-dominated behavior of conventional CPU and GPU acceleration.
Hardware Support for Sparsity: Sparse accelerators reduce work and data movement by compressing sparse representations and skipping operations involving zero weights or activations.Examples include Eyeriss activation compression, EIE sparse matrix-vector multiplication, and SpArch sparse-sparse multiplication.
Hardware Support for Sparsity: Attention-specific accelerators exploit sparsity in query, key, and value vectors, which differs fundamentally from weight pruning because attention contains no weights.A3 sorts key-vector dimensions and selects extreme elements for query-key multiplication.
Hardware Support for Quantization: INT8 inference is supported across several mobile, CPU, GPU, and FPGA platforms, while NVIDIA Turing INT4 inference provides an additional 59% speedup over INT8.Quantization therefore depends on hardware support for low-precision arithmetic.
ML-Based Hardware Design: Neural and hardware architecture choices are entangled, with correlations that vary across hardware, so future systems should jointly optimize hardware, software mapping, and neural architectures.Current frameworks often omit lower-level hardware design and mapping strategies, and frequently rely on simulation rather than complete synthesis and layout results.

7 Conclusion

Efficient deep learning addresses the computational constraints that limit DNN deployment on edge devices. The paper surveys methods, applications, and software/hardware systems while identifying broader directions for co-designed acceleration.

Efficient deep learning is needed because DNN computational complexity exceeds the form-factor, battery, and heat-dissipation constraints of many edge devices.
The paper systematically surveys model compression, AutoML, on-device training, task-specific acceleration, and supporting software/hardware systems.
Automated methods such as neural architecture search, automated pruning, and automated quantization can reduce manual design effort and outperform handcrafted solutions.
Task-specific acceleration exploits spatial sparsity and temporal or token redundancy in point cloud, video, and natural language processing.
Future progress requires accelerating training and inference across software and hardware, including co-design that exploits their combined properties.