Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Motivation

• DNN workloads can exceed mobile-device capabilities: a state-of-the-art machine-translation model requires more than 10G MACs for a 30-word sentence, while a popular LiDAR model needs more than 2000G MACs per second (10 frames).Source: §1 · p. 2

Model compression and AutoML

• Inference efficiency uses pruning, low-rank factorization, quantization, and compact architectures, increasingly automated through NAS, automated pruning, and automated quantization.AutoML reduces manual design effort but can substantially increase carbon emissions, motivating efficient architecture search for greener AI.Source: §1 · p. 2

On-device training

• Efficient on-device training combines memory-efficient algorithms, transfer learning, and federated learning to support local user customization while preserving privacy.Relevant techniques include gradient checkpointing, activation pruning, low-bit quantization, and aggregation of users’ trained models without compromising privacy.Source: §1 · pp. 2–3

Application-specific acceleration

• Task-specific acceleration targets point clouds, videos, and natural language by exploiting spatial, temporal, and token-level redundancies.These workloads are computationally expensive because of their large memory footprints but offer distinct redundancy patterns for acceleration.Source: §1 · p. 3

Efficient systems

• Specialized software and hardware bridge the gap between algorithmic sparsity or quantization and measured acceleration through parallelism, graph optimization, memory scheduling, and native support.Their joint design enables resource assignment, hardware sizing, and neural-network–accelerator co-search across platforms.Source: §1 · p. 3

Survey scope

• The survey offers a holistic view spanning manual and automated methods, training and inference, algorithms and systems, and general-purpose and application-specific optimization.Its roadmap covers model compression, AutoML, on-device training, task-specific acceleration, and efficient software/hardware design.Source: §1 · pp. 3–4

2.1 Parameter Pruning

• Pruning removes redundant weights or structures, with granularity governing the trade-off between compression, accuracy, and hardware acceleration.Fine-grained pruning can preserve accuracy at high compression, whereas structured pruning is more hardware-friendly but may reduce accuracy; fine-tuning, iterative, dynamic, and sparsity-constrained training recover performance.Source: §2.1 · pp. 4–5

2.2 Low-Rank Factorization

• Low-rank factorization decomposes convolutional or fully connected layers into smaller components, reducing computation from O(mk) to O(mt + tk) for truncated SVD.Convolutional approaches include separable kernels, channel-factorized layers, CP decomposition, and Tucker decomposition, which can provide higher compression than SVD.Source: §2.2 · pp. 5–6

2.3 Quantization

• Quantization reduces bits per weight and can accelerate inference, trading model size and speed against accuracy through precision, rounding, and training choices.INT8 supports integer-only inference, post-training quantization usually causes minor or no accuracy loss, and data-free methods address privacy-sensitive settings without training data.Source: §2.3 · pp. 6–7

2.4 Knowledge Distillation

• Knowledge distillation transfers teacher outputs or intermediate representations to smaller students, improving compact models across classification and other applications.FitNet [245] and Attention Transfer [345] use intermediate activations but require matching spatial resolution, limiting student-model choices.Source: §2.4 · p. 7

2.5 Manual Neural Architecture Design

• Manual efficient architectures reduce convolutional cost using pointwise, group, and depthwise convolutions, exemplified by SqueezeNet, MobileNets, and ShuffleNets.MobileNetV2 [250] reaches 72.0% top-1 accuracy with 300M MACs and 3.4M parameters, while ShuffleNetV2 reaches 72.6% with 299M MACs on ImageNet.Source: §2.5 · pp. 8–9

2.6 Future Directions

• Compression based on pre-defined networks may miss hardware-specific efficiency, so co-design with compilers or hardware and architecture design from scratch can provide larger gains.The paper introduces Neural Architecture Search (NAS) as the next approach to automatic network architecture design.Source: §2.6 · pp. 9–10

3.1 Automated Model Compression

• Automated compression searches layer-specific pruning ratios and mixed-precision policies to reduce the time and suboptimality of manually tuning model size, speed, accuracy, and hardware constraints.AMC uses reinforcement learning for pruning, while HAQ and HAWQ automate quantization-policy selection using hardware feedback or Hessian spectra.Source: §3.1 · p. 10

3.2 Automated Neural Architecture Design

• NAS searches architectures through design spaces, algorithms, and performance predictors, but conventional searches can require 12,800 trained models and 104 GPU hours.Cell-based and hierarchical spaces, gradient-based methods, hypernetworks, and one-shot supernets reduce search cost, while RL, evolutionary, and gradient-based methods provide competitive results.Source: §3.2 · pp. 11–13
• Fewer #MACs do not guarantee faster hardware execution: NASNet-A and AmoebaNet-A run slower than MobileNetV2-1.4 despite having fewer MACs on Google Pixel 1.Hardware-aware NAS therefore incorporates measured or predicted latency alongside accuracy, accounting for data movement, parallelism, and element-wise operations.Source: §3.2 · pp. 13–14, Figure 7 · p. 13

3.3 Joint Compression and Neural Architecture Search

• 74.1% vs. 71.8%: APQ’s joint architecture, pruning, and quantization optimization achieves better accuracy than sequential optimization under the same latency/energy constraint.APQ uses an accuracy predictor, a once-for-all network, and predictor transfer to avoid expensive fine-tuning during joint search.Source: §3.3 · p. 16

3.4 Limitations and Future Directions

• Current AutoML methods mainly search restricted spaces derived from human-designed architectures, limiting discovery of novel operations and requiring hardware-specific spaces.Future directions include broader design spaces, stronger AutoML algorithms, and integration of domain-specific insights; training activation memory also remains a bottleneck.Source: §3.4 · pp. 16–17, Figure 10 · p. 17

4 Efficient On-Device Learning

• On-device training is constrained by memory and energy, with activations as the main bottleneck because they grow with batch size and must be stored for back-propagation.MobileNetV2 training requires close to 1GB at batch size 16, while ResNet50 activations are 13.9× its parameter size under the same batch size.Source: §4 · pp. 17–18

4.1 General Efficient Training Techniques

• General efficient training reduces activation or communication costs through gradient checkpointing, activation pruning, low-bit training, and gradient compression.Checkpointing reduces activation size from O(n) to O(n/k) + O(k), activation pruning removes input-dependent non-critical neurons, mixed precision reduces memory without accuracy loss, and DGC reduces transferred bits by up to 600× with warm-up and error compensation.Source: §4.1 · p. 18

4.2 Efficient Transfer Learning

• TinyTL enables data-efficient transfer learning by freezing pretrained weights, updating biases, and adding lite residual learning, matching full fine-tuning accuracy on Cars while reducing memory cost by 4.6×.The method targets training memory rather than only trainable-parameter count and uses generalized memory-efficient bias modules to recover capacity.Source: §4.2 · p. 19

4.3 Federated Learning

• Federated learning keeps local data on clients, while federated averaging, gradient compression, quantization, and delayed updates address low bandwidth, high latency, and unstable edge connectivity.The framework supports applications including keyboard suggestions and medical treatment analysis while preserving data ownership.Source: §4.3 · p. 19

4.4 Discussions and Future Directions

• Future efficient on-device learning should combine algorithms with hardware and software to fit training in SRAM, while addressing unstable Wi-Fi, privacy risks, and multi-device communication.TinyTL reduces transfer-learning memory from more than 250MB to 16MB, supporting the goal of faster and more energy-efficient in-memory training.Source: §4.4 · p. 19

5.1 Efficient Point Cloud Processing

• SPVNAS [281] achieves 3× measured speedup and 8× computation reduction over MinkowskiNet [56] for outdoor scene segmentation, while sparse-convolution methods provide the best accuracy and efficiency.For indoor segmentation, MinkowskiNet [56] achieves the best accuracy and efficiency, while KPConv [285] is best among point-based methods; KPConv [285] and PVCNN [197] achieve state-of-the-art part-segmentation accuracy with up to 2.7× speedup and 1.9× parameter reduction over PointCNN.Source: §5.1 · pp. 23–24

5.2 Efficient Video Recognition

• Efficient video recognition decomposes spatial and temporal modeling through separable convolutions, channel operations, shifts, and temporal aggregation, improving accuracy–speed trade-offs over conventional 3D CNNs.TSM models outperform I3D and ECO at equal computation per video, while scalable design enabled Kinetics training on 1,536 GPUs in 1 hour; NAS methods such as X3D and PV-NAS automate efficient video architectures.Source: §5.2 · pp. 25–26

5.3 Efficient Natural Language Processing

• NLP efficiency methods restrict or replace attention, use compression and architecture search, and reduce decoding iterations because Transformer-Big requires 13G MACs and 20 seconds on a Raspberry Pi for a 30-token sentence.Alternatives include convolutional, hashed, low-rank, recurrent, early-exit, and non-autoregressive designs; Linformer reduces attention complexity from O(n^2) to O(n), while quantization can reduce BERT accuracy by 3%–7% at 4 bits and an additional 4%–13% at 2 bits.Source: §5.3 · pp. 26, 28, 29
• NLP efficiency exploits language redundancy through token/head activation pruning and progressive attention-input quantization, reducing computation and DRAM access while adapting precision to attention distributions.SpAtten [298] accumulates token influence and head magnitude across layers, then fetches additional low-order bits only when attention probabilities require them.Source: §5.3 · p. 29
• Hardware-aware NAS is necessary because FLOPs cannot reflect measured latency and different hardware prefers different Transformer architectures; HAT [296] searches heterogeneous designs under target-hardware latency constraints.Source: §5.3 · p. 29
• Additional automated NLP compression methods include AdaBert [38] for task-adaptive BERT models, AutoEmb [354] for popularity-based embedding sizes, and AttentionNAS [362] for efficient attention cells.AdaBert [38] combines differentiable supernet-based NAS with knowledge distillation, while AttentionNAS [362] searches a broad operation space.Source: §5.3 · p. 30

5.4 Discussions and Future Directions

• Future NLP research targets One-For-All Transformers, sparse and quantized pre-training, lower-latency generation, and cross-stack algorithm–hardware co-design.The proposed directions aim to deploy one super transformer across edge devices, reduce large-corpus pre-training costs, accelerate GPT-2 generation, and improve efficiency across the stack.Source: §5.4 · p. 30

6 Efficient System Design

• Specialized software and hardware systems bridge the gap between theoretical algorithmic benefits and measured speedup and energy efficiency.These systems also create an orthogonal design space for jointly optimizing algorithms with software and hardware.Source: §6 · p. 30

6.1 Software System

• Software systems span general training, inference, acceleration, and serving libraries, while specialized compilers and runtimes automate graph, memory, hardware, and domain-specific optimizations.TVM, FlexFlow, and TASO automate optimization, with MetaFlow and TASO achieving up to 3x speedup over existing DNN frameworks; TinyEngine targets memory-constrained microcontrollers.Source: §6.1 · pp. 30–31

6.2 Hardware System

• Hardware accelerators improve efficiency through data specialization, parallelism, optimized local memory, reduced overhead, and dataflows that exploit reuse or sparsity.Sparse designs compress data and skip zero-valued operations, while quantization hardware supports low-precision and mixed-precision arithmetic for deployment acceleration.Source: §6.2 · pp. 31–33
• Neural and hardware architectures should be co-designed because their design spaces are deeply entangled, while current machine-learning-based methods often omit low-level hardware, compiler mappings, or complete design-cycle validation.NAAS searches PE connectivities and compiler mappings, achieving 4.88× energy-delay-product improvement and 2.7% top-1 accuracy improvement on ImageNet over Eyeriss running ResNet50; future work should automate broader hardware-software co-design and synthesis-to-layout stages.Source: §6.2 · p. 34, §6.2.1 · p. 35

• Efficient deep learning addresses DNN computational complexity that limits AI deployment on edge devices constrained by form factor, battery, and heat dissipation.Source: §7 · p. 35
• The paper systematically covers compression, automated design, on-device training, task-specific acceleration, and efficient software/hardware systems for practical deep learning.Covered methods include pruning, factorization, quantization, efficient model design, neural architecture search, automated pruning and quantization, local-data customization, and exploitation of spatial or temporal/token redundancy.Source: §7 · p. 35
• Future work should pursue extreme efficiency from cloud AI to mobile and tiny AI across training, inference, software, and hardware.Source: §7 · p. 35
• Algorithm, software, and hardware co-design should exploit the unique properties of different application domains for further optimization.Source: §7 · p. 35