PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Benchmark: All About Deep Learning on Smartphones in 2019

Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, Luc Van Gool

arXiv:1910.06663v1cs.PFcs.LG

TL;DR

Mobile AI accelerators and deployment frameworks have advanced rapidly, but fragmented hardware interfaces and software reliability complicate inference on smartphones. The paper benchmarks mobile chipsets against desktop CPUs and GPUs, analyzes Android ML tooling, and reports substantially higher mobile performance across recent generations.

  • Problem

    Mobile AI hardware and Android inference tooling require broader comparative evidence because NPUs remain fragmented and NNAPI has design flaws that slow adoption and usage growth.

  • Method

    The paper benchmarks Qualcomm, HiSilicon, Samsung, MediaTek, and Unisoc chipsets using 21 deep-learning tests covering more than 50 AI-performance aspects, and compares them with desktop CPUs and GPUs.

  • Results

    More than 7.5× and 3.5× increases in floating-point and quantized inference speed over second-generation NPUs brought recent flagship SoCs to performance equivalent to or above entry-level CUDA-enabled desktop GPUs and high-end CPUs.

  • Takeaways & Limitations

    The paper concludes that forthcoming mid-range and high-end chipsets should have enough power to run most standard deep-learning models on mobile devices.

  • Takeaways & Limitations

    TensorFlow Lite releases still contained serious NNAPI, output, and memory-consumption problems, so developers were advised to test models extensively.

Abstract

from arXiv · show

The performance of mobile AI accelerators has been evolving rapidly in the past two years, nearly doubling with each new generation of SoCs. The current 4th generation of mobile NPUs is already approaching the results of CUDA-compatible Nvidia graphics cards presented not long ago, which together with the increased capabilities of mobile deep learning frameworks makes it possible to run complex and deep AI models on mobile devices. In this paper, we evaluate the performance and compare the results of all chipsets from Qualcomm, HiSilicon, Samsung, MediaTek and Unisoc that are providing hardware acceleration for AI inference. We also discuss the recent changes in the Android ML pipeline and provide an overview of the deployment of deep learning models on mobile devices. All numerical results provided in this paper can be found and are regularly updated on the official project website: http://ai-benchmark.com.

1. Introduction

Mobile AI progressed from compact, manually engineered models toward widespread deep-learning deployment as smartphones gained specialized hardware and software support. The paper benchmarks this ecosystem across mobile chipsets, desktop hardware, and Android ML tools.

  • Before 2010, mobile intelligent-processing methods relied mainly on manually designed features and compact models constrained by single-core 600 MHz Arm CPUs and 8–128 MB of RAM.
  • After 2010, multi-core processors, GPUs, DSPs, and NPUs made mobile devices more suitable for machine- and deep-learning tasks.
  • Deep-learning models now support mobile computer-vision and natural-language tasks including classification, enhancement, super-resolution, tracking, recognition, translation, and language understanding.
  • Figure 1 compares image throughput for float Inception-V3 under different mobile and desktop inference stacks, using FP16 on mobile devices and FP32 on Intel and Nvidia hardware.The caption warns that Inception-V3 is relatively small, so Nvidia’s advantage may be larger for bigger models.
  • TensorFlow Mobile enabled unmodified models on mobile CPUs in 2015, while TensorFlow Lite added hardware acceleration in 2017 and later supported vendor and GPU delegates.These changes connected mobile deployment more closely with the standard machine-learning environment.
  • The benchmark evaluates 21 deep-learning tests and more than 50 AI-performance aspects across Qualcomm, HiSilicon, Samsung, MediaTek, and Unisoc hardware, comparing mobile devices with desktop CPUs and GPUs.The updated benchmark also reports results from more than 200 Android devices and 100 mobile SoCs collected in the wild.

2. Hardware Acceleration

Mobile AI hardware evolved from limited or specialized acceleration toward broader NNAPI support across successive SoC generations. The section surveys architectures and SDKs from major vendors, while noting substantial implementation and accessibility constraints.

  • Generational overview: 2017-era mobile SoCs often required special SDKs or GPU libraries, while newer generations added Android NNAPI support and broader model acceleration.Generation 1 lacked operating-system AI acceleration; Generation 2 supported NNAPI but might accelerate only float or quantized models, whereas Generation 3 supported all model types.
  • Samsung chipsets / EDEN SDK: Samsung’s Exynos 9820 introduced an NNAPI-compatible NPU supporting only quantized inference, with two cores containing 1024 MAC units.Its NPU includes a controller, CPU, DMA unit, code SRAM and network controller.
  • Samsung chipsets / EDEN SDK: 2.1 TOPS and 6.9 TOPS are reported for 5×5 kernels with 0% and 75% zero weights, respectively, on the Exynos 9820 NPU.The corresponding energy efficiencies are 3.6 TOPS/W and 11.5 TOPS/W; Inception-V3 efficiency ranges from 2 TOPS/W to 3.4 TOPS/W with network sparsity.
  • HiSilicon chipsets / HiAI SDK: Huawei’s Da Vinci architecture uses a 3D Cube computing engine for matrix computations and powers the Kirin 810 and Kirin 990 AI accelerators.Huawei’s HiAI update supports more than 300 deep learning operations on the latest Kirin 810 and Kirin 990 5G SoCs.
  • Qualcomm chipsets / SNPE SDK: Qualcomm’s AI Engine combines Hexagon DSP, Adreno GPU and Kryo CPU cores, assigning floating-point models to the GPU and quantized inference to the DSP.The Snapdragon 855 introduced a Hexagon 690 DSP with a 1024-bit SIMD, twice as many pipelines and an additional tensor accelerator unit.
  • MediaTek chipsets / NeuroPilot SDK: 1.1 TMACs / second is the Helio P90 APU 2.0 peak computational power, four times higher than the preceding Helio P60/P70 series.The APU supports FP16, INT16 and INT8 inference and accelerates convolutions, fully connected layers, activations, pooling and other tensor operations.
  • Unisoc chipsets / UNIAI SDK: Unisoc’s Tiger T710 was its first smartphone chipset with a dedicated NPU, including separate accelerator cores for integer and floating-point models.The integer core supports INT4, INT8 and INT16 formats; the passage identifies the platform as targeting the mid-range segment.
  • Google Pixel 3 / Pixel Visual Core: Google’s Pixel Visual Core could accelerate deep learning operations, but its lack of released SDK or NNAPI drivers limited access to Google.In Pixel phones, it was used for selected HDR photography and real-time sensor-data processing tasks.

3. Deep Learning on Smartphones

The Android mobile deep-learning ecosystem gained broader hardware and software support, but fragmentation, update delays, unsupported operations, and model resource demands remain important constraints. Floating-point and quantized inference present complementary accuracy, speed, memory, and energy trade-offs.

  • Android ML ecosystem: TensorFlow Lite replaced TensorFlow Mobile and added delegates for executing supported model sub-graphs through external libraries.Unsupported operators remain on the CPU, and transfers between delegated sub-graphs and the main graph can increase inference time.
  • Android ML ecosystem: NNAPI 1.2 added 56 operators, per-axis weight quantization, IEEE Float 16, runtime accelerator queries, burst execution, compilation caching, and vendor extensions.These changes broaden supported models and provide more control over accelerator selection and repeated execution.
  • Deployment constraints: Mobile NPUs remain fragmented, while NNAPI adoption is slowed by OS-bundled updates and limited custom-operation support.New drivers may take up to a year to arrive, and unsupported operations can force CPU execution that ruins performance.
  • Deployment constraints: More than 6-8 gigabytes of RAM and enormous computational resources required by some high-resolution models are unavailable on even the latest high-end smartphones.Such models must first be modified to fit mobile hardware resources.
  • Floating-point versus quantized inference: Converting 16-bit floating-point models to int-8 reduces size and RAM consumption by a factor of 4 and potentially speeds execution by 2-3 times.Integer computation can also reduce energy use, but reducing weights from 16 to 8 bits may cause accuracy loss ranging from small degradation to accuracy close to zero.
  • Floating-point versus quantized inference: Floating-point models generally provide better accuracy, whereas quantized models provide faster inference; the suitable choice depends on task and hardware or energy constraints.Float models suit complex models and high-performance devices, while quantized inference benefits low-power, low-RAM, quantization-friendly models.
  • Floating-point versus quantized inference: Comparisons between devices should not mix floating-point inference on one device with quantized inference on another because the resulting performance numbers have different accuracy properties.Such comparisons are meaningful only when contrasting inference types alongside their corresponding accuracy results.

4. AI Benchmark 3.0

AI Benchmark 3.0 expanded the mobile benchmark into a modular, accuracy-aware suite covering 11 sections and 21 tests. It measures diverse inference modes and reports detailed timing, variability, accuracy, initialization, memory, and configurable PRO Mode results.

  • Benchmark evolution: AI Benchmark 3.0.0 increased the number of tests from 11 to 21 and introduced a modular architecture, accuracy checks, new tasks and networks, PRO mode, and an updated scoring system.Versions 3.0.1 and 3.0.2 fixed bugs and added PRO Mode features.
  • Benchmark design: Version 3.0.2 consists of 11 test sections and 21 tests using popular deep-learning architectures deployable on smartphones.The suite includes image classification, face recognition, Atari, deblurring, super-resolution, bokeh simulation, segmentation, enhancement, and memory tests.
  • Accuracy evaluation: Starting with version 3.0.0, accelerated float and quantized outputs in Test Sections 1, 2, 3, 5, and 6 receive separate L1-loss accuracy estimates.The loss is computed between target and actual model outputs.
  • Benchmark design: The benchmark evaluates int-8, float-16, float-32, parallel, CPU, memory, and model-initialization performance categories.Individual tests may process images sequentially across inference modes, run models in parallel, or continue until an out-of-memory error.
  • Scoring: Category scores use geometric means, NNAPI runtimes receive an exponential L1-error penalty with exponent 1.5, and memory results contribute multiplicatively.Normalization coefficients are based on the best results of the current SoC generation.
  • PRO Mode: PRO Mode compares CPU and NNAPI execution across five inference types where both float and quantized models are NNAPI-compatible.It reports single-image and throughput inference time, standard deviation, output accuracy, and model initialization time.
  • PRO Mode: PRO Mode additionally supports automatic execution, JSON or TXT result export, a TFLite CPU backend for debugging, and sustained performance mode.The added options were introduced under the Settings tab in version 3.0.1.

14. ICNet [90] [image segmentation]

The benchmark reports inference-time results for floating-point networks on mobile SoCs with fp-16 hardware acceleration, with desktop hardware included for reference. Results from this benchmark version are available online, and future releases will support unified mobile–desktop rankings.

  • Table 2 reports per-image inference times for floating-point networks on mobile SoCs providing fp-16 hardware acceleration.Snapdragon 835, Intel CPUs, and Nvidia GPUs are included as reference platforms.
  • Intel CPU acceleration used Intel MKL-DNN, while Nvidia GPU acceleration used CUDA and cuDNN.
  • The benchmark version’s results are available on the project webpage.
  • Upcoming releases will provide a unified ranking system for directly comparing mobile-device and desktop results.

5. Benchmark Results

The benchmark compares mobile AI accelerators across floating-point and quantized inference, including desktop CPU/GPU references. Results show major generational gains, strong NPU performance, and important software-driver effects.

  • Benchmark scope: Over 20,000 mobile devices were benchmarked alongside desktop CPUs and GPUs to estimate smartphone-versus-desktop deep-learning runtime.The comparison was added to assess how far mobile AI silicon had progressed and help developers estimate runtime relationships.
  • Floating-point performance: More than 7.5 times faster floating-point execution raised mobile results from 14% to 100% relative to the benchmark reference.The increase was reported between the best publicly available result at the end of September 2018 and later mobile chips.
  • Floating-point performance: The Kirin 990 approached the GeForce GTX 950 and significantly outperformed the Intel Core i7-9700K in floating-point inference.The GTX 950 is identified as a 2015 mid-range desktop GPU, while the i7-9700K is an octa-core Intel CPU.
  • GPU versus NPU: Third-generation GPU- and NPU-based accelerators showed roughly equal performance, while the Snapdragon 855 Plus was around 10-15% faster.Reaching fourth-generation NPU performance would require GPUs to increase inference speed by 2-3 times, likely requiring major micro-architectural changes.
  • GPU versus CPU: GPU inference consumed 2 to 4 times less power than CPU inference, although mid-range and older GPUs were inefficient in resulting speed.This trade-off can make GPU acceleration advantageous in overall energy efficiency despite lower speed.
  • Software effects: Custom vendor OpenCL drivers increased Exynos 9820 performance from 6% to 26%, achieving up to 10 times speed-ups for many architectures.Default Arm NN OpenCL drivers did not fully exploit Mali GPU potential; similar considerations applied to Qualcomm NNAPI drivers.
  • Quantized performance: Snapdragon 855 Plus and Unisoc Tiger T710 led quantized inference with nearly identical int-8 results, running 15-20% faster than several competing SoCs.The comparison includes the Kirin 990, Helio P90, and standard Snapdragon 855 among the slower alternatives.

6. Discussion

Mobile AI hardware has advanced substantially, reaching performance comparable to entry-level desktop GPUs and high-end CPUs. Software progress is more incremental, with TensorFlow Lite remaining central but affected by critical bugs and issues.

  • Hardware progress: Floating-point and quantized inference became more than 7.5 and 3.5 times faster, respectively, than on second-generation NPUs.The paper describes this as bringing smartphone AI capabilities to a substantially higher level.
  • Hardware progress: All flagship SoCs released during the preceding 12 months matched or exceeded entry-level CUDA-enabled desktop GPUs and high-end CPUs.Fourth-generation mobile AI silicon delivered still higher performance.
  • Future outlook: The authors project that mid-range and high-end chipsets will gain enough power within two-three years to run most standard deep-learning models.They associate this projection with more AI projects targeting mobile devices as the main deployment platform.
  • Software ecosystem: TensorFlow Lite remains the only major mobile deep-learning library combining reasonably high functionality, deployment ease, and a large developer community.The discussion characterizes software-stack progress as evolutionary rather than revolutionary.
  • Software ecosystem: Critical bugs and issues in new TensorFlow Lite versions prevent the authors from recommending it for some use cases.

7. Conclusions

The paper reviews Android mobile AI hardware, software-pipeline changes, benchmark modifications, and inference results across major chipset vendors. It compares mobile results with desktop systems and discusses future deployment recommendations.

  • Contributions: The paper surveys mobile chipsets that can accelerate neural-network execution on smartphones and other portable devices.
  • Contributions: It overviews recent changes in the Android machine-learning pipeline and describes modifications to the current AI Benchmark release.
  • Contributions: Floating-point and quantized inference results are reported for Qualcomm, HiSilicon, Samsung, MediaTek, and Unisoc chipsets.
  • Contributions: The paper compares mobile results with desktop CPU and GPU results to characterize the relationship between these hardware platforms.
  • Contributions: It discusses future hardware and software development and gives recommendations for deploying deep-learning models on smartphones.
Loading 1910.06663v1…