Source-linked AI summary
AI Benchmark: Running Deep Neural Networks on Android Smartphones
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, Luc Van Gool
TL;DR
Deep-learning algorithms remain computationally demanding on smartphones, while existing mobile AI support is fragmented and difficult to assess objectively. The paper surveys Android frameworks, programming interfaces, chipset acceleration resources, and limitations, then evaluates real devices with AI Benchmark. Its benchmark covers major Android hardware configurations and reports performance across more than 10,000 smartphones and tablets, while the analysis identifies substantial platform and software limitations.
Problem
Deep-learning workloads challenge mobile hardware, while AI benchmarks do not measure the acceleration available from mobile AI chips and DSPs.
Method
The paper surveys Android machine-learning frameworks, NNAPI, chipset acceleration resources, and deep-learning tests, then applies AI Benchmark to real smartphones and SoCs.
Results
AI Benchmark reports real-world deep-learning performance from over 10,000 mobile devices and more than 50 mobile SoCs, with Huawei Kirin 970 devices showing the fastest conventional float-neural-network runtime at the time.
Takeaways & Limitations
Android AI deployment must account for differences among frameworks, hardware accelerators, chipset drivers, and device limitations.
Takeaways & Limitations
Android 8.1 default NNAPI drivers can make convolutional operations twice as slow, so affected phones were omitted from average SoC runtimes.
Abstract
from arXiv · showhide
Over the last years, the computational power of mobile devices such as smartphones and tablets has grown dramatically, reaching the level of desktop computers available not long ago. While standard smartphone apps are no longer a problem for them, there is still a group of tasks that can easily challenge even high-end devices, namely running artificial intelligence algorithms. In this paper, we present a study of the current state of deep learning in the Android ecosystem and describe available frameworks, programming models and the limitations of running AI on smartphones. We give an overview of the hardware acceleration resources available on four main mobile chipset platforms: Qualcomm, HiSilicon, MediaTek and Samsung. Additionally, we present the real-world performance results of different mobile SoCs collected with AI Benchmark that are covering all main existing hardware configurations.
1 Introduction
Android devices have become powerful enough for demanding applications, yet deep-learning workloads remain difficult because they impose heavy computational and battery costs. The paper reviews mobile AI software and hardware and introduces a benchmark for measuring on-device deep-learning performance.
- Motivation: Deep-learning workloads remain challenging on high-end smartphones and tablets despite major increases in mobile SoC capabilities.Current devices combine multicore processors, dedicated GPUs, and gigabytes of RAM, but AI algorithms still exceed many mobile platforms’ practical capacity.
- Motivation: Running machine-learning models on phone CPUs creates substantial computational overhead and drains battery power.
- Application scope: Mobile deep-learning applications span computer vision tasks such as classification, enhancement, super-resolution, recognition, tracking, and scene understanding, alongside natural-language processing.
- Motivation: Many AI algorithms are unavailable on smartphones or run remotely, introducing privacy, connectivity, latency, and server-capacity concerns.
- Existing approaches: Manufacturer SDKs accelerate neural networks on selected Qualcomm, HiSilicon, and MediaTek hardware, but their chipset-specific design limits portability.
- Paper contribution: The AI Benchmark measures machine-learning performance, hardware accelerators, chipset drivers, and memory limitations through computer-vision tests executed directly on Android devices.The study reports in-the-wild results for about 200 Android devices and major chipsets, covering over 10,000 smartphones and tablets.
2 Hardware Acceleration
Mobile AI acceleration combines heterogeneous CPU, GPU, DSP, NPU, and APU resources with platform-specific and unified software interfaces. The section contrasts chipset architectures and shows that hardware access and NNAPI support remain uneven across devices.
- Mobile acceleration foundations: DSPs remained important in mobile devices because they often delivered better performance at lower power consumption than CPUs and GPUs.They evolved from voice and radio processing toward image, video, and sound workloads.
- Qualcomm: Snapdragon combines Arm CPU cores with NEON, Adreno GPUs, and programmable Hexagon DSPs as a heterogeneous architecture for AI workloads.SNPE provides runtime acceleration across Snapdragon processing components and supports multiple model frameworks.
- HiSilicon: Kirin 970 introduced a specialized NPU for vector- and matrix-based AI computation, while later Kirin 980 chipsets integrated enhanced NPUs.Huawei reports up to 25 times better performance and 50 times greater efficiency than the standard quad-core Cortex-A73 CPU cluster; the peak figure of about 1.92 TFLOPs mainly refers to quantized 8-bit computation.
- MediaTek: MediaTek’s Helio P60 integrates an APU delivering up to 280GMAC/s for 8-bit computations, while float models run on Cortex-A53 CPUs and a Mali GPU.NeuroPilot is built around TensorFlow Lite and Android NNAPI and provides tools, interpreters, and accelerated APU/GPU operations.
- Unified software access: Google’s NNAPI provides a unified Android C API, allowing higher-level machine-learning frameworks to invoke hardware-accelerated inference through device-specific drivers.The model is represented as a directed computation graph combined with model data, and proprietary SDKs remain chipset-specific and incompatible.
3 Deep Learning Mobile Frameworks
Android supports several deep-learning frameworks, with TensorFlow Mobile offering broad model compatibility and TensorFlow Lite targeting smaller, faster deployment but retaining substantial preview-stage limitations.
- Android ports include Torch, Deeplearning4j, TensorFlow Mobile, TensorFlow Lite, Caffe, Caffe2, MXNet and NNabla; TensorFlow Mobile, TensorFlow Lite and Caffe2 are most commonly used.
- TensorFlow represents models as directed computational graphs whose nodes define operators and dataflow from inputs to outputs.Trained models can be exported as .pb graphs for mobile execution.
- TensorFlow Mobile supports most standard TensorFlow operations, enabling nearly any TensorFlow model to be converted for mobile execution.SNPE, HiAI, NeuroPilot and ArmNN provide partial hardware-acceleration support for TensorFlow Mobile.
- TensorFlow Lite uses optimized kernels, pre-fused activations and fewer dependencies to provide better performance and a smaller binary size than TensorFlow Mobile.Models are converted to .tflite format using FlatBuffers serialization and can run on Android or iOS.
- TensorFlow Lite remains limited by incomplete operator support, restricted officially guaranteed models, potentially 2× higher inference time and RAM, and stability issues.The limitations particularly affect image-to-image translation problems and some mobile devices.
- Caffe2 uses a programming model similar to TensorFlow, with static computational graphs and operator-representing nodes.
4 AI Benchmark
AI Benchmark evaluates mobile deep-learning execution through nine Android computer-vision tests spanning recognition, restoration, segmentation, enhancement and memory capacity. Its design combines representative deployable architectures with TensorFlow Lite, NNAPI acceleration where supported, and CPU-based execution otherwise.
- The AI Benchmark is an Android application that measures performance and memory limitations while running neural networks directly on mobile devices.Its models represent popular architectures currently deployable on smartphones.
- The benchmark version 2.0.0 contains nine deep-learning tests.
- Recognition: Tests 1 and 2 classify ImageNet images with MobileNet-V1 and Inception-V3, contrasting a resource-efficient mobile model with a larger, more accurate network.MobileNet-V1 performs about 569M multiply-add operations, whereas Inception-V3 requires about 5,000M and occupies around 96MB.
- Recognition: Test 3 retrieves the most similar face using Inception-Resnet-V1 features trained with triplet loss on VGGFace2.
- Deployment scope: The first three classification models form a core set suitable for mobile deployment, while faster models generally sacrifice accuracy and more precise models exceed 100–150MB.Quantization can partly reduce the size problem, but quantized versions were not publicly available in the described setting.
- Image processing: Tests 4–8 cover deblurring, super-resolution, semantic segmentation and image enhancement using SRCNN, VDSR, SRGAN, ICNet and DPED-derived models.SRGAN uses adversarial training for perceptual restoration, while ICNet targets fast segmentation on low-performance devices and DPED requires device-specific training.
- Memory: Test 9 increases SRCNN input resolution until a memory exception occurs, measuring the largest processable image rather than runtime.
- Technical implementation: TensorFlow Lite is the backend for all embedded models; tests 1, 2, 4, 5, 8 and 9 use NNAPI-supported operations, while tests 3, 6 and 7 run entirely on CPU.CPU-only tests include operations absent from NNAPI, and missing NNAPI drivers make the first group fall back to CPU with Arm NEON.
5 Benchmark Results
AI Benchmark results show substantial but uneven hardware acceleration across Android SoCs, with performance shaped by model type, memory, drivers, and chipset-specific support. The benchmark also exposes important variability and software limitations affecting interpretation.
- Benchmark scope: Over 10,000 devices were evaluated using average per-image processing time, maximum SRCNN resolution, and aggregated AI scores after outlier removal.The benchmark covers computer-vision tests executed directly on device hardware.
- Neural-network results: Quantized MobileNet uses about 3–4× less RAM than its float counterpart and is generally an order of magnitude faster on CPU than Inception-V3.The benchmark results broadly match theoretical expectations for processing time and memory consumption.
- Qualcomm: Snapdragon 845 runs quantized MobileNet under 25ms on its Hexagon DSP versus 60–65ms on CPU, while commercial Qualcomm devices do not yet support float-model acceleration.Potential float acceleration on Adreno GPUs is reported as an estimate rather than an official test.
- Huawei: Kirin 970’s NPU delivers 7–21× faster float inference than its CPU and 4–7× better performance than the best overall CPU results, but lacks quantized-model acceleration.For supported tests, average per-image times are 132, 274, 240, and 193 milliseconds; quantized processing in the first test averages 160ms on CPU.
- MediaTek: MediaTek’s Helio P60 accelerates quantized models on its APU at 21ms per image and float models on its GPU at 2–5× CPU acceleration, whereas other MediaTek chipsets lack AI acceleration.The reported Helio P60 figures come from developer phones, while the Vivo V11 performs slightly worse.
- Samsung: Samsung provides no third-party AI acceleration, and Exynos 8895/9810 benchmark scores can vary by up to 50% across repeated runs and 200–800ms across devices.The variation is attributed to integrated power-management logic that automatically selects performance modes.
6 Discussion
Android AI software and hardware are evolving rapidly, but inconsistent support and unresolved quantization and power-efficiency questions complicate objective assessment and deployment decisions.
- Rapid milestone releases and missing standardized requirements or public specifications hinder objective assessment of mobile AI advantages and limitations.
- TensorFlow Mobile remains the easiest mature Android entry point, while TensorFlow Lite is an alternative when hardware acceleration is critical.
- Small communities around Caffe2 Mobile and other less widespread frameworks leave users with few tutorials and often require GitHub issue creation for problems.
- Kirin 970 currently shows the fastest conventional float-neural-network runtime, but nearly all SoC manufacturers may achieve similar results in newer chipsets.
- Significant battery drain could restrict accelerated AI to a few standard in-camera processing techniques, making power efficiency important alongside performance.
- Quantized-network applicability remains limited because standard, reliable tools are unavailable even for image-classification models.
7 Conclusions
The paper surveys Android AI hardware and software, introduces a benchmark for deep-neural-network performance, and analyzes results from a broad set of mobile devices and SoCs.
- AI Benchmark measures performance aspects of deep-neural-network execution on Android devices using results from over 10,000 devices and more than 50 mobile SoCs.
- The paper discusses future software and hardware development and provides recommendations for deploying deep-learning models on Android devices.