PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Network Inference on Mobile SoCs

Siqi Wang, Anuj Pathania, Tulika Mitra

arXiv:1908.11450v2cs.LGcs.DCstat.ML

TL;DR

Mobile devices increasingly need to run neural-network inference on-device, while fragmented SoCs make accelerator deployment difficult. The paper characterizes component power-performance behavior and uses roofline analysis to examine optimization scope. It reports substantial NPU energy efficiency alongside portability and compatibility challenges, while high-end GPUs offer a more general option.

  • Problem

    On-device neural inference matters for mobile applications, but SoC fragmentation and limited accelerator availability complicate deployment across devices.

    Source: §I · p. 1

  • Method

    The paper characterizes inference across mobile SoC components and uses roofline analysis to examine network performance and optimization scope.

    Source: §I · p. 1, A. Construction of a Roofline Model · p. 4

  • Results

    The NPU provides unmatched inference energy efficiency, while high-end GPUs can deliver comparable performance at satisfactory energy efficiency.

    Source: 4) Insights · p. 3

  • Takeaways & Limitations

    NPU deployment demands effort with proprietary APIs and limited network compatibility, whereas portable OpenCL support makes high-end GPUs a general option when performance matters.

    Source: 4) Insights · p. 3

    NPU portability and network compatibility are limited, and its proprietary, black-box interface can constrain development and optimization.

    Source: 4) Insights · p. 3

Abstract

from arXiv · show

The ever-increasing demand from mobile Machine Learning (ML) applications calls for evermore powerful on-chip computing resources. Mobile devices are empowered with heterogeneous multi-processor Systems-on-Chips (SoCs) to process ML workloads such as Convolutional Neural Network (CNN) inference. Mobile SoCs house several different types of ML capable components on-die, such as CPU, GPU, and accelerators. These different components are capable of independently performing inference but with very different power-performance characteristics. In this article, we provide a quantitative evaluation of the inference capabilities of the different components on mobile SoCs. We also present insights behind their respective power-performance behavior. Finally, we explore the performance limit of the mobile SoCs by synergistically engaging all the components concurrently. We observe that a mobile SoC provides up to 2x improvement with parallel inference when all its components are engaged, as opposed to engaging only one component.

I. INTRODUCTION

Mobile vision inference must meet high accuracy and throughput on power-constrained devices, while heterogeneous SoCs and fragmented hardware complicate deployment. The paper characterizes component capabilities and proposes engaging SoC components simultaneously.

I. INTRODUCTION

  • Mobile inference is increasingly needed for richer user experience and privacy, but must provide high accuracy and throughput on-device.Vision applications use real-time video and CNNs, including in ADAS, VR, and AR, despite substantial compute and memory demands.Source: §I · p. 1
  • Fragmented mobile SoCs make accelerator-based applications difficult to develop across devices, while CPUs remain common and favored for inference.Source: §I · p. 1
  • The paper quantitatively characterizes mobile SoC inference, explains the power-performance gap between ubiquitous CPUs and high-performance accelerators using roofline models, and proposes simultaneous component engagement.Source: §I · p. 1

A. Heterogeneous Multi-processor SoCs

Mobile SoCs vary widely in their CPU, GPU, and accelerator composition, but share CPU clusters that support broad availability. The section also describes accelerator deployment challenges and CPU optimization opportunities.

A. Heterogeneous Multi-processor SoCs

  • Over two thousand mobile SoCs differ in CPUs, GPUs, caches, memory controllers, and accelerators, while sharing one or more CPU clusters.This diversity makes standard optimizations difficult.Source: A. Heterogeneous Multi-processor SoCs · p. 1
  • ARM big.LITTLE pairs high-performance and energy-efficient CPU clusters that share an ISA, and CPUs' general availability favors device-agnostic inference optimizations.The passage reports that 99.9% of Android devices in 2019 had multiple cores.Source: 1) ARM big.LITTLE · pp. 1–2

2) Accelerators:

  • GPU and dedicated accelerators can offer strong ML runtime and energy efficiency, but the lack of standard mobile neural accelerators makes cross-device application integration difficult.Accelerator availability is limited, and this also constrains GPU use.Source: 2) Accelerators · p. 2

B. Mobile ML Framework and Optimizations

III. CHARACTERIZING INFERENCING ON MOBILE SOC

The study compares inference on two mobile SoCs from different technology nodes and years, using CPU, GPU, and NPU components where available. It describes the platforms and measurement setup.

III. CHARACTERIZING INFERENCING ON MOBILE SOC

  • The experiments compare 28 nm Exynos 5422 and 10 nm Kirin 970 SoCs, released in 2014 and 2017, respectively, as approximations of mid- and high-end mobile SoCs.Source: §III · p. 2
  • Both SoCs use ARM-CL 18.05v; Kirin NPU deployment uses HiAI DDK v100, and power is measured with on-chip sensors for Exynos or socket measurements for Kirin.Kirin's lack of integrated on-chip sensors necessitates the socket-power measurement.Source: §III · p. 2

1) CPU:

  • Both SoCs use ARM big.LITTLE CPUs; Kirin 970 has Cortex-A73 and A53 clusters, while Exynos 5422 has older Cortex-A15 and A7 clusters.All CPU cores support NEON SIMD operations.Source: 1) CPU · p. 2
  • Kirin 970 includes a Mali G72 GPU and Huawei NPU, whereas Exynos 5422 includes a Mali T628 GPU and no ML accelerator.The Kirin NPU's HiAI DDK deployment API works only with Android.Source: 1) CPU · p. 2, 3) NPU · p. 2

4) Network Structure:

The study tests several popular CNNs and compares inference throughput across heterogeneous components and two mobile SoCs. Results show generation-dependent performance differences and deployment constraints for the NPU.

4) Network Structure:

B. Individual Heterogeneous Components

1) Throughput:

  • The newer Kirin 970 components outperform their Exynos 5422 counterparts by average factors of 4.4x for Big CPUs, 2.6x for Small CPUs, and 4.2x for GPUs.Source: 1) Throughput · p. 2
  • The Big-to-Small CPU performance gap falls from 4x to 2.5x as the Big-to-Small power-consumption ratio drops from 10x to 4x.Source: 1) Throughput · pp. 2–3
  • The GPU-to-CPU-cluster performance gap is about 2x to 3x on both SoCs, and the NPU is only 1.6x better than the high-end G72 GPU on average.MobileNet could not be deployed on the NPU because of incompatible operators; proprietary tooling also complicates portability and optimization.Source: 1) Throughput · p. 3

2) Energy Efficiency:

The section compares component energy efficiency across mobile SoCs, showing that newer designs improve performance and often efficiency, while small-cluster power can limit efficiency gains.

2) Energy Efficiency:

  • NPU is the most energy-efficient component, GPUs rank second, and small clusters are efficient but have too little absolute throughput to be useful alone.Source: 2) Energy Efficiency · p. 3
  • 4.4x and 2.6x throughput improvements occur across generations for the Big and Small clusters, respectively, alongside architectural changes beyond frequency scaling.The cited comparison covers the 28 nm Exynos 5422 and 10 nm Kirin 970 platforms.Source: 3) Impact of Technology Scaling Versus… · p. 3
  • Small-cluster energy-efficiency improvement is limited for some networks: A53 consumes roughly twice the power of A7, offsetting technology scaling.A53's higher power is attributed to increased area and micro-architectural changes.Source: 3) Impact of Technology Scaling Versus… · p. 3
  • A73 is more power-efficient than A15, while measured A53 and A73 power is higher because memory power could not be separated.Source: 3) Impact of Technology Scaling Versus… · p. 3

4) Insights:

The section weighs the portability and power-performance trade-offs of mobile inference components, then uses roofline analysis to characterize compute- and memory-bound behavior.

4) Insights:

  • NPU offers unmatched inference energy efficiency, but proprietary APIs, black-box behavior, and limited network compatibility demand porting effort and constrain development.The authors note these requirements could make the NPU obsolete for future networks.Source: 4) Insights · p. 3
  • High-end GPUs can match NPU performance at satisfactory energy efficiency, while OpenCL supports portability across many GPUs and compatible CPUs.Source: 4) Insights · p. 3
  • CPUs have the worst throughput and energy efficiency, yet remain critical because they are common across mobile devices and inference on them is inevitable.Source: 4) Insights · p. 3
  • Any single component can barely meet rising inference performance requirements; co-execution can mitigate this, while network-level optimization remains necessary.Source: 4) Insights · p. 3

IV. ROOFLINE ANALYSIS

  • Roofline analysis classifies applications as memory- or compute-bound; because the two SoCs show similar CPU-cluster and GPU behavior, only Exynos 5422 is presented.Source: §IV · p. 4

A. Construction of a Roofline Model

The roofline model uses theoretical and empirical operational intensity to relate workload behavior to hardware limits; empirical intensity better reflects real-world performance.

A. Construction of a Roofline Model

  • 3.44 GB/s, 0.49 GB/s, and 6.15 GB/s are the observed peak bandwidths for the A15 cluster, A7 cluster, and T628 GPU, versus a specified memory-bus peak of 14.9 GB/s.Source: A. Construction of a Roofline Model · p. 4
  • Theoretical OI (OIt) is calculated from code, while OIe uses actual DRAM accesses measured with application-specific performance counters.OIe models the memory hierarchy and correlates better with component performance than OIt.Source: A. Construction of a Roofline Model · p. 4

B. Theoretical and Empirical OI

  • Theoretical OI points fall in the memory-bound region, whereas OIe points show achieved performance below the roofline; cache shifts OIe rightward by reducing DRAM accesses.For fully connected layers, points move along the roofline; convolutional layers become compute-bound but gain less because of insufficient or imperfect parallelization.Source: B. Theoretical and Empirical OI · p. 4
  • Empirical operational intensity (OIe) is a better indicator of real-world performance because it reflects DRAM accesses and cache effects; subsequent plots therefore use OIe.Source: B. Theoretical and Empirical OI · p. 4, A. Construction of a Roofline Model · p. 4

C. Across Different Components

Roofline analysis shows that network performance varies between compute- and memory-bound behavior across components and layers. Differences in memory hierarchy and layer characteristics help explain deviations from the roofline.

C. Across Different Components

  • Memory severely bottlenecks the A7 cluster and T628 GPU, while A15 performance spans compute- and memory-bound regions depending on the network.The component comparison appears in Fig. 3(b).Source: C. Across Different Components · p. 4, Figure 3 · p. 5
  • Different memory hierarchies produce different empirical operational intensities; the Big cluster's 2 MB L2 cache benefits more than the GPU's 128 KB L2 cache.AlexNet's large parameter set can flush caches regardless of cache size, while small filters can underutilize parallelism.Source: C. Across Different Components · p. 4

D. Major Layers in Inference

  • On the Big cluster, AlexNet's early convolutional layers are compute-bound, while its fully connected layers are memory-bound.The analysis examines major layers because convolutional and fully connected layers dominate network execution time.Source: D. Major Layers in Inference · p. 4, Figure 3 · p. 5
  • Layers within a network can occupy both compute- and memory-bound regions, motivating per-layer DVFS and fine-grained layer co-execution.The authors link these optimizations to power management and improved overall chip utilization, respectively.Source: D. Major Layers in Inference · p. 5, Figure 3 · p. 5

E. Effect of Quantization

The quantized MobileNet implementation reduces convolution latency but gains no overall performance because conversion overhead outweighs the benefit. The NPU analysis finds memory bottlenecks in tested networks, while its confidential design details limit the analysis to reverse-engineered behavior.

E. Effect of Quantization

  • Quantized MobileNet in ARM-CL 18.05v shows no overall performance improvement because de-quantization and re-quantization overheads overshadow reduced convolution latency.The implementation uses a QASYMM8 model with 8-bit weights.Source: E. Effect of Quantization · p. 5

F. Glimpse of NPU

  • Because NPU architectural and integration details are confidential, the authors can only attempt to reverse engineer its behavior.Source: F. Glimpse of NPU · p. 5
  • The NPU's performance is significantly memory-bounded for the networks tested, leaving scope to optimize toward its full processing potential.The authors construct an NPU roofline using observed memory bandwidth and the specified peak performance.Source: F. Glimpse of NPU · p. 5

A. Co-Execution of Multiple Components

The authors parallelize image inference across SoC components using a unified stream and work stealing. Co-execution improves throughput over the best isolated component, with NPU-inclusive execution offering exceptionally high throughput on Kirin 970.

A. Co-Execution of Multiple Components

  • Stream processing may require 10 to 40 images per second, and the authors report that Kirin 970 can barely sustain this while Exynos 5422 cannot.Source: A. Co-Execution of Multiple Components · p. 5
  • A unified image stream is processed in parallel by multiple components using work stealing, with a buffer reordering out-of-sync outputs.Co-execution achieves significantly higher throughput than the highest-throughput isolated component.Source: A. Co-Execution of Multiple Components · p. 6
  • On average, co-execution with CPU clusters and GPU improves throughput by 50% over GPU-only execution, though Exynos 5422 remains very low in absolute throughput.Table II compares co-execution with the best individual component, the GPU, on both platforms.Source: A. Co-Execution of Multiple Components · p. 6

B. Co-execution with NPU

  • Kirin 970 achieves exceptionally high throughput when all on-chip components, including the NPU, co-execute.The authors describe NPU and GPU parallel execution as an option for high-performance applications or simultaneous inferences.Source: B. Co-execution with NPU · p. 6, Table III · p. 6

C. Co-Execution Energy Efficiency

The paper reports up to 2x higher network throughput when all SoC components co-execute. Co-execution also improves energy efficiency over the Big CPU cluster, with Kirin 970 co-execution outperforming its power-efficient Small CPU cluster.

C. Co-Execution Energy Efficiency

  • Co-execution energy efficiency is always better than the Big CPU cluster, although engaging multiple components increases power consumption.The comparison covers all components on Exynos 5422, CPU and GPU on Kirin 970, and all Kirin 970 components including the NPU.Source: C. Co-Execution Energy Efficiency · p. 6
  • On Kirin 970, co-execution is more energy-efficient than the power-efficient Small CPU cluster.Source: C. Co-Execution Energy Efficiency · p. 6

VI. SUMMARY

  • Up to 2x network throughput improvement is possible when all SoC components engage in inference simultaneously.Source: §VI · p. 6

Related papers

Loading 1908.11450v2…