Source-linked AI summary
Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
Zongqing Lu, Swati Rallapalli, Kevin Chan, Thomas La Porta
TL;DR
CNNs achieve strong vision accuracy but are difficult to deploy on mobile devices because their growing complexity stresses time and memory. The paper measures layerwise inference on mobile CPUs and GPUs, models CNN resource requirements, and builds Augur to estimate execution costs from a model descriptor. Augur is intended to show whether and how efficiently a CNN can run before implementation and deployment, while its accuracy is limited by omitted memory operations and architectural dependencies.
Problem
The paper addresses how to understand CNN inference time and memory requirements on mobile devices despite constrained resources and diverse CNN and platform configurations.
Source: §1 · p. 1Method
The authors deploy popular CNNs on mobile CPUs and GPUs, measure and profile layerwise behavior, model matrix-multiplication costs, and build Augur from these measurements.
Source: §1 · pp. 1–2, §7 · p. 8Results
Augur estimates CNN compute time and memory usage from a model descriptor, providing guidance on mobile execution without implementation and deployment.
Source: §5.3 · p. 8, §7 · p. 8Takeaways & Limitations
Augur can support CNN design for resource-constrained mobile devices by allowing designers to estimate resource usage and tune models before deployment.
Source: §5.3 · p. 8Augur currently emphasizes matrix multiplication and needs additional factors, including memory operations and stacked or branched CNN architectures, for more precise estimates.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) have revolutionized the research in computer vision, due to their ability to capture complex patterns, resulting in high inference accuracies. However, the increasingly complex nature of these neural networks means that they are particularly suited for server computers with powerful GPUs. We envision that deep learning applications will be eventually and widely deployed on mobile devices, e.g., smartphones, self-driving cars, and drones. Therefore, in this paper, we aim to understand the resource requirements (time, memory) of CNNs on mobile devices. First, by deploying several popular CNNs on mobile CPUs and GPUs, we measure and analyze the performance and resource usage for every layer of the CNNs. Our findings point out the potential ways of optimizing the performance on mobile devices. Second, we model the resource requirements of the different CNN computations. Finally, based on the measurement, pro ling, and modeling, we build and evaluate our modeling tool, Augur, which takes a CNN configuration (descriptor) as the input and estimates the compute time and resource usage of the CNN, to give insights about whether and how e ciently a CNN can be run on a given mobile platform. In doing so Augur tackles several challenges: (i) how to overcome pro ling and measurement overhead; (ii) how to capture the variance in different mobile platforms with different processors, memory, and cache sizes; and (iii) how to account for the variance in the number, type and size of layers of the different CNN configurations.
1 INTRODUCTION
This paper examines how CNN inference can be measured, modeled, and evaluated on resource-constrained mobile devices. It combines layerwise deployment measurements with resource modeling and the Augur estimation tool.
1 INTRODUCTION
- The study focuses on inference, or the forward pass, because mobile devices are expected to perform vision inference while training remains offline on powerful GPUs.Source: §1 · p. 1
- Mobile deployment is motivated by constrained memory and computing capability, while CNNs such as VGGNet require 528M parameters and over 15G FLOPs per image.Existing approaches include parameter compression, cloud offload, and distributing computation across heterogeneous processors.Source: §1 · p. 1
- The modeling approach addresses profiling overhead by amortizing repeated measurements and abstracts diverse CNN layers through their shared matrix-multiplication core.Source: §1 · pp. 1–2
- The authors deploy AlexNet, VGGNet, GoogleNet, and ResNet on NVIDIA TK1 and TX1 CPUs and GPUs, then measure resource usage layer by layer.Source: §1 · p. 2
- Augur accepts a CNN model descriptor and estimates compute time and resource usage to assess mobile execution without implementing and deploying the model.Source: §1 · p. 2
2 BACKGROUND
The background describes CNN layer computations, mobile resource constraints, and prior efforts to accelerate or benchmark CNN inference without modeling its resource requirements.
2 BACKGROUND
- Convolutional layers dominate CNN computation and can be formulated as matrix multiplication by reshaping local input regions and kernels into matrices.This formulation underlies CPU BLAS and GPU cuBLAS implementations, although input replication increases memory use.Source: §2.1 · p. 2
- Pooling downsamples spatial dimensions, BatchNorm normalizes each input dimension during testing using learned statistics, and ReLU applies a parameter-free elementwise function without changing tensor size.Source: §2.1 · p. 2
- Fully connected layers impose heavy memory and computation costs, while newer GoogleNet and ResNet models reduce this burden by using only one fully connected classifier layer.GoogleNet has 26MB of parameters compared with 233MB in AlexNet.Source: §2.1 · p. 2
- Prior mobile-CNN work targets parameter compression, cloud offload, heterogeneous processors, and memory reduction, while existing benchmarking does not model CNN resource requirements.Source: §2.2 · pp. 2–3
- Because mobile platforms perform differently across CNNs, whether custom optimization or offloading is needed depends on the CNN and the target platform.Source: §2.2 · p. 3
3 MEASUREMENT SET-UP
The measurement setup evaluates representative CNNs on two mobile developer kits using Caffe, comparing CPU and GPU inference across architectures with different depths and connectivity patterns.
3 MEASUREMENT SET-UP
- The study uses NVIDIA TK1 and TX1 developer kits to measure CNN inference on both mobile CPUs and GPUs.TK1 has a 2.3GHz quad-core ARM Cortex-A15 CPU, 192 CUDA cores, and 2GB RAM; TX1 has a 1.9GHz quad-core ARM Cortex-A57 CPU, 256 CUDA cores, and 4GB RAM.Source: §3 · p. 3
- Caffe with OpenBLAS is used because common CNN frameworks typically rely on BLAS for CPUs and cuBLAS for GPUs.Source: §3 · p. 3
- The benchmark models are AlexNet, VGG-16, GoogleNet, and ResNet-50, spanning architectures from several layers to more than one hundred layers and from stacked to branched designs.Source: §3 · p. 3
4 INITIAL MEASUREMENT STUDY
The study measures CNN inference time and memory layer by layer on mobile CPUs and GPUs, revealing hardware- and architecture-dependent bottlenecks and measurement artifacts.
4.1 Timing
- 600 ms on TK1 CPU versus about 900 ms on TX1 CPU for AlexNet, despite TX1 having the more powerful CPU.TX1 executes convolution layers more than 4x faster but fully connected layers more than 3x slower than TK1.Source: §4.1 · p. 3
- GPUs significantly accelerate CNN computation over CPUs, but GPU layerwise timing exceeds full-pass timing because synchronization introduces measurement overhead.The mismatch is especially pronounced for models with many layers, such as GoogleNet and ResNet.Source: §4.1 · p. 4
- 55% of TX1 CPU time is consumed by GoogleNet’s LRN, while ResNet is faster than GoogleNet on TX1 GPU despite having twice as many FLOPs.ResNet benefits from larger matrix multiplications, which have higher GPU throughput.Source: §4.1 · pp. 4–5
4.2 Memory
- CNN memory use comprises parameters, intermediate data, and computation workspace, with convolution and fully connected layers supplying most parameters.cuDNN can trade GPU speed for less workspace, but workspace is not the dominant memory component.Source: §4.2 · p. 5
- TX1 uses more memory than TK1 because its 64-bit operating system increases framework overhead, while mobile GPU execution can duplicate parameters under unified memory.For AlexNet, TK1 GPU uses 560 MB, 236 MB more than TK1 CPU, while weights and biases total 233 MB.Source: §4.2 · p. 5
4.1 Timing
- FLOPs alone cannot accurately predict CNN compute time because memory operations and matrix-multiplication sizes also determine throughput.ResNet is faster than GoogleNet on GPUs despite requiring more FLOPs.Source: §4.3 · p. 5, §4.1 · p. 5
5 AUGUR
Augur models CNN resource requirements from matrix multiplications, layer configurations, and mobile-platform measurements, enabling pre-deployment estimates of compute time and memory. Its models capture CPU linearity, more complex GPU behavior, and achieve over 78% estimation accuracy for NIN and VGG19M across CPUs and GPUs.
5 AUGUR
- Augur parses CNN descriptors, derives layer dimensions and matrix multiplications, then estimates minimal memory and compute time without implementation or deployment.It includes data, parameters, and workspace in memory estimates and uses platform-specific matrix-multiplication models.Source: §5 · p. 5, §5.1 · pp. 5–6, §5.3 · p. 8
- The model does not fully account for dependencies among parallel branches, such as GoogleNet’s four-branch inception components.Handling these dependencies is identified as future work.Source: §5.1 · p. 7
- More than 60% of CNN compute time comes from matrix multiplications in most mobile-platform cases, supporting matrix-based compute-time prediction.The share varies by architecture and platform; GoogleNet is an exception in some configurations because of its branched inception components.Source: §5.1 · pp. 6–7
5.2 Modeling
- CPU matrix-multiplication time scales linearly with n, m, and k, enabling a linear model for each CPU device.This behavior is reported for both TK1 and TX1 CPUs.Source: Figure 2 · p. 6, Figure 3 · p. 6, §5.2 · p. 7
- GPU matrix-multiplication time has more complex relationships with n, m, and k, including bipartite behavior in n and piecewise linear behavior in m.The n=96 artifact is associated with CUDA-core parallelization and platform-specific matrix-size preferences.Source: §5.2 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 7
5.3 Accuracy
- Augur’s matrix-multiplication-only estimates approximate NIN and VGG19M compute time with more than 78% accuracy across CPUs and GPUs.Reported accuracy reaches up to 94% on CPUs and up to 84% on GPUs, because matrix multiplications generally occupy a larger CPU compute-time share.Source: Figure 7 · p. 7, Figure 8 · p. 7, §5.3 · p. 8
6 DISCUSSION
Augur currently estimates CNN compute time primarily through profiled matrix multiplication, but additional memory and architectural factors are needed for greater precision. The framework should also be tailored to mobile-platform characteristics.
6 DISCUSSION
- Matrix multiplication accounts for more than 90% of CNN FLOPs and commonly dominates compute time, making it Augur’s current basis for estimating CNN compute time.This focus also allows Augur to extend to additional mobile platforms by profiling matrix multiplication operations on each platform.Source: §6 · p. 8
- More precise Augur estimates require accounting for memory operations and CNN architectures, including stacked or branched designs.Source: §6 · p. 8
- Mobile CNN frameworks should optimize inference and reflect platform characteristics such as unified memory architecture.Source: §6 · p. 8
7 CONCLUSION
The paper models CNN resource requirements on mobile devices through layerwise measurements and matrix-multiplication profiling. It packages these results in Augur, which estimates compute time and memory usage before implementation or deployment.
7 CONCLUSION
- Augur estimates a CNN’s compute time and memory usage to assess whether and how efficiently it can run on a mobile platform.Source: §7 · p. 8
- The study measures and analyzes CNN inference performance and resource usage at layerwise granularity on mobile CPUs and GPUs.Source: §7 · p. 8
- The modeling approach profiles matrix multiplications because they form the core computations of CNNs.Source: §7 · p. 8