Source-linked AI summary
JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services
Amir Erfan Eshratifar, Mohammad Saeed Abrishami, Massoud Pedram
TL;DR
Mobile and cloud-only DNN execution each impose important latency, energy, communication, or resource constraints, especially for inference and adaptive training. JointDNN partitions computation at layer granularity, optimizes schedules with graph and ILP formulations, and uses lossless output compression. The paper reports substantial improvements in latency, mobile energy consumption, and cloud workload over status-quo approaches.
Problem
Mobile-only and cloud-only DNN execution face complementary latency, energy, communication, congestion, and resource constraints, while cloud-only training can impose continuing communication and mobile-energy costs.
Method
JointDNN partitions DNN layers between mobile and cloud, formulates scheduling at layer granularity using shortest-path and ILP models, and applies lossless compression to CNN layer outputs.
Results
JointDNN significantly improves latency, mobile energy consumption, and cloud workload compared with status-quo single-platform approaches, with up to 3× latency and 7× energy improvements without compression.
Takeaways & Limitations
Collaborative layer-level execution can provide an alternative to single-platform DNN inference and training while adapting schedules to mobile battery, cloud congestion, and QoS constraints.
Takeaways & Limitations
Optimal constrained scheduling is NP-Complete, and accurate analytical latency and energy modeling requires exact hardware and software specifications that are often unavailable.
Abstract
from arXiv · showhide
Deep learning models are being deployed in many mobile intelligent applications. End-side services, such as intelligent personal assistants, autonomous cars, and smart home services often employ either simple local models on the mobile or complex remote models on the cloud. However, recent studies have shown that partitioning the DNN computations between the mobile and cloud can increase the latency and energy efficiencies. In this paper, we propose an efficient, adaptive, and practical engine, JointDNN, for collaborative computation between a mobile device and cloud for DNNs in both inference and training phase. JointDNN not only provides an energy and performance efficient method of querying DNNs for the mobile side but also benefits the cloud server by reducing the amount of its workload and communications compared to the cloud-only approach. Given the DNN architecture, we investigate the efficiency of processing some layers on the mobile device and some layers on the cloud server. We provide optimization formulations at layer granularity for forward- and backward-propagations in DNNs, which can adapt to mobile battery limitations and cloud server load constraints and quality of service. JointDNN achieves up to 18 and 32 times reductions on the latency and mobile energy consumption of querying DNNs compared to the status-quo approaches, respectively.
1 INTRODUCTION
DNN applications increasingly require substantial computation, while mobile-only and cloud-only execution each face latency, energy, communication, or resource constraints. JointDNN addresses these trade-offs by partitioning layers adaptively between mobile and cloud and optimizing communication and scheduling.
- Motivation: DNN advances improve applications including computer vision, speech recognition, language modeling, and autonomous cars, but increasing computational complexity raises resource requirements.The paper situates collaborative execution among GPU development, distributed training, model compression, and cloud–fog computation.
- Motivation: Mobile-only execution can incur high inference latency and energy use, whereas cloud-only execution adds communication overhead, congestion, subscription costs, and network dependence.These limitations motivate computation across both platforms rather than relying on a single platform.
- Motivation: Cloud-only training communicates input data and model parameters, while collaborative training communicates layer outputs and only a portion of model parameters, potentially reducing transferred data.The issue is especially relevant to adaptive learning, where model parameters change through continuous environmental interaction.
- JointDNN approach: Prior collaborative DNN work used one offloading point, whereas JointDNN permits each layer to execute independently on either the mobile device or cloud server.The approach models the DNN as an ordered layer sequence and adapts scheduling to hardware, battery, network, and QoS conditions.
- JointDNN approach: JointDNN formulates layer-granular computation scheduling as a shortest-path problem and integer linear programming, covering inference and training under resource constraints.The paper also investigates lossless compression of CNN layer outputs to reduce communication costs while preserving model accuracy.
- Contributions and results: JointDNN improves latency, mobile energy consumption, and cloud workload relative to status-quo single-platform approaches.Without compression, the paper reports improvements of up to 3× in latency and 7× in energy.
2 PROBLEM DEFINITION AND MODELING
This section presents the DNN-layer architecture and profiling method, then models cost optimization as a shortest-path problem and constrained scheduling as integer linear programming.
- Architecture and profiling: The modeling section explains DNN layer architecture and the profiling method used to estimate execution costs.It frames profiling as the basis for the subsequent scheduling optimization.
- Graph optimization: JointDNN reduces cost optimization to a shortest-path problem using a graph model.The graph represents alternative computation schedules across mobile and cloud resources.
- Constrained optimization: Constrained scheduling is formulated with integer linear programming to represent limitations on the optimization problem.The section introduces ILP after presenting the shortest-path formulation.
2.1 Energy and Latency Profiling
The paper compares approaches for estimating DNN layer latency and energy, emphasizing grouped-operator profiling and application-specific runtime profiling. Analytical and statistical approaches face accuracy, specification, or measurement-cost challenges.
- Three approaches measure DNN layer latency and energy: statistical modeling, analytical modeling, and application-specific profiling.
- Statistical modeling requires many measurements and a complex regression model across consecutive operator combinations and input sizes.
- Grouped execution of consecutive convolution operators can differ substantially from summing separately measured operators, producing larger errors as operator count increases.The experiments used 25 consecutive convolutions with separate versus grouped execution while keeping parameters in GPU memory.
- Analytical modeling requires exact hardware and software specifications and can substantially underestimate layer-level delay, including by around 900% for a 4096-neuron fully connected layer.
- JointDNN uses application-specific runtime profiling because mobile neural-network applications are generally limited in number and this method promises more accurate estimates.
2.2 JointDNN Graph Model
JointDNN models layer-granular mobile-cloud execution as a graph whose path selects where consecutive layers run and accounts for computation and transfer costs. Shortest-path scheduling supports inference and online training, while constrained variants motivate ILP formulations and lookup tables.
- The model represents a linear-topology DNN as sequential layers with input and output sizes and layer-specific latency or energy costs.The initial linear formulation is later replaced because consecutive layer grouping must be considered.
- JointDNN replaces the linear topology with a tournament graph to represent all consecutive groupings of layers on the mobile device or cloud server.
- Graph nodes C_i:j and M_i:j represent layers i through j computed on the cloud and mobile, respectively.
- Edges encode mobile-to-mobile, cloud-to-cloud, mobile-to-cloud, and cloud-to-mobile transitions with computation and, when needed, upload or download costs.The transition costs are represented as ME_i,j, CE_i,j, EU_i,j = ME_i,j + UID_j+1, and ED_i,j = CE_i,j + DOD_j+1.
- Shortest-path solving obtains inference schedules, while online training mirrors the network for backpropagation and adds cloud weight-download costs.
- Constrained shortest-path scheduling is NP-Complete, so JointDNN targets optimal solutions through lookup tables for parameter settings such as bandwidth and cloud load.
2.3 ILP Setup
JointDNN formulates layer-granular inference and training scheduling as ILP, modeling computation placement, communication costs, and application-specific constraints. The formulation supports residual architectures, mobile-energy objectives, and fast schedule generation for changing parameters.
- Performance-efficient computation offloading: Inference scheduling profiles consecutive layer groups and assigns binary variables for mobile or cloud computation.The formulation includes N(N + 1)/2 consecutive layer groups and variables representing each placement choice.
- 47?
3 EVALUATION
The evaluation uses common DNN application types and describes JointDNN’s optimal-scheduling procedure for layer-partitioned computation.
- Benchmark selection: Three common DNN application types are selected according to application-dependent neural-network architectures.The benchmarks include discriminative models, generative models, and autoencoders.
- Benchmark selection: Discriminative benchmarks cover classification, regression, and speech recognition, including AlexNet, OverFeat, VGG16, Deep Speech, ResNet, and NiN.Except for Deep Speech, the listed benchmarks are used for image classification.
- Scheduling procedure: JointDNN takes the DNN layer count, layers, per-layer data sizes, network bandwidth, and mobile communication power as inputs.Its output is an optimal DNN schedule.
- Scheduling procedure: The engine profiles grouped layer placements and constructs a shortest-path graph representing candidate schedules.The graph uses start and finish nodes for the scheduling model.
10 else
The scheduling procedure applies optimization routines when battery or cloud-server constraints are present, and also includes an energy-efficient optimization formulation.
- Constraint handling: When the battery-limited constraint applies, JointDNN invokes a performance-efficient integer linear programming formulation.The formulation receives the DNN and communication parameters used by the scheduling procedure.
- Constraint handling: When the cloud-server constraint applies, JointDNN invokes the same performance-efficient integer linear programming formulation.This branch addresses the cloud-server condition in the scheduling procedure.
- Constraint handling: JointDNN also includes an energy-efficient integer linear programming formulation for schedule optimization.This formulation uses the same DNN and communication parameters.
24 end
The evaluation uses specified mobile and server hardware, benchmark model classes, and measured mobile-network characteristics to model computation and communication.
- Benchmarks: Table 2 presents the benchmark specifications used in the evaluation.The passage identifies Table 2 as the source of benchmark specifications.
- Benchmarks: The benchmark suite includes generative models and autoencoders alongside the discriminative models described elsewhere.Pix2Pix is used as the autoencoder benchmark; autoencoder applications include reconstruction, translation, and denoising.
- Experimental platforms: The mobile platform is an NVIDIA Jetson TX2 with a Pascal GPU, 256 CUDA cores, and shared 8 GB LPDDR4 memory.An INA226 power sensor measures mobile-platform power consumption.
- Experimental platforms: The server platform uses an NVIDIA K40C GPU with 12 GB of memory and over an order of magnitude more computation capability than the mobile device.The passage identifies the K40C as the server GPU.
- Communication model: Communication modeling uses average U.S. mobile-Internet upload and download speeds across 3G, 4G, and Wi-Fi networks.These network specifications are presented in Table 3.
- Communication model: Download and upload communication power are modeled as linear functions of network throughput with less than 6% error.Table 3 provides the parameter values for the download and upload equations for different networks.
4 RESULTS
JointDNN improves latency, energy consumption, and cloud workload by adaptively partitioning DNN computation across mobile and cloud resources. Results vary with batch size, architecture, communication cost, and layer-output compression.
- Inference and training: 66% and 86% improvements in latency and energy consumption, respectively, are achieved during inference across eight benchmarks.The comparison uses the best mobile-only and cloud-only cases as baselines.
- Inference and training: As batch size increases, communication cost grows linearly while computation cost grows at a much lower rate, making mobile-only increasingly preferable.
- Inference and training: Online training avoids downloading large updated weights by computing only a few back-propagation steps in the cloud.As the percentage of updated weights increases, latency and energy consumption become constant.
- Inference and training: 73% latency and 56% energy-consumption improvements are reported during inference.
- Architecture-dependent scheduling: Scheduling patterns depend on architecture: discriminative models commonly use mobile-cloud inference and mobile-cloud-mobile training, while autoencoders place first and last layers on mobile.Generative models may reverse the pattern because their last layers can have large data sizes.
- Cloud workload: 84% workload reduction and 53% average reduction are reported for the cloud server.JointDNN moves part of the computation to mobile devices.
- Communication and compression: 99%, 93%, and 81% of AlexNet execution time are communication costs over 3G, 4G, and Wi-Fi, respectively.The same relative portion applies to energy consumption in the cloud-only approach.
- Communication and compression: PNG compresses layer data by up to 5.8× and 3.5× on average, producing extra 4.9× energy and 4.6× latency improvements on average.Compression ratio correlates highly with the ratio of zero-valued neurons, which ranges from 50% to 90% after ReLU.
5 RELATED WORK AND COMPARISON
Prior offloading frameworks generally depend on programmer annotations or function-level decisions, while JointDNN schedules DNN layers at run time using topology and system conditions. Compared with Neurosurgeon, it supports richer schedules, constraints, and model architectures.
- General task offloading frameworks: Existing general task-offloading frameworks rely on programmer annotations and pre-specified functions, limiting DNN computation partitioning.
- General task offloading frameworks: Function-level offloading can miss optimal decisions because same-type layers may have different computation and data characteristics across models.
- Comparison with Neurosurgeon: Neurosurgeon assumes one transfer point with mobile-first execution, whereas JointDNN reports mobile-cloud-mobile, multi-transfer, and cloud-first schedules.These patterns occur especially in online training, generative models, and autoencoders.
- Comparison with Neurosurgeon: JointDNN additionally adapts to battery limits, cloud congestion, QoS, inter-layer optimizations, and complex architectures such as ResNet and RNNs.Neurosurgeon targets latency and energy for simpler CNN and ANN models.
7 CONCLUSIONS AND FUTURE WORK
The paper formulates mobile-cloud DNN partitioning as a graph optimization problem and provides constrained formulations for practical system limitations. It identifies architecture-dependent schedules and leaves recurrent-network extensions for future work.
- Conclusions: Cloud-only and mobile-only execution are not optimal for latency and energy, motivating collaborative DNN partitioning.
- Conclusions: DNN computation partitioning is reduced to a shortest-path problem, while added battery, congestion, and QoS constraints make it NP-Complete and motivate ILP formulations.
- Conclusions: Discriminative, generative, and autoencoder architectures imply different layer-placement patterns based on their input, output, and intermediate data sizes.
- Future work: JointDNN formulations target feed-forward networks; extending them to recurrent neural networks is future work.