PaperlayerView exact revision on arXiv

Source-linked AI summary

JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution

Hongshan Li, Chenghao Hu, Jingyan Jiang, Zhi Wang, Yonggang Wen, Wenwu Zhu

arXiv:1812.10027v1cs.DC

TL;DR

JALAD addresses the latency and resource limitations of cloud-only or edge-only deep-network deployment by splitting inference between edge and cloud. It compresses intermediate feature maps, optimizes and adapts the partition, and reports substantial latency reductions with bounded accuracy loss.

  • Problem

    Cloud-only deployment transfers large amounts of data to centralized servers, while edge devices lack the resources to run full deep models efficiently.

  • Method

    JALAD compresses intermediate feature maps, formulates partition selection as latency minimization under an accuracy constraint, and dynamically re-decouples networks as conditions change.

  • Results

    JALAD reduces execution latency by up to 90% in real-world experiments while keeping accuracy loss within 10%.

  • Takeaways & Limitations

    Edge-cloud deep-network decoupling can reduce execution latency when feature-map compression and accuracy-aware partitioning address intermediate-data amplification.

Abstract

from arXiv · show

Recent years have witnessed a rapid growth of deep-network based services and applications. A practical and critical problem thus has emerged: how to effectively deploy the deep neural network models such that they can be executed efficiently. Conventional cloud-based approaches usually run the deep models in data center servers, causing large latency because a significant amount of data has to be transferred from the edge of network to the data center. In this paper, we propose JALAD, a joint accuracy- and latency-aware execution framework, which decouples a deep neural network so that a part of it will run at edge devices and the other part inside the conventional cloud, while only a minimum amount of data has to be transferred between them. Though the idea seems straightforward, we are facing challenges including i) how to find the best partition of a deep structure; ii) how to deploy the component at an edge device that only has limited computation power; and iii) how to minimize the overall execution latency. Our answers to these questions are a set of strategies in JALAD, including 1) A normalization based in-layer data compression strategy by jointly considering compression rate and model accuracy; 2) A latency-aware deep decoupling strategy to minimize the overall execution latency; and 3) An edge-cloud structure adaptation strategy that dynamically changes the decoupling for different network conditions. Experiments demonstrate that our solution can significantly reduce the execution latency: it speeds up the overall inference execution with a guaranteed model accuracy loss.

I. INTRODUCTION

JALAD addresses inefficient cloud-only deployment by splitting deep networks between edge and cloud while jointly managing accuracy, computation, and transmission latency. It compresses intermediate data, optimizes the partition, and adapts execution to network conditions.

  • Motivation: Cloud-only deployment can incur substantial upload latency because users transmit large inputs to centralized servers, while edge devices cannot run full large models.A 2.4 MB facial-recognition image can require 2.4 seconds to upload at 1 MBps; edge devices have limited computation and storage.
  • Motivation: Pruning or quantization provides fixed accuracy-latency trade-offs that users cannot adaptively customize after retraining.The limitation concerns miniaturized edge models derived from the original network.
  • Motivation: Simple partitioning can fail because convolutional layers amplify intermediate data, leaving transmission volume too large for meaningful speedup.This motivates compressing feature maps before transferring them between edge and cloud.
  • JALAD: JALAD decouples a deep network across edge and cloud and addresses partition selection, limited edge computation, and overall execution latency.The framework jointly considers computation and transmission latency.
  • JALAD: JALAD compresses intermediate data by jointly considering compression rate and model accuracy, reducing transmission data to 1/2−1/5 of the original image size while maintaining similar accuracy.The compression strategy combines quantization, image compression, and reinforcement-learning-based channel-wise feature removal.
  • Evaluation: Up to 90% execution-latency reduction is achieved in real-world experiments while keeping accuracy loss within 10%.Experiments use the ILSVRC2012 dataset and include real-world and simulation evaluations.

II. FRAMEWORK AND CHALLENGE

The framework runs early layers at the edge, compresses the resulting feature maps, and executes later layers in the cloud. Its central challenge is that intermediate feature maps may be larger than the original input, undermining naive partitioning.

  • Framework: JALAD runs early network layers on the edge, compresses the final edge-layer output, transfers it to the cloud, and runs the remaining layers there.The partition layer changes dynamically according to network bandwidth and edge computation power.
  • Challenge: In-layer Data “Amplification”: Naive deep-network partitioning often fails to reduce latency because convolutional layers can amplify intermediate data beyond the original input size.In tested ResNet models, some early-layer outputs are 20× larger than the original input.
  • Design setup: The detailed design defines decoupling points and models the factors that determine execution under the decoupled scheme.Notation for the optimization formulation is listed in Table I.

A. Decoupling Points

JALAD defines decoupling points across sequential and branchy networks, then compresses sparse intermediate feature maps to reduce transfer size while preserving accuracy. This enables partitioning at more network locations.

  • A. Decoupling Points: JALAD treats one ResNet residual unit as a decoupling layer and uses convolution-layer granularity for sequential models.This definition accommodates both branchy architectures such as ResNet and sequential architectures such as VGGNet.
  • B. Accuracy-Aware Feature Compression: Intermediate feature maps are compressed because their sparsity creates an opportunity to reduce transmitted data while constraining accuracy loss.The framework evaluates typical image-recognition CNNs for this purpose.
  • B. Accuracy-Aware Feature Compression: JALAD converts floating-point feature-map values into small integers using an adaptive integer-bit count determined by network conditions and the accuracy constraint.The conversion maps values into [0, 2^c) to reduce feature-map size.
  • B. Accuracy-Aware Feature Compression: Huffman coding further compresses the quantized integer feature maps by exploiting their many zero values.The combined compression reduces feature maps to 1/10−1/100 of their original size.

C. Compressed accuracy and data size predictor

JALAD predicts accuracy loss and compressed data size for each compression setting from historical statistics. These predictors support selecting compression parameters under an accuracy constraint.

  • Accuracy prediction: For the evaluated deep networks, using c ≥4 integer bits provides an accuracy-loss guarantee of 10%.The trade-off is measured on the ILSVRC2012 dataset by relating compression bits to the accuracy-loss threshold.
  • Prediction: JALAD predicts accuracy loss and compressed size at a compression setting because both quantities are stable across test epochs and related to historical input statistics.The predictors are trained using statistics from ILSVRC2012 and iterating over all c ∈C.
  • Prediction: The paper illustrates an accuracy predictor Ai(c) and compressed-data-size predictors for c = 4 and c = 8.These examples are shown in Figures 6 and 3, respectively.

D. Formulation of Decoupling and Execution Latency

JALAD models decoupled inference as edge computation, compressed feature transmission, and cloud computation, then uses layer- and bit-dependent measurements to evaluate latency. The formulation selects a partition that minimizes execution time while accounting for compression and network bandwidth.

  • Compression characterization: Compression measurements characterize feature-map size and accuracy as functions of the quantization setting c, including results across layers and decoupling points.The supplied figures cover compressed performance, accuracy loss versus bits, epoch stability, and accuracy loss at different points.
  • Decoupled execution: JALAD decouples a network at layer i, executing preceding layers at the edge and remaining layers in the cloud.The partition is represented as edge layers 1 through i* and cloud layers i*+1 through N.
  • Latency components: Layer-level edge latency is profiled for the deployed device because execution time varies across devices but tends to remain stable on a specific device.The profiling procedure records execution times from the input through each layer.
  • Latency components: The overall execution latency combines edge execution, cloud execution, and transmission of compressed intermediate feature maps.Transmission cost depends on the compressed feature size and current bandwidth.
  • Latency components: Transmission latency is estimated from the predicted compressed size and the current network bandwidth.The predictor supplies the data size associated with a selected layer and quantization setting.

E. Decoupling Implementation

JALAD formulates decoupling as a constrained integer optimization over the layer and quantization-bit choices. The solver uses precomputed accuracy and size statistics with latency estimates, and can adapt the selected scheme when network conditions change.

  • Optimization formulation: The optimization chooses one decoupling layer and one quantization-bit setting through binary variables x_ic.Each variable indicates whether layer i with c-bit feature quantization is selected.
  • Accuracy constraint: A user-defined accuracy-loss boundary Δα constrains feasible layer-and-bit selections.The predicted accuracy loss A_i(c) is used in the optimization constraint.
  • Optimization formulation: The objective minimizes joint inference latency using edge, transmission, and cloud latency estimates treated as constants by the decision maker.Accuracy-loss and compressed-size statistics are stored, while edge and cloud times are estimated during initialization.
  • Solver and adaptation: The fixed-size ILP contains N · C binary variables and was solved in 1.77 ms on an Intel i7-6800K desktop in testing.The reported scale is considered polynomial-time solvable, and the algorithm selects the minimum-latency feasible partition.
  • Solver and adaptation: JALAD re-decouples the network when edge-cloud conditions change, allowing different schemes under different network conditions.The edge and cloud synchronize using the newly selected decoupling.

IV. EVALUATION

The evaluation uses real-world experiments and controlled simulations to assess the effectiveness of JALAD.

  • Evaluation setup: The evaluation combines real-world experiments with controlled simulations.Both evaluation modes are used to assess the design’s effectiveness.

A. Experiment Setup

The experiments evaluate JALAD with controlled edge-cloud conditions, representative devices, and four deep-network models, using accuracy, latency, and execution speedup metrics.

  • Real-world setup: Real-world experiments use a cloud server with an NVIDIA 1080ti GPU and an edge device with an NVIDIA Quadro K620 GPU.Network bandwidth and accuracy-loss thresholds are varied using ILSVRC2012 validation images.
  • Simulation setup: Controlled simulations model high- and low-performance edge devices to examine computation-power effects.The simulation uses 1 MBps average edge-cloud bandwidth and device-specific FLOPS assumptions.
  • Baselines: The evaluation compares Origin2Cloud, which transfers raw images, with JPEG2Cloud, which transfers lossy JPEG images before cloud inference.These represent conventional cloud-based deployment strategies.
  • Models and metrics: JALAD decouples VGGNet-16, VGGNet-19, ResNet-50, and ResNet-101 using 100 ILSVRC2012 samples per iteration and averages latency over 20 iterations.Execution speedup and accuracy performance are the primary evaluation metrics.

B. Execution Latency Speedup

JALAD improves latency against pure-cloud baselines across network conditions while adjusting its decoupling according to accuracy tolerance and transmission cost.

  • Execution speedup: At 300 KBps, JALAD reaches up to 3.6× speedup over PNG2Cloud and 6.0× over Origin2Cloud for VGGNet.For ResNet, the corresponding average speedups are 5.6× and 9.3×, with accuracy loss limited to 10%.
  • Execution speedup: At 1 MBps, JALAD achieves average speedups of 1.6× over PNG2Cloud and 2.5× over Origin2Cloud across four models.Higher bandwidth leads JALAD to place more layers in the cloud because transmission latency is lower.
  • Accuracy threshold: Increasing the allowed accuracy loss enables lower latency by changing the decoupling layer or reducing feature-map bit depth.These choices reduce the number of transmitted bytes while respecting the selected threshold.
  • Accuracy threshold: Figure 7 presents the accuracy-latency relationship used to assess how accuracy thresholds affect average execution latency and decoupling decisions.The figure title identifies accuracy and latency as its two compared quantities.

D. Impact of Cloud-Edge Bandwidth Variation

JALAD adapts decoupling to bandwidth and edge computation power, maintaining low latency when network conditions vary but depending on sufficient edge capability.

  • Bandwidth variation: Under changing bandwidth, JALAD maintains stable low latency by adaptively changing its decoupling strategy.At 1.5MBps, it tends to upload raw PNG images and matches PNG2Cloud latency; poorer networks can yield higher speedups.
  • Edge computation power: With a high-performance NVIDIA Tegra X2 edge device at 1 MBps, JALAD averages 7.6× speedup over PNG2Cloud and 12.6× over Origin2Cloud.The result covers four representative deep networks.
  • Edge computation power: Limited edge computation can prevent decoupling from reducing latency for some networks, including VGGNet.The constraint limits exploration of possible decoupling layers.
  • Related work: Earlier approaches often ignored feature-map sparsity or in-layer compression, causing partition points near the first or last layer and limiting practical latency gains.Such partitions can effectively become cloud-only or client-only deployments.

VI. CONCLUDING REMARKS

JALAD studies accuracy-latency trade-offs in edge-cloud DNN decoupling and combines feature-map compression with constrained latency optimization. Experiments show faster execution while keeping accuracy loss within a user-defined boundary.

  • Conclusion: JALAD decouples deep networks between edge and cloud while optimizing execution latency subject to a guaranteed accuracy constraint.Its design includes accuracy-aware in-layer feature-map compression and an optimization formulation for selecting the decoupling.
  • Conclusion: Real-world experiments on four representative deep neural networks demonstrate speedup while keeping accuracy loss within a user-defined boundary.The conclusion states the result without specifying a single universal speedup value.
Loading 1812.10027v1…