PaperlayerView exact revision on arXiv

Source-linked AI summary

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

En Li, Liekang Zeng, Zhi Zhou, Xu Chen

arXiv:1910.05316v1cs.NIcs.CVcs.DCcs.LG

TL;DR

Mobile devices lack sufficient resources for computation-intensive DNNs, while cloud-assisted inference suffers wide-area network latency. Edgent combines device-edge partitioning, early-exit right-sizing, and static or dynamic network configuration; experiments report higher dynamic-environment throughput than the static configurator.

  • Problem

    Mobile devices have limited computation resources for DNN inference, while cloud assistance can incur significant wide-area network latency.

  • Method

    Edgent jointly optimizes DNN partitioning and right-sizing, selecting collaborative inference plans for static and dynamic bandwidth environments.

  • Results

    At CDF 0.6, the dynamic configurator achieves 27 FPS throughput versus 17 FPS for the static configurator.

  • Takeaways & Limitations

    Edgent provides on-demand collaborative DNN inference that maximizes accuracy while meeting predefined latency requirements.

Abstract

from arXiv · show

As a key technology of enabling Artificial Intelligence (AI) applications in 5G era, Deep Neural Networks (DNNs) have quickly attracted widespread attention. However, it is challenging to run computation-intensive DNN-based tasks on mobile devices due to the limited computation resources. What's worse, traditional cloud-assisted DNN inference is heavily hindered by the significant wide-area network latency, leading to poor real-time performance as well as low quality of user experience. To address these challenges, in this paper, we propose Edgent, a framework that leverages edge computing for DNN collaborative inference through device-edge synergy. Edgent exploits two design knobs: (1) DNN partitioning that adaptively partitions computation between device and edge for purpose of coordinating the powerful cloud resource and the proximal edge resource for real-time DNN inference; (2) DNN right-sizing that further reduces computing latency via early exiting inference at an appropriate intermediate DNN layer. In addition, considering the potential network fluctuation in real-world deployment, Edgentis properly design to specialize for both static and dynamic network environment. Specifically, in a static environment where the bandwidth changes slowly, Edgent derives the best configurations with the assist of regression-based prediction models, while in a dynamic environment where the bandwidth varies dramatically, Edgent generates the best execution plan through the online change point detection algorithm that maps the current bandwidth state to the optimal configuration. We implement Edgent prototype based on the Raspberry Pi and the desktop PC and the extensive experimental evaluations demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.

I. INTRODUCTION

Edgent addresses the latency limitations of mobile and cloud-based DNN inference by coordinating device and edge resources. It combines adaptive partitioning, early exiting, and network-specific configuration to pursue accurate inference under latency requirements.

  • Mobile devices struggle to support DNN applications because DNN computation requires substantial resources.
  • Cloud-centric inference can incur intolerable latency because data repeatedly traverses a long wide-area network.
  • Edgent jointly uses DNN partitioning and right-sizing to maximize inference accuracy while meeting application latency requirements.Partitioning divides computation between device and edge, while right-sizing uses early exits.
  • The initial study focuses on execution-latency optimization, leaving energy efficiency for future work.
  • Edgent specializes its workflow for static and dynamic network environments to address varying bandwidth conditions.Static environments use regression-based latency estimation; the supplied passage introduces this tailored configuration mechanism.
  • The prototype uses a Raspberry Pi and desktop PC, with real-world network-trace evaluations demonstrating the framework’s effectiveness.

II. RELATED WORK

Prior work uses runtime management, model optimization, and hardware acceleration to address mobile DNN limitations. Edgent expands this space by jointly selecting partition and exit points for static and dynamic network conditions.

  • Mobile DNN research includes runtime management, model architecture optimization, and hardware acceleration.
  • DNN partitioning distributes successive model parts across participating devices to optimize computation offloading or workload distribution.
  • Edgent maximizes inference accuracy under a predefined latency requirement according to available computation resources.
  • Partitioning alone remains constrained by computation on the mobile device, while right-sizing alone remains dominated by the original model workload.
  • Edgent integrates partitioning and right-sizing to expand the design space and balance accuracy against latency on demand.
  • The framework is specialized for both static and dynamic network environments, whereas existing efforts mainly focus on stable networks.

III. BACKGROUND AND MOTIVATION

DNNs support many intelligent applications but require substantial computation. The background motivates examining edge-only and device-only inference and using partitioning and right-sizing for device-edge collaboration.

  • The background analyzes edge-only and device-only methods before motivating DNN partitioning and right-sizing with device-edge synergy.
  • DNNs serve as core technology across computer vision and natural language processing applications.
  • A typical DNN is represented as a directed graph of connected layers that transform weighted inputs into outputs.
  • Figure 2 compares execution runtime for edge-only and device-only AlexNet approaches under different bandwidths.
  • Typical DNN inference demands substantial computation, so this paper focuses on inference rather than delay-tolerant training.

B. Insufficiency of Device- or Edge-Only DNN Inference

Device-only and edge-only DNN inference can both suffer high end-to-end latency. Edgent addresses the bottleneck by jointly choosing a partition point and an early-exit point under a latency constraint.

  • Insufficiency of Device- or Edge-Only DNN Inference: Device-only and wholly offloaded inference may both produce high end-to-end latency, hindering real-time application requirements.
  • DNN Partitioning: Layer runtime and intermediate output size are heterogeneous, so the most computationally expensive layer need not produce the largest transmitted output.
  • DNN Partitioning: DNN partitioning divides the model and offloads computation-intensive parts at low transmission cost to use nearby device and edge resources.
  • DNN Right-Sizing: DNN right-sizing uses early exits to reduce inference latency, but introduces a latency-accuracy tradeoff.
  • Problem Definition: Edgent jointly optimizes partitioning and right-sizing to maximize accuracy without violating a predefined latency requirement.

A. Framework Overview

Edgent operates through offline configuration, online tuning, and co-inference stages to select device-edge execution plans. Its workflow jointly optimizes DNN partitioning and right-sizing under measured bandwidth and latency requirements.

  • Framework workflow: Edgent uses offline configuration, online tuning, and co-inference stages to pursue DNN inference performance across network conditions.The configurator prepares static or dynamic configurations, the optimizer selects an execution plan, and co-inference executes it.
  • Online tuning: During online tuning, Edgent jointly selects a partition point and exit point to maximize inference accuracy while satisfying the given latency requirement.The optimizer uses the measured bandwidth and the configuration obtained offline.
  • Co-inference: At co-inference, layers before the selected partition point execute on the edge server and the remaining layers execute on the device.The selected exit point determines where inference terminates, while the partition point determines computation placement.
  • Framework workflow: The static configuration combines regression models with a branchy DNN when bandwidth remains stable, while dynamic configuration selects among bandwidth-state-specific options.These configurations support different bandwidth environments within the same overall workflow.

B. Edgent for Static Environment

For stable bandwidth, Edgent predicts layer-wise latency and uses a branchy DNN to search jointly over exit and partition points. It selects the most accurate configuration that meets the execution deadline, subject to stated deployment assumptions.

  • Static configurator: The static configurator profiles layer-wise latency on the device and edge server, trains regression models, and creates a multi-exit branchy DNN.It profiles layer types separately and uses BranchyNet to enable early exits.
  • Runtime optimization: The runtime optimizer searches exit and partition points using the static configuration, measured bandwidth, and latency requirement.The search targets maximum accuracy while ensuring the execution deadline.
  • Runtime optimization: For each exit point, Edgent estimates total latency from device computation, server computation, and communication, then chooses the lowest-latency partition.The partition point can represent device-only, edge-only, or split execution.
  • Runtime optimization: Because partitioning does not affect inference accuracy, Edgent tests exit layers and selects the highest-accuracy model satisfying the latency requirement.Different exit layers provide different precision levels while partition choices determine execution latency.
  • Assumptions: The design assumes a nearby edge server is available and that device and edge computation resources are fixed and allocated beforehand.The authors state that these assumptions could be relaxed with prediction models incorporating different resource levels.

C. Edgent for Dynamic Environment

For dynamic bandwidth, Edgent builds a configuration map from historical bandwidth states and candidate co-inference strategies, then uses online change detection to select the corresponding plan.

  • Offline configuration: The dynamic configurator sketches bandwidth states from historical traces and obtains an optimal exit point and partition point for each state.Static Edgent supplies the optimal points during offline configuration.
  • Configuration map construction: The configuration-map algorithm evaluates each candidate strategy with a reward and records the best exit and partition points for every bandwidth state.Each candidate combines an exit point and partition point.
  • Configuration map construction: The reward first requires the latency requirement to be satisfied; among feasible strategies, it prioritizes accuracy and then higher throughput for similar accuracy.An infeasible strategy receives zero reward.
  • Online tuning: During online tuning, the runtime optimizer detects significant changes in bandwidth-measurement distributions and selects the matching or closest configuration state.The selected plan guides collaborative inference at runtime.

V. PERFORMANCE EVALUATION

The paper presents an Edgent implementation and reports performance evaluation results, but the supplied passage provides no specific evaluation findings.

  • Evaluation: This section presents the Edgent implementation and evaluation results.No quantitative or comparative evaluation result is stated in the supplied passage.

A. Experimental Setup

The experiments evaluate Edgent on Raspberry Pi–desktop hardware under controlled static bandwidth and dynamic traces, varying latency requirements and bandwidth to assess its configuration decisions.

  • Hardware and workloads: The prototype uses a Raspberry Pi 3 as the mobile device and a desktop PC as the edge server.The Raspberry Pi 3 has a quad-core 1.2 GHz ARM processor and 1 GB RAM; the desktop has a quad-core 3.40 GHz Intel processor and 8 GB RAM.
  • Network settings: Static bandwidth is controlled with WonderShaper, while dynamic bandwidth uses Belgium 4G/LTE logs and Oboe traces for configuration states.The configuration map contains 428 bandwidth states ranging from 0Mbps to 6Mbps.
  • Hardware and workloads: Experiments use a five-exit branchy AlexNet trained on CIFAR-10 for image classification.BranchyNet and Chainer support the multi-branch model, with each exit corresponding to a branch of standard AlexNet.
  • Configurators: The dynamic runtime optimizer maps accumulated bandwidth measurements to a bandwidth state and returns the corresponding co-inference strategy when the state changes.Its inputs include bandwidth measurements, prior strategies, bandwidth states, and a state-detection function.
  • Evaluation protocol: Performance is measured across predefined latency requirements and varying available bandwidth settings.Bandwidth experiments fix the latency requirement at 1000ms and vary bandwidth from 50kbps to 1.5Mbps.
  • Static-bandwidth findings: As bandwidth increases, Edgent selects longer exits for higher accuracy, while communication constraints below 250kbps favor exit point 3 over exit point 5.The reported latency first descends sharply and then climbs abruptly as bandwidth increases because the system bottleneck changes.
  • Latency-accuracy findings: Under a 100ms latency requirement, all four methods fail, while at requirements of 200ms and 300ms Edgent works earlier with moderate accuracy loss.When the requirement exceeds 400ms, all methods except device-only inference finish in time.

C. Experiments in Dynamic Bandwidth Environment

Dynamic-bandwidth experiments use real mobility traces to test Edgent’s online configuration, showing improved throughput over static configuration while reward remains close in the reported comparison.

  • Dynamic environment: The configuration map is generated from Oboe bandwidth traces containing 49 download-chunk tuples per trace.Mean average bandwidth per trace represents bandwidth-state fluctuation, producing 428 states from 0Mbps to 6Mbps.
  • Dynamic environment: The dynamic evaluation uses Belgium 4G/LTE traces measured while walking, cycling, riding buses, trains, or cars.Online change point detection is integrated with the runtime optimizer for bandwidth measurement.
  • Dynamic configurator: Edgent satisfies the latency requirement despite bandwidth fluctuation, while its unchanged exit point keeps inference accuracy stable.Partition-point selection approximately follows the throughput traces under the stated reward design.
  • Throughput comparison: At the same CDF level, the dynamic configurator achieves higher throughput than the static configurator.The dynamic configurator’s CDF begins at 11 FPS, compared with 1 FPS for the static configurator.
  • Reward comparison: The dynamic configurator generally performs better, although static and dynamic configurators achieve nearly the same reward in the reported comparison.The reward difference mainly comes from throughput because both usually select exit point 5.

VI. CONCLUSION

The paper concludes that Edgent combines device-edge collaboration with partitioning and early exiting, using separate configurators for static and dynamic bandwidth environments. Prototype experiments on Raspberry Pi demonstrate feasibility and effectiveness for low-latency edge intelligence, with model compression and multi-device allocation identified as future extensions.

  • Contributions: Edgent is an on-demand DNN co-inference framework based on device-edge collaboration.
  • Contributions: DNN partitioning enables collaboration, while DNN right-sizing uses early exit to optimize inference latency.
  • Configurators: Separate configurators determine collaboration strategies for static and dynamic bandwidth environments.
  • Evaluation: Prototype implementation and Raspberry Pi evaluation demonstrate Edgent’s feasibility and effectiveness for low-latency edge intelligence.
  • Future work: Future work includes combining Edgent with model compression and supporting multi-device applications through resource allocation algorithms.
Loading 1910.05316v1…