Source-linked AI summary
Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy
En Li, Zhi Zhou, Xu Chen
TL;DR
Mobile DNN inference is constrained by device resources and variable network latency. Edgent combines adaptive device-edge partitioning with early-exit right-sizing to meet deadlines while preserving accuracy, and evaluations on Raspberry Pi demonstrate low-latency edge intelligence.
Problem
Mobile devices incur high DNN inference overhead, while cloud and edge execution can suffer latency from wide-area transmission and volatile bandwidth.
Method
Edgent jointly selects a DNN partition point and an early-exit point to optimize device-edge co-inference under a latency requirement.
Results
Evaluations on Raspberry Pi show that Edgent selects increasingly accurate models as latency requirements relaxes and can meet requirements earlier than other methods at 200ms to 300ms.
Takeaways & Limitations
Device-edge synergy with partitioning and right-sizing provides an on-demand approach for low-latency DNN inference.
Abstract
from arXiv · showhide
As the backbone technology of machine learning, deep neural networks (DNNs) have have quickly ascended to the spotlight. Running DNNs on resource-constrained mobile devices is, however, by no means trivial, since it incurs high performance and energy overhead. While offloading DNNs to the cloud for execution suffers unpredictable performance, due to the uncontrolled long wide-area network latency. To address these challenges, in this paper, we propose Edgent, a collaborative and on-demand DNN co-inference framework with device-edge synergy. Edgent pursues two design knobs: (1) DNN partitioning that adaptively partitions DNN computation between device and edge, in order to leverage hybrid computation resources in proximity for real-time DNN inference. (2) DNN right-sizing that accelerates DNN inference through early-exit at a proper intermediate DNN layer to further reduce the computation latency. The prototype implementation and extensive evaluations based on Raspberry Pi demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.
I. INTRODUCTION & RELATED WORK
Edgent addresses the latency and energy challenges of mobile DNN inference by combining device-edge collaboration with adaptive partitioning and early exit under application deadlines.
- Mobile devices struggle to support DNN applications because inference requires substantial computation, creating latency and energy overhead.
- Cloud offloading introduces high end-to-end latency and energy consumption through long wide-area data transmission.
- As bandwidth falls from 1Mbps to 50Kbps, edge-inference latency rises from 0.123s to 2.317s, approaching local-device processing latency.
- Edgent adaptively partitions DNN computation between mobile devices and edge servers, then combines partitioning with early exit to reduce latency.
- Edgent jointly optimizes partitioning and right-sizing to maximize accuracy without violating predefined deadlines, complementing device-side compression and architecture optimization.
II. BACKGROUND & MOTIVATION
DNNs perform inference through layered computation but are computationally intensive, while this initial investigation focuses on execution latency rather than energy efficiency.
- The study focuses on execution latency initially and defers consideration of energy efficiency to future work.
- DNNs support applications including computer vision, speech recognition, and natural language processing, but inference can require millions of parameters.
- A DNN passes data through interconnected layers from raw input to an output category; this forward process is called model inference.
- The paper focuses on DNN inference because training is generally delay tolerant and conducted offline using powerful cloud resources.
B. Inefficiency of Device- or Edge-based DNN Inference
Both local and remote DNN execution can have poor real-time performance, motivating analysis of latency breakdowns across bandwidth conditions and deployment layers.
- Direct device execution and cloud or edge offloading may fail to satisfy real-time latency requirements for applications such as AR/VR gaming and intelligent robots.
- The Raspberry Pi and desktop-PC AlexNet experiment examines end-to-end latency under varying bandwidth between the emulated mobile device and edge server.
- Figure 2 reports the runtime breakdown of different execution approaches, including a local execution case exceeding 2s.
C. Enabling Edge Intelligence with DNN Partitioning and Right-Sizing
Edgent combines DNN partitioning and right-sizing to reduce device-edge inference latency, while jointly balancing early-exit accuracy loss against application deadlines.
- DNN Partitioning: DNN layers differ substantially in runtime and output size, so the computationally intensive portion can be offloaded when transmission overhead is low.
- DNN Right-Sizing: DNN partitioning remains limited by the runtime of layers retained on the device, motivating complementary right-sizing through multiple early-exit points.
- DNN Right-Sizing: A branchy AlexNet illustration contains five exit points, each corresponding to a different model size.
- Problem Description: Early exit reduces computation time but deteriorates inference accuracy, creating a latency-accuracy tradeoff for deadline-constrained applications.
- Problem Description: Given a predefined latency requirement, Edgent jointly optimizes the partition and exit decisions to maximize DNN inference accuracy.
III. FRAMEWORK
Edgent automatically selects a DNN partition point and an early-exit point to maximize accuracy while satisfying an end-to-end latency requirement. Its three-stage framework profiles infrastructure, trains models with exit points, optimizes online using bandwidth and latency inputs, and executes the selected plan across edge and device.
- Edgent selects the best DNN partition point and exit point to maximize accuracy while satisfying the execution-latency requirement.
- The framework comprises offline training, online optimization, and co-inference stages.Figure 5 presents this three-stage organization.
- Offline training profiles the mobile device and edge server for layer-performance models and trains Branchynet models with various exit points.Profiling is infrastructure-dependent, whereas DNN training is application-dependent.
- Online optimization uses profiled latency models, Branchynet models, observed bandwidth, and a predefined latency requirement to select partition and early-exit points.
- During co-inference, the edge server executes layers before the partition point and the mobile device executes the remaining layers.
B. Layer Latency Prediction
Edgent predicts DNN runtime by modeling individual layer latency and incorporating model loading effects. Regression models are built for typical layer types using layer-related variables.
- Edgent models per-layer latency rather than whole-DNN latency to reduce profiling overhead because DNNs contain limited classes of layers.
- The latency models use independent variables such as input data size and output data size, while also including DNN model size to capture model-loading time.
- Regression models are provided for each type of DNN layer, with size measured in bytes and latency measured in milliseconds.
C. Joint Optimization on DNN Partition and DNN Right-Sizing
Edgent jointly searches over exit points and partition points using predicted device, edge, and transfer latencies. It chooses the largest exit model whose optimized partition satisfies the user’s latency requirement.
- For each exit point and partition point, Edgent estimates total runtime from device latency, edge latency, input transfer, and partition-output transfer.
- The optimizer considers branchy models with M exit points, where larger exit-point indices represent more accurate and larger inference models.
- Partitioning does not affect inference accuracy, so Edgent searches exit points separately after finding the minimum-latency partition for each model.
- The search algorithm iterates through exit points from M down to 1 and evaluates every possible partition point.
- The algorithm returns an exit point and partition point when the predicted latency is within the target latency requirement; otherwise, it returns NULL.
IV. EVALUATION
The paper presents preliminary implementation and evaluation results for Edgent.
- The paper reports preliminary implementation and evaluation results.
A. Prototype
Edgent is prototyped with a desktop PC emulating the edge server and a Raspberry Pi 3 acting as the mobile device. The evaluation uses a branchy AlexNet and regression-based latency models, examining selections under bandwidth and latency variation.
- Prototype: A desktop PC emulates the edge server, while a Raspberry Pi 3 represents the mobile device.The PC has a quad-core 3.4 GHz Intel processor and 8 GB RAM; the Raspberry Pi 3 has a quad-core 1.2 GHz ARM processor and 1 GB RAM.
- Prototype: The branchy AlexNet model is trained for Cifar-10 image recognition with five exit points.Its five sub-models contain 12, 16, 19, 20, and 22 layers, respectively.
- Prototype: The prototype examines selection under different bandwidths and latency requirements, alongside model runtime under varying bandwidths.The referenced figures cover partition and exit selection, runtime, and latency-requirement conditions.
- Prototype: Regression-based latency prediction models are built for each DNN layer.The independent variables and resulting regression models are reported in Tables I and II.
B. Results
The evaluation varies bandwidth and latency requirements because both affect Edgent’s optimization. Higher bandwidth or a relaxed latency requirement generally permits later exits and higher accuracy, while Edgent satisfies moderate latency goals earlier than the other methods.
- Bandwidth results: Higher bandwidth leads to higher exit points and therefore higher accuracy, although the best partition points may fluctuate.Bandwidth is varied from 50kbps to 1.5Mbps with the latency requirement fixed at 1000ms.
- Bandwidth results: As bandwidth increases, model runtime first drops substantially and then rises when improved accuracy remains within the latency requirement.The runtime increase occurs as bandwidth changes from 1.2Mbps to 2Mbps; the regression approach estimates actual runtime latency well.
- Latency results: A larger latency requirement selects a higher exit point, giving more room for accuracy improvement.This experiment fixes bandwidth at 500kbps and varies latency requirements from 100ms to 1000ms.
- Latency results: At 200ms to 300ms requirements, Edgent satisfies the latency goal earlier than other methods by selecting a small model with moderate accuracy.At 100ms, all four methods fail to satisfy the requirement; Edgent’s selected-model accuracy increases as the requirement relaxes.
V. CONCLUSION
The paper presents Edgent as a collaborative, on-demand DNN co-inference framework using device-edge synergy. Its partitioning and right-sizing knobs tune latency, and Raspberry Pi evaluations demonstrate low-latency edge intelligence.
- Conclusion: Edgent is a collaborative and on-demand DNN co-inference framework with device-edge synergy.The framework targets low-latency edge intelligence.
- Conclusion: DNN partitioning enables collaboration between the edge and mobile device, while DNN right-sizing shapes the computation requirement.Together, the two knobs tune the latency of a DNN model.
- Conclusion: A preliminary implementation and Raspberry Pi evaluations demonstrate Edgent’s effectiveness in enabling low-latency edge intelligence.The conclusion identifies further community discussion and effort as a direction for accomplishing intelligent edge services.