Source-linked AI summary
CoEdge: Cooperative DNN Inference with Adaptive Workload Partitioning over Heterogeneous Edge Devices
Liekang Zeng, Xu Chen, Zhi Zhou, Lei Yang, Junshan Zhang
TL;DR
Edge devices lack sufficient computing capability for timely DNN inference, while remote offloading faces network delay and reliability concerns. CoEdge coordinates heterogeneous edge devices through adaptive workload partitioning, reducing energy while maintaining close inference latency; it achieves up to 25.5%–66.9% energy savings across four CNN models.
Problem
Resource-constrained edge devices struggle to execute DNNs within latency requirements, while remote cloud offloading incurs unreliable and delay-significant wide-area network transmission.
Method
CoEdge orchestrates cooperative DNN inference across heterogeneous edge devices and uses constrained optimization with an efficient approximation algorithm to adapt workload partitioning to computation and communication conditions.
Results
Up to 25.5%–66.9% energy savings were achieved for four CNN models, while CoEdge provided 7.21×–4.49× latency speedup over local inference and remained close to other cooperative approaches in latency.
Takeaways & Limitations
Cooperative inference can reduce system energy by harvesting nearby edge resources while satisfying execution latency requirements under favorable network conditions.
Takeaways & Limitations
The workflow may not fit more complicated model architectures, and execution on minitype devices may still cause precision loss despite preserving model accuracy theoretically.
Abstract
from arXiv · showhide
Recent advances in artificial intelligence have driven increasing intelligent applications at the network edge, such as smart home, smart factory, and smart city. To deploy computationally intensive Deep Neural Networks (DNNs) on resource-constrained edge devices, traditional approaches have relied on either offloading workload to the remote cloud or optimizing computation at the end device locally. However, the cloud-assisted approaches suffer from the unreliable and delay-significant wide-area network, and the local computing approaches are limited by the constrained computing capability. Towards high-performance edge intelligence, the cooperative execution mechanism offers a new paradigm, which has attracted growing research interest recently. In this paper, we propose CoEdge, a distributed DNN computing system that orchestrates cooperative DNN inference over heterogeneous edge devices. CoEdge utilizes available computation and communication resources at the edge and dynamically partitions the DNN inference workload adaptive to devices' computing capabilities and network conditions. Experimental evaluations based on a realistic prototype show that CoEdge outperforms status-quo approaches in saving energy with close inference latency, achieving up to 25.5%~66.9% energy reduction for four widely-adopted CNN models.
I. INTRODUCTION
CoEdge addresses latency and energy challenges in edge DNN inference by coordinating heterogeneous nearby devices instead of relying solely on remote cloud or local execution. It dynamically partitions workloads according to device capabilities and network conditions, with a real-time optimization approach evaluated on a multi-device prototype.
- Motivation: Edge applications require millisecond-scale responses, but DNN workloads are computation-intensive and edge devices have limited capability.Local execution can take several seconds for smart-home camera face recognition, making the service unusable.
- Motivation: Cooperative edge inference uses nearby computation resources to improve responsiveness while preserving privacy compared with remote-cloud execution.The approach distributes computation within a manageable proximity to the data source.
- CoEdge: CoEdge orchestrates cooperative DNN inference across heterogeneous devices without modifying the given model or sacrificing its accuracy.Its workload partitions match device computing capabilities and network conditions while optimizing latency and energy.
- CoEdge: CoEdge formulates workload distribution as a constrained optimization problem, proves it NP-hard, and develops a fast approximation algorithm for real-time partitioning.The algorithm is tailored to diverse computing capabilities and network conditions.
- Evaluation: A multi-device prototype evaluates CoEdge on four widely adopted DNN models to assess its performance.The prototype uses Raspberry Pi 3, Jetson TX2, and desktop PC hardware.
A. Deep Neural Network Inference
CNN inference extracts feature representations and then classifies them, while cooperative execution partitions image workloads across edge devices. A two-device case study shows that offloading can reduce latency and energy under favorable network conditions, with workload allocation shaped by device heterogeneity.
- A. Deep Neural Network Inference: CNN inference consists of feature extraction through operations such as convolution and pooling, followed by classification through fully connected layers.The first stage generates multidimensional feature maps; the second produces the inference result.
- A. Deep Neural Network Inference: The edge-deployment challenge is the gap between intensive CNN computation and limited device capability.Cooperative inference addresses this gap by exploiting available resources across edge devices.
- B. Case Study: Cooperative Inference with Two Devices: A Raspberry Pi 3 and Jetson TX2 parallelize inference after part of an input image is offloaded, then aggregate results at the Raspberry Pi.The experiment measures end-to-end latency and energy for this process.
- B. Case Study: Cooperative Inference with Two Devices: The offloading ratio specifies the proportion of image data transferred from the Raspberry Pi 3 to the Jetson TX2.A ratio of 0.5 splits the image along its height into two equal parts and transfers one part.
- B. Case Study: Cooperative Inference with Two Devices: Fully offloading to the Jetson TX2 does not necessarily minimize latency or energy when the network fluctuates.Offloading and result-transfer overheads can offset the Jetson’s performance advantage.
- B. Case Study: Cooperative Inference with Two Devices: Under favorable network conditions, cooperative inference reduces both latency and energy as the offloading ratio increases.The system benefits from harvesting the cooperator’s computing resources.
- B. Case Study: Cooperative Inference with Two Devices: Latency drops faster with increasing offloading because parallel execution is constrained by the slower device.Higher bandwidth makes assigning more workload to the faster Jetson TX2 more beneficial.
C. Merits and Challenges
CoEdge coordinates heterogeneous edge devices to keep cooperative DNN inference within the edge while adapting workload placement to device capabilities and network conditions. Its two-phase runtime profiles devices, plans partitions, distributes work, and aggregates cooperative execution.
- Merits: Cooperative inference can reduce latency and energy by harvesting idle computing resources at the edge while keeping data within users’ control scope.It also avoids delay-significant wide-area network transmission and associated privacy issues.
- Challenges: CoEdge addresses heterogeneous capabilities and dynamic networks by orchestrating computation, communication, and workload allocation across participating devices.The system decides which devices participate and how much workload each device handles.
- CoEdge Design: In setup, devices record execution profiles; at runtime, the master uses profiling results and network status to create workload partitions and assignments.The master then distributes partitions to workers and manages cooperative execution.
- Cooperative Inference Workflow: CoEdge uses model parallelism: devices process assigned input feature-map portions in parallel, producing output portions that are concatenated into complete layer outputs.Its workflow splits images into differentiated partitions, executes feature extraction cooperatively, and aggregates results during classification.
- Cooperative Inference Workflow: Convolution across partition boundaries requires devices to pull padding data from neighboring partitions before producing complete feature maps.The example shows device A fetching a 1 × 6 margin row from device B for a 3×3 kernel.
C. Impact of Workload Partitioning
Workload partitioning directly shapes cooperative inference cost because partition boundaries induce communication, while assignments must match heterogeneous computation and communication resources. CoEdge formulates allocation under latency and resource constraints to minimize energy.
- Impact of Workload Partitioning: Partition boundaries create convolution communication: devices pull padding data of ⌊k/2⌋ along the split dimension from neighboring devices when kernel size k exceeds 1.A neighboring partition must provide the boundary data needed to compute across the partition edge.
- Impact of Workload Partitioning: CoEdge requires neighboring allocated partitions to be at least as large as the padding size, unless the neighboring device owns no partition.This principle ensures padding data can be acquired from only one neighboring device.
- Impact of Workload Partitioning: During convolution, devices exchange padding data; after convolutions, feature-map partitions are aggregated for fully-connected computation and the result is returned to a selected device.The communication pattern distributes input, exchanges boundary data, aggregates classification inputs, and transfers the final result to a specified location.
- Impact of Workload Partitioning: A partition assignment must balance computation and communication resources because high bandwidth cannot compensate for poor computing capability in every case.The paper gives large offloading to a high-bandwidth but weak-compute device as a counterexample.
- Problem Formulation: The optimization allocates partition sizes to minimize total energy while satisfying an execution deadline, given available computation and communication resources.The formulation models CNN layers progressively from single-layer constraints to whole-workflow behavior.
1) Single-Layer Formulation:
The single-layer formulation constrains partition sizes, estimates computation and communication costs, and incorporates device memory and resource characteristics. It assumes computation latency scales with processing cycles and excludes static energy from the model.
- Single-Layer Formulation: Partition sizes are nonnegative integers whose concatenation along the split dimension equals the input height H; experiments split along height without loss of generality.The formulation also imposes padding-related size restrictions.
- Single-Layer Formulation: Each partition’s workload size is constrained by the device’s available memory capacity, although the model limits memory footprint to per-layer inputs for simplicity.Detailed runtime memory and platform-related footprint can be added as an enhancement for deployment.
- Single-Layer Formulation: Computation latency is estimated by dividing processing cycles by device frequency, while computation energy is computed from latency and computation power.The cycle estimate is based on input size and the model’s profiled computing intensity.
- Single-Layer Formulation: The energy formulation counts dynamic computation and communication energy but excludes static energy used to maintain basic system-level services.This exclusion limits the model’s accounting of total device energy consumption.
- Single-Layer Formulation: Communication latency is modeled from available bandwidth between devices, including initial input distribution and padding-data transfers from neighboring devices.The formulation also treats a device’s self-bandwidth as memory bandwidth.
2) Multi-Layer Formulation:
The multi-layer formulation extends single-layer costs across synchronized computation and communication under a Bulk Synchronous Parallel model. It defines an NP-hard deadline-constrained energy minimization problem, motivating an efficient solver.
- Multi-Layer Formulation: CoEdge devices alternate computation and communication jobs, triggering synchronization after communication jobs except the initial communication.Synchronization transfers the padding data required for convolutional computation.
- Multi-Layer Formulation: Total computation and communication energy are obtained by summing the corresponding energy across all devices and layers.The formulation denotes these totals as Ec and Ex.
- Multi-Layer Formulation: Under BSP, total physical latency sums interval durations based on the maximum latency across devices.This total includes fully-connected-layer latency, whose maximum is effectively the selected classification device’s latency.
- Multi-Layer Formulation: The optimization chooses partitioning solution π to minimize Ec + Ex without violating execution deadline D and subject to the partition constraints.The decision variable is π = [a1, a2, · · · , aN].
- Multi-Layer Formulation: Problem P1 is NP-hard, and its exponentially growing decision space motivates an efficient method for real-time solution generation.The paper identifies P1 as an Integer Linear Programming problem and notes the challenge of deployments with many edge devices.
B. Problem Transformation
The integer workload-partitioning problem is relaxed into a linear program by replacing discrete partition sizes with continuous proportions, enabling efficient approximation while managing threshold constraints.
- P1 is an integer linear program whose discrete partition-size variables make it difficult to solve directly.The optimization minimizes computation and communication energy subject to an execution deadline.
- λi approximates partition size ai through ai = λiH, where H is input height and λi is the proportion assigned to partition i.For typical large CNN inputs, the approximation error is described as tiny and tolerated.
- Relaxing the threshold constraint transforms P1 into P2, which minimizes Ec + Ex but may assign some devices workload below the required padding threshold.P1 requires each nonzero assignment to meet the padding-size threshold, whereas P2 permits smaller positive assignments.
- P2 is a linear programming problem that can be solved efficiently with mature programming solvers to approximate P1.The paper identifies CPLEX as an example of such a solver.
C. Workload Partitioning Algorithm Design
CoEdge uses a threshold-based recursive algorithm to obtain feasible workload partitions from a linear-programming relaxation, then distributes image segments across heterogeneous devices.
- Algorithm 1 iteratively narrows participating devices by checking the threshold constraint and approaching a feasible solution to P1.It removes zero-workload devices and then removes the minimum workload element when the threshold is violated.
- The algorithm takes CNN layer configurations, device resources, bandwidths, and an execution deadline as inputs, and returns a workload-partitioning solution.An empty device set produces no feasible solution.
- The partitioning solver runs in under 10 ms because recursion occurs no more than once per available device.The implementation uses CPLEX, whose individual LP runtime is reported as under 1 ms.
- If a feasible plan is returned, CoEdge segments the input image and sends partitions to corresponding devices; otherwise, it treats the deadline as too strict and offloads all workload to the devic
- During runtime, devices fetch padding data from neighboring devices, blocking requests until the required feature-map data is prepared.The paper states that this situation is rare because workload allocation is optimized to match device capabilities and network conditions.
VI. PERFORMANCE EVALUATION
The evaluation uses a six-device heterogeneous prototype to measure CoEdge on four CNN models, comparing latency and dynamic energy against cooperative and local baselines.
- The prototype contains four Raspberry Pi 3 devices, one Jetson TX2, and one desktop PC.The devices represent weak IoT devices, a mobile AI platform, and a small edge server, respectively.
- CoEdge is evaluated on AlexNet, VGG-f, GoogLeNet, and MobileNet using image classification on one ImageNet image.The models are implemented with TensorFlow Lite and measured over repeated inference runs.
- The comparison includes MoDNN, Musical Chair, and a local approach that runs inference solely on a Raspberry Pi 3 master device.MoDNN partitions according to computing capability, Musical Chair uses equal-proportion data or model parallelism, and the local approach is the baseline.
- Latency is measured under model-specific deadlines of 100 ms, 100 ms, 200 ms, and 100 ms for AlexNet, VGG-f, GoogLeNet, and MobileNet, respectively.
- Dynamic energy is measured on the same six-device testbed under the experimental settings used for latency evaluation.Energy measurements use a Monsoon High Voltage Power Monitor.
B. Performance Comparison
CoEdge meets the tested latency deadlines while reducing energy relative to cooperative and local baselines, and it adapts workload allocation to deadlines, cluster composition, and bandwidth changes.
- Performance Comparison: 7.21×∼4.49× latency speedup is achieved by CoEdge over the local approach, while CoEdge, Musical Chair, and MoDNN meet the tested deadlines.CoEdge and MoDNN have similar latency, whereas Musical Chair is slower because it ignores resource heterogeneity.
- Performance Comparison: 66.9%, 64.9%, 46.0%, and 25.5% energy savings are achieved by CoEdge over Musical Chair for the four tested models, respectively.Against the local baseline, the corresponding savings are 39.2%, 37.8%, 11.5%, and 10.9%.
- Performance Comparison: CoEdge achieves the lowest energy costs by jointly optimizing computation and communication tradeoffs, while model structure affects the available improvement space.Frequent exchanges increase GoogLeNet’s communication burden, while MobileNet’s local optimization limits cooperative gains.
- Deadline Sensitivity: When deadlines are very stringent, CoEdge prioritizes latency over energy; as deadlines loosen, it shifts emphasis toward energy optimization and eventually stabilizes.All approaches miss deadlines of 50 ms or less in the AlexNet experiment.
- Scalability: Adding more devices generally lowers CoEdge latency and energy, with distinctive drops when a desktop PC or Jetson TX2 joins the cluster.Adding weaker Raspberry Pis after a powerful device produces approximately stable results because CoEdge assigns most workload to stronger devices.
- Network Adaptation: CoEdge reprograms workload partitions when bandwidth changes, with under 10 ms reprogramming overhead and lower energy costs as bandwidth increases.The experiment uses AlexNet on the six-device cluster with a 100 ms deadline.
VII. RELATED WORK
Prior edge-intelligence approaches exploit cloud execution, local resources, or multi-device collaboration, but CoEdge combines cooperative workflow and adaptive partitioning across heterogeneous devices. Its scope includes energy-focused optimization, robustness, generalization, and alternative objectives.
- Prior edge-intelligence research spans cloud-assisted execution, local resource exploitation, and multi-device collaboration.
- Multi-device collaboration: CoEdge combines layer-fusion parallelism with workload-size adjustment and jointly adapts allocation to computation and communication resources.
- Robustness and Generalization: A participant crash or network timeout can break cooperative inference, motivating modularity or periodic intermediate-result backups.
- Robustness and Generalization: Applying workload partitioning across the whole network may not fit sophisticated architectures whose deeper-layer feature maps have smaller height and width.
- Other optimizing objectives: CoEdge’s optimization focuses on dynamic energy consumption under preset deadlines for CNN inference.
- Other optimizing objectives: CoEdge can be extended toward weighted latency-energy objectives, static-energy accounting, accuracy optimization, and edge-oriented accelerators.
APPENDIX A PROOF OF THEOREM 1
The appendix establishes that the workload-allocation formulation is an integer linear program and proves it is NP-hard by reducing P||Cmax to a special case. A relaxed formulation remains a linear program.
- P1 is NP-hard by reducing the NP-hard P||Cmax scheduling problem to a special case with all power parameters set to 1.
- P1 is an integer linear programming problem because its expressions are linear and its allocation variables have integer constraints.
- The continuous formulation P2 remains a linear programming problem after substituting λ_iH for the integer allocation variables.