Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

• Mobile devices struggle to support DNN applications because DNN computation requires substantial resources.Source: §I · p. 1
• Cloud-centric inference can incur intolerable latency because data repeatedly traverses a long wide-area network.Source: §I · p. 1
• Edgent jointly uses DNN partitioning and right-sizing to maximize inference accuracy while meeting application latency requirements.Partitioning divides computation between device and edge, while right-sizing uses early exits.Source: §I · pp. 1–2
• The initial study focuses on execution-latency optimization, leaving energy efficiency for future work.Source: §I · p. 1
• Edgent specializes its workflow for static and dynamic network environments to address varying bandwidth conditions.Static environments use regression-based latency estimation; the supplied passage introduces this tailored configuration mechanism.Source: §I · p. 2
• The prototype uses a Raspberry Pi and desktop PC, with real-world network-trace evaluations demonstrating the framework’s effectiveness.Source: §I · p. 2

• Mobile DNN research includes runtime management, model architecture optimization, and hardware acceleration.Source: §II · p. 2
• DNN partitioning distributes successive model parts across participating devices to optimize computation offloading or workload distribution.Source: §II · p. 2
• Edgent maximizes inference accuracy under a predefined latency requirement according to available computation resources.Source: §II · p. 2
• Partitioning alone remains constrained by computation on the mobile device, while right-sizing alone remains dominated by the original model workload.Source: §II · pp. 2–3
• Edgent integrates partitioning and right-sizing to expand the design space and balance accuracy against latency on demand.Source: §II · p. 3
• The framework is specialized for both static and dynamic network environments, whereas existing efforts mainly focus on stable networks.Source: §II · p. 3

• The background analyzes edge-only and device-only methods before motivating DNN partitioning and right-sizing with device-edge synergy.Source: §III · p. 3
• DNNs serve as core technology across computer vision and natural language processing applications.Source: A. A Brief Introduction on DNN · p. 3
• A typical DNN is represented as a directed graph of connected layers that transform weighted inputs into outputs.Source: A. A Brief Introduction on DNN · p. 3
• Figure 2 compares execution runtime for edge-only and device-only AlexNet approaches under different bandwidths.Source: Figure 2 · p. 3
• Typical DNN inference demands substantial computation, so this paper focuses on inference rather than delay-tolerant training.Source: A. A Brief Introduction on DNN · p. 3

Insufficiency of Device- or Edge-Only DNN Inference

• Device-only and wholly offloaded inference may both produce high end-to-end latency, hindering real-time application requirements.Source: B. Insufficiency of Device- or… · p. 3

DNN Partitioning

• Layer runtime and intermediate output size are heterogeneous, so the most computationally expensive layer need not produce the largest transmitted output.Source: C. DNN Partitioning and Right-Sizing… · p. 3, Figure 3 · p. 4
• DNN partitioning divides the model and offloads computation-intensive parts at low transmission cost to use nearby device and edge resources.Source: C. DNN Partitioning and Right-Sizing… · p. 4

DNN Right-Sizing

• DNN right-sizing uses early exits to reduce inference latency, but introduces a latency-accuracy tradeoff.Source: C. DNN Partitioning and Right-Sizing… · p. 4

Problem Definition

• Edgent jointly optimizes partitioning and right-sizing to maximize accuracy without violating a predefined latency requirement.Source: C. DNN Partitioning and Right-Sizing… · p. 4, §IV · p. 4

Framework workflow

• Edgent uses offline configuration, online tuning, and co-inference stages to pursue DNN inference performance across network conditions.The configurator prepares static or dynamic configurations, the optimizer selects an execution plan, and co-inference executes it.Source: A. Framework Overview · p. 4

Online tuning

• During online tuning, Edgent jointly selects a partition point and exit point to maximize inference accuracy while satisfying the given latency requirement.The optimizer uses the measured bandwidth and the configuration obtained offline.Source: A. Framework Overview · p. 4

Co-inference

• At co-inference, layers before the selected partition point execute on the edge server and the remaining layers execute on the device.The selected exit point determines where inference terminates, while the partition point determines computation placement.Source: A. Framework Overview · p. 4

Framework workflow

• The static configuration combines regression models with a branchy DNN when bandwidth remains stable, while dynamic configuration selects among bandwidth-state-specific options.These configurations support different bandwidth environments within the same overall workflow.Source: A. Framework Overview · p. 4

Static configurator

• The static configurator profiles layer-wise latency on the device and edge server, trains regression models, and creates a multi-exit branchy DNN.It profiles layer types separately and uses BranchyNet to enable early exits.Source: B. Edgent for Static Environment · p. 5

Runtime optimization

• The runtime optimizer searches exit and partition points using the static configuration, measured bandwidth, and latency requirement.The search targets maximum accuracy while ensuring the execution deadline.Source: B. Edgent for Static Environment · p. 5
• For each exit point, Edgent estimates total latency from device computation, server computation, and communication, then chooses the lowest-latency partition.The partition point can represent device-only, edge-only, or split execution.Source: B. Edgent for Static Environment · p. 5
• Because partitioning does not affect inference accuracy, Edgent tests exit layers and selects the highest-accuracy model satisfying the latency requirement.Different exit layers provide different precision levels while partition choices determine execution latency.Source: B. Edgent for Static Environment · p. 5

Assumptions

• The design assumes a nearby edge server is available and that device and edge computation resources are fixed and allocated beforehand.The authors state that these assumptions could be relaxed with prediction models incorporating different resource levels.Source: B. Edgent for Static Environment · pp. 5–6

Offline configuration

• The dynamic configurator sketches bandwidth states from historical traces and obtains an optimal exit point and partition point for each state.Static Edgent supplies the optimal points during offline configuration.Source: C. Edgent for Dynamic Environment · p. 6

Configuration map construction

• The configuration-map algorithm evaluates each candidate strategy with a reward and records the best exit and partition points for every bandwidth state.Each candidate combines an exit point and partition point.Source: C. Edgent for Dynamic Environment · pp. 6–7
• The reward first requires the latency requirement to be satisfied; among feasible strategies, it prioritizes accuracy and then higher throughput for similar accuracy.An infeasible strategy receives zero reward.Source: C. Edgent for Dynamic Environment · p. 6

Online tuning

• During online tuning, the runtime optimizer detects significant changes in bandwidth-measurement distributions and selects the matching or closest configuration state.The selected plan guides collaborative inference at runtime.Source: C. Edgent for Dynamic Environment · pp. 6–7

Evaluation

• This section presents the Edgent implementation and evaluation results.No quantitative or comparative evaluation result is stated in the supplied passage.Source: §V · p. 7

Hardware and workloads

• The prototype uses a Raspberry Pi 3 as the mobile device and a desktop PC as the edge server.The Raspberry Pi 3 has a quad-core 1.2 GHz ARM processor and 1 GB RAM; the desktop has a quad-core 3.40 GHz Intel processor and 8 GB RAM.Source: A. Experimental Setup · p. 7

Network settings

• Static bandwidth is controlled with WonderShaper, while dynamic bandwidth uses Belgium 4G/LTE logs and Oboe traces for configuration states.The configuration map contains 428 bandwidth states ranging from 0Mbps to 6Mbps.Source: A. Experimental Setup · p. 7

Hardware and workloads

• Experiments use a five-exit branchy AlexNet trained on CIFAR-10 for image classification.BranchyNet and Chainer support the multi-branch model, with each exit corresponding to a branch of standard AlexNet.Source: A. Experimental Setup · p. 7

Configurators

• The dynamic runtime optimizer maps accumulated bandwidth measurements to a bandwidth state and returns the corresponding co-inference strategy when the state changes.Its inputs include bandwidth measurements, prior strategies, bandwidth states, and a state-detection function.Source: B. Experiments in Static Bandwidth… · p. 7

Evaluation protocol

• Performance is measured across predefined latency requirements and varying available bandwidth settings.Bandwidth experiments fix the latency requirement at 1000ms and vary bandwidth from 50kbps to 1.5Mbps.Source: B. Experiments in Static Bandwidth… · p. 7

Static-bandwidth findings

• As bandwidth increases, Edgent selects longer exits for higher accuracy, while communication constraints below 250kbps favor exit point 3 over exit point 5.The reported latency first descends sharply and then climbs abruptly as bandwidth increases because the system bottleneck changes.Source: B. Experiments in Static Bandwidth… · p. 7

Latency-accuracy findings

• Under a 100ms latency requirement, all four methods fail, while at requirements of 200ms and 300ms Edgent works earlier with moderate accuracy loss.When the requirement exceeds 400ms, all methods except device-only inference finish in time.Source: B. Experiments in Static Bandwidth… · pp. 7–8, Figure 8 · p. 8, Figure 9 · p. 8

Dynamic environment

• The configuration map is generated from Oboe bandwidth traces containing 49 download-chunk tuples per trace.Mean average bandwidth per trace represents bandwidth-state fluctuation, producing 428 states from 0Mbps to 6Mbps.Source: C. Experiments in Dynamic Bandwidth… · p. 8
• The dynamic evaluation uses Belgium 4G/LTE traces measured while walking, cycling, riding buses, trains, or cars.Online change point detection is integrated with the runtime optimizer for bandwidth measurement.Source: C. Experiments in Dynamic Bandwidth… · p. 8

Dynamic configurator

• Edgent satisfies the latency requirement despite bandwidth fluctuation, while its unchanged exit point keeps inference accuracy stable.Partition-point selection approximately follows the throughput traces under the stated reward design.Source: C. Experiments in Dynamic Bandwidth… · p. 8

Throughput comparison

• At the same CDF level, the dynamic configurator achieves higher throughput than the static configurator.The dynamic configurator’s CDF begins at 11 FPS, compared with 1 FPS for the static configurator.Source: C. Experiments in Dynamic Bandwidth… · p. 8, Figure 10 · p. 9, Figure 11 · p. 9

Reward comparison

• The dynamic configurator generally performs better, although static and dynamic configurators achieve nearly the same reward in the reported comparison.The reward difference mainly comes from throughput because both usually select exit point 5.Source: C. Experiments in Dynamic Bandwidth… · pp. 8–9

Contributions

• Edgent is an on-demand DNN co-inference framework based on device-edge collaboration.Source: §VI · p. 9
• DNN partitioning enables collaboration, while DNN right-sizing uses early exit to optimize inference latency.Source: §VI · p. 9

Configurators

• Separate configurators determine collaboration strategies for static and dynamic bandwidth environments.Source: §VI · p. 9

Evaluation

• Prototype implementation and Raspberry Pi evaluation demonstrate Edgent’s feasibility and effectiveness for low-latency edge intelligence.Source: §VI · p. 9

Future work

• Future work includes combining Edgent with model compression and supporting multi-device applications through resource allocation algorithms.Source: §VI · p. 9