Source-linked AI summary
DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager
TL;DR
Layer-wise edge–cloud inference lets resource-constrained devices contribute to LLM execution, but heterogeneous cuts create idle gaps and make conventional batching difficult. DySCo uses resident layer ranges and depth-synchronized batching to run shared cloud suffixes. In the reported tests, DSB improved throughput over exact-match batching while reducing mean per-session latency, and idle gaps added cloud-side decode latency.
Problem
Layer-wise edge–cloud inference faces idle gaps between cloud invocations and difficulty conventionally batching requests that arrive at different model depths.
Source: §I · pp. 1–2Method
DySCo keeps KV caches local, uses dyForward to execute configurable contiguous ranges from resident shards, and uses DSB to batch heterogeneous requests at a common depth.
Source: §I · p. 2Results
At mean concurrency eight, DSB improves throughput by 48% over exact-match batching while reducing mean per-session latency, and idle gaps add up to approximately 25 ms of cloud-side decode latency per token.
Source: §I · p. 2Takeaways & Limitations
Requests with different edge–cloud cuts can reuse resident cloud weights and share batched suffix computation when concurrency and suffix depth permit.
Source: C. Discussion · p. 9, §VI · p. 10DSB’s benefit is not universal: shallow shared tails, clustered fleets, or very long contexts can reduce or eliminate the gain.
Source: C. Discussion · p. 9
Abstract
from arXiv · showhide
Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models (LLMs) are increasingly used to support these applications. Yet, due to their high resource demands, LLMs are mostly deployed in the cloud. Layer-wise edge-cloud inference lets resource-constrained edge devices contribute computation to LLMs they cannot host in full. However, heterogeneous split points introduce two coupled inefficiencies. First, edge execution and communication create idle gaps between cloud invocations. Second, requests arriving at different model depths cannot be conventionally batched. We present DySCo, a collaborative runtime that keeps KV caches local and introduces dyForward, a model-aware layer-range executor that runs configurable contiguous layer ranges from resident model shards without reloading weights. For multi-edge serving settings, we introduce depth-synchronized batching (DSB), which advances heterogeneous requests to the deepest cut and batches their common suffix. Experiments across heterogeneous devices, two model families, and local and wide-area links show that idle gaps increase the latency of subsequent GPU forward calls even when waiting time is excluded, adding up to 25 ms of additional cloud-side suffix latency per decoding step in our measurements. At an average concurrency of eight, DSB improves throughput by 275% over FIFO, 48% over exact-match batching, and 79% over round-robin interleaving while reducing mean per-session latency. Together, these results show that requests with different edge-cloud splits can reuse resident cloud weights and share batched suffix computation. The artifact repository for this work is publicly available at: https://github.com/Large-scale-Sustainable-Computing-LSC/dysco-artifact
I. INTRODUCTION
DySCo addresses the idle gaps and cut-layer diversity that complicate cloud execution when heterogeneous edge devices execute different model prefixes. It supports resident-shard execution and batches requests at a shared model depth; the reported tests show added cloud latency from idle gaps and higher throughput with DSB than exact-match batching.
I. INTRODUCTION
- Layer-wise edge–cloud inference lets devices contribute computation when they cannot host the full model, unlike fully cloud or fully local deployment.Prior systems often route complete requests or inference phases; DySCo partitions each request at a configurable layer boundary.Source: §I · p. 1
- DySCo’s dyForward selects contiguous layer ranges from loaded shards, enabling different session cuts without reloading weights.The runtime keeps KV caches with their respective shards.Source: §I · p. 2
- Controlled replay isolates cloud-side latency caused by inter-token idle gaps, showing that layer count alone does not determine cloud latency.The measurements span heterogeneous edge devices and network settings.Source: §I · p. 2
- Depth-synchronized batching advances requests with different cuts to a common model depth and batches their remaining shared suffix.This targets the loss of conventional batching across heterogeneous cuts.Source: §I · pp. 1–2
- At mean concurrency eight, DSB improves throughput by 48% over exact-match batching while reducing mean per-session latency; idle gaps add up to approximately 25 ms of decode latency per token.The evaluation uses four Qwen and Llama configurations and prompts from two source datasets.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
The paper distinguishes its per-request layer-wise split from prior approaches that route complete requests, divide inference phases, or place distributed shards. It frames two problems: measuring how cut placement and idle gaps affect latency, and batching requests that reach the cloud at different model depths.
II. BACKGROUND AND RELATED WORK
- Prior systems route requests, separate prefill and decode, or place distributed shards; DySCo instead studies fixed per-request cuts and cloud batching across different cuts.Its setting partitions each request between edge prefix and cloud suffix without changing model weights.Source: §II · pp. 2–3, §III · p. 3
III. PROBLEM SETTING AND OVERVIEW
- The execution model fixes each session’s cut during prefill and decoding, keeps each device’s KV-cache entries local, and leaves joint cut selection and scheduler adaptation for future work.The single-edge evaluation characterizes temporal effects; the multi-edge evaluation isolates scheduling and batch formation under controlled hardware placement.Source: §III · p. 3
- RQ1 asks how cut placement and idle gaps jointly affect latency, using heterogeneous-device measurements and controlled hidden-state replay to isolate idle-interval effects.A deeper cut shifts work to the edge and can change communication and the interval between GPU invocations.Source: §III · p. 3
- RQ2 concerns batching requests at different model depths: exact-match batching requires identical cuts, while DSB advances requests to the deepest cut in a scheduling window and batches their common tail.Different cuts can fragment requests even when they share a long suffix.Source: §III · p. 3
IV. SYSTEM DESIGN AND IMPLEMENTATION
DySCo connects edge clients to a shared cloud backend, with per-session state and a scheduler managing ready cloud work. Session setup establishes the cut and generation configuration; prefill initializes caches, and decoding repeatedly alternates edge and cloud execution.
A. Architecture and Execution Flow
- Each edge executes a request’s prefix and maintains its KV cache, while the shared cloud executes the suffix, maintains its cache, and performs token selection.A cloud session manager creates per-session state, and a scheduler queues and dispatches ready work.Source: A. Architecture and Execution Flow · p. 3, Figure 3 · p. 4
- Session initialization sends the cut layer and generation configuration to the cloud; the cloud validates the request, creates state, and acknowledges setup.The control path is separated from the repeated inference data path.Source: A. Architecture and Execution Flow · p. 3
- During prefill, the edge processes the prompt through its prefix and sends cut-layer hidden states to initialize the cloud suffix cache; during decode, both sides advance local caches and exchange the hidden state and next token ID.The scheduler can dispatch ready work individually, interleave sessions, or form compatible batches.Source: A. Architecture and Execution Flow · pp. 3–4, Figure 4 · p. 4
B. Sharding and Communication
DySCo’s dyForward executes selected contiguous layer ranges from resident edge and cloud shards, while each device retains the cache for its own layers. Normal decoding exchanges a single-token hidden state and returned token ID; prefill also sends the sequence and metadata needed to initialize cloud state.
B. Sharding and Communication
- DyForward selects a contiguous decoder-layer range from a resident shard, updates that range’s KV cache, and applies final normalization only when execution reaches the model’s end.This avoids separate model instances or reconstruction for each cut while allowing sessions to use different edge prefix depths.Source: B. Sharding and Communication · p. 4, Figure 5 · p. 4
- During cached decoding, the edge sends one cut-layer hidden state and the cloud returns one token ID; caches and intermediate layer state remain local.Prefill instead sends the cut-layer hidden sequence with the attention mask and position IDs needed to initialize cloud state.Source: B. Sharding and Communication · pp. 4–5
- The tensor-size estimates define B as batch size, Tin as tokens processed in the current call, and Tctx as the context length represented by request state.In prefill, Tin equals Tctx and both equal prompt length; in cached decoding, Tin is one while Tctx includes prompt and generated tokens.Source: B. Sharding and Communication · pp. 4–5
C. Cloud-Side Scheduling
DSB synchronizes requests with heterogeneous cuts at the deepest arrival layer, then batches their shared suffix. It preserves session-local KV caches and reduces to exact-match batching when cuts coincide.
C. Cloud-Side Scheduling
- DSB addresses fragmentation because conventional batched calls apply the same layer range to every row, so exact-match batching combines only sessions with identical cuts.Different cuts otherwise prevent batching requests that share a suffix.Source: C. Cloud-Side Scheduling · p. 5
- DSB advances each session over its private range to the deepest cut, then merges hidden states and suffix KV-cache rows for one batched call over the common tail.Updated cache rows and token outputs are split back to their corresponding sessions.Source: C. Cloud-Side Scheduling · p. 5
- The synchronization depth is the deepest cut in the ready window; when all cuts match, private ranges vanish and DSB is exactly exact-match batching.No fleet-wide cut-layer coordination is required when devices join or leave; only a subsequent window's composition and checkpoint change.Source: C. Cloud-Side Scheduling · p. 5
- KV caches remain session-local outside merged calls; padding and row maps reconcile context lengths temporarily, and prefill remains unbatched.DSB changes only cloud-side decode dispatch, not edge execution.Source: C. Cloud-Side Scheduling · p. 5
V. PERFORMANCE EVALUATION
The evaluation pairs single-edge latency characterization across edge hardware with multi-edge scheduling tests that vary concurrent-session cuts. The setup uses distinct testbeds, model and prompt pools, and execution protocols for those two scales.
V. PERFORMANCE EVALUATION
- The evaluation has two scales: single-edge experiments characterize partition depth, heterogeneity, and idle gaps, while multi-edge experiments test DSB throughput when sessions enter at different layers.The single-edge study varies edge hardware; the multi-edge study varies session count and cuts while fixing hardware.Source: §V · p. 5, A. Experimental Setup · p. 6
A. Experimental Setup
- Single-edge tests use an A6000 cloud with heterogeneous edge devices and gRPC; the Jetson–cloud pair communicates over a WAN, while datacenter GPU pairs use a private intra-cluster network.The multi-edge testbed uses one cloud process and edge processes on a multi-GPU server to isolate scheduling from wide-area network variation.Source: A. Experimental Setup · p. 6
- Single-edge results use Qwen3.5-9B and replicate with Llama-3.1-8B; multi-edge tests use Llama-3.2-3B and replicate the main trend with Qwen2.5-3B.The models have 32, 32, 28, and 36 layers, respectively.Source: A. Experimental Setup · p. 6
- Workloads draw short prompts from WildGPT-4 for decode and scheduling sweeps, and long prompts from LongReason for prefill and context-length tests.Short prompts are 16–128 tokens; long prompts are approximately 4,000–6,000 tokens before truncation to target lengths.Source: A. Experimental Setup · p. 6
- Unless stated otherwise, tests use bfloat16 and greedy decoding; dyForward varies cuts without reloading weights, and each session keeps its cut from prefill through decoding.Single-edge sessions generate 64 tokens and multi-edge sessions 128; multi-edge cut heterogeneity is emulated on identical physical edge GPUs.Source: A. Experimental Setup · p. 6
1) Single-Edge Single-Cloud Characterization:
Single-edge measurements show that cloud latency depends on more than suffix depth: edge-side idle gaps can raise the next cloud forward-pass latency. Controlled replay isolates this effect and explains the observed nonlinearity in split-inference latency.
1) Single-Edge Single-Cloud Characterization
- Cloud latency can first rise and exceed the full-model baseline as edge layers increase, before declining after a device-dependent turning point; this pattern also appears with Llama-3.1-8B.The Orin starts in the high-overhead regime, while the other tested edges show an initial sharp increase.Source: 1) Single-Edge Single-Cloud… · p. 6, Figure 8 · p. 7
- Controlled replay isolates idle-gap effects by replaying cut-layer hidden states on the A6000 and inserting 0–25 ms gaps, excluding injected wait and using a smoothed zero-gap reference.The study measures latency rather than throughput for a single request; short-input decode is the focus because it captures repeated per-token execution.Source: 1) Single-Edge Single-Cloud… · pp. 6–7
- Up to approximately 25 ms of additional execution latency occurs at cut layer 1 and 13 ms at cut layer 12, beyond the injected wait; overhead rises over a narrow gap range then plateaus.A larger suffix has a higher ceiling and generally needs a longer gap to reach it; prefix-only tests show the same qualitative effect at smaller absolute cost for few edge layers.Source: 1) Single-Edge Single-Cloud… · p. 7
- Idle-gap overhead exceeds the computation saved by moving layers to the edge only under some conditions, explaining why cloud latency can rise before falling as suffix depth shrinks.Once overhead reaches its ceiling, moving additional layers to the edge again reduces cloud latency.Source: 1) Single-Edge Single-Cloud… · p. 7
2) Multi-Edge Serving with Depth-Synchronized Batching:
The section evaluates schedulers under heterogeneous cuts and finds that DSB’s throughput gains depend on shared-tail depth and fleet composition.
Compared schedulers
- DSB uses a shared dispatch window and batch-size limit, while the four schedulers differ in how they dispatch or combine decode steps.Exact-match batching groups only identical cuts; interleaving serves sessions singly, while DSB combines heterogeneous sessions.Source: 2) Multi-Edge Serving with… · p. 7
Measurement protocol
- Measurement conditions were randomized, and differences below approximately six percentage points were not interpreted.Welch’s |t| > 2.5 was the stated significance criterion.Source: 2) Multi-Edge Serving with… · pp. 7–8
Exact-match batching under heterogeneity
- Exact-match batching improves throughput by +377.0% for a shared cut, +180.3% for two cut tiers, and +80.9% when all eight cuts differ.With all cuts different, no sessions merge, so the remaining gain comes from consolidating dispatch in one scheduling thread.Source: 2) Multi-Edge Serving with… · p. 8
Performance under Poisson arrivals
- At mean concurrency 8, DSB improves throughput by +275% over FIFO, +48% over exact-match batching, and +79% over interleaving.It also achieves the lowest mean latency at every tested load, from 4.93 to 6.45 s versus 26.56 to 28.34 s for FIFO.Source: 2) Multi-Edge Serving with… · p. 8, Table III · p. 8
Determinants of the throughput gain
- DSB’s gain over exact-match batching rises from +78.1% at a 20-layer tail to −0.2% at a single-layer tail, and from −6.7% at two distinct cuts to +81.3% at eight.Throughput gain is determined by the deepest cut in the window; reducing forward calls alone does not predict it.Source: 2) Multi-Edge Serving with… · p. 8, Figure 10 · p. 9
Sensitivity to fleet composition
- Among clustered fleets, all eight single-session outlier fleets gain +2.2% to +16.6%, while all six fleets with a larger minority lose −8.9% to −1.5%.The measurements do not identify the mechanism behind the different outcomes.Source: 2) Multi-Edge Serving with… · p. 8
Operating envelope
- DSB is beneficial with at least four concurrent sessions, four distinct cuts, a shared tail of about five or more layers, and contexts up to about 2048 tokens.At 4096 tokens its gain is no longer significant; on clustered fleets it can underperform exact-match batching by up to 8.9%.Source: 2) Multi-Edge Serving with… · p. 8
- Under synchronized bursts, interleaving gains +67% to +114% over FIFO despite executing the same single-session calls, so that gain stems from its single-thread dispatcher rather than batching.The crossover with DSB depends on session count and arrival pattern.Source: 2) Multi-Edge Serving with… · pp. 8–9
C. Discussion
The discussion connects idle gaps and cut-layer diversity to cloud-side latency and batching, while emphasizing that DSB’s benefits depend on workload and setting.
Discussion
- Edge computation and communication can make a subsequent cloud forward pass slower after an idle gap, even when the wait itself is excluded.DVFS and power-state transitions are plausible contributors, but the A6000 test did not eliminate the effect with clock locking.Source: C. Discussion · p. 9
- DSB improves throughput and latency when concurrency and cut-layer diversity are sufficient, but shallow shared tails, clustered fleets, or long contexts can reduce or eliminate its gain.A practical scheduler should consider both temporal readiness and structural compatibility.Source: C. Discussion · p. 9
- The measurements use heterogeneous fixed cuts and unbatched multi-edge prefill; wide-area multi-edge operation and online cut selection remain unevaluated.The single-edge study includes heterogeneous hardware and network links, while the multi-edge study uses a controlled server.Source: C. Discussion · pp. 9–10
VI. CONCLUSION
DySCo couples configurable edge–cloud cuts with cloud scheduling: idle gaps can raise subsequent cloud-forward latency, while depth-synchronized batching (DSB) improves throughput and per-session latency in evaluated workloads. Future work will evaluate geographically distributed fleets and jointly adapt cuts and cloud scheduling as conditions change.
VI. CONCLUSION
- DSB improves throughput and per-session latency in the evaluated multi-edge workloads.Different cuts can separate requests that share cloud suffix weights into distinct conventional batches; DSB recovers shared-suffix batching when concurrency and suffix depth permit.Source: §VI · p. 10
- Idle intervals can increase the latency of the cloud’s next forward pass, even when more layers are moved to the edge.The effect is reported in the single-edge measurements.Source: §VI · p. 10
- Cut placement and cloud scheduling are coupled because a cut determines when a session reaches the cloud and how much computation it can share with others.Source: §VI · p. 10
- Future work will evaluate geographically distributed edge fleets and jointly choose cuts and schedule cloud work as device speed, network conditions, and request load change.Source: §VI · p. 10