Source-linked AI summary
AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference
Yida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang
TL;DR
Edge-cloud LLM inference remains limited by WAN communication and rejection-driven pipeline rollbacks despite the promise of speculative decoding. AceSpec uses asymmetric communication, probabilistic multi-branch caching, and optimized branch allocation to address these costs, achieving up to 3.52× throughput speedup under WAN conditions down to 50 Kbps.
Problem
WAN latency, bandwidth limits, and rejection-driven rollbacks constrain edge-cloud speculative decoding and can erase its collaborative throughput benefits.
Method
AceSpec uses edge-side probabilistic caching, asymmetric main-chain and sparse-distribution transmission, and network-aware Lagrangian branch allocation.
Results
3.52× throughput speedup over traditional baselines is reported, with efficient generation sustained under WAN conditions down to 50 Kbps.
Takeaways & Limitations
AceSpec provides a training-free edge-cloud inference framework designed to remain efficient under constrained and volatile WAN conditions.
Abstract
from arXiv · showhide
Deploying Large Language Models (LLMs) on edge devices typically relies on model compression or split inference. However, compression degrades reasoning capabilities, while split inference suffers from severe Wide Area Network (WAN) communication bottlenecks. Edge-cloud speculative decoding emerges as a promising alternative, leveraging an edge small model to draft tokens for cloud verification. Yet, over volatile WANs, inevitable prediction rejections trigger catastrophic pipeline stalls and network-wide rollbacks, neutralizing collaborative gains. To overcome this, we propose AceSpec, an asymmetric edge-cloud collaborative framework. AceSpec utilizes un-saturated edge compute to proactively construct a probabilistic state cache, effectively transforming network-wide pipeline flushes into $\mathcal{O}(1)$ local memory lookups. To preserve bandwidth, it employs an asymmetric communication protocol that transmits minimal main-chain indices uplink and compact sparse distributions downlink. Furthermore, we introduce a network-aware, Lagrangian-optimized resource allocation strategy that dynamically maximizes the local cache hit rate. Evaluations demonstrate that AceSpec achieves up to a 3.52$\times$ throughput speedup and exhibits exceptional bandwidth immunity, sustaining near-peak inference performance even under severely constrained 50 Kbps WAN conditions.
I. INTRODUCTION
AceSpec addresses WAN bottlenecks and rollback stalls in edge-cloud LLM inference with a training-free, asymmetric framework that uses edge computation for probabilistic caching. It reports up to 3.52× throughput speedup while sustaining efficient generation down to 50 Kbps WAN conditions.
- Compression degrades reasoning and generative capabilities, while split inference encounters a communication wall from transmitting large hidden states over WANs.
- Speculative decoding reduces communication by having an edge SLM draft tokens for parallel cloud verification, but volatile WANs make rejection-induced stalls costly.
- AceSpec proactively expands a probabilistic multi-branch token tree during cloud verification and converts WAN rollbacks into O(1) local memory lookups.
- Its two-level optimization balances edge compute and network delay through bandwidth-aware budgeting and Lagrangian-optimized non-uniform branch allocation.
- 3.52× throughput speedup is achieved over traditional baselines, with efficient generation sustained under WAN conditions down to 50 Kbps.
II. RELATED WORK
Prior edge-cloud and speculative-decoding systems reduce some computation or latency, but remain constrained by WAN synchronization, bandwidth, hardware, or deployment requirements. AceSpec uses local compute and optimized probabilistic caching to target these limitations without inflating communication overhead.
- Speculative Decoding: Existing speculative-decoding methods assume near-zero communication costs and degrade when frequent state synchronization meets high-latency, bandwidth-constrained WANs.
- Edge-Cloud Collaborative Inference: Spatial splitting and routing schemes reduce local computation but leave edge devices idle during remote processing and transmission, preserving the WAN communication wall.
- Distributed Speculative Decoding: Recent distributed speculative systems impose heavy edge resource burdens, enlarge transmission payloads, or require invasive model modifications.
- Pipelined Architectures: Pipelined approaches overlap computation and communication, but their practical low-bandwidth limitations motivate a different architecture.
- AceSpec Motivation: AceSpec uses available local compute to build probabilistic multi-branch caches and optimizes allocation to hide WAN latency without inflating communication overhead.
III. METHOD
Edge-cloud speculative decoding drafts tokens locally and verifies them in the cloud, but synchronous execution and rejection bubbles expose inference to WAN latency. AceSpec overlaps transmission with local multi-branch expansion and recovers rejected states from an edge cache.
- Rollback Bottleneck: An edge SLM drafts γ tokens while a cloud Target LLM performs parallel verification, with expected accepted tokens determined by the average acceptance rate α.
- Rollback Bottleneck: The synchronous stop-and-wait cycle combines drafting, WAN RTT, and cloud verification, leaving edge compute underutilized when RTT is high.
- Rollback Bottleneck: Asynchronous pipelines can mask WAN delay under perfect acceptance, but single-chain synchronization makes rejection bubbles include downlink, redrafting, and uplink delays.
- Asymmetric Architecture: AceSpec shifts computation toward the edge and uses an asymmetric architecture with an edge Tree-based Drafter and cloud server.
- Asymmetric Execution: The edge uploads only main-chain token indices, expands branches during cloud turnaround, and stores resulting state sequences in a probabilistic cache.
- Local Recovery: On rejection, the cloud returns an index and partial target distribution, enabling local resampling and cache-based state-pointer updates instead of sequential redrafting.
C. Asymmetric Tree Construction and O(1) State Recovery
AceSpec constructs a multi-branch token tree during the asynchronous window and packs branches into one masked forward pass. Rejected states are recovered through local cache lookup and O(1) KV compaction.
- Asymmetric Tree Construction: The edge expands branches at potential rejection depths using its local draft distribution as a proxy for the unknown cloud target distribution.
- Asymmetric Tree Construction: The Tree Attention Mask packs all branches into one parallel pass using Shared Prefix, Main-Path Branch, and Suffix Diagonal blocks.
- O(1) State Recovery: After receiving a sparsified target distribution and acceptance length, the edge resamples locally and queries the cached tree with the resulting state key.
- O(1) State Recovery: O(1) memory compaction aligns the hit branch behind the verified prefix and invalidates unselected branches, eliminating sequential redrafting.
IV. THEORETICAL OPTIMAL BRANCH ALLOCATION
AceSpec models multi-branch caching through theoretical cycle-time components and derives a speedup lower bound that depends on ideal masking and rollback penalties.
- Expected Time and Speedup Analysis: Tnaive combines drafting, WAN round-trip, and verification, while Tideal takes the maximum of tree drafting and cloud turnaround time.These define the synchronous baseline and perfectly masked pipeline cycle times.
- Expected Time and Speedup Analysis: E[Ttree] adds the cache-miss penalty to Tideal according to the probability of a tree-cache miss.
- Expected Time and Speedup Analysis: S ≥ Sideal / [1 + (1 − P tree hit(T)) · ρ], linking achievable speedup to cache hits and rollback overhead.The bound follows because multi-branch caching provides at least the single-chain expected accepted length.
- Expected Time and Speedup Analysis: Sideal = Tnaive/Tideal is the maximum theoretical speedup when the pipeline is perfectly masked.
- Expected Time and Speedup Analysis: ρ = Tbubble/Tideal measures the rollback penalty imposed by WAN conditions and hardware constraints.Tbubble is defined as RTT + Tredraft(γ).
- Expected Time and Speedup Analysis: Equation (4) separates hardware and network limits from the algorithm’s cache-masking performance.
B. Problem Formulation with Two-Level Constraints
AceSpec formulates cache construction as a two-level constrained optimization: network conditions determine the available tree budget, while hardware limits prevent edge computation from becoming the bottleneck.
- B. Problem Formulation with Two-Level Constraints: Maximizing the tree cache hit rate P tree hit(T) is identified as necessary for hiding latency.
- B. Problem Formulation with Two-Level Constraints: The resulting formulation is a two-level constrained optimization problem over network and algorithmic constraints.
- 1) Network-Aware Budgeting:: Tree drafting must finish within the cloud turnaround window: Ttree draft(B) ≤ RTT(t) + Tverify.This prevents multi-branch drafting from becoming a new computational bottleneck.
- 1) Network-Aware Budgeting:: AceSpec continuously probes RTT(t) to derive a network-aware computation budget.The supplied passage introduces Bnet(t) as the resulting budget.
- 1) Network-Aware Budgeting:: The effective budget B∗(t) is bounded by both the network-derived limit and the edge device’s VRAM and parallel-capacity limit Bmax.Ttree draft(·) is the inverse latency function of the edge SLM.
- 1) Network-Aware Budgeting:: B∗(t) dynamically adapts tree size to current network delay while respecting edge hardware capacity.
- 1) Network-Aware Budgeting:: AceSpec enlarges the tree under network fluctuations and reduces it under favorable conditions to conserve edge energy.This expresses the compute-energy tradeoff of network-aware budgeting.
2) Task-Aware Shape Allocation:
Task-aware allocation shapes the alternative branches to maximize cache hits under a fixed budget, accounting for acceptance probabilities, sequential survival, and long-tail residual distributions.
- 2) Task-Aware Shape Allocation:: The main draft chain consumes γ branches, leaving Balloc(t) = B∗(t) − γ for alternatives.
- 2) Task-Aware Shape Allocation:: F = [F0, F1, …, Fγ−1] assigns alternative branches across depths, with nonnegative allocations constrained by the remaining budget.
- 2) Task-Aware Shape Allocation:: At depth k, hit probability combines main-token acceptance α with residual-candidate coverage: p(Fk) = α + (1 − α)R(Fk).
- 2) Task-Aware Shape Allocation:: Assuming conditional independence across sequence steps, the overall tree hit rate is the product of depth-wise hit probabilities.This assumption makes the cache-hit analysis tractable.
- 2) Task-Aware Shape Allocation:: Taking logarithms converts the product objective into a computationally tractable constrained nonlinear optimization problem.
- 2) Task-Aware Shape Allocation:: The optimization is task-aware and constrained by the network-aware allocation limit, yielding a non-uniform geometric decay strategy.
- 2) Task-Aware Shape Allocation:: Sequential verification weights deeper branches by prefix survival probability αk, which decreases with depth.
- 2) Task-Aware Shape Allocation:: The residual distribution is modeled with a power-law tail, approximating R(Fk) by 1 − cF_k^-β and the hit probability accordingly.The exponent β is positive and controls the decay.
A. Experimental Setup
AceSpec is evaluated on a simulated edge-cloud testbed spanning multiple model pairs, task types, network conditions, and inference baselines.
- A. Experimental Setup: The edge runs on an NVIDIA Jetson AGX Orin with 32GB, while the cloud uses 4× NVIDIA A100 GPUs with 40GB each.Linux tc emulates WAN throughput limits and latency.
- A. Experimental Setup: The evaluation therefore varies hardware scale, model architecture, parameter size, task complexity, and token acceptance rate.
- A. Experimental Setup: Model pairs cover Qwen-0.6B/32B, Qwen-1.7B/32B, and LLaMA-1B/70B configurations.
- A. Experimental Setup: Datasets include GSM8K for mathematical reasoning, HumanEval for code generation, and Alpaca for instruction following.
- A. Experimental Setup: AceSpec is compared with autoregressive decoding, Vanilla Speculative Decoding, Split Inference, and additional inference paradigms.The supplied passage states that five paradigms are compared, but truncates the final list.
B. End-to-End Performance Comparison
AceSpec delivers strong throughput across model pairs and datasets, remains near-peak as bandwidth falls to 50 Kbps, and benefits from non-uniform branch allocation. Its main limitations are task-specific regression and computational overhead from multi-branch drafting.
- Throughput comparison: 3.52x throughput speedup is achieved with LLaMA-1B/70B on HumanEval, while Qwen-0.6B/32B reaches 1.75x on GSM8K.On GSM8K, AceSpec exceeds PicoSpec at 1.45x and DSSD at 0.85x.
- Limitations: AceSpec reaches 0.98x speedup with Qwen-1.7B/32B on HumanEval, below the 1.00x cloud-only Autoregressive and PicoSpec baselines.The regression is associated with rigid code-generation precision demands and increased edge latency from the larger draft model.
- Bandwidth robustness: AceSpec maintains a stable throughput plateau as bandwidth decreases from 100 Mbps to 50 Kbps.Performance degradation appears only at 25 Kbps and 10 Kbps.
- Branch allocation ablation: Non-uniform branch allocation reaches 24.37 tokens/s on GSM8K, compared with 20.14 tokens/s for single-branch and 21.58 tokens/s for uniform allocation.The strategy shifts budget toward shallower, high-confidence nodes to maximize cache hits under a fixed compute budget.
- Branch allocation ablation: Non-uniform allocation reduces GSM8K TTFT to 436.62 ms and raises draft hit rate to 0.97, whereas uniform expansion reaches 450.79 ms TTFT.Uniform dense-tree construction delays main-chain dispatch by saturating edge compute.
- Limitations: The current multi-branch drafting implementation incurs computational overhead that future fused attention kernels and memory management could reduce.This overhead can offset latency-hiding benefits when edge tree drafting exceeds cloud turnaround time.
E. Sensitivity Analysis
AceSpec remains effective across changes in edge branch budget and sampling temperature, with diminishing returns at larger budgets and hit rates above 81% under diverse generation.
- Branch budget B: As B increases from 20 to 40, draft hit rate rises from approximately 0.750 to over 0.850, then flattens toward B=80.The flattening indicates diminishing marginal returns as the edge explores more alternative tokens.
- Branch budget B: The branch-budget curve guides selecting a threshold that maximizes draft hit rate without saturating edge computation.A smaller budget captures highly probable tokens, while further expansion yields less additional benefit.
- Sampling temperature T: At temperatures T ≥1.5, AceSpec maintains a draft hit rate above 81% despite increased generation randomness.Across T from 0.3 to 1.7, hit rate declines from 0.885 at low temperatures to 0.811 at high temperatures.
- Sampling temperature T: The hit-rate decline becomes steeper above T=1.0, but the multi-branch cache remains effective for deterministic and creative generation.The cache captures alternative tokens across diverse sampling conditions.