PaperlayerView exact revision on arXiv

Source-linked AI summary

AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference

Yida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang

arXiv:2609.02514v1cs.DC

TL;DR

Edge-cloud LLM inference remains limited by WAN communication and rejection-driven pipeline rollbacks despite the promise of speculative decoding. AceSpec uses asymmetric communication, probabilistic multi-branch caching, and optimized branch allocation to address these costs, achieving up to 3.52× throughput speedup under WAN conditions down to 50 Kbps.

  • Problem

    WAN latency, bandwidth limits, and rejection-driven rollbacks constrain edge-cloud speculative decoding and can erase its collaborative throughput benefits.

    Source: A. Speculative Decoding · p. 2, A. The Rollback Bottleneck in… · p. 3

  • Method

    AceSpec uses edge-side probabilistic caching, asymmetric main-chain and sparse-distribution transmission, and network-aware Lagrangian branch allocation.

    Source: §I · p. 2

  • Results

    3.52× throughput speedup over traditional baselines is reported, with efficient generation sustained under WAN conditions down to 50 Kbps.

    Source: §I · p. 2, §VI · p. 9

  • Takeaways & Limitations

    AceSpec provides a training-free edge-cloud inference framework designed to remain efficient under constrained and volatile WAN conditions.

    Source: §I · p. 2, §VI · p. 9

Abstract

from arXiv · show

Deploying Large Language Models (LLMs) on edge devices typically relies on model compression or split inference. However, compression degrades reasoning capabilities, while split inference suffers from severe Wide Area Network (WAN) communication bottlenecks. Edge-cloud speculative decoding emerges as a promising alternative, leveraging an edge small model to draft tokens for cloud verification. Yet, over volatile WANs, inevitable prediction rejections trigger catastrophic pipeline stalls and network-wide rollbacks, neutralizing collaborative gains. To overcome this, we propose AceSpec, an asymmetric edge-cloud collaborative framework. AceSpec utilizes un-saturated edge compute to proactively construct a probabilistic state cache, effectively transforming network-wide pipeline flushes into $\mathcal{O}(1)$ local memory lookups. To preserve bandwidth, it employs an asymmetric communication protocol that transmits minimal main-chain indices uplink and compact sparse distributions downlink. Furthermore, we introduce a network-aware, Lagrangian-optimized resource allocation strategy that dynamically maximizes the local cache hit rate. Evaluations demonstrate that AceSpec achieves up to a 3.52$\times$ throughput speedup and exhibits exceptional bandwidth immunity, sustaining near-peak inference performance even under severely constrained 50 Kbps WAN conditions.

I. INTRODUCTION

AceSpec addresses WAN bottlenecks and rollback stalls in edge-cloud LLM inference with a training-free, asymmetric framework that uses edge computation for probabilistic caching. It reports up to 3.52× throughput speedup while sustaining efficient generation down to 50 Kbps WAN conditions.

  • Compression degrades reasoning and generative capabilities, while split inference encounters a communication wall from transmitting large hidden states over WANs.Source: §I · p. 1
  • Speculative decoding reduces communication by having an edge SLM draft tokens for parallel cloud verification, but volatile WANs make rejection-induced stalls costly.Source: §I · pp. 1–2
  • AceSpec proactively expands a probabilistic multi-branch token tree during cloud verification and converts WAN rollbacks into O(1) local memory lookups.Source: §I · p. 2
  • Its two-level optimization balances edge compute and network delay through bandwidth-aware budgeting and Lagrangian-optimized non-uniform branch allocation.Source: §I · p. 2
  • 3.52× throughput speedup is achieved over traditional baselines, with efficient generation sustained under WAN conditions down to 50 Kbps.Source: §I · p. 2

II. RELATED WORK

Prior edge-cloud and speculative-decoding systems reduce some computation or latency, but remain constrained by WAN synchronization, bandwidth, hardware, or deployment requirements. AceSpec uses local compute and optimized probabilistic caching to target these limitations without inflating communication overhead.

Speculative Decoding

  • Existing speculative-decoding methods assume near-zero communication costs and degrade when frequent state synchronization meets high-latency, bandwidth-constrained WANs.Source: A. Speculative Decoding · p. 2

Edge-Cloud Collaborative Inference

  • Spatial splitting and routing schemes reduce local computation but leave edge devices idle during remote processing and transmission, preserving the WAN communication wall.Source: B. Edge-Cloud Inference · p. 3

Distributed Speculative Decoding

Pipelined Architectures

AceSpec Motivation

  • AceSpec uses available local compute to build probabilistic multi-branch caches and optimizes allocation to hide WAN latency without inflating communication overhead.Source: C. Distributed Speculative Decoding · p. 3

III. METHOD

Edge-cloud speculative decoding drafts tokens locally and verifies them in the cloud, but synchronous execution and rejection bubbles expose inference to WAN latency. AceSpec overlaps transmission with local multi-branch expansion and recovers rejected states from an edge cache.

Rollback Bottleneck

  • An edge SLM drafts γ tokens while a cloud Target LLM performs parallel verification, with expected accepted tokens determined by the average acceptance rate α.Source: A. The Rollback Bottleneck in… · p. 3
  • The synchronous stop-and-wait cycle combines drafting, WAN RTT, and cloud verification, leaving edge compute underutilized when RTT is high.Source: A. The Rollback Bottleneck in… · p. 3
  • Asynchronous pipelines can mask WAN delay under perfect acceptance, but single-chain synchronization makes rejection bubbles include downlink, redrafting, and uplink delays.Source: A. The Rollback Bottleneck in… · p. 3

Asymmetric Architecture

  • AceSpec shifts computation toward the edge and uses an asymmetric architecture with an edge Tree-based Drafter and cloud server.Source: B. AceSpec · p. 4

Asymmetric Execution

  • The edge uploads only main-chain token indices, expands branches during cloud turnaround, and stores resulting state sequences in a probabilistic cache.Source: B. AceSpec · p. 4

Local Recovery

  • On rejection, the cloud returns an index and partial target distribution, enabling local resampling and cache-based state-pointer updates instead of sequential redrafting.Source: B. AceSpec · p. 4

C. Asymmetric Tree Construction and O(1) State Recovery

AceSpec constructs a multi-branch token tree during the asynchronous window and packs branches into one masked forward pass. Rejected states are recovered through local cache lookup and O(1) KV compaction.

Asymmetric Tree Construction

O(1) State Recovery

IV. THEORETICAL OPTIMAL BRANCH ALLOCATION

AceSpec models multi-branch caching through theoretical cycle-time components and derives a speedup lower bound that depends on ideal masking and rollback penalties.

Expected Time and Speedup Analysis

B. Problem Formulation with Two-Level Constraints

AceSpec formulates cache construction as a two-level constrained optimization: network conditions determine the available tree budget, while hardware limits prevent edge computation from becoming the bottleneck.

B. Problem Formulation with Two-Level Constraints

1) Network-Aware Budgeting:

  • Tree drafting must finish within the cloud turnaround window: Ttree draft(B) ≤ RTT(t) + Tverify.This prevents multi-branch drafting from becoming a new computational bottleneck.Source: 1) Network-Aware Budgeting · p. 6
  • AceSpec continuously probes RTT(t) to derive a network-aware computation budget.The supplied passage introduces Bnet(t) as the resulting budget.Source: 1) Network-Aware Budgeting · p. 6
  • The effective budget B∗(t) is bounded by both the network-derived limit and the edge device’s VRAM and parallel-capacity limit Bmax.Ttree draft(·) is the inverse latency function of the edge SLM.Source: 1) Network-Aware Budgeting · p. 6
  • B∗(t) dynamically adapts tree size to current network delay while respecting edge hardware capacity.Source: 1) Network-Aware Budgeting · p. 6
  • AceSpec enlarges the tree under network fluctuations and reduces it under favorable conditions to conserve edge energy.This expresses the compute-energy tradeoff of network-aware budgeting.Source: 1) Network-Aware Budgeting · p. 6

2) Task-Aware Shape Allocation:

Task-aware allocation shapes the alternative branches to maximize cache hits under a fixed budget, accounting for acceptance probabilities, sequential survival, and long-tail residual distributions.

2) Task-Aware Shape Allocation:

A. Experimental Setup

AceSpec is evaluated on a simulated edge-cloud testbed spanning multiple model pairs, task types, network conditions, and inference baselines.

A. Experimental Setup

  • The edge runs on an NVIDIA Jetson AGX Orin with 32GB, while the cloud uses 4× NVIDIA A100 GPUs with 40GB each.Linux tc emulates WAN throughput limits and latency.Source: A. Experimental Setup · p. 7
  • The evaluation therefore varies hardware scale, model architecture, parameter size, task complexity, and token acceptance rate.Source: A. Experimental Setup · p. 7
  • Model pairs cover Qwen-0.6B/32B, Qwen-1.7B/32B, and LLaMA-1B/70B configurations.Source: A. Experimental Setup · p. 7
  • Datasets include GSM8K for mathematical reasoning, HumanEval for code generation, and Alpaca for instruction following.Source: A. Experimental Setup · p. 7
  • AceSpec is compared with autoregressive decoding, Vanilla Speculative Decoding, Split Inference, and additional inference paradigms.The supplied passage states that five paradigms are compared, but truncates the final list.Source: A. Experimental Setup · p. 7

B. End-to-End Performance Comparison

AceSpec delivers strong throughput across model pairs and datasets, remains near-peak as bandwidth falls to 50 Kbps, and benefits from non-uniform branch allocation. Its main limitations are task-specific regression and computational overhead from multi-branch drafting.

Throughput comparison

  • 3.52x throughput speedup is achieved with LLaMA-1B/70B on HumanEval, while Qwen-0.6B/32B reaches 1.75x on GSM8K.On GSM8K, AceSpec exceeds PicoSpec at 1.45x and DSSD at 0.85x.Source: B. End-to-End Performance Comparison · p. 7

Limitations

  • AceSpec reaches 0.98x speedup with Qwen-1.7B/32B on HumanEval, below the 1.00x cloud-only Autoregressive and PicoSpec baselines.The regression is associated with rigid code-generation precision demands and increased edge latency from the larger draft model.Source: B. End-to-End Performance Comparison · pp. 7–8

Bandwidth robustness

Branch allocation ablation

  • Non-uniform branch allocation reaches 24.37 tokens/s on GSM8K, compared with 20.14 tokens/s for single-branch and 21.58 tokens/s for uniform allocation.The strategy shifts budget toward shallower, high-confidence nodes to maximize cache hits under a fixed compute budget.Source: D. Ablation Study on Branch Allocation… · p. 8
  • Non-uniform allocation reduces GSM8K TTFT to 436.62 ms and raises draft hit rate to 0.97, whereas uniform expansion reaches 450.79 ms TTFT.Uniform dense-tree construction delays main-chain dispatch by saturating edge compute.Source: D. Ablation Study on Branch Allocation… · p. 9

Limitations

E. Sensitivity Analysis

AceSpec remains effective across changes in edge branch budget and sampling temperature, with diminishing returns at larger budgets and hit rates above 81% under diverse generation.

Branch budget B

  • As B increases from 20 to 40, draft hit rate rises from approximately 0.750 to over 0.850, then flattens toward B=80.The flattening indicates diminishing marginal returns as the edge explores more alternative tokens.Source: E. Sensitivity Analysis · p. 9, Figure 6 · p. 9
  • The branch-budget curve guides selecting a threshold that maximizes draft hit rate without saturating edge computation.A smaller budget captures highly probable tokens, while further expansion yields less additional benefit.Source: E. Sensitivity Analysis · p. 9

Sampling temperature T

  • At temperatures T ≥1.5, AceSpec maintains a draft hit rate above 81% despite increased generation randomness.Across T from 0.3 to 1.7, hit rate declines from 0.885 at low temperatures to 0.811 at high temperatures.Source: E. Sensitivity Analysis · p. 9, Figure 7 · p. 9
  • The hit-rate decline becomes steeper above T=1.0, but the multi-branch cache remains effective for deterministic and creative generation.The cache captures alternative tokens across diverse sampling conditions.Source: E. Sensitivity Analysis · p. 9

Related papers

Loading 2609.02514v1…