Distributed LLM Inference via Communication-Efficient Speculative Decoding

I. INTRODUCTION

• Existing distributed methods transmit full draft or target-token distributions, creating substantial communication overhead when vocabularies are large.DSSD reduces uplink payload but still sends the full target distribution to the device upon rejection.Source: §I · p. 1
• DISCO uses a server-side replica of the draft SLM to localize the distribution needed for rejection sampling, exchanging only token indices and scalar probabilities.This removes vocabulary-dependent communication from resampling while preserving exact target sampling.Source: §I · p. 1
• The paper derives a latency formulation for the communication–computation trade-off and identifies the break-even bandwidth at which DISCO becomes advantageous.The analysis is presented as a contribution alongside the framework.Source: §I · p. 1
• DISCO achieves higher speedup than existing distributed speculative decoding methods across diverse bandwidth conditions.The introduction’s result statement is truncated after noting especially pronounced gains in low-… conditions.Source: §I · p. 1

II. BACKGROUND ON DISTRIBUTED SD

• The device-side draft SLM and target LLM are denoted M_d^q and M_p, and are assumed to share a vocabulary V of size |V|.The draft SLM autoregressively generates γ candidate tokens from the current prefix.Source: §II · p. 2, 1) Draft Phase · p. 2

1) Draft Phase:

• At each draft step, the SLM samples a token from its next-token distribution q_i.The draft length γ is the number of candidate tokens generated in one round.Source: 1) Draft Phase · p. 2

2) Verification Phase:

• The target LLM accepts a drafted token with probability min(1, p_j(x_j)/q_j(x_j)); otherwise, it resamples from the normalized positive part of the target-minus-draft distribution.If all γ drafts are accepted, the target LLM samples one additional token; the procedure preserves the target LLM distribution.Source: 2) Verification Phase · p. 2

B. Distributed speculative decoding

• In distributed speculative decoding, the device generates γ drafts and sends them to the server for target-LLM verification; accepted tokens extend the prefix and rejected tokens are replaced.Prior methods differ in where resampling occurs and what information passes between device and server.Source: B. Distributed speculative decoding · p. 2

1) DSD:

• DSD has the server perform both verification and resampling, requiring the device to send drafted token indices and full draft-token distributions.The full distributions create substantial uplink communication overhead.Source: 1) DSD · p. 2

2) DSSD:

• DSSD moves resampling to the device and sends drafted token indices with scalar probabilities, but the server must send the full target-token distribution when a draft is rejected.This reduces uplink communication during drafting while retaining a costly rejection-time downlink transfer.Source: 2) DSSD · p. 2

III. DISCO

• The device sends only drafted token indices and scalar probabilities; upon rejection, the server replica reproduces the draft-side distribution for local resampling.If all drafts are accepted, the server-side LLM samples an additional token and sends it to the device.Source: Figure 2 · p. 3, §III · p. 3
• DISCO performs rejection resampling locally at the server using an additional draft SLM identical to the device-side SLM.The replicas are assumed to share tokenizer, parameters, and decoding pipeline so they produce matched next-token distributions for the same prefix.Source: §III · pp. 2–3

• For OPT (|V| = 50,272) with γ = 4, DSD uses approximately 402.2 kB uplink per round, while DISCO uses 24 B.Upon rejection, DSSD uses approximately 100.5 kB downlink, versus 8 B for DISCO.Source: §IV · p. 3, Table I · p. 3
• DSD sends full draft-token distributions uplink; DSSD sends drafted indices and scalar probabilities but sends a full target distribution downlink upon rejection; DISCO avoids vocabulary-scale transmission both ways.The methods differ by where resampling occurs and whether a vocabulary-scale distribution must be transmitted uplink or downlink.Source: §IV · p. 3
• A round’s base latency comprises device drafting, server verification, and communication; the communication model separates RTT from payload-dependent transmission delay.The payload-dependent terms are B/C, while RTT represents bandwidth-independent round-trip delay.Source: §IV · p. 3

• DSSD sends γ drafted token indices and their scalar probabilities over the uplink.Source: T DSD · p. 3
• DISCO consistently outperforms DSSD and DSD across datasets and bandwidth settings, with especially clear benefits at low bandwidth.Source: Figure 3 · p. 4
• Upon rejection, DSSD sends the full target-token distribution over the downlink.Source: T DSD · p. 4

• DISCO’s uplink and downlink payloads are both independent of vocabulary size.Source: T DSSD · p. 4

T DISCO

• Compared with DSD, DISCO eliminates vocabulary-scale uplink communication; compared with DSSD, it eliminates the vocabulary-scale target distribution sent downlink upon rejection.Source: T DISCO · p. 4
• Because DSSD and DISCO have the same uplink payload, their latency difference depends on rejection-time downlink transmission and DISCO’s additional server-side SLM computation.The comparison is derived under the assumption btok ≪|V|bprob.Source: T DISCO · p. 4

E[T DSSD

• DISCO has lower expected round latency than DSSD when BW < BW ∗, and BW ∗ increases approximately linearly with |V|.Source: E[T DSSD · p. 4

A. Experimental Setup

• The evaluation uses OPT-125m/OPT-30B and Qwen2.5-0.5B/Qwen2.5-32B on three datasets, varying bandwidth from 1 to 100 Mbps and γ from 2 to 8.Experiments use two A100 GPUs, 100 samples per dataset, and averages over three seeds; RTT is fixed at 20 ms.Source: A. Experimental Setup · p. 4

B. Performance Under Varying Network Conditions

• Across GSM8K, CNN/DailyMail, and HumanEval, DISCO maintains approximately 1.5× speedup under extreme bandwidth restrictions, while DSD and DSSD degrade in low-bandwidth regimes.The speedup curves use a fixed draft length of γ = 4.Source: B. Performance Under Varying Network… · pp. 4–5
• DISCO achieves the best speedup across evaluated GSM8K settings and maintains more than 1.3× speedup for both model families at 1 and 5 Mbps.The comparison varies bandwidth, draft length, and model family.Source: B. Performance Under Varying Network… · p. 5, Table II · p. 5
• DISCO maintains high speedup across draft lengths, especially γ = 4–8, while DSD speedup decreases as γ increases.The passage attributes DSD's decline to transmitting more full draft-token distributions for larger γ.Source: B. Performance Under Varying Network… · p. 5

C. Communication–Computation Trade-off Analysis

• The server-side SLM requires 0.36 GFLOPs/token versus approximately 143.36 GFLOPs/token for the target LLM.The SLM performs one forward pass only when resampling is required upon rejection.Source: C. Communication–Computation Trade-off… · p. 5
• DISCO reduces communication latency compared with DSD and DSSD, while computation latency remains similar across methods.The per-token breakdown is for GSM8K at BW = 10 Mbps and RTT = 20 ms.Source: Figure 4 · p. 5, C. Communication–Computation Trade-off… · p. 5
• At 10 Mbps, DISCO's communication latency is about 16.4 ms/token lower than DSSD's, against approximately 1.5 ms/token of additional computation time.The communication term scales approximately with 1/BW, while device- and server-side computation times remain nearly constant.Source: C. Communication–Computation Trade-off… · p. 5
• The DSSD–DISCO speedup gap becomes small around 100 Mbps, and the break-even bandwidth is expected to widen approximately linearly with vocabulary size.DISCO's communication payload is vocabulary-independent.Source: C. Communication–Computation Trade-off… · pp. 5–6

D. Heterogeneous Device–Server Deployment

• DISCO achieves the highest speedup across the tested heterogeneous GPU settings, reaching up to 1.92× with an RTX 4090 device and NVIDIA A100 server.The evaluated device GPUs are RTX 3090 or RTX 4090, with two A100 GPUs at the server.Source: D. Heterogeneous Device–Server… · p. 6
• Measured speedups are slightly higher than simulation results, consistent with the LAN's 84.3 Mbps bandwidth and 0.5 ms RTT.The conclusion also reports higher speedup than autoregressive generation and existing distributed speculative decoding methods across simulation and heterogeneous deployments.Source: D. Heterogeneous Device–Server… · p. 6, §VI · p. 6

VI. CONCLUSION

• DISCO localizes the draft-side distribution for resampling without changing the target model or exact acceptance rule, supporting extension to other model families and exact speculative decoding variants.Source: §VI · p. 6
• The evaluation covers single-request settings; extending DISCO to concurrent multi-user requests is left for future work.Source: §VI · p. 6