PaperlayerView exact revision on arXiv

Source-linked AI summary

Spexis: Speculative Lookahead Scheduling for LLM Inference

Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi, Sungkyun Kim, Jinho Lee, Jiwon Seo

arXiv:2609.34370v1cs.LGcs.DC

TL;DR

Multi-GPU LLM inference faces pipeline-parallel memory pressure and tensor-parallel communication overhead. Spexis overlaps speculative and normal execution without increasing KV-cache memory, and uses lookahead scheduling; it achieves speedups of up to 34% over a baseline using optimal pipeline and tensor parallel execution.

  • Problem

    Pipeline parallelism faces micro-batch KV-cache memory pressure, while tensor parallelism incurs synchronization overhead that can leave GPUs idle.

    Source: §3 · pp. 2–3

  • Method

    Spexis overlaps speculative and normal execution without increasing KV-cache memory, and uses lookahead scheduling based on predicted speculation quality and near-future memory pressure.

    Source: §8 · p. 9

  • Results

    Up to 34% speedups over the baseline using the optimal combination of pipeline and tensor parallel execution.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Speculative decoding can serve as a systems mechanism for improving distributed LLM inference performance, not only as an algorithmic acceleration technique.

    Source: §8 · p. 9

    Spexis's relative advantage over TP depends on hardware interconnects; with exceptionally fast inter-GPU communication, such as pure NVLink, TP alone might suffice.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Spexis is a multi-GPU LLM inference framework that improves the efficiency of pipeline and tensor parallelism through speculative parallelism. Rather than using speculative decoding only to accelerate token generation, Spexis runs speculation in parallel with normal execution, introducing a new parallelism axis without increasing KV-cache memory usage. This improves memory efficiency and helps mitigate the bottlenecks of multi-GPU inference. Spexis further uses lookahead scheduling to predict speculation quality and future memory pressure, allowing it to reduce wasted speculation, KV-cache eviction, and recomputation. Built on top of vLLM, Spexis largely improves serving performance across a range of GPU configurations, achieving speedups of up to 34% over a baseline that uses the optimal combination of pipeline and tensor parallelism. Spexis's source code is publicly available at https://github.com/mlsys-seo/spexis.

1 Introduction

Spexis addresses the bottlenecks of multi-GPU LLM inference with speculative-parallel scheduling and prediction-based admission. Its evaluation reports speedups of up to 34%.

1 Introduction

  • Pipeline parallelism faces micro-batch memory pressure, while tensor parallelism incurs synchronization overhead; both bottlenecks worsen as GPU count grows.Adding pipeline stages increases memory pressure, while using more GPUs for tensor parallelism increases communication overhead.Source: §1 · p. 1
  • Speculative-parallel scheduling runs speculation alongside normal execution, adding a parallelism axis without increasing KV-cache memory usage.The approach is intended to make multi-GPU execution more memory-efficient.Source: §1 · p. 1
  • Prediction-based scheduling estimates future memory pressure to decide when to run requests, reducing eviction and recomputation while improving batch efficiency.The framework also develops cost models for pipeline, tensor, and speculative parallelism, including recomputation costs for admission decisions.Source: §1 · p. 1
  • Across NVIDIA A40, RTX PRO 6000, H100, A100 and L40S GPUs, Spexis achieves speedups of up to 34%.The framework is implemented on top of vLLM.Source: §1 · p. 1

2 Related Work

Related work addresses single-node scheduling, distributed parallelism, length-aware scheduling, and speculative decoding. Spexis applies these ideas to multi-GPU inference bottlenecks.

2 Related Work

  • Prior single-node scheduling work improves performance but does not address bottlenecks from distributing LLM inference across multiple GPUs or nodes.The cited work addresses low decode utilization, fluctuating prefill overhead, and KV-cache memory inefficiency (Agrawal et al., 2024; Kwon et al., 2023).Source: §2 · p. 2
  • Hybrid pipeline and tensor parallelism is common, but synchronization and memory bottlenecks remain; Spexis targets them with speculative-parallel execution.The cited approaches are TP (Shoeybi et al., 2020), PP (Huang et al., 2019), and hybrid PP+TP (vLLM Team, 2025).Source: §2 · p. 2
  • Prior length-aware schedulers use predictions to increase batch size or prioritize shorter requests, while Spexis targets eviction overhead and throughput.When predicted lengths indicate that admitting a request would exceed memory capacity, its prefill can be deferred; the cost model balances request delay against recomputation cost.Source: §2 · p. 2
  • Self-speculative decoding reuses parts of the base model for drafting; Spexis applies this approach to pipeline execution to fill idle time.The cited self-speculative approaches include Zhang et al. (2024), Elhoushi et al. (2024), and Liu et al. (2024).Source: §2 · p. 2

3 Motivation: Bottleneck in LLM Serving

Pipeline parallelism is constrained by micro-batch KV-cache memory, while tensor parallelism is constrained by communication and synchronization. Increasing either parallelism degree can worsen these bottlenecks.

3 Motivation: Bottleneck in LLM Serving

  • PP micro-batches require separate KV-cache entries for their sequences, limiting micro-batch size and lowering GPU utilization and throughput.The passage identifies micro-batch KV-cache memory as PP's main bottleneck.Source: §3 · p. 2
  • TP synchronization twice per transformer layer leaves GPUs idle; NVLink helps, but non-uniform bandwidth and local all-reduce collectives remain bottlenecks.TP partitions each layer and its parameters across GPUs, which must synchronize partial outputs after computation.Source: §3 · pp. 2–3
  • The PP and TP bottlenecks worsen as their degree increases, so adding model parallelism may fail to improve throughput or can reduce it.This motivates another parallel execution strategy that alleviates both bottlenecks while retaining multi-GPU scalability.Source: §3 · p. 3

4 Spexis: Speculative-Parallel Execution

Spexis adds speculative parallelism (SP) alongside pipeline and tensor parallelism, running speculation concurrently with normal execution. Its cost models estimate effective batch size and decoding throughput to select a parallel configuration.

4 Spexis: Speculative-Parallel Execution

  • SP runs speculation in parallel with normal execution, using a self-speculative model’s earlier layers to draft tokens and the rest to verify them.The execution can run across pipeline- and/or tensor-parallel GPUs.Source: §4 · p. 3
  • Sharing the normal execution’s KV cache lets SP support larger batches, while potentially reducing the TP degree and its communication overhead.PP micro-batching instead uses separate KV caches for each micro-batch.Source: §4 · p. 3
  • SP’s effectiveness depends on draft-token acceptance; the first few tokens, especially in self-speculative models, tend to have relatively high acceptance rates.The paper reports a maximum 78% acceptance rate for its first draft token and cites a prior-work maximum of 85% for EAGLE [Li et al., 2024b].Source: §4 · p. 3
  • The cost model estimates SP’s effective batch size as the expected number of accepted tokens, using a recurrence relation.For simplicity, the model assumes equal sequence lengths and time-invariant speculation accuracy.Source: §4 · p. 4
  • Profiling supplies PP, TP, and SP decoding-latency functions; the model combines these with effective batch sizes and execution times to estimate throughput and select a parallel configuration.The estimates capture PP micro-batching memory overhead and TP synchronization overhead, with only small errors mostly due to batch-size variation.Source: §4 · p. 4

5 Lookahead Scheduling Optimization

Spexis uses lookahead scheduling to improve speculative-parallel execution by selecting draft tokens and predicting future memory pressure. It combines remaining-length estimates with an explicit trade-off between admitting requests sooner and avoiding eviction and recomputation.

5 Lookahead Scheduling Optimization

  • Lookahead scheduling predicts speculation quality and future memory pressure to reduce wasted speculation, KV-cache eviction, and recomputation.It selects tokens for speculation and can delay incoming requests when necessary.Source: §5 · p. 4

5.1 Acceptance-Guided Selective Speculation

  • Spexis ranks draft tokens by confidence, drops a fixed fraction of low-ranked tokens, and speculates only on the retained tokens.The drop ratio is chosen by profiling retained-token acceptance and speculation latency; evaluation used α values of 0.15–0.3.Source: §5.1 · p. 4
  • The selective-speculation cost model estimates steady-state batch size using the drop ratio and speculation accuracy, assuming dropping affects only speculative execution.The expected effective batch size counts accepted speculative tokens and normal executions; throughput divides it by total-batch processing latency.Source: §5.1 · pp. 4–5

5.2 Remaining-Length-Aware Scheduling

  • Remaining-length-aware scheduling delays new requests until the earliest expected iteration that avoids eviction, balancing earlier admission against eviction and recomputation costs.The policy compares aggressive immediate admission with delayed admission; the model accounts for both recomputation overhead and delay cost, and the gain can be below one when evictions are rare and delay is much longer than waiting.Source: §5.2 · pp. 5–6
  • A three-layer MLP predicts whether remaining output length falls below thresholds from 4 to 64 tokens, targeting 83% precision with reasonable recall.The resulting predictions estimate KV-cache usage over the next 64 decoding iterations and inform whether admitting a new request would require eviction.Source: §5.2 · p. 5
  • The scheduling cost model estimates throughput between request arrivals using the prefill-to-decode latency ratio and the number of running decode requests.It considers evictions under aggressive scheduling and the time evicted sequences remain unscheduled before recomputation.Source: §5.2 · p. 5

6 Implementation

Spexis extends vLLM and Ray to support speculative-parallel execution while retaining compatibility with existing optimizations. Its implementation uses a separate IPC mechanism for first-stage drafting output and moves input embedding lookup to the CPU to mitigate pipeline imbalance.

6 Implementation

  • The prototype extends vLLM’s GPURunner and uses a separate IPC mechanism to obtain first-stage output for drafting because Ray’s compiled DAG exposes stage outputs only after full model execution.The implementation remains compatible with existing vLLM and Ray optimizations, including compiled DAG optimization.Source: §6 · p. 6
  • Spexis places an LM head at the first and last pipeline stages and offloads input embeddings to host memory for CPU lookup to mitigate stage imbalance.The added lookup incurs a few hundred microseconds of overhead and has little impact on overall performance.Source: §6 · p. 6
  • The drafting adapter uses a linear layer, an attention layer, and a pretrained LM head, and is trained on ShareGPT for five epochs.For Llama-3.3-70B, training took about 12 hours on one H200 GPU; the length predictor is a three-layer MLP trained on ShareGPT for 20 epochs, taking about 15 hours on one H100 GPU.Source: §6 · p. 6

7 Evaluation

Spexis is evaluated against pipeline and tensor-parallel baselines across private-cluster and cloud GPU settings. Results cover serving latency, decoding throughput, component breakdowns, and the accuracy of its speculation and remaining-length predictions.

7 Evaluation

  • The evaluation compares Spexis with PP, TP, and their optimal combinations across private-cluster and cloud configurations, using Llama and Qwen with SpecBench, LMSYS-Chat, and UltraChat.Private-cluster tests include inter-node and intra-node setups; cloud tests use four eight-GPU configurations. Load-scaling tests use Poisson request arrivals and mean normalized latency, while cloud results report fixed-batch decoding throughput.Source: §7 · p. 6, §7.1 · p. 6

7.1 Overall Performance Improvements

  • Spexis serves up to 34% higher request rates than the two-stage pipeline baseline; Qwen on LMSYS-Chat has the largest reported gain.The inter-node load-scaling tests use Poisson arrivals and report mean normalized latency. The baseline and Spexis both use two pipeline stages; Spexis enables all Section 5 optimizations.Source: Figure 5 · p. 7, §7.1 · p. 7
  • Up to 54% higher decoding throughput is reported for Spexis at batch sizes 64–100 for Llama and 128–256 for Qwen.Spexis outperforms the baseline across batch sizes 16–256, with larger gains at higher batch sizes.Source: Figure 6 · p. 7, §7.1 · p. 7
  • On eight GPUs, Spexis outperforms TP and PP by 8% and 16%; on four GPUs, TP-only remains faster than Spexis.The intra-node comparison is for Llama-3.3-70B; the paper attributes TP-4’s slight advantage to its lower modeled throughput overhead.Source: Figure 7 · p. 7, §7.1 · p. 7

7.2 Performance Breakdown

  • Speculative-parallel execution alone improves performance by 10–15%; selective speculation adds 3 pp for Llama and 12 pp for Qwen, and remaining-length-aware scheduling adds 3 pp for both.Selective speculation drops low-confidence draft tokens; the scheduling refinement delays admissions that would otherwise evict running requests and require KV-cache recomputation.Source: Figure 8 · p. 8, §7.2 · p. 8

7.3 Performance on Cloud GPUs

  • Spexis beats TP-only in five of eight cloud model–hardware pairs, including all four PCIe cases at 1.02–1.21× speedups and Llama-3.3-70B on A100 SXM at 1.27×.TP is faster in the other three SXM cases, where its communication overhead is lower than Spexis’s speculation-failure overhead.Source: Table 2 · p. 8, §7.3 · p. 8

7.4 Accuracy of Drafting Adapter

  • Selective speculation—dropping the lowest-confidence fraction of sequences—improves speculation accuracy by 8–14%.With the first half of the model used for drafting, accuracy ranges from 0.60 to 0.79; using the first quarter yields 0.51–0.63.Source: §7.4 · pp. 8–9, Table 3 · p. 9

7.5 Accuracy of Length Prediction

  • The remaining-length predictor achieves generally high precision, mostly 70%–90.7%, and recall of 40%–71% across evaluated datasets and thresholds.The three-layer MLP is trained on a ShareGPT subset and evaluated on the remaining ShareGPT data, SpecBench, and UltraChat.Source: Table 4 · p. 9, §7.5 · p. 9

8 Conclusion

Spexis combines speculative decoding with distributed execution to add parallelism without increasing KV-cache memory use. Its lookahead scheduling uses predicted speculation quality and near-future memory pressure to reduce wasted speculation, eviction, and recomputation; evaluation reports up to 34% speedups over the optimal PP/TP baseline.

8 Conclusion

  • Spexis overlaps speculative and normal execution, adding a parallelism axis without increasing KV-cache memory usage.The framework integrates speculative decoding with distributed multi-GPU execution.Source: §8 · p. 9
  • Spexis consistently improves serving performance across diverse GPU configurations, with up to 34% speedups over the baseline using the optimal combination of pipeline and tensor parallel execution.Source: §8 · p. 9

Limitations

Spexis requires training auxiliary components, and its performance depends on speculation accuracy and hardware interconnects. Domain shifts may reduce throughput gains, while exceptionally fast interconnects may make TP alone sufficient.

Limitations

  • Deployment requires training a drafting adapter and length predictor, adding a preliminary step to the pipeline.The paper describes this overhead as relatively small.Source: Limitations · p. 9
  • Significant domain shifts may reduce expected throughput gains because Spexis is sensitive to speculation accuracy.Fallback mechanisms mitigate the impact of low-accuracy scenarios.Source: Limitations · p. 9
  • SP's advantage over TP depends on interconnects; with exceptionally fast communication such as pure NVLink, TP alone might suffice.The paper states that SP remains essential for scaling large models across diverse hardware when communication latency is a bottleneck.Source: Limitations · p. 10

A Appendix

The appendix derives steady-state batch-size limits and the throughput gain of remaining-length-aware scheduling. It begins with the saturated limit of Equation (1).

A Appendix

  • The appendix derives steady-state batch-size limits and the throughput gain of remaining-length-aware scheduling.Source: A Appendix · p. 11
  • Appendix subsection A.1 addresses the saturated limit of Equation (1).Source: A Appendix · p. 11
  • The derivation sets initial conditions with n_0 = B and states that the total quantity s^a_t + s^r_t + n_t remains B.Source: A Appendix · p. 11

A.2. Saturated Limit of Equation (3)

Appendix subsection A.2 derives a saturated limit from a recurrence and initial conditions. Subsection A.3 derives throughput ratio G between consecutive request arrivals and defines its notation.

A.2. Saturated Limit of Equation (3)

  • With initial conditions B_i = B(1 − α)^i, subsection A.2 rearranges the recurrence for B_t and sums it to derive the saturated limit.The passage states that the summation telescopes and uses the initial condition for a geometric sum.Source: §A.2 · p. 11

A.3. Derivation of Throughput Ratio G

  • Subsection A.3 derives throughput ratio G between consecutive request arrivals, defining Tok_x as decoded-token count and T_x as execution time for x ∈ {aggr, delay}.Source: §A.3 · pp. 11–12
Loading 2609.34370v1…