PaperlayerView exact revision on arXiv

Source-linked AI summary

PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation

Zihao An, Huajun Bai, Ziqiong Liu, Dong Li, Emad Barsoum

arXiv:2504.18583v4cs.LGcs.PF

TL;DR

Autoregressive generation limits LLM inference speed, while target-dependent draft methods require separate adaptation for each target model. PARD uses a target-independent draft model that predicts multiple future tokens in one pass, with COD reducing adaptation training cost. On vLLM, it reaches 264.88 tokens per second on LLaMA3.1-8B, 1.15× faster than EAGLE-3.

  • Problem

    Target-dependent speculative-decoding methods require a separately trained draft model for each target, increasing adaptation and deployment costs.

    Source: §1 · p. 1

  • Method

    PARD uses one target-independent draft model to predict multiple future tokens in a single forward pass, and COD adapts autoregressive draft models into parallel ones.

    Source: §1 · pp. 2–3

  • Results

    On vLLM, PARD reaches 264.88 tokens per second on LLaMA3.1-8B, a 3.67× speedup over standard autoregressive inference and 1.15× over EAGLE-3.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    A single PARD draft model can accelerate a family of target models without model-specific adaptation, lowering deployment and adaptation costs.

    Source: §6 · p. 9

Abstract

from arXiv · show

The autoregressive nature of large language models (LLMs) fundamentally limits inference speed, as each forward pass generates only a single token and is often bottlenecked by memory bandwidth. Speculative decoding has emerged as a promising solution, adopting a draft-then-verify strategy to accelerate token generation. While the EAGLE series achieves strong acceleration, its requirement of training a separate draft head for each target model introduces substantial adaptation costs. In this work, we propose \textbf{PARD (PARallel Draft)}, a novel speculative decoding method featuring \textit{target-independence} and \textit{parallel token prediction}. Specifically, PARD enables a single draft model to be applied across an entire family of target models without requiring separate training for each variant, thereby minimizing adaptation costs. Meanwhile, PARD substantially accelerates inference by predicting multiple future tokens within a single forward pass of the draft phase. To further reduce the training adaptation cost of PARD, we propose a COnditional Drop-token (COD) mechanism based on the integrity of prefix key-value states, enabling autoregressive draft models to be adapted into parallel draft models at low-cost. Our experiments show that the proposed COD method improves draft model training efficiency by \textbf{3$\times$} compared with traditional masked prediction training. On the \texttt{vLLM} inference framework, PARD achieves up to \textbf{3.67$\times$} speedup on LLaMA3.1-8B, reaching \textbf{264.88} tokens per second, which is \textbf{1.15$\times$} faster than EAGLE-3. Our code is available at https://github.com/AMD-AIG-AIMA/PARD.

1 INTRODUCTION

PARD is a target-independent speculative decoding method that predicts multiple future tokens in one draft-model forward pass. It aims to reduce adaptation costs while improving inference efficiency; COD adapts autoregressive draft models into parallel ones at lower training cost.

1 INTRODUCTION

  • Target-dependent approaches such as Medusa and EAGLE require a separately trained draft model for each target, increasing adaptation and deployment costs.PARD instead uses one draft model across a family of target models.Source: §1 · pp. 1–3
  • Vanilla speculative decoding is target-independent, but its draft model’s computational cost can lower overall speedup despite higher accuracy than the EAGLE head in the reported comparison.That comparison uses LLaMA3.2-1B as the draft and LLaMA3.1-8B as the target.Source: §1 · p. 2
  • PARD predicts multiple future tokens in a single draft-phase forward pass, while vanilla speculative decoding drafts autoregressively over multiple passes.The method introduces mask tokens for parallel prediction.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • COD uses the integrity of prefix key-value states to adapt autoregressive draft models into parallel ones, improving training efficiency by up to 3× while maintaining accuracy.The paper presents COD as a low-cost training strategy for PARD.Source: §1 · pp. 2–3
  • 3.67× speedup on LLaMA3.1-8B, reaching 264.88 tokens per second on an A100-40GB GPU, and 1.15× faster than EAGLE-3.The result is reported with PARD integrated into vLLM.Source: §1 · p. 3

2 PRELIMINARIES

Autoregressive decoding generates tokens sequentially, limiting inference efficiency through repeated memory access. Speculative decoding reduces this burden by drafting candidate tokens and verifying them in parallel.

2.1 AUTO-REGRESSIVE NATURE OF LLMS

  • Autoregressive inference generates each token from preceding tokens, making decoding memory-bound through repeated loading of model weights and the KV cache.This limits GPU utilization for parallel computation and contributes to high decoding latency.Source: §2.1 · p. 3

2.2 SPECULATIVE DECODING

  • Speculative decoding drafts K candidate tokens with a smaller model, then has the target model verify them in parallel.Vanilla speculative decoding generates the draft candidates autoregressively; speculative sampling preserves performance.Source: §2.2 · p. 4
  • PARD reduces the draft model’s work from K forward passes to one, reducing its total time to 1/K of the original.The target model continues to verify candidate tokens in parallel.Source: §2.2 · p. 4

3 PARD FRAMEWORK

PARD uses mask tokens to predict multiple future tokens in one draft-model pass and adapts autoregressive draft models for parallel prediction. Its Conditional Drop-token strategy reduces training work while preserving complete prefix key–value states.

3.1 PARD INFERENCE

  • PARD replaces sequential draft-token dependencies with mask-token placeholders, allowing all K predictions in one forward pass.The predictions at each step do not depend on each other, reducing draft-model forward passes from K to one.Source: §3.1 · p. 5, Figure 4 · p. 5

3.2.1 MASK TOKENS BASED TRAINING

  • Mask-token training adapts a standard autoregressive draft model into a parallel draft model, with independent subtasks predicting future tokens.The subtasks can be trained simultaneously with minimal preprocessing, using a cross-entropy loss.Source: §3.2 · p. 5, §3.2.1 · pp. 5–6

3.2.2 CONDITIONAL DROP OF TOKENS

  • Mask-token training increases training tokens from N to K × N, motivating Conditional Drop-token (COD) to reduce computation while maintaining prediction accuracy.COD selectively drops tokens from later subtasks, which are less critical than earlier ones.Source: §3.2.2 · p. 6
  • COD retains tokens so each remaining position has complete preceding key–value states for attention computation.A retention parameter and minimum retention ratio limit how many tokens are dropped in later subtasks; with r = 0.5, training tokens fall from K × N to 2N.Source: §3.2.2 · p. 6

3.2.3 TRAINING EFFICIENCY

  • 7× higher training efficiency than EAGLE and 10× higher than EAGLE-3 are reported for PARD with LLaMA3.3-70B as the target.The comparison measures training cost in Pflops per 1M tokens.Source: §3.2.3 · p. 6
  • One PARD draft model can accelerate a series of target models, whereas EAGLE-style methods require a separately trained draft model for each target.The paper reports evaluating three LLaMA3 targets and three Qwen targets.Source: §3.2.3 · p. 6, Table 1 · p. 7

4 EVALUATION

On vLLM, PARD reports speedups of 3.20×–4.44× on code tasks and outpaces EAGLE-3 on LLaMA3.1-8B; ablations examine training retention, shared mask IDs, draft length, and bandwidth. The evaluation uses model-family-specific training data and reasoning and code benchmarks.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses LLaMA3, DeepSeek-R1-Qwen, and Qwen models, their series-specific training datasets, and HumanEval, GSM8K, MATH500, and SpecBench benchmarks.For each model series, the smallest variant is selected for PARD training; LLaMA 3 and Qwen training answers are regenerated to improve accuracy.Source: §4.1 · p. 7
  • Final comparisons use vLLM on an A100-40GB GPU; training uses eight MI250X cards for four epochs.The reported training settings include k = 8, r = 0.7, and rmin = 0.2.Source: §4.1 · p. 7

4.2 EXPERIMENTAL RESULTS

  • LLaMA3.1-8B runs 1.9× faster than EAGLE and 1.15× faster than EAGLE-3; code-task speedups range from 3.20× to 4.44×, with average speedups between 2.65× and 3.71×.The results are reported for PARD in the experimental comparison.Source: §4.2 · pp. 7–8
  • One PARD draft model accelerates three LLaMA3 targets and three Qwen targets without model-specific adaptation, unlike EAGLE-style methods that require separate training per target.The authors report this target-independence as lowering the deployment barrier.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

  • 3× faster training with original accuracy is achieved at r = 0.7 and rmin = 0.2, balancing the acceleration from lower retention against possible performance degradation.All experiments adopt these retention parameters.Source: §4.3 · p. 8
  • 221.97 tokens/s with a shared mask token ID exceeds 218.05 tokens/s with distinct IDs and supports inference with a larger K than used in training.The authors associate the shared-ID strategy with prediction consistency and extrapolation capability.Source: §4.3 · p. 8
  • Kinfer = 12 gives the best performance, while results remain stable when Ktrain ≥ 8; the study selects Ktrain = 8.PARD’s shared mask token ID enables Kinfer to exceed Ktrain.Source: §4.3 · p. 8, Figure 5 · p. 9
  • PARD achieves a higher acceptance ratio than the EAGLE series while consuming less draft-phase bandwidth; at batch sizes 1–16, its speedups range from 1.33× to 3.63×.The batch-size increase shifts the inference bottleneck from memory-bound to compute-bound.Source: §4.3 · p. 8, Table 2 · p. 8, Table 3 · p. 8

5 RELATED WORK

Related work addresses inference efficiency through model, decoding, and system techniques. Speculative decoding methods differ in how they construct drafts, introduce parallel prediction, or improve verification.

5 RELATED WORK

  • Quantization and KV-cache methods target computation, memory use, or long-context management, while system techniques optimize kernels and scheduling for hardware utilization and throughput.Examples include GPTQ, AWQ, GQA, MLA, FlashAttention, and vLLM.Source: §5 · p. 9
  • Speculative decoding generates candidate tokens with a draft model for target verification; other approaches use text retrieval or reuse selected target-model layers to construct drafts.The passage lists LOOKAHEAD, PLD+, REST, and SuffixDecoding as retrieval approaches, and LayerSkip, Kangaroo, and SWIFT as layer-reuse methods.Source: §5 · p. 9
  • Draft-accuracy methods incorporate target representations, parallel speculative decoding introduces mask tokens, and tree-based verification methods optimize token acceptance rates.The passage lists Medusa and EAGLE among target-representation methods, BiTA and PaSS among mask-token methods, and Sequoia and EAGLE-2 among tree-based methods.Source: §5 · p. 9

6 CONCLUSION

PARD combines target-independent drafting with parallel token prediction, and COD adapts autoregressive draft models into parallel ones using prefix key-value-state integrity. Experiments report consistent speedups over vanilla speculative decoding and EAGLE-based methods, including on LLaMA3.1-8B.

6 CONCLUSION

  • PARD lets one draft model serve a family of targets while reducing adaptation and deployment costs; COD uses prefix key-value-state integrity to adapt autoregressive drafts into parallel ones at lower cost.PARD supports parallel token prediction.Source: §6 · pp. 9–10
  • 264.88 tokens per second on LLaMA3.1-8B corresponds to a 3.67× speedup over standard autoregressive inference and a 1.15× speedup over EAGLE-3.The authors also report consistent outperformance of vanilla speculative decoding and EAGLE-based methods on LLaMA3 and Qwen families.Source: §6 · p. 10

A TRAINING COST

The training-cost comparison reports higher PARD training efficiency than EAGLE and EAGLE-3 for a LLaMA3-70B target. The appendix describes the cost-accounting assumptions for PARD and EAGLE-family draft training.

A TRAINING COST

  • PARD’s non-long-text forward-pass cost is approximated as twice model size, and its backward pass requires twice the forward cost.The stated forward-pass factor accounts for multiply and add operations; the effective input token count after COD mask prediction is 3.37M.Source: A TRAINING COST · p. 15
  • EAGLE’s forward pass includes the target model and draft components, while its backward pass excludes target-model gradients and accounts separately for draft-layer and lm-head costs.The passage specifies the EAGLE forward cost as 143.74 Pflops per 1M tokens.Source: A TRAINING COST · p. 15
  • 7× higher training efficiency than EAGLE and 10× higher than EAGLE-3 are reported for training draft models on LLaMA3-70B.Training cost is measured in Pflops per 1M tokens.Source: A TRAINING COST · p. 15
  • PARD’s target-independent draft can serve a model series, whereas EAGLE-style methods require a separate draft for each target.The passage gives LLaMA3-8B, 70B, and 405B as examples of a series.Source: A TRAINING COST · p. 15

B CONDITIONAL DROP TOKENS ALGORITHM

COD adapts autoregressive training data for PARD by dropping tokens while preserving complete prefix key-value context. Its ablation reports threefold faster training with the same target-model inference acceleration.

B CONDITIONAL DROP TOKENS ALGORITHM

  • With retention rate r = 0.5, the example drops no tokens for position +1, 50% for +2, and 75% for +3.For position +3, only tokens with intact prefixes are valid candidates; one is selected randomly.Source: B CONDITIONAL DROP TOKENS ALGORITHM · p. 16
  • 3× faster training with the same target-model inference acceleration is reported for COD in the ablation.The comparison is against training without COD.Source: B CONDITIONAL DROP TOKENS ALGORITHM · p. 16
  • COD selects tokens to drop while ensuring retained tokens have complete preceding key-value context.For each prediction position, the procedure uses a decaying retention rate with a minimum floor, then updates and merges the training sequences and attention mask.Source: B CONDITIONAL DROP TOKENS ALGORITHM · p. 16

C TRAINING HYPERPARAMETERS

Table 4 summarizes the hyperparameters used for training.

C TRAINING HYPERPARAMETERS

D LARGE BATCH SIZE INFERENCE

On LLaMA3.1-8B HumanEval inference, PARD retains speedups across batch sizes as the bottleneck shifts from memory-bound to compute-bound. The DeepSeek-Qwen evaluation covers mathematics and code tasks.

D LARGE BATCH SIZE INFERENCE

E PERFORMANCE ON DEEPSEEK QWEN SERIES

F PERFORMANCE DIFFERENCES ACROSS DIFFERENT INFERENCE

Inference throughput with Transformers is reported as significantly lower than with vLLM, so final experiments use vLLM. The paper also compares optimized Transformers+ and vLLM on HumanEval and GSM8K.

F PERFORMANCE DIFFERENCES ACROSS DIFFERENT INFERENCE

  • Transformers inference throughput is significantly lower than vLLM, and final experiments therefore use vLLM.The comparison is motivated as better reflecting real-world usage scenarios.Source: F PERFORMANCE DIFFERENCES ACROSS… · p. 17
  • For vanilla SPD, Transformers-framework relative speedups are reported as lower than those measured on vLLM; examples are 1.36× and 2.26×.Source: F PERFORMANCE DIFFERENCES ACROSS… · p. 17
  • Transformers+ uses torch.compile and a static key-value cache, is described as lightweight while approaching vLLM performance, and is used for development and testing.Table 7 compares Transformers, Transformers+, and vLLM on HumanEval and GSM8K for LLaMA3.1-8B.Source: F PERFORMANCE DIFFERENCES ACROSS… · p. 18, Table 7 · p. 18
Loading 2504.18583v4…