Source-linked AI summary
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Mingkang Liu, Weiming Meng, Peipei Jiao, Jiawei Zhang, Tianyuan Liu, Junyan Li, Lichen Pang
TL;DR
Soft-label distillation matches the target’s full output distribution, although speculative decoding acceptance depends on the ranking of likely tokens and draft models have limited capacity. RCSD uses accepted and rejected sequences from verification to optimize draft models for acceptance length. It improves average acceptance length by 35% over HASS and achieves 1.35× faster end-to-end inference on practical workloads without compromising generation quality.
Problem
Soft-label distillation minimizes distribution mismatch, but limited draft-model capacity makes exact target replication impractical even though acceptance depends on top-token ranking.
Source: §1 · p. 1Method
RCSD trains draft models with DPO or GRPO using accepted and rejected sequences generated during speculative inference as implicit preference or reward signals.
Source: §1 · p. 2Results
35% higher average acceptance length and 1.35× faster end-to-end inference on practical workloads than HASS, without compromising generation quality.
Source: §1 · p. 2Takeaways & Limitations
RCSD aligns draft-model optimization with maximizing expected acceptance length rather than matching the target’s full distribution.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.
1 Introduction
Soft-label distillation optimizes distribution matching even though speculative decoding depends on top-token ranking and accepted-token count. RCSD instead uses verification feedback to train draft models toward acceptance length, improving acceptance and practical inference speed.
1 Introduction
- Soft-label distillation minimizes forward KL by matching the target distribution, but limited draft-model capacity makes exact replication impractical and can leave acceptance misaligned with low KL.Acceptance depends on top-candidate ranking rather than exact probabilities; a draft can rank top tokens correctly yet incur large distillation loss.Source: §1 · p. 1, Figure 1 · p. 2
- RCSD reframes draft-model training as maximizing expected acceptance length rather than covering low-probability tails of the target distribution.The proposed mode-seeking behavior concentrates capacity on sequences with high acceptance probability.Source: §1 · p. 1
- RCSD derives implicit preference signals from accepted and rejected sequences during speculative inference and uses DPO or GRPO to optimize the draft model.DPO uses chosen and rejected sequences; GRPO uses acceptance rates across groups of sequences with different drafting intervals.Source: §1 · p. 2
- 35% higher average acceptance length and 1.35× faster end-to-end inference on practical workloads than HASS, without compromising generation quality.Reported across Qwen3-1.7B and Llama-3.1-8B.Source: §1 · p. 2
2 Related Work
Prior work advances speculative decoding through proposal and inference-time techniques, while draft-model training has largely emphasized distribution matching or preference optimization for other alignment goals. This section positions RCSD as applying preference optimization to draft prediction accuracy and acceptance efficiency.
2.1 Speculative Decoding for Autoregressive Inference
- Speculative decoding uses a draft model to propose tokens for parallel target verification; later methods expand proposals, while Figure 2 reports SFT+DPO and SFT+GRPO outperforming SFT on throughput and mean accept length.The figure evaluates Llama-3.1-8B-Instruct with SGLang 5.5.0; prior decoding methods include Medusa, lookahead, multi-token prediction, and EAGLE.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
2.2 Draft Model Training via Knowledge Distillation
- Forward-KL distillation encourages mode coverage, allocating draft capacity to low-probability regions despite acceptance length depending mainly on high-probability token proposals.Temperature scaling and truncated distillation mitigate the issue, but retain distribution matching as the objective; Figure 3 depicts RCSD’s accepted/rejected sequence training routes.Source: §2.2 · p. 3, Figure 3 · p. 4
2.3 Reinforcement Learning Methods
- Prior DPO and group-wise optimization methods use preference comparisons, but RCSD applies pairwise and group-wise optimization to acceptance efficiency using target-derived signals.Existing applications primarily target response quality or human-preference alignment; DPO avoids an explicit reward model, while GRPO-style methods use relative comparisons.Source: §2.3 · p. 4
2.4 Reinforcement Learning in Speculative Decoding
- Earlier RL work tunes inference-time hyperparameters, whereas RCSD optimizes draft prediction accuracy using target-model feedback as an implicit reward, without external reward models or human annotations.The approaches are described as orthogonal; the target provides reference outputs, verifies tokens, and supplies preference signals.Source: §2.4 · pp. 4–5
3 Methodology
RCSD trains speculative-decoding draft models with target-model feedback, using pairwise DPO and group-wise GRPO to align optimization with token verification acceptance. The methods derive preference or quality signals from target and speculative generation paths.
3 Methodology
- RCSD combines pairwise DPO and group-wise GRPO to train draft models to maximize token verification acceptance rates.Both methods use the target model as an implicit reward signal to align the draft policy with target generation behavior.Source: §3 · p. 5
3.1 Direct Preference Optimization for Eagle Training
- DPO pairs target-generated chosen sequences with speculative sequences generated by alternating target tokens and draft blocks, establishing an implicit preference for the target sequence.The sequence representation includes target-model hidden states as well as tokens.Source: §3.1 · p. 5
3.1.1 DPO Objective
- DPO defines a policy-versus-reference log-ratio over post-prompt tokens and minimizes a sigmoid loss on the chosen–rejected score difference, with β controlling optimization strength.The reference draft model is frozen and initialized from pretrained weights.Source: §3.1.1 · p. 5
3.1.2 Training Protocol
- DPO calculates sequence log-probabilities only for post-prompt tokens and masks target-generated tokens in rejected sequences to reduce penalties on already aligned tokens.It applies identical masking patterns to chosen and rejected sequences because DPO is length-sensitive.Source: §3.1.2 · p. 6
3.2 Group-wise Relative Policy Optimization for Eagle Training
- GRPO groups target-generated chosen sequences with speculative paths in which target tokens alternate with unverified draft blocks, then uses target-model quality scores to compare paths.The groups can include chosen and speculative steps 1–3.Source: §3.2 · p. 6
3.2.1 GRPO Objective
- GRPO defines a probability ratio against a frozen reference draft model and uses normalized group rewards in its loss.The passages identify Z-score normalization and a loss minimized over groups.Source: §3.2.1 · p. 6
3.2.2 Training Protocol
- GRPO computes quality scores over post-prompt tokens and applies consistent masking across groups, excluding target-generated tokens in speculative sequences from log-probability calculations.Policy and reference log-probabilities use next-token prediction alignment.Source: §3.2.2 · p. 7
4 Experiments
Across Qwen3-1.7B and Llama-3.1-8B, preference optimization improves acceptance length over HASS, with GRPO generally leading. The gains grow with longer drafting horizons.
4.1 Setup
- Experiments use Qwen3-1.7B and Llama-3.1-8B, training on 90% of ShareGPT and evaluating on four datasets.The compared variants are HASS, HASS+DPO, and HASS+GRPO, tested at speculative steps 3–6.Source: §4.1 · p. 7
4.2 Main Results
- At step 6, Qwen3-1.7B averages 2.26 (HASS), 2.97 (HASS+DPO), and 3.00 (HASS+GRPO); GRPO is highest or tied-best on Llama-3.1-8B across most settings.Table 1 reports Qwen3-1.7B results for steps 3–6; Table 2 reports Llama-3.1-8B results.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
4.3 Analysis
- Across both model families, preference-based optimization improves acceptance length over HASS, and the relative gain grows with longer speculative horizons.For Qwen3-1.7B on GSM8K, the gain rises from 31% at step 3 to 49% at step 6.Source: §4.3 · p. 8
5 Conclusion
RCSD trains draft models to maximize acceptance length using preference signals from accepted and rejected sequences. Experiments report improved acceptance length and faster practical inference while preserving generation quality.
5 Conclusion
- RCSD shifts draft-model training from distribution matching to acceptance-length maximization, using naturally occurring accepted and rejected sequences as implicit preference signals.Its formulations include pairwise DPO and group-wise relative policy optimization via GRPO.Source: §5 · p. 8
- Masked loss excludes target-generated tokens from gradient updates.Source: §5 · p. 9
- RCSD improves average acceptance length by 35–40% over conventional distillation baselines and achieves 1.2–1.35× end-to-end inference speedup on practical workloads while preserving generation quality.The passage also reports that GRPO accelerates early-stage convergence by using relative quality signals across multiple speculative paths.Source: §5 · p. 9