Source-linked AI summary
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang, Fan Yin, Shuai Shi, Yefei Chen, Xiaomin Dong, Qiaoling Chen, Jin Pan, Ji Li, Laixin Xie, Yineng Zhang, Lei Yu, Yonggang Wen, Ivor Tsang, Tianwei Zhang
TL;DR
LLM inference is slowed by sequential autoregressive decoding, while speculative decoding remains constrained by scarce high-quality drafts and scalable training infrastructure. SpecForge addresses these gaps with production-oriented EAGLE-3 training and SpecBundle draft models, achieving up to 9.9× speedup over existing training approaches. The study also derives practical training guidance for deploying speculative decoding.
Problem
Speculative decoding adoption is limited by scarce high-quality draft models and insufficient scalable training infrastructure for large target models.
Method
SpecForge combines target-draft decoupling, hybrid parallelism, optimized EAGLE-3 Training-Time Test kernels, and production inference-engine integration, while SpecBundle supplies trained draft models.
Results
Up to 9.9× speedup over existing approaches is achieved for speculative-decoding draft-model training, alongside SpecBundle models with up to 1.3× speedup over existing open checkpoints.
Takeaways & Limitations
SpecForge and SpecBundle provide a practical foundation for building and deploying high-quality speculative-decoding draft models.
Takeaways & Limitations
Increasing Training-Time Test length proportionally increases draft-model training time and memory consumption, and dynamic length strategies remain future work.
Abstract
from arXiv · showhide
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
1. Introduction
Speculative decoding addresses LLM inference latency, but adoption is constrained by scarce high-quality draft models and inadequate large-scale training tools. SpecForge provides production-oriented training infrastructure and SpecBundle draft models to improve practical deployment.
- LLM autoregressive generation requires a full forward pass per token, creating memory-bound inference latency that hinders real-time and high-throughput applications.
- Speculative decoding pairs a lightweight draft model with a large target model, enabling parallel verification of multiple proposed tokens.
- EAGLE-3 achieves up to 4.79× speedup on LLaMA-3.3-70B without quality degradation through hybrid features, dynamic tree generation, and Training-Time Test.
- High-quality draft models remain scarce, while publicly available drafts are often trained on small research-oriented datasets and are unsuitable for production-level use.
- SpecForge offers configurable EAGLE-3 training with target-draft decoupling, hybrid parallelism, and optimized Training-Time Test kernels for scalable long-context training.
- SpecBundle provides draft models for mainstream open-source LLM families, delivering up to 4.8x speedup without speculative decoding and 1.3× over public draft checkpoints.
- SpecForge’s training analyses examine architecture, dataset quality, and Training-Time Test configuration to provide practical guidance for draft-model development and deployment.
2. Preliminaries
Speculative decoding converts sequential token generation into draft proposal and parallel target verification, with efficiency determined by draft quality, cost, and speculation length. EAGLE-3 extends this approach through feature-level extrapolation and Training-Time Test, which simulates multi-step decoding during training.
- Speculative decoding: Speculative decoding uses an inexpensive draft model to propose token sequences that the target model verifies in one forward pass.This changes generation from serial token production to parallel verification, increasing arithmetic intensity and hardware utilization.
- Theoretical speedup: The expected tokens generated per target-model cycle depend on acceptance probability α and follow a truncated geometric distribution.The acceptance probability reflects draft-target alignment, while the target-model cycle length is random.
- Theoretical speedup: As α approaches 1, the expected cycle length approaches γ + 1, where γ is the number of speculative draft tokens.Theoretical speedup also depends on the draft-to-target cost ratio c.
- Theoretical speedup: Efficiency requires high acceptance rates, a draft cost much smaller than the target cost, and an appropriately chosen speculation length.Increasing γ raises potential tokens per step but also increases draft overhead, so modern systems may tune it dynamically.
- Draft-model evolution: Draft-model designs progressed from independent smaller models to multi-token prediction heads and feature-level extrapolation.The progression addresses architectural coupling, drafting overhead, and the difficulty of modeling long-range dependencies in token space.
- Training-Time Test: EAGLE-3 uses Training-Time Test to autoregressively generate several tokens during training, reducing error accumulation in multi-token prediction.At each step, attention combines the original training prefix with representations predicted at earlier TTT steps.
3. Challenges
EAGLE-3 training remains difficult because existing implementations use rigid parallelism and general-purpose model code, while target-model prefill can dominate large-scale training cost. These limitations motivate training pipelines aligned with inference-optimized execution.
- Training challenges: EAGLE-3 training is non-trivial despite having far fewer trainable parameters than conventional large-scale model training.The draft model is often a one-layer Transformer, but training still involves substantial systems complexity.
- Rigid parallelism strategies: Existing implementations wrap target and draft models together under fully sharded data parallelism, creating a rigid unified parallelism strategy.This approach is sub-optimal because target models vary greatly in size while draft models are typically small.
- Prefill performance: The EAGLE-3 pipeline includes a target-model prefill stage that processes the full input sequence before draft decoding.This stage is analogous to inference prefill and exposes a major throughput opportunity.
- Prefill performance: General-purpose or naively implemented models fail to exploit optimizations developed for production inference engines.These pipelines prioritize correctness and general training rather than high-throughput inference workloads.
- Prefill performance: The prefill stage can become a dominant bottleneck, increasing training time and resource consumption in large-scale draft-model training.Addressing this inefficiency requires aligning the training pipeline with inference-optimized execution.
4. SpecForge
SpecForge decouples target and draft models so each can use execution strategies suited to its role, while optimizing EAGLE-3 Training-Time Testing for scalable draft-model training. Its design combines specialized parallelism, inference-engine integration, sparse attention, and memory-efficient kernels.
- Hybrid Parallelism: SpecForge decouples target and draft models, enabling each component to use execution backends and parallelization strategies suited to its role.The draft model uses training-oriented frameworks, while the target model uses SGLang for inference.
- Hybrid Parallelism: The framework uses SGLang for target-model inference and lightweight sharding for the smaller draft model to reduce communication overhead.The draft model is typically 3–5% the size of the target model, so only optimizer states and gradients are sharded.
- Hybrid Parallelism: SpecForge supports co-located or disaggregated target–draft execution, with experiments conducted under the co-located setting.The architecture permits placement on the same GPU or on distinct GPUs.
- Sparse Tree Attention: Training-Time Testing simulates multiple autoregressive steps, but step length 7 creates substantial GPU-memory pressure because attention logits accumulate across steps.Stored attention logits account for 80% of total activation memory in the profiled implementation.
- Sparse Tree Attention: SpecForge reduces TTT memory use with FlexAttention’s streaming and BlockMask execution plus an in-place Triton backward kernel for masked softmax loss.FlexAttention avoids saving intermediate activations, while the backward kernel reuses the logits tensor to store gradients.
5. Evaluation of SpecForge
SpecForge is evaluated against an established EAGLE3 implementation across models and execution backends. It consistently improves end-to-end training performance, with SGLang providing the strongest backend results and a separation of responsibilities that reduces duplicated integration work.
- End-to-end Performance: SpecForge consistently outperforms the baseline across four model scales, achieving a maximum training speedup of 9.99×.The comparison uses token-throughput measurements and reports the best baseline setting across ZeRO Stage 2 and Stage 3.
- Impact of Target Model Backends: SGLang achieves up to 6.8× speedup over the Hugging Face and custom execution backends in target-model training.Its advantages include chunked prefill, torch.compile, CUDA Graphs, and high-performance kernels.
- Impact of Target Model Backends: The Hugging Face backend encounters runtime errors and does not robustly support large-scale MoE models.The limitation is especially visible in the Qwen3 experiments, where its throughput is substantially lower than SGLang’s.
- Impact of Target Model Backends: Integrating SGLang separates model support and low-level inference optimization from SpecForge’s training-specific system work.This lets the framework avoid duplicating model integration and maintenance efforts.
- Kernel Micro-benchmark: The optimized attention kernel reduces execution time by 62.1% and peak memory usage by 93.5% at sequence length 4096.The micro-benchmark uses TTT length 7 and measures the final TTT step against a native SDPA implementation.
6. SpecBundle
SpecBundle is a collection of EAGLE3 draft models for mainstream open-source LLMs, evaluated across general, mathematical, and coding benchmarks. It improves speculative-decoding throughput over standard inference and existing draft-model baselines, especially on math and coding tasks, while showing an MT-Bench trade-off for LLaMA-3 models.
- SpecBundle provides EAGLE3 draft models for mainstream open-source models including Llama, Qwen, and Kimi.
- SpecBundle was trained on the 1.4M-conversation Open-PerfectBlend dataset with regenerated target-model responses.The responses used temperature 0.8, and models were trained from scratch for two epochs.
- SpecBundle was evaluated on instruction-following, math, coding, and other-subject benchmarks using SGLang on NVIDIA H200 GPUs.Comparisons included standard target-model inference and existing open-source draft models where available.
- 4.48× is the maximum speedup over inference without speculative decoding, while the maximum speedup over an existing draft model is 1.35×.The reported gains apply across general, math, and coding benchmarks and dense and MoE models.
- 1.61× to 4.48× speedups over baselines occur particularly on coding and mathematics benchmarks.The passage attributes this gap to greater math- and code-sample coverage in the training data, while noting a slight MT-Bench decrease for LLaMA-3 8B and 70B.
- SpecBundle currently focuses on instruction models, with planned extensions to reasoning and vision–language models.
7. Training Insights
The training studies examine data regeneration, TTT length, and draft-model architecture as determinants of speculative-decoding quality and efficiency. They find gains from regenerated data and task-dependent TTT scaling, but longer TTT and MoE drafts introduce practical efficiency or performance trade-offs.
- 7.1. Impact of Data Regeneration: Data regeneration increases acceptance length on nearly all benchmarks except FinanceQA and improves average throughput by 5.3%.The experiment compared LLaMA-3.1-8B EAGLE3 models trained on original and regenerated PerfectBlend data.
- 7.2. Impact of Training-Time Test: TTT length strongly improves acceptance length, with a sharp gain from 1 to 3, but the best length depends on the task.MT-Bench performs strongly at length 3, whereas Math500, GSM8K, and HumanEval favor approximately 13.
- 7.2. Impact of Training-Time Test: Increasing TTT length proportionally increases draft-model training time and memory consumption.The paper recommends scaling experiments on small data subsets and reducing TTT to 3 or 5 under memory constraints.
- 7.3. Choice of Draft Models: Dense draft models consistently outperform the evaluated MoE variants across settings.The MoE models are harder to train; under the Same Params setting, smaller experts act as weaker learners, while routing top-k of 1 exposes experts to fewer tokens.
8. Conclusion
SpecForge is presented as an efficient, scalable framework for training EAGLE3 draft models, combining target–draft decoupling with optimized kernels. The paper also releases SpecBundle and uses training analyses to support practical speculative-decoding adoption.
- SpecForge achieves up to 9.9× speedup over existing approaches while reducing memory consumption and improving training throughput.
- SpecBundle provides production-grade, high-performance EAGLE3 draft models for speculative decoding.
- The paper reports systematic training analyses intended to facilitate real-world adoption of speculative decoding.