Source-linked AI summary
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang
TL;DR
Large language model inference is slow and costly, while scaling training data provides limited improvement for EAGLE’s acceleration. EAGLE-3 removes feature prediction constraints, uses direct token prediction with multi-layer feature fusion, and achieves up to 6.5x speedup while benefiting from more training data.
Problem
LLM inference is slow and costly, and scaling training data provides limited inference-acceleration gains for EAGLE because of its feature prediction constraints.
Method
EAGLE-3 directly predicts draft tokens through training-time test and fuses lower-, middle-, and upper-layer target-model features instead of using only top-layer features.
Results
EAGLE-3 achieves a maximum speedup of 6.5x, including a 1.4x latency speedup over EAGLE-2 at batch size 1 and 40% higher SGLang throughput at batch size 64.
Takeaways & Limitations
EAGLE-3 enables the draft model to benefit from increased training data, with observed speedup increasing proportionally as training data grows.
Abstract
from arXiv · showhide
The sequential nature of modern LLMs makes them expensive and slow, and speculative sampling has proven to be an effective solution to this problem. Methods like EAGLE perform autoregression at the feature level, reusing top-layer features from the target model to achieve better results than vanilla speculative sampling. A growing trend in the LLM community is scaling up training data to improve model intelligence without increasing inference costs. However, we observe that scaling up data provides limited improvements for EAGLE. We identify that this limitation arises from EAGLE's feature prediction constraints. In this paper, we introduce EAGLE-3, which abandons feature prediction in favor of direct token prediction and replaces reliance on top-layer features with multi-layer feature fusion via a technique named training-time test. These improvements significantly enhance performance and enable the draft model to fully benefit from scaling up training data. Our experiments include both chat models and reasoning models, evaluated on five tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with about 1.4x improvement over EAGLE-2. In the SGLang framework, EAGLE-3 achieves a 1.38x throughput improvement at a batch size of 64. The code is available at https://github.com/SafeAILab/EAGLE.
1 Introduction
EAGLE-3 addresses the limited benefit of scaling draft-model training data by removing feature-prediction constraints and fusing multi-layer target features. It reports stronger scaling and improved acceleration over prior approaches.
- LLM autoregressive inference is slow and costly because each token requires accessing all model parameters.
- Speculative sampling reduces latency by generating draft tokens rapidly and verifying them in parallel.
- EAGLE-3 directly predicts tokens during simulated multi-step training instead of predicting features.
- EAGLE-3 fuses low-, mid-, and high-level target features rather than relying only on top-layer features.
- Increasing draft-model training data proportionally increases EAGLE-3’s speedup ratio, unlike the original EAGLE architecture.
- 1.4x latency speedup over EAGLE-2 is achieved at batch size 1, while throughput improves by 40% in SGLang at batch size 64.
2 Preliminaries
Speculative sampling alternates low-cost drafting with parallel target-model verification, while EAGLE accelerates drafting by autoregressing over target-model features. EAGLE-2 further adapts the draft tree using draft-model confidence.
- 2.1 Speculative Sampling: Speculative sampling alternates inexpensive draft generation with parallel verification by the target model.
- 2.1 Speculative Sampling: The draft model autoregressively generates k tokens and records their probabilities before verification.
- 2.1 Speculative Sampling: During verification, draft tokens are accepted sequentially using a probability-ratio rule, with rejected tokens replaced by a corrected sample.
- 2.2 EAGLE and EAGLE-2: EAGLE reuses target-model top-layer features, predicts the next feature autoregressively, and uses the target LM head to obtain draft tokens.
- 2.2 EAGLE and EAGLE-2: EAGLE and Medusa use predefined static draft trees, whereas EAGLE-2 dynamically generates and prunes trees using estimated acceptance confidence.
3 EAGLE-3
EAGLE-3 alternates drafting and verification, using fused multi-layer target features and draft outputs to support token generation. Training-time test simulates this feedback process during training, with modified attention masks for tree-like dependencies.
- 3 EAGLE-3: EAGLE-3 alternates between drafting and verification stages, differing from EAGLE primarily in how the drafting stage is performed.
- 3.1 Inference Pipeline: Low-, middle-, and high-level target features are concatenated and projected into a fused feature g for the draft model.The three k-dimensional feature vectors form a 3k-dimensional vector that an FC layer reduces to k dimensions.
- 3.1 Inference Pipeline: The draft model combines fused target features with sampled-token embeddings and feeds the resulting representation through an FC layer and single-layer decoder to produce draft outputs.The sampled token embedding supplies information unavailable from target features alone because the draft model cannot access the random sampling process.
- 3.1 Inference Pipeline: At later draft steps, unavailable target features are replaced by outputs from the draft model's preceding step.This lets the draft sequence continue even though newly generated tokens have not yet been checked by the target model.
- 3.2 Draft Model Training: Training-time test generates draft outputs and feeds them back into the draft model so training covers both target features and prior draft outputs as inputs.EAGLE-3 therefore adapts the draft model to the different inputs encountered during inference.
- 3.2 Draft Model Training: EAGLE-3 uses a Transformer decoder layer and adjusts self-attention masks to represent tree-like dependencies during simulated testing.The resulting masks are mostly diagonal, with the original training data used as the key; vector dot products reduce computational waste for this case.
4 Experiments
EAGLE-3 is evaluated on chat and reasoning models across five tasks using speedup, acceptance length, and acceptance-rate metrics. It consistently outperforms prior methods, while ablations and production-framework tests examine its design and throughput behavior.
- Experimental setup: Experiments cover four chat or reasoning models and five tasks without task-specific fine-tuning.Models include Vicuna 13B, LLaMA-Instruct 3.1 8B, LLaMA-Instruct 3.3 70B, and DeepSeek-R1-Distill-LLaMA 8B.
- Effectiveness: Speedup reaches 3.0x-6.5x over vanilla autoregressive generation, with 20%-40% improvement over EAGLE-2 across all tasks and target models.HumanEval achieves the highest reported speedup, up to 6.5x, with average acceptance length up to 7.5.
- Effectiveness: EAGLE-3’s acceptance rate remains almost unchanged as self-predicted inputs increase, unlike EAGLE’s substantially declining acceptance rate.The comparison uses MT-bench with LLaMA-Instruct 3.1 8B as the target model.
- Ablation Study: Removing the feature prediction constraint and fusing low-, middle-, and high-level features both significantly improve acceptance length and speedup ratio.The ablation study uses LLaMA-Instruct 3.1 8B as the target model.
- EAGLE-3 in SGLang: EAGLE-3 improves SGLang throughput by 38% at batch size 64 despite reduced speculative-sampling effectiveness at larger batch sizes.The evaluation used an H100 GPU, LLaMA-Instruct 3.1 8B, MT-Bench, and SGLang.
- EAGLE-3 in vLLM: In vLLM on RTX3090 with LLaMA-Instruct 3.1 8B, EAGLE-3 reaches its maximum throughput improvement at batch size 56 rather than batch size 24 for EAGLE.This experiment used maximum chain length 2 and did not use the tree structure.
5 Related Work
LLM inference acceleration includes quantization, distillation, and speculative sampling. Speculative sampling uses the target model for verification, while later methods improve its draft-and-verify framework.
- Quantization and distillation accelerate LLM inference but generally involve trade-offs between model performance and acceleration benefits.
- Speculative sampling uses the target model for verification to provide lossless acceleration.
- Early speculative decoding targeted greedy generation, while later speculative sampling extended draft verification to non-greedy generation.
6 Conclusion
EAGLE-3 removes feature prediction constraints and fuses features from multiple target-model layers through Training-time test. These changes preserve benefits from more training data and produce speedups up to 6.5x.
- EAGLE-3 directly predicts draft tokens through Training-time test instead of imposing a feature prediction constraint.
- EAGLE-3 fuses lower-, middle-, and upper-layer target-model features instead of relying only on top-layer features.
- 6.5x maximum speedup is achieved while EAGLE-3 continues to benefit from training-data augmentation.
A Implementation Details
The implementation compares EAGLE-3 with multiple acceleration baselines using specified models, weights, draft-tree settings, and official library configurations. EAGLE-3 uses a deeper draft tree while keeping its node count unchanged from EAGLE-2.
- Vanilla generation uses Hugging Face Transformers models with the PyTorch backend and a pre-allocated KV cache.
- Standard speculative sampling uses Hugging Face Transformers’ assisted-generation feature.
- PLD, Lookahead, Medusa, and Hydra use their default settings and officially released weights.
- EAGLE uses officially released Vicuna and LLaMA2-Chat draft weights, while its LLaMA3-Instruct draft model is trained on ShareGPT.
- EAGLE-3 increases draft-tree depth from 6 to 8 while keeping the number of nodes the same as EAGLE-2.