Source-linked AI summary
ConFu: Contemplate the Future for Better Speculative Sampling
Zongyue Qin, Raghavv Goel, Mukul Gagrani, Risheek Garrepalli, Mingu Lee, Yizhou Sun
TL;DR
Existing draft models condition only on the current prefix, allowing errors to accumulate and their distributions to drift from the target model. ConFu adds future-oriented target signals through contemplate tokens, soft prompts, dynamic MoE, and specialized training, improving acceptance rates and generation speed over EAGLE-3 by 8–11% on Llama-3 3B and 8B models.
Problem
Existing draft models condition solely on the current prefix, causing error accumulation, distribution drift, and declining token acceptance rates during decoding.
Method
ConFu provides draft models with future-oriented target signals using contemplate tokens, soft prompts, dynamic MoE, and anchor-token-based future-prediction training.
Results
8–11% average gains in token acceptance rates and generation speed over EAGLE-3 are reported for Llama-3 3B and 8B models across task categories, temperatures, and computation budgets.
Takeaways & Limitations
Future-aware signals are presented as an effective way to mitigate error accumulation and improve speculative decoding effectiveness.
Takeaways & Limitations
ConFu is an inference-time optimization that does not introduce capabilities beyond the underlying target model or alter its sampling distribution.
Abstract
from arXiv · showhide
Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose \textbf{ConFu} (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate tokens and soft prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor token sampling and future prediction replication that learns robust future prediction. ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8--11\% on Llama-3 3B/8B and by approximately 20\% on Qwen-3 4B across downstream tasks. We believe our work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.
1. Introduction
Speculative decoding depends on draft-model quality, but prefix-only drafting accumulates errors and drifts from the target model. ConFu addresses this by providing future-oriented target signals to the draft model and reports higher acceptance and speed than EAGLE-3.
- Motivation: Prefix-only draft models accumulate errors, causing their distributions to drift from the target and reducing token acceptance rates.The resulting misalignment undermines speculative decoding’s efficiency gains.
- Motivation: ConFu conditions drafting on the target model’s future generation direction so candidate tokens remain closer to its semantic trajectory.The approach aims to make draft tokens less likely to be rejected during verification.
- Contributions: ConFu introduces contemplate tokens, soft prompts, dynamic MoE-based contemplate tokens, and training with anchor token sampling and future prediction replication.These components provide auxiliary future-oriented signals and train robust future prediction.
- Results: 8–11%: ConFu improves token acceptance rates and generation speed over EAGLE-3 with Llama-3 3B and 8B models.The gains are reported across downstream tasks and diverse decoding conditions.
- Broader significance: ConFu is presented as a bridge between speculative decoding and continuous latent thought representations.The authors describe this as a new direction for accelerating LLM inference.
2. Preliminaries
Speculative decoding uses a lightweight draft model to propose tokens that the target model verifies in parallel. Its speedup comes from accepting multiple draft tokens per verification step, while tree drafting and EAGLE-style target-hidden-state guidance improve candidate quality.
- Speculative decoding: A draft model generates candidate tokens, and the target model verifies them in one parallel forward pass.This replaces repeated target-model computation for each proposed token with batched verification.
- Speculative decoding: The standard pipeline drafts K tokens autoregressively from the current sequence before verification.The draft sequence is appended to the existing prefix.
- Speculative decoding: Draft tokens are accepted sequentially when they match target-model samples; after rejection, subsequent drafts are discarded and the first rejected token is corrected.The accepted sequence becomes the input for the next drafting step.
- Speculative decoding: Speculative decoding speedup depends on how many tokens are accepted during one target-model verification step.A longer accepted path replaces multiple sequential target-model forward passes with one.
- Tree-based drafting: Tree drafting proposes multiple tokens at each step, and tree attention lets the target model validate all paths in parallel before accepting the longest consistent path.The tree structure increases the set of candidate continuations considered per verification pass.
- EAGLE: EAGLE integrates lightweight drafting into the target model, using target hidden representations rather than a separate draft model.EAGLE-3 concatenates hidden states from initial, middle, and final target layers for autoregressive drafting.
3. ConFu: The Methodology
ConFu guides speculative draft generation with future-oriented signals extracted by contemplate tokens and soft prompts. It adds context-adaptive MoE contemplate tokens and integrates them into verification and lightweight draft generation.
- 3.1. Capture Future with Contemplate Tokens: ConFu uses soft prompts and contemplate tokens to make the target model expose future-oriented signals that guide the draft model.The target model produces a future prediction vector, which the draft model consumes as an auxiliary future token.
- 3.3. Training Pipeline: The target model remains frozen during training while soft prompts and contemplate embeddings are optimized with an attention mask restricting prompt access to contemplate tokens.ConFu uses contemplate tokens as pause-like inputs that can be processed in parallel, avoiding the multiple target-model passes required by autoregressive thought-token generation.
- 3.1. Capture Future with Contemplate Tokens: The target model verifies draft candidates while generating future predictions by inserting one contemplate token after each draft node.After verification, the prediction associated with the last accepted token is passed to the draft model for the next iteration.
- 3.1. Capture Future with Contemplate Tokens: ConFu keeps additional inference overhead modest by processing contemplate tokens alongside speculative-tree nodes, with a typical tree size of T = 30.During verification, each draft node is paired with a contemplate token, so the target model processes 2T tokens in parallel.
- 3.2. Dynamic Contemplate Tokens with MoE: A fixed contemplate embedding may not summarize diverse contexts faithfully, so ConFu uses MoE modules conditioned on recently accepted-token representations.The MoE router selects and combines learnable expert embeddings to produce dynamic contemplate and future-token embeddings.
4. Experiment
ConFu is evaluated against EAGLE-3 across models, tasks, temperatures, and draft-tree budgets, showing consistent efficiency gains. Ablations attribute benefits to future-prediction replication and dynamic MoE contemplate tokens, while longer-draft tests indicate stronger tail acceptance.
- Main Results: ConFu consistently outperforms EAGLE-3 across both Llama target models and all evaluated temperatures and draft-node configurations.It achieves higher average accepted draft length and speed-up ratio under the compared settings.
- Temperature: At greedy decoding with 30 draft nodes, ConFu improves speed-up ratio by approximately 1.14× for 8B and 1.15× for 3B targets.Accept length also rises by 9.2% and 12.8% for the 8B and 3B target models, respectively.
- Ablations: Adding future prediction replication increases average accept length by about 0.17, while dynamic MoE tokens increase accept length by 0.05 and speed-up ratio by 0.02.These ablations support the contributions of robust future-prediction training and context-aware contemplate tokens.
- Qwen3 Evaluation: ConFu improves Qwen3-4B average accepted tokens by 21.0% and speed-up ratio by 19.8% over EAGLE-3.The advantage persists across draft-node and temperature configurations.
- Long-Draft Robustness: Under deterministic long-draft decoding, ConFu has higher tail acceptance than EAGLE-3 across the entire accepted-length range.The stress test uses draft length DL=12 and complements mean acceptance and speed-up measurements.
5. Related Work
Related work accelerates LLM inference through model, input, architectural, and speculative-decoding techniques. Within speculative decoding, ConFu’s focus on draft-model architecture complements algorithmic changes to drafting and verification.
- LLM Inference Acceleration: LLM inference acceleration includes quantization, pruning, distillation, KV-cache compression, and alternative architectures beyond standard Transformers.These directions target model computation, inputs, or architectural design.
- Speculative Decoding: Speculative decoding evolved from linear draft verification toward tree-structured drafting and modified drafting or verification procedures.These methods seek greater efficiency while retaining parallel target-model verification.
- Draft-Model Architectures: Algorithmic speculative-decoding methods remain orthogonal to approaches that improve the draft model itself.They modify drafting and verification strategies without depending on specific draft or target architectures.
- Draft-Model Architectures: Draft-model research includes standalone small drafters, Medusa’s lightweight prediction heads, and the EAGLE family of stronger drafting architectures.These approaches directly address the draft model’s role in speculative-decoding effectiveness.
6. Conclusion
ConFu improves speculative decoding by giving draft models future-oriented signals from the target model. Its dynamic MoE mechanism and training framework support stable future prediction, improving acceptance rates and inference efficiency over EAGLE-3.
- ConFu captures the target model’s current “thought” to improve draft model quality.Contemplate tokens and soft prompts provide lightweight, future-oriented signals at negligible inference cost.
- A dynamic Mixture-of-Experts mechanism adapts future prediction to diverse generation contexts.
- Anchor token sampling and prediction replication train stable future representations.
- ConFu consistently improves token acceptance rates and inference efficiency over EAGLE-3 across tasks and decoding configurations.
- Future-aware signals help mitigate error accumulation and improve speculative decoding effectiveness.
7. Impact Statement
ConFu improves speculative decoding without modifying or fine-tuning the target model, supporting faster generation with reduced computational cost and energy consumption. The framework preserves the target model’s original behavior and safety characteristics.
- ConFu improves speculative decoding without modifying or fine-tuning the target model.
- The framework enables faster text generation with reduced computational cost and energy consumption.
- ConFu does not introduce new model capabilities or alter the target model’s sampling distribution.
- The framework is an inference-time optimization orthogonal to data collection, model alignment, and training-time bias.
A. Confu Overhead Compared to Eagle3
The overhead analysis measures peak GPU memory usage when ConFu runs with 60 draft nodes, averaging results across 10 SpecBench samples.
- Peak GPU memory usage is reported in gigabytes for runs with 60 draft nodes.
- The overhead measure is peak GPU memory usage during execution.
- Results are averaged over 10 SpecBench samples.
B. Ablation Results
The ablation and acceptance analyses examine ConFu’s acceptance rate across training lengths, temperatures, draft-node counts, and downstream task categories, while Figure 5 compares long accepted draft trajectories with EAGLE-3.
- ConFu’s average acceptance rate is evaluated across temperatures 0.0, 0.7, and 1.0 and draft-node counts of 30 and 60.
- The ablation varies training block lengths of 4, 7, and 10 across multiple task categories.
- The evaluated task categories include writing, question-answering, summarization, translation, coding, and math/reasoning.
- Figure 5 shows that ConFu consistently has higher tail acceptance than EAGLE-3 for consecutive draft-token lengths.