Source-linked AI summary
DLite: History-free Speculative Decoding for Agentic AI
Jiaming Xu, Hanzhen Wang, Tianlang Zhao, Jiewen Xiao, Chengze Yuan, Jiayi Pan, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai
TL;DR
Agentic workloads' growing contexts challenge speculative acceptance and require costly historical draft-cache maintenance. DLite reuses target hidden states for local drafting and uses a lightweight sequential head and two-stage training; across general-purpose, long-context, and agentic workloads, it reports 2 ∼6 acceptance length and up to 70% improvement over DSpark.
Problem
In agentic workloads, growing contexts and frequent turns coincide with degraded speculative acceptance and substantial historical draft KV-cache maintenance.
Source: §1 · p. 2Method
DLite reuses target-model contextual hidden states for local speculative prediction without maintaining historical draft KV cache, with a sequential head and tailored two-stage training.
Source: §1 · p. 2Results
2 ∼6 acceptance length and up to 70% improvement over DSpark are reported across general-purpose, long-context, and agentic workloads, while historical draft KV-cache maintenance is eliminated.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The results show that removing redundant historical modeling from the drafter can improve speculative decoding quality and system efficiency for agentic workloads.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.
1 INTRODUCTION
Agentic workloads combine growing, multi-turn contexts with degraded speculative acceptance and costly draft-cache maintenance. DLite avoids modeling that history directly, reuses target-model hidden states for local prediction, and reports improved acceptance with no historical draft-cache maintenance.
1 INTRODUCTION
- Agentic workloads bring long inputs, short outputs, frequent multi-turn interactions, and growing contexts, where existing methods can fall to 1 ∼2 accepted tokens and cache work can consume 10% ∼20% of latency.The paper attributes the acceptance degradation to lightweight drafters struggling to model long histories.Source: §1 · p. 2
- DLite reuses target-model hidden states for local prediction without historical draft KV cache, adds a lightweight sequential head for intra-block causal dependencies, and uses two-stage training.Training first uses an isomorphic history-aware teacher to bridge structural mismatch, then aligns DLite directly with the target model.Source: §1 · p. 2
- 2 ∼6 acceptance length and up to 70% improvement over DSpark, with historical draft KV-cache maintenance eliminated, are reported across general-purpose, long-context, and agentic workloads.The models were trained for Qwen3-4B and Qwen3-8B.Source: §1 · p. 2
2 BACKGROUND
Speculative decoding drafts candidate tokens for parallel target verification, while draft architectures and distillation objectives differ in how they handle causal dependencies and supervision. These approaches still depend on historical context, unlike DLite's local prediction using target contextual features.
2.1 SPECULATIVE DECODING
- Speculative decoding drafts N candidate tokens for parallel target verification, and its performance mainly depends on acceptance length τ and draft-model latency TimeMD(N).The pipeline accepts multiple tokens from a verification round to accelerate inference.Source: §2.1 · pp. 2–3
- Autoregressive, parallel, and semi-autoregressive drafters trade off sequential drafting, intra-block causal dependency, and latency, while all retain growing historical-context dependence.DLite instead reuses target contextual features for local prediction.Source: §2.1 · p. 3
2.2 MODEL DISTILLATION
- EAGLE uses soft-target cross-entropy, DFlash primarily uses token-level CE, DSpark combines CE and TV distance, and Domino supervises parallel and corrected predictions with a curriculum.DLite's history-free structure differs from the autoregressive target, making direct distribution-level distillation non-trivial.Source: §2.2 · p. 3
3 MOTIVATION
Growing contexts in agentic workloads are associated with both lower speculative acceptance and greater historical KV-cache maintenance costs. The paper identifies repeated reconstruction of history already processed by the target model as a shared inefficiency.
3.1 TWO IMPORTANT OBSERVATIONS
- Existing methods can fall to 1 ∼2 accepted tokens on growing-context agentic workloads such as SWE-Bench and AgentLongBench.They achieve higher acceptance lengths on general workloads such as GSM8K and MBPP.Source: §3.1 · p. 3
- KV-cache updates after verification and preservation across agent turns can account for 10% ∼20% of end-to-end inference latency in the evaluated multi-turn setting.The measurement uses SGLang to evaluate DeepSeek-V4's multi-token prediction module.Source: §3.1 · p. 4
3.2 ANALYSIS AND FINDINGS
- Both modeling difficulty and cache costs stem from requiring the drafter to process and maintain growing history that the target model has already processed.The paper proposes reusing target contextual representations so limited draft capacity can focus on local speculative prediction.Source: §3.2 · p. 4
4 DLite FRAMEWORK
DLite delegates historical-context modeling to the target model, then drafts locally from its contextual representations. Its history-free architecture combines a parallel backbone, a lightweight sequential head, and two-stage training to address the missing intra-block dependency and structural mismatch with the target model.
4.1 HISTORY-FREE SEMI-AUTOREGRESSIVE ARCHITECTURE
- DLite uses target hidden states, the current anchor token, and draft placeholders to generate candidates, which the target model then verifies without maintaining historical draft KV cache.Each speculative round is reconstructed from the latest target representations and current anchor token.Source: §4.1 · p. 4
4.1.1 HISTORY-FREE PARALLEL BACKBONE
- The parallel backbone combines compact target hidden states with local token information and local positions to predict around the current decoding position.Target representations retain historical information, letting the drafter avoid explicitly reconstructing the growing history.Source: §4.1.1 · pp. 4–5
4.1.2 LIGHTWEIGHT SEQUENTIAL HEAD
- A low-rank RNN-style sequential head restores only the missing intra-block dependency by recursively summarizing earlier draft tokens, while avoiding repeated intermediate vocabulary projections.The head combines target-contextual hidden states with a recurrent summary of the intra-block prefix for autoregressive prediction.Source: §4.1.2 · pp. 5–6
4.2 TWO-STAGE TRAINING STRATEGY
- DLite is trained in two stages: first against an isomorphic history-aware teacher, then directly against the target model after the teacher is removed.The structurally matched teacher provides an intermediate target because direct distribution alignment between the parallel backbone and autoregressive target is difficult.Source: §4.2 · p. 6
5 EXPERIMENTS
DLite is evaluated on general-purpose, long-context, and agentic workloads, with ablations examining its training strategy, KV-cache use, and design choices. Results report advantages over existing drafters on longer-context workloads and higher throughput than DSpark.
5.1 EXPERIMENTAL SETUP
- The evaluation uses Qwen3-4B and Qwen3-8B against vanilla autoregressive decoding and representative speculative drafters across general-purpose, long-context, and agentic tasks.Serving experiments use H800 GPUs, block size K = 7, and report average acceptance length and end-to-end speedup.Source: §5.1 · p. 7
5.2 PERFORMANCE EVALUATION
- DLite consistently improves acceptance length and end-to-end speedup over DFlash and DSpark on LongBench v2.The reported advantage becomes more pronounced on long-context workloads.Source: §5.2 · p. 7, Table 2 · p. 8
- On SWE-Bench, DLite improves acceptance length over DFlash and DSpark while achieving around 2× end-to-end speedup.The paper identifies this as its largest relative gain under agentic workloads.Source: §5.2 · p. 8, Table 3 · p. 8
- DLite achieves higher throughput than DSpark across model sizes and evaluated workloads at concurrency levels from 1 to 64.The gain remains visible when average acceptance lengths are close.Source: §5.2 · p. 8, Table 4 · p. 9
5.3 ABLATION STUDY AND DESIGN SPACE EXPLORATION
- The two-stage training strategy achieves higher acceptance length through the stable stage and improves final performance over direct single-stage training.The paper attributes the optimization bridge to a history-aware isomorphic teacher before direct alignment.Source: §5.3 · p. 8, Figure 5 · p. 9
- After sufficient training, DLite and variants with 128- or 512-token sliding-window caches achieve comparable acceptance lengths across evaluated benchmarks.The reported advantage of historical KV states diminishes as DLite learns to use contextual information transferred through target hidden states.Source: §5.3 · p. 8, Figure 5 · p. 9
- The explored design choices are target hidden-state count, feature-wise versus sequence-wise composition, and sequential-head rank, which jointly trade off representation capacity, drafting quality, and overhead.The exploration informs the final configurations reported in Section 5.1.Source: §5.3 · p. 9
6 CONCLUSION
DLite reuses target-model contextual hidden states for local speculative prediction, eliminating historical draft KV-cache maintenance. Across general-purpose, long-context, and agentic workloads, it achieves 2 ∼6 acceptance length and improves over DSpark by up to 70%.
6 CONCLUSION
- DLite combines a history-free parallel backbone, a lightweight sequential head, and two-stage training to support local prediction using target hidden states.The sequential head restores intra-block causal dependencies, while training addresses the differing prediction structures of draft and target models.Source: §6 · p. 9
- 2 ∼6 acceptance length and up to 70% improvement over DSpark are reported across general-purpose, long-context, and agentic workloads, while eliminating historical draft KV-cache maintenance.The paper concludes that removing redundant historical modeling can simultaneously improve speculative decoding quality and system efficiency for agentic workloads.Source: §6 · p. 10
A.1 TRAINING DETAILS
DLite training uses target-model-generated responses and a two-stage process, with model configurations and optimization settings specified for Qwen3-4B and Qwen3-8B.
A.1 TRAINING DETAILS
- Training uses original prompts from the described datasets and regenerates responses with the corresponding target model to align supervision with its distribution.The corpus contains approximately 1.9M samples, with maximum sequence length 4096 and an average of 512 speculative blocks or anchors per sample.Source: §A.1 · p. 12
- For Qwen3-4B and Qwen3-8B, DLite uses five Transformer layers, 12 or 14 sequence-wise composed target hidden states, and sequential-head rank 320 or 512, respectively.The speculative block length is K = 7; hidden-state selection retains the first and last layers and uniformly samples intermediate layers.Source: §A.1 · p. 12
- Training uses AdamW with learning rate 4 × 10^-4, gradient clipping 1.0, global batch size 32, cosine decay, and 0.04 warm-up ratio on 32 H800 GPUs.Gradient accumulation is one step.Source: §A.1 · p. 12
- The warm-up trains an isomorphic history-aware teacher for eight epochs, then trains DLite against it with forward KL divergence for two epochs.The stable stage removes the teacher and directly optimizes DLite against the target model for six epochs.Source: §A.1 · p. 12
A.2 DESIGN SPACE EXPLORATION
The design-space exploration varies DLite’s target hidden-state features, their composition, and sequential-head rank on four workloads. It finds that sequence-wise composition improves average acceptance length, while additional hidden states and larger head ranks yield diminishing returns.
Design-space setup
- The exploration evaluates average acceptance length on four workloads using variants trained on the same 80K-sample subset.The workloads are Alpaca, GSM8K, MBPP, and Multi-Document QA; other training configurations remain unchanged.Source: §A.2 · p. 13, Table 5 · p. 13
Number of target hidden states
- Increasing target hidden states from 7 to 14 improves acceptance length across all four workloads, while 20 or 36 provide little or no further benefit.The authors attribute the diminishing returns to redundant representations and additional computation.Source: §A.2 · p. 13, Table 5 · p. 13
Hidden-state composition
- 3.30 average acceptance length with sequence-wise composition versus 3.04 with feature-wise composition across evaluated workloads.DLite adopts sequence-wise composition, which treats hidden states from different target layers as separate input elements.Source: §A.2 · p. 13, Table 5 · p. 13
Sequential-head rank
- A rank-512 sequential head raises average acceptance length from 3.16 at rank 256 to 3.30, with only 0.3% additional end-to-end latency.The full 4096-dimensional head reaches 3.33, while larger ranks add substantial computation for marginal gains.Source: §A.2 · p. 13, Table 5 · p. 13