Source-linked AI summary
ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, Chuan-Wei Kuo
TL;DR
Feature-based speculative drafters rely on target hidden features that may not emphasize what later draft positions need, while token-wise cache retrieval grows with the prefix. ReaDy adds position-specific reads from committed HLA recurrent states to drafting, improving decoding results over both autoregressive decoding and DSpark across seven benchmarks.
Problem
Target hidden features may not emphasize relationships needed by later draft positions, and retrieving additional context from token-wise KV memory incurs I/O cost that grows with the prefix.
Source: §1 · pp. 1–2Method
ReaDy augments DSpark’s parallel backbone with a reader that uses each draft representation to query committed recurrent states and injects retrieved context through residual updates.
Source: §1 · p. 2Results
Takeaways & Limitations
The results demonstrate the value of recurrent-state retrieval for improving draft agreement and decoding efficiency within DSpark’s semi-autoregressive architecture.
Source: §6 · p. 8The authors leave training the complete drafter from scratch and learning to allocate state access and readout capacity across draft positions to future work.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average $3.16\times$ decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.
1 Introduction
Feature-based drafting may not emphasize the context needed by later draft positions, while token-wise cache retrieval grows with the prefix. ReaDy retrieves complementary context from committed recurrent states and improves decoding results across seven benchmarks.
1 Introduction
- Target hidden features may not emphasize relationships needed by later draft positions, while KV-cache retrieval reads token-wise memory whose size grows with the prefix.Semi-autoregressive drafting must prepare representations for several future positions at once.Source: §1 · p. 1
- ReaDy augments DSpark’s parallel backbone with learned, position-specific queries to committed recurrent states and integrates retrieved context through residual updates.It retains DSpark’s sequential prediction head and allows only committed states to inform subsequent drafting rounds.Source: §1 · p. 2
- ReaDy’s stated novelty is using HLA recurrent states as complementary context for speculative drafting, with a theoretical analysis motivating position-specific retrieval.Low-rank reads, fused operators, and two-stage adaptation support efficient integration with a pretrained drafter.Source: §1 · p. 2
- 3.16× average speedup over autoregressive decoding, with throughput and acceptance length improving over DSpark by 7.97% and 8.50% on average, respectively.The results use Qwen3.8-27B on a single NVIDIA B200 GPU across seven benchmarks.Source: §1 · p. 2, Figure 1 · p. 2
2 Theoretical Motivation
The analysis distinguishes a recurrent state from the query-specific readouts exposed by an interface. It motivates position-specific reads from committed states while clarifying that the idealized result does not establish that complete drafter features are insufficient.
2 Theoretical Motivation
- The analysis models target readouts as O = SQ for a fixed state S and known queries Q, with the query span projected by ΠQ = QQ†.The argument focuses on which query directions the interface exposes, rather than the number of readouts alone.Source: §2 · p. 2
- When a new query has a component outside the exposed query span, different states can agree on existing readouts yet produce different answers to that query.If the exposed queries have full row rank, this ambiguity disappears; the result concerns idealized state-query readouts, not complete multilayer drafter features.Source: §2 · p. 2
- For a fixed continuation, the committed-prefix contribution remains expressible as a new query to the committed state, but future writes are not recovered by reading that state alone.The effective query depends on the continuation and is unavailable during drafting.Source: §2 · p. 3
- A learned reader maps each draft representation to a query of the same committed state, varying retrieved context across the block without advancing the target state.Residual integration retains the backbone’s role in modeling the continuation and combining context with hidden features.Source: §2 · p. 3
3 Method
ReaDy augments DSpark’s drafting backbone with position-conditioned reads from the target’s committed recurrent states, while retaining its sequential prediction head and verification rule. Low-rank reads and fused operations reduce the added state-access cost.
3.1 State-Conditioned Block Drafting
- ReaDy adds a state-reading path to DSpark’s parallel backbone, retrieving position-specific context from the target’s committed recurrent states while retaining DSpark’s sequential prediction head.The reader is placed in the final draft layer; subsequent rounds use only states associated with committed tokens.Source: §3 · p. 3, §3.1 · p. 4
- All draft positions read the same committed states, but their representations determine the context retrieved through a learned rank-r query bottleneck.The reader’s state-side projection also compresses the value dimension; the target’s recurrent states remain unchanged.Source: §3.1 · pp. 3–4
- Under greedy decoding, the target accepts the longest candidate prefix matching its argmax predictions; rejected proposals are discarded and corrected by the target.If the entire block is accepted, the target supplies a bonus token unless generation has terminated.Source: §3.1 · p. 4
3.2 Efficient State Access
- The reader’s source-state footprint is independent of prefix length, and low-rank projections reduce the value width of each subsequent read.Fused gathering and state preparation avoid intermediate tensor materialization and redundant data movement.Source: Figure 3 · p. 4, §3.2 · p. 5
- 71.8% and 28.1%: optimized state gathering and state reading reduce operator latency by these amounts, respectively.The reported latencies fall from 16.388 to 4.622 µs for gathering and from 65.519 to 47.094 µs for reading.Source: Figure 4 · p. 5, §3.2 · p. 5
4 Experiments
Across seven benchmarks, ReaDy improves throughput and acceptance length over DSpark and reaches 3.16× the throughput of autoregressive decoding. The experiments also examine draft-prefix survival and the trade-offs of reader capacity, state sources, and residual placement.
4.1 Training and Evaluation Setup
- ReaDy starts from a pretrained DSpark drafter and uses two adaptation stages: reader-only training on 20k samples, then joint reader and drafter-LoRA training on 50k samples.Evaluation covers seven benchmarks in mathematics, code, and dialogue; inference uses greedy decoding on one NVIDIA B200 GPU, with a seven-token draft block.Source: §4.1 · pp. 5–6
4.2 Main Results
- 302.78 tokens/s versus 95.86 for autoregressive decoding and 280.42 for DSpark; ReaDy achieves 3.16× the autoregressive speedup and gains 7.97% in throughput and 8.50% in acceptance length over DSpark.Throughput improves on every benchmark, with gains over DSpark ranging from 6.43% to 9.83% across mathematics, code, and dialogue.Source: Table 1 · p. 6, §4.2 · p. 6
4.3 Draft-Prefix Survival
- At draft position seven, prefix survival rises from 45.98% to 58.17% on GSM8K, from 25.89% to 31.19% on HumanEval, and from 12.59% to 16.11% on MT-Bench.Figure 5 reports cumulative survival: the fraction of eligible rounds in which all proposals through position j are accepted, not conditional acceptance.Source: Figure 5 · p. 6, §4.3 · p. 7
4.4 Ablation Studies
- 302.78 tokens/s versus 286.50 for fine-tuned DSpark without state access; ReaDy adds 5.68% throughput and 5.97% acceptance length over that variant.Fine-tuning alone raises DSpark throughput from 280.42 to 286.50 tokens/s and acceptance length from 4.1739 to 4.2732.Source: §4.4 · p. 7, Table 2 · p. 7
- Rank 8 gives the highest measured throughput; increasing rank to 128 raises acceptance length from 4.5285 to 4.5404 but reduces the throughput gain over DSpark from 7.97% to 7.81%.Table 3 compares ranks 8, 16, 32, and 128 in both key and value spaces.Source: §4.4 · p. 7, Table 3 · p. 7
- Layer 30 plus layer 62 yields 8.50% acceptance and 7.97% throughput gains, while adding layer 2 reduces those gains to 6.89% and 6.22%.Layer 62 alone yields a 7.48% acceptance gain, compared with 2.45% for layer 2 and 2.85% for layer 30.Source: §4.4 · p. 7, Table 4 · p. 7
- 302.78 tokens/s with one final-layer injection compares with 299.93 using layers 0, 2, and 4 and 297.35 using all five layers, with similar acceptance lengths.Mean acceptance lengths are 4.5285, 4.5288, and 4.5284, respectively.Source: §4.4 · p. 7, Table 5 · p. 8
- After-attention injection raises mean acceptance length from 4.4993 to 4.5285 and throughput from 301.94 to 302.78 tokens/s versus before-attention injection.The query is formed after the draft representation incorporates attention context, and the retrieved information can then be processed by the MLP.Source: §4.4 · p. 8, Table 6 · p. 8
5 Related Work
Prior drafting approaches reuse target features or read token-wise KV memory, while ReaDy reads fixed-size recurrent states. Related work on recurrent and hybrid targets also addresses verification and state recovery.
5 Related Work
- GliDe and LongSpec use cross-attention to access the target KV cache; ReaDy instead queries fixed-size recurrent states with low-rank reads and fused operators to limit additional I/O.Token-wise KV memory traffic grows with the number of attended tokens.Source: §5 · p. 8
- Prior recurrent and hybrid-model work addresses dependency-aware verification and accepted-state recovery, including Mamba and hybrid models, tree verification [Wu et al., 2025], and topology-aware verification with state recovery [Wang et al., 2026].ReaDy focuses on position-conditioned reads of committed states for block drafting.Source: §5 · p. 8
6 Conclusion
ReaDy retrieves position-specific context from committed recurrent states and integrates it into draft representations with residual updates. Across seven benchmarks, it improves decoding speed over autoregressive decoding and throughput over DSpark; future work includes training the complete drafter from scratch and learning state-access allocation.
6 Conclusion
- 3.16× average speedup over autoregressive decoding and 7.97% throughput improvement over DSpark across seven benchmarks on Qwen3.8-27B using one NVIDIA B200 GPU.Low-rank reads and fused operators control the additional I/O overhead.Source: §6 · p. 8
- ReaDy retrieves position-specific context from committed recurrent states and integrates it with draft representations through residual updates.It is DSpark-based, and low-rank reads and fused operators control additional I/O overhead.Source: §6 · p. 8
- Future work will explore training the complete drafter from scratch and learning to allocate state access and readout capacity across draft positions.Source: §6 · p. 8
A Analysis of Recurrent-State Readouts
The analysis characterizes which recurrent-state readouts can be determined from existing queries and how committed-prefix information contributes to future readouts. Its results motivate learned queries to committed states while distinguishing this access from exact recovery of future readouts.
A.1 Information Available from Existing Queries
- A new readout Sq* is uniquely determined by existing readouts (O, Q) exactly when q* lies in the column space of Q.Outside that span, different states can agree on exposed readouts while differing on the new one; the proposition concerns a fixed state and does not show that the full target-feature interface necessarily loses drafting-relevant information.Source: §A.1 · p. 10
A.2 Prefix Contributions to Future Readouts
- For each j ≥ 1, the future state is StPj + Wj, separating the committed-prefix contribution from subsequent writes.The raw attention readout follows from this state identity; the decomposition is along a fixed continuation.Source: §A.2 · p. 10
- The effective query depends on the continuation and is unavailable to the drafter, while reading St alone does not recover future writes.ReaDy instead learns position-conditioned queries through the drafting objective, with normalization and low-rank projections restricting the practical reader; the analysis does not assert exact recovery of future readouts.Source: §A.2 · p. 10
B Training Details
The appendix describes the data, training setting, and adaptation choices used for ReaDy. Its dataset-card counts characterize the public mixture, not the proportions of the adaptation subsets.
B Training Details
- Open-PerfectBlend covers mathematics, code, conversation, and instruction following, and its dataset card reports 88.1k examples removed during deduplication.The listed source-corpus counts are before mixture deduplication and are not per-source counts for ReaDy’s training subsets.Source: B Training Details · p. 10, Table 7 · p. 11
- ReaDy’s two-stage adaptation targets thinking-mode generation, unlike the non-thinking mode used in the original DSpark paper’s offline data generation and evaluation.The non-thinking setting refers to the original paper, not an assumption about the RadixArk checkpoint’s training data.Source: B Training Details · pp. 10–11
- The adaptation reuses a pretrained DSpark drafter but does not reproduce its original training recipe.The published DSpark objective uses cross-entropy, probability-distribution L1, and confidence losses weighted 0.1, 0.9, and 1.0; the adaptation instead uses the described position-weighted objective, with no confidence, KL-divergence, feature-reconstruction, or reward loss.Source: B Training Details · p. 11
- The two adaptation stages use AdamW with FP32 master parameters and optimizer states, with peak learning rates of 1 × 10−4 for reader-only adaptation and 5 × 10−5 for joint adaptation.Both stages use cosine decay after warmup; Table 8 lists the optimizer settings.Source: B Training Details · p. 11, Table 8 · p. 11
C Qualitative Example
In the qualitative example, ReaDy matches all seven target tokens while DSpark’s early mismatch limits its accepted proposal prefix. The appendix explains recurrent-state context and the speculative-decoding procedure used to evaluate proposals.
C Qualitative Example
- ReaDy matches all seven proposed tokens in the Alpaca example, while DSpark matches two before its first mismatch.DSpark’s later displayed tokens still match the target continuation, but its early mismatch ends the accepted prefix; the figure counts accepted proposals, excluding target-supplied tokens.Source: C Qualitative Example · p. 11, Figure 6 · p. 12
D.1 Linear and Hybrid Linear Attention
- At fixed feature dimensions, recurrent linear-attention state size and update/readout cost are independent of prefix length, while the state stores aggregated associations selected by a query.HLA interleaves recurrent layers with full-attention layers, so it combines compact state-based context with access to individual cached tokens; ReaDy uses recurrent states as supplementary context without changing the target architecture.Source: §D.1 · pp. 11–12
- Gated DeltaNet combines retention gating with a delta-rule update that corrects the difference between the incoming value and the value retrieved from the decayed state.The decay gate attenuates the previous state, while the delta term supports a targeted update to the current key–value association.Source: §D.1 · p. 12
D.2 Speculative Decoding
- Speculative decoding checks draft candidates from left to right and discards remaining proposals at the first rejection; accepting all candidates adds a target-sampled token and preserves the target distribution.In greedy evaluation, the accepted prefix ends at the first mismatch with the target’s argmax predictions.Source: §D.2 · pp. 12–13
D.3 Target and Drafter Architectures
The target combines recurrent and full-attention layers, while ReaDy adds a read-only interface to committed recurrent states. The drafter uses DSpark’s parallel backbone and sequential prediction head, with distinct hidden-feature and recurrent-state paths.
D.3 Target and Drafter Architectures
- Qwen3.8-27B has 48 recurrent layers with fixed-size matrix states and 16 full-attention layers with token-wise KV caches; ReaDy leaves this hybrid architecture unchanged.Its selected source layers 30 and 62 are recurrent layers, with 128 × 128 per-head states.Source: §D.3 · p. 13
- DSpark’s parallel backbone produces position-wise representations and base logits, while its sequential head adds transition biases; ReaDy supplies additional target-state context through residual updates.The head may use a preceding-token low-rank transition map or an RNN within-block hidden state.Source: §D.3 · p. 13
- The released RadixArk Qwen3.8-27B-DSpark drafter has five full-attention layers, width 5,120, 32 query heads, eight KV heads, five auxiliary target-feature layers, and a rank-256 Markov head.These are checkpoint properties; the rank-eight setting belongs to ReaDy’s state reader.Source: §D.3 · p. 13
E Draft-Prefix Survival Protocol
The protocol compares methods on identical greedy-generation prompts and estimates draft-prefix survival from fully observable rounds. Censoring and differing drafting boundaries limit interpretation of the resulting curves.
E Draft-Prefix Survival Protocol
- Both methods decode identical prompts with greedy generation, concurrency one, at most 1,024 new tokens, and natural EOS termination; traced and uninstrumented runs produce identical output sequences.Eligible drafting rounds are pooled separately by method and benchmark.Source: E Draft-Prefix Survival Protocol · p. 13
- For each j from 1 to 7, the plotted estimator is the fraction of N eligible rounds with at least j accepted proposals.Terminal rounds with incomplete observations are censored rather than counted as rejections.Source: E Draft-Prefix Survival Protocol · p. 13
- Because methods have different drafting boundaries, their curves aggregate different rounds along identical output sequences and do not isolate a semantic retrieval mechanism.Training/evaluation overlap means the measurements characterize checkpoint decoding behavior but do not establish held-out generalization.Source: E Draft-Prefix Survival Protocol · p. 13