Source-linked AI summary
Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier
Linye Wei, Shutian Zheng, Haoyu Zeng, Meng Li
TL;DR
Self-speculation’s throughput frontier remains underexplored because sequential draft–verify execution limits efficiency at low concurrency, while longer drafts become costly at high concurrency. RecGuide uses reciprocal signals to overlap or adapt drafting across serving loads, and reports consistent throughput gains of up to 1.8× over vanilla LinearSS.
Problem
The aggregate and per-request throughput frontier for self-speculation remains underexplored amid low-concurrency sequential-pass costs and high-concurrency draft-budget constraints.
Source: §1 · p. 2Method
RecGuide uses draft logits to anticipate verification mismatches and verification history to guide low-concurrency overlap and request-specific block sizing.
Source: §1 · p. 2Results
Up to 1.8× speedup over vanilla LinearSS, with consistent throughput improvements across a wide range of concurrency levels.
Source: §7 · p. 10Takeaways & Limitations
The results present full-target-backbone self-speculation as a promising alternative for efficient speculative decoding across diverse serving loads and concurrency regimes.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Diffusion drafting with autoregressive (AR) verification has emerged as a promising paradigm for efficient speculative decoding. Recent self-speculation models, represented by Nemotron-Labs-Diffusion, further simplify the speculative pipeline by unifying drafting and verification within a shared backbone, while enabling longer acceptance lengths. However, the Pareto frontier between aggregate and per-request throughput remains underexplored. At low concurrency, sequential draft-verify execution requires two model forward passes per round, limiting the effective tokens per forward (TPF). By contrast, at high concurrency, longer drafts incur increasingly expensive computation, forcing individual requests to operate under constrained speculation budgets and preventing full exploitation of the full-backbone drafter. Our key observation indicates that drafting and verification exhibit reciprocal predictability. Draft logits can anticipate likely verification mismatches, while recent verification outcomes predict future drafting utility and suitable block sizes. Building on this observation, we introduce Reciprocal Guidance (RecGuide), a runtime draft-verify orchestration framework that adapts speculative decoding to varying serving loads. RecGuide exploits spare compute capacity through verification-overlapped drafting at low concurrency, while dynamically allocating request-specific draft block sizes as the workload becomes increasingly compute-intensive. Experiments across a wide range of concurrency levels demonstrate consistent throughput improvements over vanilla self-speculation, achieving up to $1.8\times$ speedup.
1 INTRODUCTION
Self-speculation uses one shared backbone for diffusion drafting and autoregressive verification, but its fixed draft budgets leave throughput trade-offs across serving loads unresolved. RecGuide uses reciprocal draft–verification signals to adapt its execution, improving throughput over vanilla self-speculation by up to 1.8×.
1 INTRODUCTION
- At low concurrency, sequential draft-then-verify requires two forward passes per round; at high concurrency, large blocks raise latency while small blocks underuse long-acceptance potential.These opposing constraints motivate budget orchestration across aggregate and per-request throughput.Source: §1 · p. 2
- RecGuide uses draft logits to anticipate verification mismatches and recent verification outcomes to estimate drafting utility and select the next block size.Its strategies overlap drafting with verification at low concurrency and adapt request-specific block sizes as workloads grow.Source: §1 · p. 2
- The paper characterizes self-speculation bottlenecks and identifies reciprocal predictability between drafting and verification as the basis for orchestration.Diffusion-based speculative decoding uses bidirectional drafting followed by autoregressive verification; self-speculation performs both with a shared backbone.Source: §1 · pp. 1–2
- Up to 1.8× speedup over vanilla self-speculation is reported for RecGuide, with consistent throughput improvements across experiments.The evaluation covers diverse serving regimes.Source: §1 · p. 2
2 RELATED WORK
Prior speculative-decoding work develops faster drafters and runtime strategies for parallel execution or adaptive budgets. RecGuide instead couples drafting and verification signals for shared-backbone self-speculation.
Drafting Paradigms for Speculative Decoding
- Diffusion drafters use bidirectional attention to propose blocks in one forward pass, while shared-backbone self-speculation unifies drafting and verification at increased drafting cost.The related work frames this paper around runtime execution and efficiency.Source: §2 · p. 3
Runtime Orchestration for Speculative Decoding
- Runtime work separately explores overlap, adaptive drafting, and selective verification, whereas RecGuide treats drafting and verification as mutually informative stages.For shared-backbone self-speculation, it co-batches future drafts with ongoing verification without model switching or additional devices.Source: §2 · p. 3
3 PRELIMINARY AND BOTTLENECK
Shared-backbone self-speculation switches one model between parallel diffusion drafting and causal autoregressive verification. Its acceptance variability and load-dependent forward cost create distinct low- and high-concurrency bottlenecks.
3.1 SELF-SPECULATION WITH A SHARED BACKBONE
- Diffusion drafting lets draft positions attend to the prefix and full draft region, while autoregressive verification restricts each position to earlier positions.Changing the attention pattern switches modes, allowing the shared backbone to serve as drafter without a separate draft model.Source: §3.1 · pp. 3–4
3.2 BOTTLENECKS ACROSS SERVING LOADS
- The profile shows variable acceptance potential: nearly 30% of rounds accept more than 16 tokens, while over half fail within the first 8 positions.Forward time grows with concurrency, and larger blocks incur increasingly higher latency under heavier workloads.Source: §3.2 · p. 4
- At low concurrency, sequential drafting and verification cause mutual waiting, while early mismatches can waste computation on large blocks.The memory-bound regime leaves spare compute that can support overlap and proactive correction.Source: §3.2 · p. 4
- At high concurrency, larger blocks increasingly constrain compute and raise forward latency, but uniformly small blocks sacrifice long-acceptance potential.The paper motivates request-specific block sizes and bucketed serving for differing computation lengths.Source: §3.2 · p. 4
4 RECIPROCAL PREDICTABILITY BETWEEN DRAFTING AND VERIFICATION
Draft distributions contain cues about likely verification failures, while verification history helps predict the next round’s drafting utility. These signals support anticipating mismatches and adapting draft budgets.
4 RECIPROCAL PREDICTABILITY BETWEEN DRAFTING AND VERIFICATION
- The analysis links draft-distribution signals and verification outcomes to runtime decisions about likely mismatches and future block sizes.Adjacent rounds show both local persistence and regime transitions, so fixed block sizes cannot consistently match local drafting utility.Source: §4 · p. 4, §4.1 · p. 5, §4.2 · p. 5
4.1 DRAFT DISTRIBUTIONS PREDICT VERIFICATION FAILURES
- Draft confidence is lower and more broadly distributed at first-rejected positions than at passed positions, indicating it can anticipate likely verification failures.The paper also finds a similar separation in the top-2 probability margin; a small margin leaves the second-ranked candidate as a possible correction.Source: §4.1 · p. 5, Figure 3 · p. 5
4.2 VERIFICATION HISTORY PREDICTS FUTURE DRAFTING UTILITY
- Short and long acceptance intervals most often persist into the next round, making verification history a useful prior for future drafting budgets.Acceptance is not static: only 22.42% of medium-acceptance rounds remain in the same interval, and adjacent-round transitions are frequent.Source: §4.2 · p. 5, Table 1 · p. 5
5 RECGUIDE: RUNTIME DRAFT-VERIFY ORCHESTRATION
RecGuide uses reciprocal signals between drafting and verification to adapt its orchestration to serving load. It overlaps likely future drafts with verification at low concurrency and selects request-specific block sizes under heavier workloads.
5 RECGUIDE: RUNTIME DRAFT-VERIFY ORCHESTRATION
- RecGuide overlaps verification with likely next-round drafts at low load, then uses verification history and hardware profiles to allocate request-specific block sizes as compute becomes constrained.The runtime combines predictive draft-verify overlap with adaptive drafting.Source: §5 · p. 5, Figure 4 · p. 6
5.1 PREDICTIVE DRAFT-VERIFY OVERLAP
- The overlap strategy marks low-margin draft positions as potential mismatches, prepares correction branches using second candidates, and may add a continuation branch after full-block acceptance.It retains only the earliest two risky positions, keeping each round to at most four logical rows with the verifier.Source: §5.1 · pp. 5–6
- Prepared branches run alongside current verification and are reused only when verifier outcomes match their predicted correction or continuation conditions.Unmatched branches are discarded, while a confirmed draft can repeat the procedure in the next round.Source: §5.1 · p. 6
5.2 VERIFICATION GUIDED ADAPTIVE DRAFTING
- Adaptive drafting chooses each request’s next block size from verifier entropy and the previous block outcome, balancing estimated accepted progress against profiled execution cost at the current load.A larger block is selected only when its expected acceptance gain justifies its additional computation; runtime selection uses lightweight table lookup.Source: §5.2 · p. 6
5.3 SYSTEM IMPLEMENTATION
- The system implementation extends SGLang with shared-prefix KV access, padding-free execution, GPU workload remapping, and block-size-bucketed batches.Requests in a batch share a planned block-size bucket, while request states and canonical KV remain independent.Source: §5.3 · p. 7
6 EXPERIMENTS
Across low- and high-concurrency experiments, RecGuide improves throughput over fixed or vanilla self-speculation approaches. Its overlap strategy boosts single-request decoding, while adaptive block sizing advances aggregate throughput across higher serving loads.
6.2 LOW-CONCURRENCY PERFORMANCE
- At C = 1, RecGuide averages 4.06× speedup on NLD-8B and 4.40× on NLD-14B over autoregressive decoding, and reaches up to 1.8× speedup over vanilla LinearSS on AIME25.Under the same two-forward accounting, effective acceptance length exceeds 14 tokens.Source: §6.2 · p. 7, Table 2 · p. 8
6.3 HIGH-CONCURRENCY PERFORMANCE
- RecGuide consistently improves aggregate throughput over fixed B ∈{8, 16, 32} on NLD-8B and NLD-14B across concurrency C ∈{2, 4, 8, 16, 32, 64, 128}.It favors larger blocks when compute is available and smaller blocks as execution becomes compute limited.Source: §6.3 · p. 8, Figure 5 · p. 9
- On NLD-8B, the B = 32 request share falls from 88.13% at C = 2 to 9.75% at C = 128, while B = 8 rises from 9.33% to 78.21%.NLD-14B exhibits the same transition; QA & IF shifts toward smaller blocks earlier than MATH and CODE.Source: §6.3 · p. 8, Table 4 · p. 9
- At intermediate concurrency, diverse block assignments yield greater per-request throughput gains, while endpoint assignments approach the corresponding fixed-block baselines.RecGuide avoids the high-concurrency aggregate-throughput disadvantage of globally fixed B = 8 by selecting short blocks only when beneficial and regrouping requests by block size.Source: §6.3 · p. 9
6.4 ABLATION STUDY
- At C = 1, adding a continuation branch raises the overlap improvement to 51.76% on NLD-8B and 49.36% on NLD-14B.The results indicate that three prospective branches capture most overlap benefit while bounding computation.Source: §6.4 · pp. 9–10, Table 5 · p. 9
- Across concurrency levels, adaptive selection with bucketed execution improves TPS by 39.84% on NLD-8B and 32.80% on NLD-14B.Two-round entropy alone improves TPS by 10.49% / 7.60% on NLD-8B / 14B.Source: §6.4 · p. 10
7 CONCLUSION
RecGuide applies reciprocal predictability to overlap drafting and verification at low concurrency and adapt request-specific draft budgets under heavier workloads. It consistently improves throughput over vanilla LinearSS across a wide range of concurrency levels, with up to 1.8× speedup.
7 CONCLUSION
- Table 6 presents an ablation of verification-guided adaptive drafting, with TPS averaged across all benchmarks and concurrency levels.Source: Table 6 · p. 10
- RecGuide is a runtime draft-verify orchestration framework for diffusion-AR self-speculation that adapts its strategies to serving load.It overlaps drafting and verification at low concurrency and adjusts request-specific draft budgets under heavier workloads.Source: §7 · p. 10
- Across a wide range of concurrency levels, RecGuide consistently improves throughput over vanilla LinearSS, achieving up to 1.8× speedup.The conclusion identifies full-target-backbone self-speculation as a promising alternative across diverse serving loads and concurrency regimes.Source: §7 · p. 10