DLoop: Looped Speculative Decoding

• A substantial fraction of drafting stages are fully accepted, yet standard speculative decoding verifies after every stage.Verification requires a full target-model forward pass and dominates decoding time.Source: §1 · p. 1, Figure 1 · p. 2
• Adaptive draft-length methods raise speedup only for autoregressive drafters; parallel drafters produce each stage in one forward pass, so early stopping leaves that cost unchanged.Further drafting for parallel models also lacks target-model hidden states for unverified tokens, leaving the draft model to use states it was not trained on.Source: §1 · pp. 1–2
• DLoop repeats drafting while the draft model remains confident, then verifies all accumulated draft tokens together.The loop scores each stage by summing its draft-token log probabilities and compares the score with a threshold.Source: §1 · p. 2
• Loop-aware training exposes the draft model to its own hidden states for unverified tokens across additional drafting stages.This extends EAGLE-3's training-time unrolling across stages and applies to parallel draft models.Source: §1 · p. 2
• Across EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop raises acceptance length and wall-clock speedup.It spends additional draft-model forward passes to reduce target-model forward passes; the draft count before verification is decided during decoding.Source: §1 · p. 2

• In the illustrated case, standard decoding verifies twice, while DLoop skips the first verification and verifies both stages together.With 7 tokens accepted from the first stage and 3 from the second, both approaches yield 10 tokens.Source: Figure 2 · p. 3
• DLoop changes how many full drafting blocks precede verification, rather than how a draft model produces a block.The same loop runs on EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, and composes with tree verification.Source: §2 · pp. 2–3
• Unlike adaptive methods that choose where one draft ends, DLoop treats a full drafting stage as the minimum and gates whether another follows.Its gate uses the drafter's confidence and loop-aware training, without an additional decision module.Source: §2 · p. 3

3.1 BACKGROUND

• Standard speculative decoding drafts a block of k tokens from target-side context and an anchor token, then verifies the block in one target-model pass.Its fixed alternation permits at most k + 1 tokens per verification even when all k draft tokens are accepted, motivating multiple drafting stages per verification.Source: §3.1 · pp. 3–4

3.2 CONFIDENCE-GATED DRAFTING LOOP

• DLoop repeats whole drafting stages while the latest block’s summed log probabilities exceed a threshold, then verifies all accumulated draft tokens in one target-model pass.Under full acceptance, one verification can yield up to Sk + 1 tokens instead of up to S(k + 1) across S separate verifications.Source: §3.2 · p. 4
• The loop’s gate sums the latest block’s token log probabilities and compares the score with a fixed threshold, requiring no additional module or forward pass.A single uncertain token lowers the sum, consistent with the acceptance rule discarding tokens after the first rejection.Source: §3.2 · pp. 4–5
• The accumulated draft sequence is verified as one block, preserving the target model’s output distribution and lossless decoding.If an earlier token is rejected, verification also discards every later block.Source: §3.2 · p. 5

3.3 LOOP-AWARE TRAINING

• Loop-aware training samples additional-stage counts and exposes the draft model to its own hidden-state chain, training the first and last drafting stages.Intermediate stages supply hidden states but do not contribute to the loss; the draft model runs S stages per iteration without additional target-model passes.Source: §3.3 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• Evaluation covers eight math, code, and chat benchmarks, with four draft models, two Qwen3 targets, and MTP modules for Qwen3.5-9B and Gemma-4-12B.The study reports acceptance length and wall-clock speedup, using greedy decoding unless stated otherwise.Source: §4.1 · p. 6

4.2 EXPERIMENTAL RESULTS

• 12–102% higher average acceptance length and 5–41% higher wall-clock speedup across draft-model, target-model, and task-category settings.The reported results cover four draft models and MTP modules; DSpark has the largest gain among the four draft models evaluated with Qwen3-4B and Qwen3-8B.Source: §4.2 · p. 6, Table 1 · p. 7
• Combining the confidence-gated loop with loop-aware training improves acceptance length and speedup in every setting, with gains exceeding the sum of either component alone.Loop-aware training alone changes both metrics by less than 7%; the loop alone can increase acceptance length while speedup changes little or decreases in some settings.Source: §4.2 · p. 7, Table 2 · p. 7
• On EAGLE-3 and DSpark, DLoop achieves the highest average acceptance length and speedup among the compared adaptive draft-length methods.No compared method beats the DSpark baseline speedup; on EAGLE-3, the seven methods raise it by at most 0.14×.Source: §4.2 · pp. 7–8, Table 3 · p. 8
• 4.9–24.3% less total decode time across four draft models accompanies 34–71% of output tokens generated in loops with S > 1.Across loops, time per output token decreases from 4.60–6.42 ms at S = 1 to 0.68–2.97 ms at S ≥5; DLoop reduces verifications by 7.6–34.1%.Source: §4.2 · p. 8, Table 4 · p. 8, Figure 3 · p. 9
• With tree verification, DLoop further raises average acceptance length and speedup for every draft model, with speedup gains varying by method.Over tree verification alone, average speedup rises 12–13% for DFlash with DDTree, 7–8% for Domino with DARTree, and 22–25% for DSpark with DARTree.Source: §4.2 · p. 9, Table 5 · p. 9

5 CONCLUSION

• 5 to 41 percent: DLoop improves wall-clock speedup across EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules while preserving lossless decoding.It spends additional draft-model forward passes to reduce target-model forward passes.Source: §5 · pp. 9–10
• DLoop uses a confidence-gated drafting loop and loop-aware training, while leaving the draft-model architecture and acceptance rule unchanged.The loop-aware training exposes the draft model to its own hidden states for unverified draft tokens.Source: §5 · pp. 9–10
• DLoop raises speedup for parallel draft models as well as autoregressive ones, and further raises acceptance length and speedup when applied on top of tree verification.The results show that the number of drafting stages before verification can be decided during decoding rather than fixed by the schedule.Source: §5 · p. 10

A APPENDIX

• The appendix reports benchmark results under greedy decoding and sampling at temperature T = 1, alongside evaluation protocol details.It describes benchmark sources, sample counts, and prompt templates.Source: A APPENDIX · p. 14
• It evaluates DLoop throughput under concurrent serving in SGLang and analyzes changes in drafting-stage and verification counts.It also measures one-loop time and peak GPU memory by number of drafting stages.Source: A APPENDIX · p. 14
• The appendix presents representative loop generations, compares gate and loop-aware-training design choices, and details adaptive draft length applications to EAGLE-3 and DSpark.Source: A APPENDIX · p. 14

A.1 EVALUATION PROTOCOL

• The eight benchmark subsets use samples selected after shuffling each source dataset with seed 0, following the official DFlash evaluation code.Each subset takes the first samples up to its specified sample count.Source: §A.1 · p. 14, Table A1 · p. 14
• Prompts are placed in a user message and use the target model's chat template, with no system prompt and thinking mode disabled.MBPP and MT-Bench use each sample's text without a template.Source: §A.1 · p. 14
• The LiveCodeBench template's opening instruction remains part of the user message, and the stated protocol applies to every result on the eight benchmarks.Source: §A.1 · p. 14

A.2 PER-BENCHMARK RESULTS

• DLoop raises acceptance length on every benchmark for every draft model and target model.Table A9 reports acceptance length and speedup for settings in Tables 1, 2, and 5 on each benchmark.Source: §A.2 · p. 14
• Acceptance-length gains are largest on math and smallest on chat, matching the ordering of full acceptance ratios in Fig. 1.Source: §A.2 · p. 14
• At sampling temperature T = 1, DLoop also raises average acceptance length.Table A10 reports the same settings with sampling.Source: §A.2 · p. 14

A.3 SERVING THROUGHPUT UNDER CONCURRENCY.

• At c = 32, DFlash+DLoop slightly underperforms DFlash, while DSpark+DLoop continues improving throughput on MATH-500 and remains within 1% of DSpark on HumanEval, as DLoop’s advantage diminishes at higher load.Source: §A.3 · p. 15
• Gains generally shrink with concurrency, consistently with speculative work consuming scarce batch capacity as the target model saturates.Sample-dependent loop depths also create ragged draft execution, requiring padding or dynamic regrouping with scheduling overhead.Source: §A.3 · p. 15
• At c = 1 on MATH-500, speedup rises from 5.81× to 6.23× with DFlash and from 4.45× to 4.94× with DSpark.Table A2 reports throughput in tokens per second and speedup over the autoregressive baseline.Source: §A.3 · p. 14, Table A2 · p. 15

A.4 CHANGE IN THE NUMBER OF DRAFTING STAGES AND VERIFICATIONS

• Without tree verification, drafting stages rise 3.5–24.5% and verifications fall 4.3–22.7%; with tree verification, stages rise 4.4–37.9% and verifications fall 4.6–26.9%.The counts are measured over eight benchmarks.Source: §A.4 · p. 15
• Verification reductions are largest for DSpark and smallest for Domino, and acceptance-length and speedup gains rank the draft models in the same order.For DSpark and DFlash with tree verification, added drafting stages outnumber removed verifications, yet speedup still rises 10.2–27.6%.Source: §A.4 · p. 15
• Each additional drafting stage takes less time than one verification for every draft model and S, so fewer verifications outweigh the added drafting stages.Source: §A.4 · p. 15

A.5 COST OF A LOOP BY THE NUMBER OF DRAFTING STAGES

• At S = 1, the loop reduces to standard speculative decoding, and drafting time increases almost linearly with S for every draft model.Source: §A.5 · p. 15
• Verification time remains within 21.9–25.0 ms as S grows, even though verification covers more draft tokens.One drafting stage costs less than one verification, even for EAGLE-3.Source: §A.5 · p. 15
• Peak GPU memory increases by less than 0.1 GB from S = 1 to S = 4.Source: §A.5 · p. 15

A.6 EXAMPLES OF THE DRAFTING LOOP.

• DLoop varies drafting depth within generation: confident loops continue for two to four stages before one verification, while others stop after one.The examples report lower end-to-end latency than the same DSpark model without the loop.Source: §A.6 · pp. 15–16, Figure A1 · p. 17

A.7.1 GATE THRESHOLD.

• The gate-threshold sweep reports average acceptance length and speedup over eight benchmarks at T = 0.Source: Table A5 · p. 16, §A.7.1 · p. 16

A.7.2 GATE FUNCTION.

• Across four draft models, log probability yields the highest AUROC and precision and the highest speedup among practical gate functions.Entropy is the strongest parameter-free alternative but trails log probability in AUROC and precision; learned heads require extra parameters and training.Source: §A.7.2 · pp. 16–17, Table A6 · p. 17

A.7.3 LOOP-AWARE TRAINING.

• Loop-aware training with N = 1 raises DSpark acceptance length from 5.75 to 8.13 in τ; N = 3 adds gains for DFlash and DSpark, while N = 5 adds none.Intermediate supervision lowers acceptance length and speedup while increasing training cost; the adopted setting is N = 3 without mid-CE.Source: §A.7.3 · p. 17, Table A7 · p. 18

A.8.1 DECISION RULES.

• Compared adaptive rules stop on draft probabilities, entropy, accumulated confidence, learned stop heads, predicted acceptance, or preset tree criteria.Some methods also adjust thresholds after verification or set draft length before drafting begins.Source: §A.8.1 · p. 18

A.8.2 APPLICATION TO EAGLE-3.

• For EAGLE-3, the compared rules are evaluated at their specified depths or after blocks, with default maximum depth and each method’s maximum draft length.BiLD’s fallback is ported, but its rollback policy is omitted because rollback can change the target model’s output.Source: §A.8.2 · p. 18

A.8.3 APPLICATION TO DSPARK.

• For parallel DSpark, most rules score the full draft block, while PACER and DDD decide after each drafting stage whether to continue.The comparisons use released DSpark weights and permit another stage when a rule has not stopped within a block.Source: §A.8.3 · p. 18

A.8.4 TRAINING AND TUNING.

• Decision heads are trained for one epoch on loop-aware-training data, and thresholds plus PACER block size are swept to select each method’s highest-average-speedup setting.The appendix also reports benchmark results with sampling at T = 1.Source: §A.8.4 · p. 19, Table A9 · p. 21, Table A10 · p. 22