PaperlayerView exact revision on arXiv

Source-linked AI summary

CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

Jungseob Lee, Sugyeong Eo

arXiv:2610.00321v2cs.CLcs.LG

TL;DR

Block drafters score many future-token candidates in one pass, but standard speculative decoding verifies only a chain and discards the alternatives. CAST packs candidates into a tree, chooses its width using verification cost, and is faster than the standard chain in all eight evaluated settings.

  • Problem

    Standard decoding discards already-scored block-drafter alternatives, and deployment-dependent verification costs make tree width costly to tune by sweeping.

    Source: §1 · pp. 1–2

  • Method

    CAST ranks block-drafter candidates into a prefix-closed tree and chooses its width with a cost-aware stopping rule, verifying the tree in one target pass.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    CAST is faster than the standard DFlash chain in all eight hardware and model settings, with average gains of 20–36% and up to 43% on one domain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CAST leaves the target, drafter weights, and decoding rule unchanged while preserving the target output distribution under greedy and sampled decoding.

    Source: §1 · p. 2

    The width rule assumes locally affine cost, and its experimental slope estimate underestimates total cost because it omits overhead.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Speculative decoding accelerates large language model inference by drafting future tokens cheaply and verifying them with the target model in parallel. Block drafters score a whole block of future tokens in one forward pass, yet standard decoding verifies only the top-scoring chain and discards the other candidates. Because these candidates are already scored, verifying more of them adds target computation but no extra drafting. We introduce CAST (Cost-Aware Speculative Trees), which packs these candidates into a tree and verifies it in a single target pass, leaving the target model, drafter weights, and decoding rule untouched. To decide how wide the tree should be, CAST adds candidates while the expected gain from the next one outweighs the verification time it adds. The width therefore adapts to each deployment from a latency measurement, without sweeping over widths. We evaluate CAST across five domains on three GPU generations and two model families. At its predicted width, CAST is faster than the standard chain in all eight settings, by up to 43%. We also find that the best width depends strongly on the deployment. Where verification cost jumps at a kernel boundary, a 128-token tree is only 2% faster than the standard chain, whereas the tree at the predicted width is 20% faster. Furthermore, we prove that CAST leaves the target output distribution unchanged under both greedy and sampled decoding. Code is available at https://github.com/js-lee-AI/CAST.

1 INTRODUCTION

Block drafters score many candidate tokens at once, but standard decoding verifies only one chain and discards the alternatives. CAST turns those candidates into a verified tree and chooses its width using deployment-specific verification cost.

1 INTRODUCTION

  • Standard decoding verifies the most-likely chain and discards other already-scored candidates, including alternatives the target might have selected after an early rejection.Tree verification retains alternatives and checks the tree in one target pass.Source: §1 · p. 1
  • CAST ranks candidate sequences using the product of their drafter probabilities, then adds candidates while expected acceptance outweighs the verification time they add.The width adapts from drafter scores and a short latency measurement, without sweeping widths.Source: §1 · p. 2
  • CAST keeps the target, drafter weights, and decoding rule unchanged while proving distribution preservation under greedy and sampled decoding.For any fixed width, keeping the highest-scoring candidates maximizes expected accepted tokens under the drafter’s scores.Source: §1 · p. 2
  • CAST is faster than the standard DFlash chain in all eight hardware and model settings, with average gains of 20–36% and up to 43% on one domain.Its predicted width matches the best swept width in six settings.Source: §1 · p. 2
  • At a kernel boundary, a 128-token tree is 2.4% faster than the standard chain, while the predicted-width tree is 19.8% faster.Under serving load, the best static width shrinks toward the chain as concurrency grows, while best-first shared-budget allocation lifts goodput by up to 13.7%.Source: §1 · p. 2

2 BACKGROUND AND SETUP

Speculative decoding drafts future tokens and verifies them in parallel, committing accepted draft tokens plus a target-selected token each round. CAST uses a one-pass block drafter and verifies a prefix-closed candidate tree with an ancestor-masked target pass.

2 BACKGROUND AND SETUP

  • Each speculative round commits accepted draft tokens plus one token selected by the target, and throughput is expected committed length divided by round time.If A draft tokens are accepted, the round commits R = A + 1 tokens.Source: §2 · p. 3
  • A round begins with a cache-backed prefix and a pending root token, which starts the next draft block and candidate tree.The prefix holds already-computed target KV and DFlash feature caches.Source: §2 · p. 3
  • DFlash produces a vocabulary distribution for each future position in one pass, while standard decoding keeps each position’s most-likely token as a chain.CAST ranks candidate sequences by the product of the corresponding position probabilities; this is a ranking score, not a target-factorization assumption.Source: §2 · p. 3
  • A budget-N tree packs N+1 tokens, including the pending root, into one target pass where each node attends only to the prefix and its ancestors.Under greedy decoding, the round commits the longest tree path matching the target, then appends the target’s next token.Source: §2 · p. 3
  • For temperature T>0, a proposal-aware stochastic verifier samples retained siblings recursively, preserving autoregressive target sampling by Theorem 3.The optimality analysis described here uses greedy verification.Source: §2 · p. 3

3 THE CAST WIDTH POLICY

CAST changes the verifier input while keeping the target, drafter weights, cache contract, and decoding rule fixed. Its policy selects high-scoring prefix-closed trees and stops widening when the expected acceptance gain no longer justifies verification cost.

3.1 THE COST OF WIDTH

  • From 16 to 64 packed tokens, target-forward latency rises only 12% on Blackwell-8B and a few percent on other GPU-target pairs, then jumps at Blackwell-8B’s 128-token boundary.Figure 2 reports batch-one target-forward latency relative to 16 packed tokens, with packed count n = N + 1.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 BUDGET-OPTIMAL CANDIDATE TREES

  • For any fixed budget, the N highest-mass strings in a finite prefix-closed universe maximize expected accepted length under nonincreasing prefix beliefs.Ties favor prefixes, and the selected set is itself prefix-closed.Source: §3.2 · p. 4
  • The deployed top-K=8, best-first tree matches full-rank accepted length within 1.1% at evaluated budgets, while the oracle-to-plug-in gap is bounded by prefix-belief errors.The error bound sums estimation errors over the symmetric difference between the two trees.Source: §3.2 · p. 4

3.3 COST-AWARE WIDTH

  • Theorem 2 makes throughput unimodal under a locally affine, nonnegative cost slope; add the next node while its prefix mass meets the token-time exchange rate.The experimental slope uses forward-only measurements, which underestimate the total slope because they omit overhead.Source: §3.3 · p. 5
  • Under load, global best-first expansion maximizes expected accepted length across concurrent requests sharing a verification budget, allowing unequal widths.The batched stopping rule reduces to the batch-one rule when B=1.Source: §3.3 · p. 5

3.4 THE CAST DECODER

  • Each CAST round runs the unchanged drafter once, verifies a packed tree in one ancestor-masked pass, then emits the accepted path and next pending root.The local affine-cost assumption has a boundary: hybrid linear-attention targets may pay a shared chunk-kernel entry cost before costs become locally affine again.Source: §3.4 · p. 5, §3.3 · p. 5
  • Greedy decoding reproduces the target continuation by construction; sampled decoding is distribution-preserving, with measured total variation below the fp64 Monte-Carlo floor and about 0.004 in bf16.In bf16, greedy departures from autoregressive decoding occur only at near-ties of the top two target logits.Source: §3.4 · p. 5

4 EXPERIMENTS

Across evaluated hardware and model settings, CAST’s predicted width stays close to sweep-oracle gains, while performance depends on verification-cost shape and tree width. Additional tests cover matched-budget tree shape, sampling, longer generations, another target family, and another one-pass drafter.

4 EXPERIMENTS

  • H100’s nearly flat cost curve favors the largest tested widths, while Blackwell-8B gains plateau at N=63–95 and fall at N=127.Flatter A6000 and Blackwell-4B curves support wider trees.Source: §4 · p. 6
  • Both Qwen3 targets average 5.62× speedup over AR on H100 SXM, with longer committed rounds throughout.Table 1 reports speedup over AR and committed round length R.Source: Table 1 · p. 6, §4 · p. 6
  • Across eight settings, predicted-width gains are within 1.7 percentage points of the sweep oracle.The settings span three GPUs, two model families, two block sizes, and a 30B sparse mixture-of-experts target.Source: §4 · p. 6, Table 2 · p. 7
  • With sampling at T=1, CAST’s gain over DFlash stays between +22% and +37% in every domain on both targets.On Blackwell with Qwen3-8B, the gain exceeds 15% under each of three sampling seeds.Source: §4 · p. 7, Table 3
  • At 16 packed tokens, changing tree shape alone makes CAST 5.9–29.2% faster than DFlash across all 12 Blackwell and A6000 cells.Both methods use the same block-16 drafter and 15 nonroot nodes.Source: Figure 5 · p. 7, §4 · p. 7
  • On LLaMA-3.1-8B, CAST averages 3.87× speedup over AR versus EAGLE-3’s 2.59×, and is 30–60% faster in every domain.On GSM8K, both trees commit R=6.08, while EAGLE-3 uses eight draft passes per round and CAST uses one.Source: Table 4 · p. 8, §4 · p. 8
  • On MATH-500, CAST is 21% faster at 512 new tokens and 20% faster at 2048; on GovReport, it is 28% faster for Qwen3-8B and 27% for Qwen3-4B.The GovReport results are within 3 percentage points of the sweep oracle.Source: §4 · p. 8
  • For DART’s one-pass logits, a 63-node best-first tree lengthens committed rounds over the eight-node chain by 40% on GSM8K and 34% on HumanEval.Timed on an A6000 with Qwen3-8B, DART’s 60-node tree reaches 2.14× speedup over autoregressive decoding on GSM8K and 2.34× on HumanEval.Source: §4 · p. 8

5 SERVING UNDER LOAD

Under serving load, a single static width does not suit the full load range; allocating a shared verifier budget by global best-first expansion improves goodput over the best static width.

5 SERVING UNDER LOAD

  • At B=16 and B=32, global best-first allocation lifts Blackwell goodput over the best static width by median 7.0% and 13.7%, respectively.Across 32-request batches, it also raises accepted tokens over a uniform split by 16.8–22.5%. The best static width switches from a fixed wide tree up to B=8 to the plain chain from B=16.Source: §5 · p. 8

6 RELATED WORK

CAST differs from prior tree-decoding approaches by building a tree from already-scored one-pass candidates and pricing width using measured verification cost. The conclusion emphasizes deployment-specific width selection, shared-budget allocation, and retaining the base model’s safeguards.

6 RELATED WORK

  • SMART (Wang & Zhou, 2026) and OPT-Tree use marginal benefit–cost rules, but CAST prices measured verification time because one-pass drafting adds no draft step within the block.The related-work discussion also distinguishes width prediction and shared-budget allocation across requests.Source: §6 · p. 8
  • DART prunes one-pass logits using an external n-gram signal (Liu et al., 2026a), whereas CAST derives its budget-optimal tree from the drafter’s product score and measured hardware cost.JetSpec retrains the head, and Weaver adds an autoregressive adapter; CAST retains the checkpoint.Source: §6 · p. 9
  • CAST’s width allocation differs from D-cut’s profiled chain-depth pruning (Liu et al., 2026b): it splits a shared budget by best-first mass at measured goodput.ECHO (Hu et al., 2026b) shares depth and width across requests with autoregressive drafters.Source: §6 · p. 9
  • Unlike approaches that control reasoning amount, CAST controls verification work without changing the target output distribution; deployments should retain the base model’s safeguards.The paper states that faster inference can reduce harmful-generation costs.Source: §6 · p. 9, ETHICS STATEMENT · p. 9

A.1 CONCLUSION

  • CAST recovers a prefix-closed tree from one computed block and chooses its width from measured verifier costs, with predicted widths spanning 64 to 128 packed tokens.The conclusion says deployments can recompute width when the GPU or target changes.Source: §A.1 · p. 13

A.2 LIMITATIONS

The appendix states CAST’s deployment and tree-scope limits, then proves exact verification, optimal candidate selection, a marginal stopping rule, and shared-budget allocation results.

A.2 LIMITATIONS

  • CAST is evaluated through controlled within-setting decoder comparisons, with dense targets up to 8B parameters and one 30B mixture-of-experts target with 3B active parameters.Timing is decode-only steady state, not a serving-throughput or cost claim; the free-width argument requires a one-pass drafter.Source: §A.2 · p. 14

B.1 DRAFTER INTERFACE AND EXACT VERIFICATION

  • An ancestor-only attention mask makes each packed node’s logits identical to separate autoregressive forwards, and recursive speculative sampling preserves the target sequence distribution.The proof relies on exact arithmetic and the packed-logit equivalence.Source: §B.1 · pp. 14–15

B.2 OPTIMAL CANDIDATE TREES

  • For a fixed prefix belief, the N highest-mass strings form a prefix-closed tree maximizing expected accepted length; the plug-in guarantee applies only within the chosen feasible universe.Under a rank cap, omitted tokens are outside the exactness claim.Source: §B.2 · pp. 15–16

B.3 THE MARGINAL STOPPING RULE

  • Under a locally affine, nonnegative verification-cost slope, throughput is unimodal, and adding the next node helps exactly when its mass meets the current-throughput cost threshold.The first budget where the condition fails is a maximizer, subject to boundary cases.Source: §B.3 · p. 17

B.4 SHARED BUDGET UNDER LOAD

  • With a shared verifier budget, selecting the globally highest-mass nodes across requests is optimal, and the batched stopping rule is unimodal under its stated local cost assumption.The batch cost’s non-drafter portion is assumed locally affine in total budget with nonnegative slope.Source: §B.4 · pp. 17–18

C IMPLEMENTATION DETAILS

CAST builds and verifies a candidate tree each round using one unchanged DFlash call, then updates the target and feature caches along the accepted path. Its stochastic verifier also preserves the target sampling law.

The round

  • Each round uses one unchanged DFlash call, verifies a packed candidate tree under an ancestor-only mask, and follows matching children to emit the accepted path and next pending root.The target cache is compacted to the committed path, while the next root stays outside both caches until the next round.Source: C IMPLEMENTATION DETAILS · p. 18

Candidate construction

  • Best-first expansion ranks sequences by the product of their original position marginals, retaining top-K tokens per position without renormalizing their scores.The implementation uses a heap and performs at most K+NK pushes and N pops.Source: C IMPLEMENTATION DETAILS · p. 18

Packed verification

  • Packed verification assigns positions by tree depth and masks each packed token to the prefix and its ancestor chain.The ancestor mask is built from parent pointers and copied to the device each round.Source: C IMPLEMENTATION DETAILS · p. 18

KV gather and feature cache

  • After acceptance, the target cache gathers the prefix and committed packed indices, and the DFlash feature cache appends features for exactly those committed positions.The timer window is shared with AR and DFlash.Source: C IMPLEMENTATION DETAILS · p. 18

Stochastic verifier

  • The direct sampled walk preserves the sampling distribution, with bf16 total variation 0.004, matching the proposal-aware verifier.AR, DFlash, and CAST use bf16 in a shared software environment within each setting.Source: C IMPLEMENTATION DETAILS · p. 18

D MEASUREMENT PROTOCOL AND CROSS-HARDWARE RESULTS

The section measures decode-only steady-state performance and examines how verification costs, hardware, and decoding mode affect CAST’s gains. It also checks output equivalence and measurement variability.

D MEASUREMENT PROTOCOL AND CROSS-HARDWARE RESULTS

  • Timing excludes prefill and the first speculative draft, but includes later drafting, verification, tree construction, packing, cache gathering, and feature-cache updates.Headline rows use otherwise idle GPUs; shared-GPU load inflates relative gains, so idle-GPU timing is the conservative operating point.Source: D MEASUREMENT PROTOCOL AND… · p. 19
  • Verification costs vary by target and hardware: Blackwell-8B reaches 30.3 ms at 128 packed tokens, while H100 SXM costs rise 6.7% for 8B and 7.4% for 4B from 16 to 128.On H100 SXM, the rule predicts N*=127 across tested prefix lengths; on the second A6000 host, N=127 gains trail N=95 for both targets.Source: D MEASUREMENT PROTOCOL AND… · pp. 19–20, Table 5 · p. 19, Table 6 · p. 20
  • 5.9% to 29.2% faster across all twelve cells, a 16-packed-token CAST tree also raises committed round length by 0.69 to 1.17 over standard DFlash.The comparison holds packed length fixed, isolating tree shape from width.Source: D MEASUREMENT PROTOCOL AND… · p. 20, Table 9 · p. 21
  • At N*=63 on Blackwell-8B, sampled decoding gains 16.7% to 30.1% over the chain across three seeds.Table 10 reports committed round length, decode-only ms/token, mean gain, and seed-level gain range.Source: D MEASUREMENT PROTOCOL AND… · p. 21, Table 10 · p. 21

D.1 HYBRID LINEAR-ATTENTION TARGETS

  • On hybrid linear-attention targets, multi-token verification incurs a fixed kernel-entry cost, after which latency grows locally; at a 1k prefix, a 63-node tree matches the 16-token chain within noise on the dense hybrid.The MoE hybrid's per-token slope remains 0.39 to 0.40 ms.Source: §D.1 · p. 21, Table 11 · p. 22

D.2 STATISTICS AND OUTPUT EQUIVALENCE

  • Expanded-prompt bootstrap confidence intervals for every Blackwell cell lie above zero, and three-run gain variation is at most 2.3 points per domain at N*=63.The bootstrap uses paired prompt-level resampling with 10,000 resamples.Source: §D.2 · p. 22
  • In fp32, CAST and standard DFlash match AR greedy decoding token for token across evaluated domains and budgets; bf16 differences occur only at near-ties in the top target logits.The bf16 comparison covers over 600 prompts.Source: §D.2 · p. 22
  • For T=1 sampling, the proposal-aware verifier has total variation 0.003 in fp64 and 0.004 in bf16 versus autoregressive sampling, with the fp64 value below the 0.013 Monte-Carlo floor.Appendix F reports the same comparison under serving load.Source: §D.2 · p. 22

E H100 SXM PANELS IN FULL

The H100 SXM panels compare CAST width sweeps and reported baselines across Qwen3 targets, domains, and generation settings. The results include gains across tasks and robustness checks across repetitions.

E H100 SXM PANELS IN FULL

  • Two additional repetitions reproduce every chain-to-CAST gain within 3.7 points, and no cell shows the Blackwell-8B sweep’s cliff reversal.The full sweep is reported in Table 12.Source: E H100 SXM PANELS IN FULL · p. 22, Table 12 · p. 23
  • At T=1, CAST beats the chain by 22% to 37% across ten cells, with each seed’s mean tree decode time within 3% of the three-seed cell mean.Table 12 reports Qwen3 greedy width sweeps using the public block-16 DFlash head.Source: E H100 SXM PANELS IN FULL · p. 22, Table 12 · p. 23
  • The EAGLE-3 comparison runs official EAGLE code on the same node, prompts, and decode-only timer, using draft trees of 16 and 60 tokens, seven draft steps, and top-k 10.Table 13 compares Qwen3-8B heads for greedy decoding on H100 SXM, reporting decode-only ms/token and committed round length R for 16- and 60-token trees.Source: E H100 SXM PANELS IN FULL · p. 22, Table 13 · p. 23
  • For MATH-500 at 2048 generated tokens and GovReport with an 8k prefix, gains stay within 4 points from N=47 to 127, with the best width at 95.The long-prefix rows generate 256 new tokens; Table 14 reports speedup over AR, committed round length, gain over DFlash, and the sweep oracle.Source: E H100 SXM PANELS IN FULL · p. 22, Table 14 · p. 23

F SERVING HARNESS

The serving harness tests allocation policies under concurrent requests, including a dense prototype and a single-pass variable-length verifier. Results show that fixed widths depend on load and that request-level best-first allocation can improve goodput in the tested settings.

F SERVING HARNESS

  • At matched eight-token packed budgets, a seven-node tree from the block-16 drafter is fastest in every domain, ahead of both the block-8 chain and block-8 tree.The comparison indicates that shrinking tree width costs less than shrinking the drafter block.Source: F SERVING HARNESS · p. 24, Table 15 · p. 24
  • At B≥16, the fixed wide tree collapses and the chain is best, while dense request-level allocation’s acceptance gains do not translate to goodput because padding charges every request the maximum width.Figure 7 compares the block chain, fixed N=48 tree, uniform W=8, and dense request-level allocation on Blackwell Qwen3-8B.Source: F SERVING HARNESS · p. 24, Figure 7 · p. 24
  • Under ragged packing, online global best-first allocation raises accepted tokens by 16.8% to 22.5% over a uniform split at a scarce budget, but only 1% to 2% when budget is abundant.Bucketing requests incurs extra passes that re-stream model weights, while padding rows still cost 9% to 15% of the verify forward.Source: F SERVING HARNESS · p. 24
  • The varlen verifier improves goodput over the batch-wise best static chain by median 7.0% at B=16 and 13.7% at B=32, with gains in every run.Against uniform width, its median gains are 8.6% at B=16 and 13.9% at B=32.Source: F SERVING HARNESS · pp. 24–25
  • With continuous batching at 200-request occupancy, request-level best-first beats uniform width by 0.9% to 3.6% across active batch sizes 8 to 32; at T=1, served samples match autoregressive sampling within the 0.013 Monte-Carlo floor.The served-token total variation is 0.006 in fp64.Source: F SERVING HARNESS · p. 25

G EXTENDED ANALYSIS

The extended analyses test adaptive trees against fixed shapes, examine candidate ranking and stopping behavior, and assess width transfer across a different drafter and model family.

G EXTENDED ANALYSIS

  • At fixed budget N=47, the adaptive top-N path-product tree reaches R=7.7/4.3/7.7, versus the chain’s 6.3/3.2/6.3 on GSM8K/MT-Bench/HumanEval.The fixed schedule and uniform binary branching do not beat the standard chain on these domains.Source: G EXTENDED ANALYSIS · p. 25
  • At first chain rejection, the target correction ranks 2–4 in 56–77% of events and lies outside the top 32 in only 2–11%.The rank-2–4 rate is 77% on GSM8K and HumanEval, and 56% on MT-Bench.Source: G EXTENDED ANALYSIS · p. 25
  • At depth one, the binned reliability is close to diagonal: empirical hit rates are 0.98–0.99 in the [0.9, 1.0] bin, with mid-range bins mildly underconfident.For the [0.6, 0.7] bin, the empirical hit rate is 0.66–0.77.Source: G EXTENDED ANALYSIS · p. 25
  • For GSM8K on Blackwell, the stopping threshold is about 0.015; candidate masses fall from 0.86 at rank 1 to 0.016 at rank 48 and 0.011 at rank 64.The rule predicts an optimum between N=47 and N=63, the flat region of Table 8.Source: G EXTENDED ANALYSIS · p. 25

G.1 RANK-CAP ABLATION

  • At budget 32, the K=8 tree comes within 0.5% of the full-rank tree’s committed accepted length in every domain; at N*=63, the gap stays within 1.1%.The realized target token falls beyond rank 8 only 9.5–15.2% of the time, while the cap scores eight candidates per position.Source: §G.1 · p. 25

G.2 WIDTH TRANSFER TO DART

  • At budgets of 8 and 60, DART’s tree decoder is 1.91× and 2.14× faster on GSM8K and 2.02× and 2.34× faster on HumanEval than autoregressive decoding.At budget 63, the tree exceeds the top-1 chain by 39.9% on GSM8K and 34.2% on HumanEval.Source: §G.2 · p. 26

H.1 LLAMA-3.1-8B, BLOCK 10

  • 30–60% faster than the 60-token EAGLE-3 tree in every domain, CAST at N*=127 on LLaMA-3.1-8B; the tree beats the chain at every width and domain.The width sweep is within 2.1% from 63 to 127; on GSM8K, EAGLE-3 at tree 60 matches CAST’s accepted length but uses eight draft passes rather than one.Source: Table 16 · p. 26, §H.1 · p. 26

H.2 QWEN3-CODER-30B-A3B, BLOCK 16

On Qwen3-Coder-30B-A3B, CAST’s latency-based width selection matches the mean sweep oracle. The related-work discussion distinguishes CAST’s width choice from prior tree construction, verification, and adaptive-budget approaches.

H.2 QWEN3-CODER-30B-A3B, BLOCK 16

  • 35.6% mean gain over DFlash; the latency rule selects N*=127, matching the mean sweep oracle on Qwen3-Coder-30B-A3B.Gains rise through 127 on GSM8K and HumanEval, while MBPP peaks at 95; mean gain improves only 0.7 points at the final step, consistent with a broad plateau.Source: §H.2 · pp. 26–27, Table 17 · p. 27

I EXTENDED RELATED WORK

  • DART prunes one-pass candidates with an external n-gram signal and trie; CAST instead chooses how many nodes to pack on measured hardware.DART addresses which candidates to keep under a supplied construction rule or budget, whereas CAST targets width selection.Source: I EXTENDED RELATED WORK · p. 27
  • JetSpec trains a tree-causal draft head, while Weaver trains an adapter over DFlash marginals; both target the acceptance ceiling of branch-agnostic marginals.JetSpec pairs its head with best-first expansion and tree attention; Weaver adds rollback-free tree verification for gated linear-attention layers.Source: I EXTENDED RELATED WORK · p. 27
  • SpecInfer, Medusa, and Sequoia study tree verification, while EAGLE-2 and OPT-Tree optimize adaptive trees for autoregressive or EAGLE-style drafters before verification.Block verification, HSD, Traversal Verification, SpecTr, and Towards Optimal Multi-draft SD address verification rules, traversal, or distribution-preserving acceptance.Source: I EXTENDED RELATED WORK · p. 27
  • Serving-level controllers manage speculation or batch budgets, while D-cut, AdaFlash, and BlockPilot adapt the depth or granularity of one chain.The cited approaches include MagicDec, TETRIS, FASER, a utility gate, and MoESD for serving-level control.Source: I EXTENDED RELATED WORK · pp. 27–28
  • Acceptance-aligned objectives change the trained drafter; DistillSpec and HASS target draft–target alignment or training–decoding mismatch, while Yin et al. analyze unbiased rejection-based decoding limits.Surveys of speculative decoding round out the related work.Source: I EXTENDED RELATED WORK · p. 28

Related papers

Loading 2610.00321v2…