PaperlayerView exact revision on arXiv

Source-linked AI summary

Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Yuhao Shen, Tianyu Liu, Xinyi Hu, Quan Kong, Baolin Zhang, Jun Dai, Jun Zhang, Shuang Ge, Lei Chen, Yue Li, Mingcheng Wan, Cong Wang

arXiv:2605.20104v1cs.LGcs.AI

TL;DR

Speculative decoding faces a latency–acceptance tradeoff: pruning saves draft computation but can discard valid candidates. Graft combines pruning with retrieval to refill pruned slots, achieving cross-setting speedups of up to 5.41× while maintaining or increasing accepted length.

  • Problem

    Dynamic pruning reduces draft cost but can discard valid continuations, limiting accepted length below dense-tree performance.

  • Method

    Graft uses training-free, lossless prune-then-graft construction to insert retrieval-based branches into slots released by confidence-based pruning.

  • Results

    Across short-context, long-context, and high-batch settings, Graft improves speed while maintaining or increasing MAT, reaching up to 5.41× speedup.

  • Takeaways & Limitations

    Graft establishes a new Pareto frontier by treating pruned slots as reusable budget for retrieval rather than discarded candidates.

  • Takeaways & Limitations

    Retrieval works best when prompt or generation history contains useful local transition structure, while the DFlash extension remains preliminary.

Abstract

from arXiv · show

Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41$\times$ speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.

1. Introduction

Graft addresses the latency–acceptance tradeoff in speculative decoding by combining dynamic pruning with retrieval under a training-free, lossless prune-then-graft framework. It releases computation through pruning, restores candidate coverage through retrieval, and improves speed across short-context, long-context, and large-scale settings.

  • Motivation: Speculative decoding reduces autoregressive latency by having a draft model propose candidates that the target model verifies in parallel, while tree drafting increases mean accepted length.Large draft trees can still incur substantial search, memory-bandwidth, and verification costs that limit end-to-end speedups.
  • Motivation: Dynamic-depth pruning lowers wasted computation but restricts candidates to subtrees, so noisy confidence decisions can remove valid branches and reduce mean accepted length.Pruning-only methods therefore face a latency–MAT tradeoff rather than reaching the dense-tree acceptance bound.
  • Graft: Graft treats pruned slots as reusable computational budget: pruning removes low-confidence branches, then retrieval fills released positions with additional candidates.This couples pruning and retrieval so that pruning supplies budget while retrieval compensates for pruning-induced candidate loss.
  • Graft: Graft is a training-free, lossless, GPU-friendly prune-then-graft framework that packs retrieved candidates into the standard verification path while preserving the target verification budget.Its implementation uses calibrated pruning checkpoints, root-centered parallel retrieval, a GPU-resident adjacency matrix, and online target-guided updates.
  • Long-context extension: Extended contexts make pruning more valuable because autoregressive drafting grows costlier, while richer local transition patterns improve retrieval hit rates without additional overhead.The framework is therefore designed to scale naturally to long-context generation.
  • Evaluation and extension: 5.41× speedup is achieved on short-context tasks, while average speedup over EAGLE-3 improves by up to 21.8% on large-scale Qwen3-235B.For long-context generation, Graft reaches 3.22× average speedup on LLaMA3.1-8B and outperforms EAGLE3-64K by 16.6% on Qwen3-14B; a preliminary DFlash-style exploration extends grafting beyond autoregressive trees.

2. Preliminary Study

The preliminary study identifies a strict latency–MAT trade-off in dynamic-depth pruning: reducing draft cost also caps candidate coverage and accepted length. It motivates Graft’s prune-then-graft strategy, which uses released budget to insert retrieved continuations and repair pruning-induced coverage loss.

  • Dense and Pruned Drafting: EAGLE-3 uses a dense tree to maximize MAT through broader candidate coverage, whereas dynamic-tree methods reduce overhead through dynamic-depth pruning.Pruning lowers the cost denominator but constrains pruned trees structurally.
  • Dense and Pruned Drafting: Pruning-only designs cannot introduce novel candidate paths, so MAT remains capped by the dense-tree upper bound.Removing nodes accelerates drafting but limits accepted-token coverage.
  • Latency–MAT Trade-off: The latency–MAT frontier reflects a trade-off: pruning saves time by eliminating expensive draft computation but can lose valid continuations when confidence-based controllers over-prune.The probability of at least one harmful pruning decision grows across path depth as 1 − ∏_d(1 − ε_d).
  • Budget Release and Retrieval: Pruning should be viewed as budget release, because freed candidate slots can be repopulated by cheaper retrieval rather than left empty or reinvested in uncertain autoregressive drafting.Local continuations from the prompt or generation history can be reused with minimal additional computation.
  • Retrieval Insertion: Under a fixed candidate budget, Graft(ROOT) displaces strong root candidates, while Graft(TAIL) offers only marginal, task-dependent gains.These limitations show that retrieval should be inserted where pruning creates space rather than simply beside or behind the dense tree.
  • Graft Motivation: Graft prunes unreliable drafting and grafts retrieved candidates into the released slots, preserving pruned-drafting latency while recovering coverage within one fixed-budget verification pass.By adding nodes outside the original subtree, it bypasses the dynamic-tree constraint and breaks the pruning-only Pareto frontier.

3. Graft: Hybrid Tree Construction for Speculative Decoding

Graft combines pruning with retrieval: it removes low-confidence branches from a base draft tree, then uses the released budget to graft retrieval-based branches. Its pruning policy favors shallow, reliable decisions to limit accumulated errors and preserve valid candidates.

  • Hybrid construction: Graft first prunes low-confidence branches from the base tree and then grafts retrieval-based branches into the budget released by pruning.The design uses EAGLE-3 as its base tree drafter but assumes a fixed-budget tree-style speculative verifier.
  • Pruning policy: Unlike dense dynamic-tree methods, Graft avoids frequent intermediate-depth decisions that can accumulate over-pruning and prematurely remove valid branches.The method addresses the challenge of deciding how far to expand the base draft tree without excessive decision error.
  • Pruning policy: At each pruning checkpoint, Graft compares the highest-scoring path confidence with a calibrated threshold and prunes when the checkpoint decision is zero.For a node, confidence is derived from the highest-scoring cumulative path, and pruning skips deeper draft expansion.
  • Pruning policy: Graft sets stricter thresholds at shallow checkpoints so uncertain branches are removed before drafting errors compound.This policy is intentionally biased toward shallow and reliable pruning decisions.

Graft: Hybrid Tree Construction for Speculative Decoding

Graft couples dynamic-depth pruning with GPU-resident retrieval to reallocate a fixed verification budget from unreliable draft nodes to context-aware candidates. It merges both candidate types into one lossless hybrid-tree verification pass and continuously updates retrieval from target-model signals.

  • Budget allocation: Dynamic-depth pruning reallocates, rather than increases, the fixed candidate budget: shallower pruning grants more slots to retrieval, while deeper pruning preserves more draft nodes.This reshapes budget composition without inflating target-side verification cost.
  • Retrieval mechanism: Graft uses a GPU-resident adjacency matrix whose O(|V|k) token-ID storage enables retrieved-node generation through a single parent-row gather without host-device communication.Batched GPU gathers make retrieval critical-path depth scale with d_s rather than N_s.
  • Retrieval mechanism: Stage-adaptive retrieval templates match the base-tree envelope and vary branch size with pruning depth, from large retrieval branches after root pruning to minimal branches when pruning is absent.Templates intentionally favor high-rank successors with greater depth while retaining narrower sibling coverage.
  • Hybrid verification: Retained draft and retrieved nodes are merged into a single hybrid tree with total verification budget |T_s| = K_max, then verified in one parallel target-model pass using the unchanged tree-attention interface.Retrieval changes candidate identities while preserving the verification budget and lossless speculative verification.
  • Online adaptation: 0 extra cost: the adjacency matrix is refreshed from target distributions over the whole verified tree, allowing rejected candidates to provide successor information.A warm-up phase initializes the matrix, while online updates keep retrieval aligned with the target model.

4. Experiments

Across short- and long-context settings, Graft consistently accelerates decoding and improves over pruning, static-tree, and long-context baselines. Its gains persist on Qwen3-235B and under greedy decoding, while retrieval uses context-specific verification signals to recover pruned branches.

  • Experimental settings: Graft is evaluated on short-context generation, long-context generation, and high-concurrency serving across diverse benchmarks and model configurations.Short-context tasks cover code generation, mathematical reasoning, summarization, and open-ended chat.
  • Short-context case: 5.41× maximum speedup: short-context Graft reaches 1.83×–5.41× speedups across evaluated model families.Average speedup over EAGLE-3 improves by 7.4% on Vicuna-13B, 5.9% on LLaMA3.1-8B, 15.7% on Qwen3-8B, 17.5% on Qwen3-32B, and 21.8% on Qwen3-235B.
  • Scalability to Large Models: 2.09× average speedup: on Qwen3-235B, Graft gains 21.8% over EAGLE3-2K, 19.4% over TR, and 8.9% over ECHO.The improvement over EAGLE3-2K reaches 26.2% on CNN/DM.
  • Long-context case: 3.22× average speedup: long-context Graft outperforms MagicDec at 0.74×, TokenSwift at 1.67×, TriForce at 1.81×, and EAGLE3-64K at 2.92×.Graft’s speedups range from 1.76× to 3.37× and improve over EAGLE3-64K under the same verification budget.
  • Long-context case: Long-context retrieval benefits from repeated phrases, document-specific entities, and local code patterns, while verification updates the adjacency matrix with accepted and rejected draft-node transitions.This supplies increasingly context-specific evidence for later root-centered retrieval.

5. Related Works

The related work spans speculative decoding, retrieval-based candidate generation, dynamic token trees, and long-context optimizations. Graft distinguishes itself by integrating retrieval with pruning within the existing tree-attention verification path.

  • Speculative Decoding: Speculative decoding accelerates LLM inference by drafting candidate tokens and verifying them in parallel with the target model.Existing approaches use separate lightweight draft models or auxiliary-head and self-speculative drafters such as Medusa and EAGLE.
  • Retrieval-based Speculative Decoding: Retrieval-based speculative decoding proposes candidates with little additional draft-model computation while preserving lossless verification.Representative methods retrieve candidates from prompt-local matches, external datastores, and cached transitions.
  • Retrieval-based Speculative Decoding: Graft grafts retrieval branches into pruning-released budget, stores transitions in a GPU-friendly adjacency matrix, and verifies candidates through the same tree-attention path.This design aligns retrieval-based speculative decoding more closely with industrial serving requirements.
  • Dynamic Token Tree: Dynamic token tree methods adapt candidate-tree topology using token probability or confidence to improve budget utilization, unlike static structures that spend budget on uncertain deep branches.Tree-based speculative decoding expands multiple candidate paths to increase accepted length, while static methods such as EAGLE-3 use fixed structures with low overhead.
  • Speculative Decoding for Long Context: Long-context speculative decoding is constrained by increasing KV-cache traffic and attention overhead, motivating sparse or partial KV caching, hierarchical speculation, and long-context draft adaptation.These approaches primarily optimize the KV-cache bottleneck associated with longer contexts.

6. Conclusion … A.1. Retrieval Template Details

Graft treats pruning as budget reallocation: retrieval refills pruned tree capacity, preserving or improving speculative decoding acceptance and speed across multiple deployment settings. The appendices detail its generation procedure, retrieval templates, theoretical analyses, evaluation settings, implementation, profiling, and preliminary DFlash extension.

  • 6. Conclusion: Graft couples dynamic pruning with retrieval, using freed candidate slots to compensate for pruning-induced coverage loss rather than treating pruning solely as candidate removal.Retrieval reuses prompt, history, and verification signals with little draft-model computation, while pruning-only trees remain bounded by the dense base tree.
  • 6. Conclusion: Graft’s current limitations include dependence on useful local transition structure, incomplete optimization of high-concurrency retrieval kernels and scheduler placement, and preliminary DFlash adaptation.The paper notes that block drafters differ from tree drafters in topology and confidence calibration.
  • Appendix Contents: The appendices cover Graft’s generation procedure and retrieval templates, theoretical pruning and verification analyses, evaluation details, concurrency, runtime profiling, and DFlash implementation.Appendix B includes pruning regret, retrieval coverage, speedup condition, and lossless verification; Appendix C includes baseline settings and cross-task threshold generalization.
  • A. Procedure and Pseudocode: The generation pipeline combines confidence-based pruning, GPU-resident adjacency-matrix retrieval, and online matrix updates from target-model verification logits.The merged proposal tree is packed into the standard tree-attention verification path, so retrieval changes the proposal set but not the target-model acceptance rule.
  • A. Procedure and Pseudocode: Pruning checkpoints d0, d1, and d5 retain 8, 24, and 40 draft-tree nodes, respectively, allocating remaining fixed-budget slots to retrieval.If pruning is not triggered, Graft degenerates to the original base tree; otherwise, the algorithm retains top draft nodes and appends retrieved candidates before rebuilding tree metadata.
  • A.1. Retrieval Template Details: The retrieval template is a static, unbalanced rank-path tree whose higher-ranked successors receive more children and greater depth while preserving alternative local continuations.The full template contains 80 non-root nodes across 9 retrieval depths, and decoding selects a stage-specific prefix based on pruning location.
  • A.1. Retrieval Template Details: After draft and retrieved nodes are collected, Graft rebuilds tree masks, position IDs, and candidate-path indices before sending the merged sequence through the original tree-attention verification path.Templates are filled in BFS-like order using GPU matrix lookup from successor ranks, and shallower pruning stages allocate more fixed budget to retrieval.

B. Theoretical Analysis · B.1. Pruning Regret

This analysis characterizes pruning regret relative to dense drafting, explains how over-pruning and accumulated misjudgments reduce accepted length, and relates regret to latency savings. It also frames retrieval grafting and final target verification as mechanisms for recovering regret without sacrificing losslessness.

  • B. Theoretical Analysis: Graft’s analysis studies pruning regret, retrieval-based recovery under the same verification budget, and losslessness after target-model verification.These are the three stated theoretical properties examined in the appendix.
  • B.1. Pruning Regret: Pruning retains a sparse tree T_π⊆T_0 from the original dense tree T_0, with expected accepted length defined as A(T)=𝔼[L(T)].The notation establishes the dense-versus-pruned comparison used throughout the analysis.
  • B.1. Pruning Regret: Accepted length and expected accepted length cannot increase when candidates are removed through pruning.This follows from verification monotonicity: adding candidates cannot invalidate an already valid accepted prefix.
  • B.1. Pruning Regret: The resulting acceptance regret is the cost of reducing draft latency through pruning.The analysis identifies this regret as the central tradeoff between tree compactness and accepted length.
  • B.1. Pruning Regret: A pruned tree improves the speed proxy only when latency savings sufficiently offset the MAT loss caused by pruning.This condition is expressed through the ratio in Eq. 3.
  • B.1. Pruning Regret: Over-pruning drives much of the regret by removing a depth-d branch even when the dense tree would have accepted beyond depth d.The appendix formalizes this mechanism using the event F_d.
  • B.1. Pruning Regret: Even mildly noisy pruning decisions can accumulate harmful errors along a path as more decision points are introduced.The harmful-pruning probability increases with the number of decision points, formalizing misjudgment accumulation.

B.2. Coverage Gain from Retrieval Grafting … C. Evaluation Details

Graft uses pruning-released candidate slots to improve frontier coverage and preserve the latency benefits of pruning, while standard target-model verification guarantees losslessness. The evaluation section provides additional experimental-setup details for reproducibility.

  • B.2. Coverage Gain from Retrieval Grafting: Pruning releases candidate slots that retrieval grafting fills, while keeping the hybrid tree’s verification budget no larger than the dense tree’s.The released budget is ΔK = |T0| − |Tπ|, and the grafted retrieval tree satisfies |G| ≤ ΔK.
  • B.2. Coverage Gain from Retrieval Grafting: Grafting cannot reduce accepted length for a fixed pruned tree because the pruned tree is contained in the hybrid tree.The construction gives Tπ ⊆ Tg, so grafting adds proposals without removing the pruned tree’s candidates.
  • B.2. Coverage Gain from Retrieval Grafting: Retrieval increases frontier coverage whenever it adds a candidate with non-zero target probability that was absent from the remaining draft candidates.Only retrieved tokens not already covered by the frontier candidate set contribute additional probability mass.
  • B.3. Speedup Condition with Grafted Retrieval: Graft improves over pure pruning when retrieved candidates recover part of the removed valid continuation, yielding Γg > 0.Retrieval uses released slots without another draft-model pass; GPU-resident matrix indexing is overlapped with drafting, making Tret small.
  • B.4. Lossless Verification: Graft preserves the target model’s output distribution because retrieval changes only the proposal set, while every proposed token undergoes the standard speculative acceptance rule.Rejected positions are resampled from the target correction distribution, regardless of whether proposals came from the drafter or adjacency-matrix retrieval.
  • B.4. Lossless Verification: The paper summarizes Graft’s theoretical role as reducing drafting cost through pruning, recovering coverage through retrieval, and preserving losslessness through target verification.This combines the stated effects of pruning, grafting, and verification into the paper’s overall takeaway.
  • C. Evaluation Details: The evaluation section provides additional experimental-setup details for reproducibility, with source code planned for later release.These details supplement the setup described in Section 4.

C.1. Data Configurations · C.2. Model Configurations · C.3. Detailed Baselines

The evaluation spans short-context, long-context, and high-concurrency decoding, using established benchmarks and matched target–draft configurations. Graft is compared with diverse speculative-decoding baselines under controlled serving, timing, and warm-up protocols.

  • C.1. Data Configurations: Short-context evaluation covers code generation, mathematical reasoning, summarization, instruction following, and chat using HumanEval, GSM8K, CNN/DM, Alpaca, and MT-Bench.These experiments follow the standard EAGLE-3 and Spec-Bench setup.
  • C.1. Data Configurations: Long-context evaluation uses five generation-heavy LongBench benchmarks spanning meeting, government-report, and multi-document summarization plus long-context code completion.The supplied passage identifies QMSum, GovReport, MultiNews, and LCC among these benchmarks.
  • C.2. Model Configurations: Short-context experiments use official EAGLE/ECHO-compatible checkpoints, load all weights in bfloat16 without quantization, and leave target and draft parameters unchanged.This preserves Graft’s training-free evaluation setting.
  • C.2. Model Configurations: Long-context experiments pair each target model with a YaRN-adapted EAGLE3-64K draft module.The target–draft pairings are summarized in Table 9.
  • C.2. Model Configurations: The model configurations include Vicuna 68M, the lightweight draft module used for the Vicuna-13B tree-draft pair.The broader architecture settings are listed in Table 10.
  • C.3. Detailed Baselines: Short-context comparisons cover standard, retrieval-based, and other speculative-decoding families, including Sps, Lookahead, PLD, SAMD, and Token Recycling.The passage describes these methods as representative baselines and identifies their cited works.
  • C.3. Detailed Baselines: Long-context comparisons include EAGLE3-64K, MagicDec, TokenSwift, and TriForce, representing direct tree drafting and sparse, partial, or hierarchical-cache approaches.EAGLE3-64K is the direct baseline under the same target model and context window.
  • C.3. Detailed Baselines: Experiments run on 8× NVIDIA H20 GPUs with greedy decoding, while high-concurrency ablations use SGLang and low-concurrency comparisons share depth 8, top-k=10, and candidate budget 60.Graft preserves the per-request verification budget and static serving envelope for high-concurrency evaluation.

C.4. Cross-Task Threshold Generalization

Graft’s pruning locations and gating thresholds generalize across downstream datasets: thresholds vary mostly within 0.05, while HumanEval-calibrated thresholds cause only about 1%–3% relative degradation elsewhere and still outperform EAGLE3.

  • Calibration stability: Graft’s pruning locations are primarily determined by draft-model capability and are only weakly sensitive to the downstream dataset.Warm-up calibrates both pruning locations and gating thresholds.
  • Calibration stability: Gating thresholds remain stable across five benchmarks, with variations mostly within 0.05 despite dataset-specific acceptance distributions.The observed threshold fluctuations do not significantly affect the calibration outcome.
  • Fixed-threshold generalization: About 1%–3% relative degradation results when HumanEval-calibrated thresholds are applied to other datasets instead of using per-dataset calibration.This fixed-threshold evaluation directly tests cross-task robustness.
  • Fixed-threshold generalization: Fixed thresholds still substantially outperform EAGLE3 on all evaluated benchmarks, demonstrating strong robustness and generalization ability.The comparison uses thresholds calibrated on HumanEval and transferred to other benchmarks.

D. High-Concurrency Implementation Details

The high-concurrency implementation of Graft on SGLang v0.5.4 preserves static per-request tree construction and applies pruning after the full batch is available. It uses fixed verification shapes, shared GPU-resident retrieval structures, and batched updates without changing core interfaces.

  • Batch Construction: Graft’s batched implementation completes standard static tree construction for every request before applying depth-based processing to the available batch.Unlike the single-request implementation, it does not change per-request tree depth during drafting.
  • Interface Compatibility: The implementation preserves fixed per-request verification shapes, changing only token identities from low-confidence draft candidates to retrieved candidates.It therefore avoids modifying CUDA-graph capture, the scheduler interface, or the tree-attention kernel API.
  • GPU-Resident Retrieval: All requests share one GPU-resident adjacency matrix, while verified nodes and target logits are flattened for batched top-k successor updates.GPU indexing performs retrieval from the shared matrix without CPU-side datastore synchronization or inter-request communication.

E. Runtime Overhead

Graft’s retrieval construction overlaps with tree drafting, while post-pruning node merging adds almost no latency. Runtime is dominated by target verification and standard per-round KV/input updates rather than retrieval or graft integration.

  • Retrieval and integration: GPU-resident matrix lookup builds the retrieval branch in parallel with tree construction.Its latency is reported separately because it does not add to the critical path.
  • Retrieval and integration: 0.015 ms is required to merge retrieved nodes into the retained draft tree after pruning.
  • Retrieval and integration: 0.570 ms is required to rebuild the tree mask, position IDs, and candidate paths.
  • Dominant costs: 26.275 ms for target verification and 8.036 ms for standard per-round KV/input updates dominate the runtime.
  • Critical path: 43.19 ms is the measured critical-path latency used to normalize the runtime breakdown percentages.

F. DFlash Demo Implementation · G. Detailed Related Work

The DFlash demo adapts Graft through tail retrieval while preserving the original verification budget and losslessness, but remains preliminary. Related work situates Graft among adaptive trees, retrieval, parallel, long-context, block, and multimodal speculative decoding.

  • F. DFlash Demo Implementation: DFlash drafts 16-token blocks, and the demo tests whether grafting extends to block drafters rather than providing a fully optimized deployment.The implementation uses HuggingFace transformers instead of a full SGLang integration.
  • F. DFlash Demo Implementation: Graft(TAIL) retains the reliable DFlash prefix after confidence pruning and appends retrieved tokens to the remaining chain tail without changing the pipeline.Tail attachment reduces prefix-dependency risk and avoids additional tree-verification cost.
  • F. DFlash Demo Implementation: The method keeps the total verified-token budget at B, reuses Graft’s GPU-resident adjacency matrix, and remains lossless under standard speculative acceptance.Retrieved tokens fill B_ret = B − B_df slots while DFlash retains B_df tokens.
  • F. DFlash Demo Implementation: Future DFlash work requires high-concurrency optimization, broader model and task testing, and alternative retrieval sources or grafting positions.These directions aim to preserve the low-latency chain structure while allocating released token budget more effectively.
  • G. Detailed Related Work: Related work develops speculative decoding through draft models, blockwise prediction, tree verification, feature-level drafting, and adaptive tree construction.Adaptive methods adjust draft length or topology using confidence, acceptance history, or token probabilities.
  • G. Detailed Related Work: Retrieval-based methods reuse repeated local patterns for candidate generation, but retrieval-only approaches often depend on prompt-local matches and can lack broader coverage.The cited approaches include Lookahead decoding, PLD, REST, Token Recycling, LogitSpec, SAMD, and Ouroboros.
  • G. Detailed Related Work: Parallel, long-context, and block-level drafters address drafting dependencies, KV-cache traffic, attention bandwidth, and serial proposal costs through concurrency, cache methods, or parallel blocks.The related work includes parallel speculative decoding, LongSpec, SpecPV, KVShot, and DFlash.
  • G. Detailed Related Work: Multimodal speculative decoding introduces modality-specific proposal, visual-token compression, and cross-modal cache challenges, while Graft is currently evaluated only on text generation.Recent systems address vision- and video-language inference through visual-alignment-aware or vision-aware verification.
Loading 2605.20104v1…