PaperlayerView exact revision on arXiv

Source-linked AI summary

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

Yuchen Xian, Yang He, Yunqiu Xu, Yi Yang

arXiv:2606.12243v1cs.CLcs.AI

TL;DR

Speculative decoding’s binary accept-or-recompute rule sends intermediate-confidence tokens to the full verifier unnecessarily. VIA-SD introduces a routed slim verifier for staged verification, reducing rejection rates by 0.10–0.22 and delivering 10–20% speedups over strong speculative-decoding baselines.

  • Problem

    Binary draft–verify decisions leave intermediate-confidence tokens to the full verifier despite their requiring less verification capacity.

  • Method

    VIA-SD reframes speculative decoding as staged verification with an intra-model routed slim verifier between direct acceptance and full-model verification.

  • Results

    VIA-SD reduces rejection rates by 0.10–0.22 and delivers 10–20% speedups over state-of-the-art speculative-decoding baselines.

  • Takeaways & Limitations

    Multi-tier speculative decoding enables finer-grained allocation of verification effort and reduces unnecessary full-model invocations without sacrificing generation quality.

  • Takeaways & Limitations

    Reported speedups measure online decoding acceleration under the same model pair and exclude amortized end-to-end speedup from offline search.

Abstract

from arXiv · show

Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/

1. Introduction

The introduction reframes speculative decoding as multi-tier verification: intermediate slim-verifiers handle medium-confidence tokens, while the full model is reserved for the hardest cases. VIA-SD operationalizes this idea with a dynamically routed intra-model verifier and reports lower rejection rates and 10–20% speedups over state-of-the-art speculative decoding baselines.

  • Motivation: Speculative decoding typically amortizes large-model forward passes through a two-tier draft–verify process, but improvements largely retain a binary allocation rule.Existing efforts strengthen the drafter or accelerate the verifier without changing the accept-or-recompute decision structure.
  • Motivation: Many tokens occupy a middle zone where the drafter is close but unreliable, motivating lightweight intermediate verifiers that generate tokens while reserving the full model for hardest cases.The proposed hierarchical verification assigns generation responsibility progressively to stronger verifiers rather than enforcing binary acceptance or full recomputation.
  • Contributions: A KL-based information-theoretic view motivates adding intermediate verification stages according to alignment between drafter and verifier distributions.This analysis provides the design principle for moving beyond the conventional two-tier draft–verify mechanism.
  • Contributions: VIA-SD uses a dynamically routed intra-model slim-verifier to approximate the large model, handle medium-confidence tokens, and reduce redundant full-model calls.The framework empirically identifies a three-tier instantiation balancing efficiency and accuracy while alleviating the binary limitation of traditional draft–verify methods.
  • Results: 0.1–0.22 lower rejection rates and 10–20% speedup over state-of-the-art speculative decoding baselines are reported across summarization, translation, reasoning, QA, and coding tasks.The experiments cover five task categories listed in the introduction.

2. Related Work

Related work develops independent and adaptive drafting strategies alongside strict, lossy, cascading, and tree-based verification methods. VIA-SD instead accelerates verification through intermediate intra-model routed verifiers for hierarchical verification, reducing rejection while preserving drafter–verifier modularity.

  • Drafting Strategies: Drafting methods divide into independent drafters using small or non-autoregressive models and adaptive drafters reusing the target model’s structure.Adaptive designs use auxiliary heads for parallel generation, layer skipping, or early exiting.
  • Verification Strategies: Verification methods progress from strict, lossless validation toward lossy or cascading policies and token-tree parallelization.Strict verification guarantees equivalence but incurs rollback overhead, whereas later approaches relax acceptance or defer decisions to improve efficiency.
  • Our Perspective: VIA-SD accelerates verification through intermediate, intra-model routed verifiers that enable hierarchical verification.The approach adopts a distributional geometry perspective rather than focusing on drafting-side acceleration.
  • Our Perspective: The design reduces rejection while preserving modularity between drafter and verifier models.This positioning distinguishes VIA-SD from methods that primarily change drafting strategies.

3. Methodology

VIA-SD reframes speculative decoding as hierarchical verification: direct acceptance handles easy tokens, while intermediate stages address tokens that do not require the full verifier. It uses KL-based staged distributions and an intra-model routed submodel q′ as the slim verifier.

  • Draft–Verify and Lossy Acceptance: Speculative decoding drafts γ tokens from p_t and checks them sequentially under q_t, stopping at the first rejection or emitting the full block when all tokens are accepted.In lossy decoding, acceptance depends on relative likelihoods under p_t and q_t with tolerance α, and rejected tokens are sampled from a q_t-aligned residual distribution.
  • Rejection Rate and Its Limitation: In lossless speculative decoding, the single-step rejection rate satisfies ρ_t = D_TV(p_t, q_t), so reducing rejection requires changing the drafter or verifier distributions.This makes the distributional gap between p_t and q_t the fundamental determinant of speculative-decoding efficiency.
  • Rejection Rate and Its Limitation: Many drafted tokens occupy a middle zone that does not require the largest verifier’s full capacity, but binary accept–reject verification sends them to that verifier.This motivates adding intermediate verification stages without increasing the distributional gap.
  • KL Geometry and Π-Space: KL divergence supports staged verification because its directional, additive mismatch can be decomposed along intermediate distributions rather than mapped directly from p to q.The proposed path p → u_1 → u_2 → ··· → u_n → q provides principled intermediate anchors for additional verification stages.
  • KL Geometry and Π-Space: The lossy mechanism defines a global Π-space with mixture target distributions that keep p_t, u_t, and q_t consistent while routing tokens across progressively stronger stages.The mixture weighting is controlled by δ, which balances the drafter and verifier contributions.
  • Intra-Model Slim-Verifier: The intermediate verifier is a routed submodel q′ projected from q, chosen over scaled drafter variants and independent same-family models because it better preserves distributional consistency with q.Adjusting the intra-model routing ratio controls the trade-off between decoding acceleration and accuracy, yielding the routed path p → q′ → q.

4. Experiments

Experiments across multiple model families and benchmarks show that VIA-SD reduces rejection rates and improves decoding speed while preserving task quality. Ablations and integration studies attribute these gains to carefully routed intermediate verification that remains compatible with existing speculative-decoding frameworks.

  • Setups and Metrics: VIA-SD is evaluated on T5, Gemma2, LLaMA2, and Qwen using paired small-to-large configurations across summarization, translation, QA, and reasoning benchmarks.Experiments use encoder–decoder and decoder-only families, with efficiency measured by rejection rate and speedup over greedy decoding on A100 GPUs.
  • Decoder-Only Models: 30–45% lower rejection rates than speculative decoding yield 0.3×–0.8× speedup improvements over the strongest cascade baselines on decoder-only benchmarks, while maintaining or slightly improving accuracy.In large-gap settings, rejection rates fall from approximately 0.22–0.30 to 0.14–0.16, enabling speedups above 2.3×–2.7×.
  • Encoder-Decoder Models: 8–12 percentage-point rejection-rate reductions versus speculative decoding and cascade-based baselines produce 1.2×–1.4× additional speedup without degrading ROUGE scores on XSum and CNN/DailyMail.The results indicate that summarization’s concentrated token uncertainty allows many drafted tokens to resolve during early routed verification.
  • Search Cost: 18–68 minutes, or 0.30–1.13 GPU-hours, is the one-time DIMR search cost per model pair, with no routing-mask search during inference.Reported speedups correspond to online decoding acceleration rather than amortized end-to-end speedup including offline search.
  • Slim-Verifier Construction: 1.55× to 2.08× speed improves when replacing two-layer speculative decoding with an independent 13B intermediate model, but peak memory increases to 1.38×.Random skip-layer routing avoids extra model loading, while DIMR-based construction is favored because random masks reduce both accuracy and speedup.
  • Compatibility with Existing Frameworks: 20–31% lower rejection rates and 8–26% higher decoding speedups result when VIA-SD is added to EAGLE and PEARL across Gemma2-9B and Gemma2-27B backbones.The integration preserves the standard draft–verify interface while inserting a lightweight routed verifier between direct acceptance and full-model verification.

5. Conclusion

VIA-SD reframes speculative decoding as multi-tier verification rather than a binary draft–verify process. Its routed slim verifier allocates verification effort by token confidence, reducing unnecessary full-model invocations while preserving generation quality.

  • VIA-SD reframes speculative decoding as a multi-tier verification problem beyond the conventional binary draft–verify paradigm.
  • Hierarchical verification with an intra-model routed slim verifier allocates verification effort across tokens with varying confidence.
  • The framework reduces unnecessary full-model invocations without sacrificing generation quality.
  • A KL-based theoretical view motivates intermediate distributions for hierarchical verification.

Impact Statement · A. Notation and Details of Lossy Speculative Decoding

VIA-SD reduces unnecessary full-model verification through an intra-model routed slim verifier, with potential latency, compute, and energy benefits alongside deployment and misuse considerations. The lossy speculative-decoding formulation defines drafter and verifier distributions, sequential drafting and verification, rejection resampling, and a lossless special case.

  • Impact Statement: VIA-SD inserts an intra-model routed slim verifier between the drafter and full verifier to reduce unnecessary full-model invocations.The framework may maintain or slightly improve task quality while lowering latency, compute cost, and energy consumption.
  • Impact Statement: Speed measurements use A100 GPUs under controlled decoding settings, so absolute speedups may vary with deployment conditions.Batch size, sequence length, KV-cache implementation, kernel fusion, memory bandwidth, and hardware generation can affect measured speed.
  • Impact Statement: Rejection-rate reduction remains a model-level indicator of fewer full-verifier invocations and potential serving-efficiency gains.The passage distinguishes this indicator from deployment-dependent absolute speedups.
  • Impact Statement: Cheaper, faster generation can increase the throughput of both benign and harmful uses, including spam, misinformation, and automated social engineering.The method does not alter training or the underlying model’s unsafe-content propensity, but it can reduce the marginal cost of generating harmful content.
  • A. Notation and Details of Lossy Speculative Decoding: At decoding step t, the drafter Mp and verifier Mq define conditional next-token distributions pt(· | x<t) and qt(· | x<t) in the probability simplex ∆V.The notation uses V for the vocabulary and ∆V for the probability simplex over V.
  • A. Notation and Details of Lossy Speculative Decoding: Given block size γ, the drafter sequentially samples γ tokens from its conditional distributions, and the verifier evaluates them in order until the first rejection.The drafted sequence is xt through xt+γ−1.
  • A. Notation and Details of Lossy Speculative Decoding: When a drafted token is rejected, it is resampled from a residual distribution controlled by α and scaled by β ≥ 1 − α.The construction normalizes replacement probabilities and keeps replacement tokens aligned with the verifier distribution.
  • A. Notation and Details of Lossy Speculative Decoding: When α = 0 and β = 1, lossy speculative decoding reduces to standard autoregressive sampling from q and becomes lossless.This recovers the lossless decoding process described in the cited prior work.

B. Hierarchical Verification Pipeline: Derivation of δ1, δ2 and Their Relation to α1, α2 · B.1. Full Pseudocode for DIMR Pipeline · B.2. Probabilistic Derivation of δ1, δ2

The appendix specifies a hierarchical DIMR verification pipeline in which drafted blocks are processed by a slim verifier and escalated to the full verifier when confidence is insufficient. It formalizes the resulting three-way mixture through δ1 and δ2, relates these weights to α1 and α2, and notes that the thresholds may be learned or tuned.

  • B.1. Full Pseudocode for DIMR Pipeline: At each decoding step, the drafter proposes a γ-token block, DIMR builds q′ from q at skip ratio r, and q′ verifies prefixes in parallel.The procedure tracks the earliest rewrite or rejection position for rollback.
  • B.1. Full Pseudocode for DIMR Pipeline: Confident q′ decisions either accept the drafted token or rewrite it, while insufficient confidence escalates verification to q under the selected acceptance rule.The pseudocode distinguishes q′ acceptance, q′ replacement, and full-verifier escalation.
  • B.1. Full Pseudocode for DIMR Pipeline: The committed output is the longest valid prefix: κ=γ when the whole block is accepted, otherwise κ=j∗, followed by fallback to q at position t+κ.Decoding then appends the accepted prefix and continues from the updated position.
  • B.2. Probabilistic Derivation of δ1, δ2: With 0<α2<α1<1 and τ1=1−α1, τ2=1−α2, the analysis partitions drafted tokens into q′ acceptance, q′ rewriting, and escalation events.The three events are mutually exclusive and exhaustive.
  • B.2. Probabilistic Derivation of δ1, δ2: δ2 is defined as the probability of escalation event Ct(v), while ¯δ1 is the probability of q′ rewriting event Bt(v) under v∼pt.Together with the drafted-token probability δp, these weights satisfy δp+¯δ1+δ2=1.
  • B.2. Probabilistic Derivation of δ1, δ2: The resulting output distribution is a three-way mixture of acceptance with p, rewriting with q′, and escalation to q, with theoretical and practical confidence-gated formulations.The practical formulation matches Eq. (14) in the main text and corresponds to Algorithm 2.
  • B.2. Probabilistic Derivation of δ1, δ2: A stricter α1 increases q′’s weight within the non-escalation branch, whereas a looser α2 lowers δ2 and reduces escalation to q.The appendix also notes that α1 and α2 can be parameterized as learnable gates and optimized through the mixture distribution.

B.3. Analysis of Threshold Parameters α1, α2 … D.2. Shortest Piecewise Path via a Chain of Intermediates

The paper shows how routing thresholds regulate verifier reliance and efficiency, while KL-based intermediate distributions support flexible, theoretically minimal verification paths. It further establishes that beneficial intermediates are generally non-unique and that successive I-projections minimize additive piecewise KL action under nested constraints.

  • B.3. Analysis of Threshold Parameters α1, α2: Increasing α2 makes escalation probability Rt non-increasing, thereby reducing reliance on the large verifier q.The relationship is α2 ↑ ⇒ Eα2 ↓ ⇒ Rt = Pr(Eα2) is non-increasing.
  • B.3. Analysis of Threshold Parameters α1, α2: Increasing α1 makes q′ less strict and δ1 = Pr(Aα1) non-decreasing, accepting more tokens before rewriting or escalation.Conversely, smaller α1 enforces stricter early filtering.
  • B.3. Analysis of Threshold Parameters α1, α2: Increasing α2 improves speed by suppressing escalation to q, whereas increasing α1 mildly raises cost by shifting more tokens from p to q′.Smaller α1 can decrease cost but may increase rollback or escalation rates depending on α2.
  • C. Comparison of Divergence Measures in Π-Space: TV exactly matches rejection rate when α = 0 but is rigid because the direct path p →q is always shortest and intermediates cannot reduce effective distance.TV is symmetric and satisfies the triangle inequality.
  • C. Comparison of Divergence Measures in Π-Space: KL is asymmetric, lacks the triangle inequality, and permits an intermediate r∗ to strictly reduce effective discrepancy through information projection.This flexibility motivates KL as the primary divergence measure for hierarchical or lossy verification pathways.
  • D.1. An Infinite Family of Beneficial Intermediates: Under a non-singleton closed convex feasible set U containing q, the beneficial intermediate set Uben is typically infinite.I-orthogonal submanifolds through the I-projection u∗ or local perturbation families can generate infinitely many beneficial intermediates.
  • D.1. An Infinite Family of Beneficial Intermediates: The result shows that piecewise-path improvement is generally not tied to a unique intermediate, motivating families of deployable intermediates in hierarchical pipelines.The feasible family U may reflect task priors, model structure, and deployment constraints.
  • D.2. Shortest Piecewise Path via a Chain of Intermediates: Successive I-projections onto nested non-empty closed convex sets minimize additive piecewise KL action among chains respecting the same nest.With increasingly fine discretization along a dual-flat geodesic, the polygonal chain converges to a straight dual-coordinate trajectory while retaining action DKL(p∥q).

D.3. Practical Choices of u in Engineering Implementation

Practical choices for the intermediate distribution u are constrained by computational cost, latency, hardware, and deployment complexity, and fall into three broad categories. The paper selects a skip-layer variant of the large model q as the final intermediate layer.

  • Practical u designs: Three practical u designs are identified: scaling up the small model p, using a same-family intermediate-size model, and skipping layers in the large model q.These options reflect real-world constraints on computational cost, inference latency, hardware resources, and deployment complexity.
  • Scale-up of the small model p: Scale-up of p offers low additional inference cost and rapid integration through parameter sharing, but its expressive power remains limited relative to q.Examples include LoRA, adapters, prefix-tuning, or adding a few attention or feedforward layers.
  • Intermediate-size model: A same-family intermediate-size model provides broader verification accuracy and easy architectural integration, but requires maintaining a separate model.Natural checkpoints such as 2B–9B–27B series can serve as u.
  • Skip-layer variant of the large model q: A skip-layer variant of q balances theoretical soundness and engineering feasibility through parameter-space consistency and low overhead.It is constructed by skipping certain layers or extracting lightweight sub-networks from q.
  • Final choice: The paper chooses solution 3, the skip-layer variant of q, as the final intermediate layer.This selection follows its balance between distributional alignment and practical deployment considerations.

E. Why the Drafter Must Be an Independent Small Model p Instead of a Layer-Skipped q′′

The drafter must be an independently trained small model p rather than a layer-skipped q′′ because truncating q causes distribution collapse and substantially larger KL divergence. This raises rejection rates and can erase speculative decoding’s acceleration, whereas p preserves smoother distributions and lower cost.

  • Summary: Within hierarchical verification, q′′ is appropriate for the intermediate verifier u but not the drafter; the drafter must be an independent small model p.This design avoids KL blow-up from distribution collapse while preserving efficiency, smooth distributions, and speculative decoding acceleration gains.
  • Distribution Mismatch: Layer-skipping q to construct a drafter q′′ often causes distribution collapse, with significantly larger KL distance from q than an independently trained small model p.This makes q′′ poorly suited to generating many low-cost candidates while maintaining distributional consistency with q.
  • Underlying Reason: Removing critical layers damages q′′’s representational capacity, causing poor probability calibration and distorted token likelihoods.As the number of retained layers decreases, the divergence gap tends to grow rapidly, reflecting systematic bias rather than structured compression.
  • Effect on Rejection Rate: Because rejection-rate bounds are governed by distributional distance, replacing p with q′′ makes the bound dominated by the substantially larger D_KL(q′′∥q).The resulting rejection rate is therefore expected to be unacceptably high.
  • Practical Implication: Using q′′ as the drafter causes much more frequent rejection by q, negating speculative decoding’s acceleration benefit, while p yields smaller KL divergence and lower overall cost.The independent model p aligns better with q in distribution space and maintains more stable rejection rates.

F. Alternative Margin Penalties ϕ and Detailed Derivations … H. Additional Tables and Figures

The appendix unifies the acceptance gates as log-margin violations and generalizes their ReLU cost to convex, nondecreasing penalty functions. It also derives token-wise and block-level costs, while describing alternative penalties and their calibration properties.

  • F. Alternative Margin Penalties ϕ and Detailed Derivations: Acceptance-gate violations are represented by z1(v) > 0 for insufficient acceptance and z2(v) > 0 for residual replacement pressure.These margins correspond to the two distinct failure modes in the acceptance inequalities.
  • F. Alternative Margin Penalties ϕ and Detailed Derivations: A general ϕ-style single-step cost uses any convex, nondecreasing positive-part surrogate to penalize the two margin violations.ReLU and smooth alternatives are examples of admissible surrogates.
  • 2. Squared hinge: ϕ(z) =: Squared hinge, Huber-hinge, softplus, power hinge, and exponential penalties provide alternative curvature and smoothness choices for violation costs.Larger curvature penalizes large violations more aggressively, while softplus and Huber-hinge provide stable gradients.
  • 4. Softplus / smooth hinge (temperature τ > 0): ϕτ(z) =: Softplus satisfies z+ ≤ϕτ(z) ≤z+ + τ log 2 for all z, linking its smooth approximation to the positive-part penalty.Its temperature controls the approximation gap.
  • 5. Power hinge (aggressive): ϕp(z) =: Power-hinge penalties use an aggressive exponential form whose gradient increases with positive violations.This makes the surrogate emphasize larger positive margins.
  • 6. Exponential positive-part (heavy-tail): ϕexp: Alternative penalties preserve the three-tier folded-path construction, reducing to Eq. (7) for ReLU or ϕ = log(1 + ez).The same ϕ-style cost is reused on each segment of p→u→q.
  • G. From LLR Gates to Token-wise ϕ-Costs: The derivation averages acceptance shortfall under p and residual replacement cost under q because drafts originate from p while replacement borrows mass from q.This distribution-specific averaging yields the single-step ϕ-cost.
  • G. From LLR Gates to Token-wise ϕ-Costs: Token-wise summation and temporal accumulation recover the main-text equations, including Eq. (9), Eq. (10), and Eq. (11), under ReLU or softplus.The construction first expands expectations over vocabulary tokens, then sums single-step costs over decoding steps.

H.1. All Results on T5 and Gemma

Across Gemma and T5 experiments, the proposed hierarchical verification method consistently improves the quality–rejection–speedup trade-off across model scales and tasks. It reduces rejection while preserving or improving quality and delivers substantial acceleration over decoding baselines.

  • Hyperparameters: With γ = 5, α1 = 0.5, and α2 = 0.3, the proposed approach outperforms all baselines in the quality–rejection–speedup trade-off.This configuration balances intermediate-verifier acceptance tolerance with final-verifier replacement flexibility.
  • Gemma Results Analysis: On GSM8K, quality increases from 0.70 with baseline Speculative Decoding to 0.73 for Gemma2-2B→9B and 0.78 for Gemma2-2B→27B.Similar improvements on WebQuestions and NaturalQA indicate that lower rejection need not compromise output quality.
  • Gemma Results Analysis: 0.45→0.30 rejection rate on NaturalQA and 0.24→0.15 on TriviaQA show substantial reductions for Gemma2-2B→27B versus baseline Speculative Decoding.The reductions indicate more effective filtering of drafted candidates and fewer rollbacks.
  • Gemma Results Analysis: 2.50× speedup on TriviaQA with Gemma2-2B→27B surpasses Faster Cascades at 2.30× and CLaSP at 1.99×.On NaturalQA, the maximum speedup reaches 2.61×, with larger-model advantages becoming more pronounced.
  • T5 Results Analysis: For T5-S→XL, speedup reaches 1.95× on XSum, 2.75× on CNNDM, and 3.35× on WMT14, while WMT14 achieves 23.10 BLEU.On XSum, 18.9 ROUGE-2 pairs with a 0.26 rejection rate; the method also exceeds Cascade SD at 1.40× and Faster Cascades at 1.60×.

H.2. Additional Ablation Studies

The ablations show that α2 chiefly controls reliance on the full verifier, while α1 has a milder filtering effect. Across model scales and datasets, α1 ≈ 0.5 and α2 = 0.3 provide a robust trade-off, while moderate draft lengths maximize speedup.

  • Ablation on α1 and α2: α2 dominates full-verifier reliance: increasing it raises reliance above 0.6, whereas α2 = 0.1–0.3 keeps reliance around 0.1–0.2.Higher α2 stabilizes quality through more frequent full-verifier calls but reduces efficiency.
  • Ablation on α1 and α2: α1 has a milder effect, primarily tuning early-stage filtering while overall full-verifier reliance is chiefly governed by α2.Increasing α1 makes the pre-verifier more selective and modestly increases large-model invocation.
  • Ablation on α1 and α2: α1 ≈ 0.5 maximizes efficiency across model scales, while α2 = 0.30 or 0.40 generally outperforms α2 = 0.50; MBPP instead favors α1 = 0.6 with α2 = 0.30/0.40.NaturalQA and TriviaQA are more sensitive to α2, whereas SQuAD2 has flatter curves.
  • Ablation on α1 and α2: α1 = 0.5 and α2 = 0.3 provide the most stable trade-off, consistently achieving the highest or near-highest speedup across evaluated datasets.Both parameters set high approach near-lossless quality but almost eliminate computational benefits.
  • Ablation on Draft Length γ: γ = 5 is optimal for Gemma2-2B→9B, reaching about 2.1× on GSM8K, 2.2× on MBPP, and 2.6× on SQuAD2.Larger γ increases rejection probability and verification overhead after the optimum, rapidly reducing speedup.
  • Ablation on Draft Length γ: Gemma2-2B→27B has a broader optimum at γ = 6–7, reaching approximately 2.4× on GSM8K, 2.6× on MBPP, and nearly 3.0× on SQuAD2.Moderate γ values balance drafting efficiency and verification stability; larger models tolerate longer blocks better.
Loading 2606.12243v1…