PaperlayerView exact revision on arXiv

Source-linked AI summary

ProVe: Proposer-Verifier Decoding

Ziqi Jin, Weiwen Xu, Jiaxi Li, Yujie Zhong, Qi She, Aixin Sun

arXiv:2610.27815v1cs.CLcs.LG

TL;DR

Speculative decoding reduces large-model passes by having a small model draft serially, but its roles are tied to the cost of serial drafting. ProVe instead has a large model propose a block and a verifier read it in a second pass. Across six benchmarks, it offers lossless acceleration or a smaller-verifier speed–accuracy tradeoff.

  • Problem

    Speculative decoding reduces large-model passes with a small model that drafts tokens serially, but serial drafting must be cheap, fixing the roles by model cost.

    Source: §1 · p. 1

  • Method

    ProVe has a large model propose a block in one pass and a verifier read the proposed tokens in a second pass; the proposer can verify itself for lossless decoding.

    Source: §1 · p. 2

  • Results

    Across six benchmarks, 32B ProVe-lossless reaches 2.5× average AR throughput; with a 3B verifier, ProVe reaches 2.7× at 2.6 accuracy points below AR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ProVe provides a controllable speed–accuracy tradeoff with a smaller verifier, while self-verification reproduces the proposer’s greedy AR decoding.

    Source: §1 · p. 2

    The cost analysis ignores the small additional cost of processing a block and serving overheads.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Speculative decoding accelerates autoregressive (AR) language models by letting a small model draft tokens serially and a large model verify them in parallel. We reverse these roles. Starting from a standard AR checkpoint, we fine-tune the large model into a proposer that predicts a block of future tokens in a single forward pass, then use one additional forward pass to verify the proposed block. This gives ProVe, Proposer-Verifier decoding. The verifier can be a smaller AR model or the proposer itself; the latter gives ProVe-lossless, which reproduces the proposer's greedy AR decoding. Block proposals already provide substantial room for acceleration: with a block size of 16, the proposed block has an average AR-matching prefix of 6.6 tokens on GSM8K. Across six benchmarks with a 32B proposer, ProVe-lossless reaches 2.5x average AR throughput without training a separate draft head. With a small 3B verifier, ProVe reaches 2.7x average throughput at 2.6 accuracy points below AR on the six-task average, providing a controllable tradeoff between speed and accuracy.

1 Introduction

Speculative decoding assigns serial drafting to a small model and parallel verification to a large one; ProVe reverses those roles by having a large model propose a block and a verifier read it in a second pass. Across six benchmarks, ProVe offers lossless acceleration or a smaller-verifier speed–accuracy tradeoff.

1 Introduction

  • ProVe reverses speculative decoding’s roles: the large model proposes a block in one pass, and a verifier reads it in a second pass.Source: §1 · pp. 1–2
  • 6.6 tokens is the average AR-matching prefix on GSM8K, compared with 3.0 tokens committed by the default confidence rule.Source: §1 · p. 2
  • The proposer’s confidence rule uses placeholder-conditioned predictions, whereas verification reads the realized proposed tokens to decide how much to commit.Source: §1 · p. 2
  • Across six benchmarks, 32B ProVe-lossless reaches 2.5× average AR throughput, while a 3B verifier reaches 2.7× with accuracy 2.6 points below AR.Source: §1 · p. 2

2 Preliminaries: Block-Proposing AR Models

A block-proposing AR model uses one causal forward pass to predict multiple future tokens from placeholder inputs. Its predictions become less reliable with depth, and training combines ordinary next-token and block-prediction objectives.

2 Preliminaries: Block-Proposing AR Models

  • A block-proposing AR model is an ordinary decoder-only model additionally trained to predict the next B tokens in one forward pass.Source: §2 · p. 3
  • Given a context followed by B placeholders, one causal pass produces B distributions for the next B tokens.Source: §2 · p. 3
  • Deeper positions omit more intervening future tokens from their conditioning context, and both individual-token and full-prefix match probabilities decrease with position.Source: §2 · p. 3, Figure 3 · p. 4
  • Training combines ordinary next-token prediction with block prediction on placeholder-corrupted responses, using the same transformer and output head.Source: §2 · p. 3
  • The original MARS rule commits the first token, then continues left to right while proposer confidence exceeds a threshold.Source: §2 · p. 3

3 Proposer-Verifier Decoding

ProVe keeps the MARS block proposer fixed and spends one additional forward pass for a verifier to judge the realized proposed tokens. Verification can use a smaller autoregressive model or the proposer itself, with the latter preserving its greedy decoding exactly.

3.1 Why Verify the Proposed Block?

  • 6.6 tokens is the average AR-matching prefix, versus 3.0 tokens committed by MARS’s default free-confidence rule.Lowering the confidence threshold commits more tokens but also increases non-matching commits.Source: §3.1 · p. 4
  • Verifier probability distinguishes matching from non-matching proposals better than proposer confidence (AUC 0.960 versus 0.910); their product reaches AUC 0.964.The paper uses the parameter-free product as ProVe’s gate score.Source: Figure 4 · p. 5, §3.1 · p. 5

3.2 Verification

  • Each cycle proposes a block in one pass, then teacher-forces it through the verifier to score the realized prefix and commit the longest prefix passing the product threshold.The first proposed token is always committed; lowering the threshold can extend the committed prefix but may admit tokens that diverge from greedy AR decoding.Source: §3.1 · p. 4, §3.2 · p. 5
  • ProVe-lossless verifies the block with the proposer itself, replacing the first disagreement with its greedy prediction and thereby reproducing its greedy AR sequence exactly in two proposer passes per cycle.If all proposed tokens match, it also commits the next greedy token after the block.Source: §3.2 · p. 6

3.3 Cost

  • ProVe’s idealized per-cycle cost is 1 + ρ proposer-pass equivalents, versus 1 + kρ for speculative decoding with k serial draft steps; actual speedup also depends on tokens committed.The cost model omits small block-processing and serving overheads.Source: §3.3 · p. 6

4 Experiments

The experiments compare block-proposing decoding with autoregressive and speculative baselines across six general benchmarks, then test a reasoning proposer on GSM8K and MATH-500. ProVe-lossless preserves greedy AR output while smaller verifiers provide a speed–accuracy tradeoff.

4 Experiments

  • The main proposer is Qwen2.5-32B-Instruct adapted with block size 16; the evaluation covers six benchmarks using same-node AR throughput references.The 3B and 7B verifiers are evaluated as stock checkpoints or after fine-tuning on proposer generations.Source: §4 · p. 7

4.1 Main results

  • 2.54× average AR throughput is achieved by ProVe-lossless while reproducing the proposer’s greedy AR decoding, without a separate draft head.The reported range is 1.43× on IFEval to 3.98× on BBH; a proposer-specific EAGLE-3 head reaches 2.58× average.Source: §4.1 · p. 7, Table 1 · p. 8
  • 2.30× average throughput with a stock 7B verifier at θ=0.05 comes with −0.3 points relative to AR; a stock 3B verifier at θ=0.01 reaches 2.72× at −2.6 points.On GSM8K, the latter configuration reaches 2.89× throughput at −3.0 points relative to AR.Source: §4.1 · pp. 7–8
  • At matched non-matching-token error rates, verifiers fine-tuned on proposer generations commit 6–10% more tokens per cycle than stock verifiers.Matching error rates controls for probability-scale shifts that make fixed thresholds differ in conservativeness.Source: §4.1 · p. 9
  • At θ=0.10, ProVe reaches 2.28× average throughput with a stock 3B verifier and 2.19× with a stock 7B, exceeding speculative decoding with the same models.Speculative decoding reaches 1.63× and 1.39×, respectively; the difference is largest on longer structured generations such as BBH and GSM8K.Source: §4.1 · p. 9
  • Across 27 BBH subtasks, ProVe-lossless speedup rises from about 1.9× on the shortest responses to 5.5× on the longest.Within GSM8K responses, average AR-matching prefix length increases from 4.2 to 8.9 tokens, while committed tokens per cycle rise from 4.4 to 7.5.Source: §4.1 · p. 9, Figure 6 · p. 9

4.2 A reasoning proposer

  • On GSM8K and MATH-500, ProVe-lossless achieves 2.47× and 2.79× throughput relative to autoregressive (AR) decoding, respectively.At θ=0.10, ProVe reaches 2.34× on GSM8K at −0.4 points relative to AR and 3.16× on MATH-500 at −0.2 points.Source: §4.2 · p. 10

5 Related Work

Prior speculative decoding generally keeps the small-drafter, large-verifier division of labor, while block-proposal methods let a model propose multiple positions in one pass. ProVe adds a verifier pass that reads the realized proposed block.

Speculative decoding and draft heads

  • Most speculative decoding preserves the target distribution and fixed small-drafter/large-verifier roles, while Judge Decoding and Big Little Decoder relax verification rules.Prior block-draft systems retain exact target verification; MARS and related methods instead commit using proposer confidence, whereas ProVe adds a pass that reads the proposed block.Source: §5 · p. 10

Block proposals

  • Block-proposal methods range from added look-ahead embeddings, prompts, or heads to fine-tuning for masked future-token prediction, as in MARS.Unlike these approaches, ProVe adds verification of the proposed block; the paper reports that this step is essential to committing more tokens.Source: §5 · p. 10

6 Conclusion

ProVe uses a verifier to read proposed tokens, while ProVe-lossless verifies with the proposer itself and reproduces greedy AR decoding. The paper concludes that block proposal and verification need not follow the traditional division of labor.

6 Conclusion

  • On GSM8K, proposed blocks average 6.6 AR-matching prefix tokens, while the original confidence rule commits only 3.0.The paper attributes this gap to confidence being computed from placeholders rather than the realized proposed prefix.Source: §6 · p. 11
  • A verifier reads the realized proposed tokens, providing a second signal for deciding how much of the block to commit.With a smaller verifier, ProVe offers a speed–accuracy tradeoff; proposer self-verification yields exact greedy AR decoding and 2.5× average AR throughput.Source: §6 · p. 11
  • Block proposal and verification need not follow the traditional speculative-decoding division of labor: a large model can propose in parallel and verification can take one additional pass.The proposer can also serve as its own verifier in ProVe-lossless.Source: §6 · p. 11

A Implementation Details

The implementation uses a shared serving stack and controlled comparisons to measure throughput, block-size behavior, and the effects of training and speculative draft length.

Serving

  • The serving setup uses one SGLang-based stack, two data-center GPUs for the 32B model, bf16 weights, and one request in flight.ProVe uses block size 16; an undersized verifier memory pool silently degrades toward one accepted token per cycle.Source: A Implementation Details · p. 13

Measured pass costs

  • On two GPUs, measured pass costs are 14.8 ms for AR, 15.6 ms for a 32B window pass, and 5.0 ms or 5.8 ms for the 3B or 7B verifier.Scheduling and cache-trimming overhead is 8.7 ms for ProVe-lossless and 4 ms for ProVe.Source: A Implementation Details · p. 13

Block size

  • At block sizes 8/16/24/32, ProVe-lossless commits 5.9/7.5/8.1/8.6 tokens per cycle and reaches 2.44/2.79/2.84/2.84× speedup.Speedup saturates from block size 24 because both passes pay for every row of the window; other experiments use 16, the training block size.Source: A Implementation Details · p. 13

Effect of block training on ordinary AR quality

  • Block training changes six-benchmark mean greedy-AR accuracy from 73.1 to 72.5, a −0.6-point difference, with every task change within 1.8 points.The controlled comparison is the stage-1 AR-SFT checkpoint against the final proposer, not the original checkpoint.Source: A Implementation Details · p. 14, Table 3 · p. 14

Speculative draft length

  • Speculative draft lengths k=3 and k=5 do not dominate across tasks: the 3B six-task averages are 1.58× and 1.63×, while the 7B averages are 1.39× and 1.38×.Short constrained generations favor the shorter draft, while BBH benefits from the longer one.Source: A Implementation Details · p. 14, Table 4 · p. 14

Harness

  • The evaluation harness records generated tokens and wall-clock time per request, with throughput defined as total tokens over total in-flight seconds.Configurations undergo a fixed-prompt speed probe before use.Source: A Implementation Details · p. 14

B Absolute Evaluation Scores

Tables 5 and 6 provide absolute task scores underlying the main results tables; Table 5’s average is the unweighted mean across six tasks. The main tables report score changes relative to greedy-AR references, with evaluation uncertainty reported separately.

Absolute Evaluation Scores

  • Main tables report score changes relative to the corresponding greedy-AR reference, while evaluation uncertainty is reported in Appendix D.Positive deltas indicate direction only.Source: B Absolute Evaluation Scores · p. 15
  • Tables 5 and 6 report the underlying absolute scores for the lossy block and the scores underlying Table 2, respectively.Exact decoders are omitted from Table 5 because their algorithmic target is the AR reference; Table 6 notes that bf16 execution can make ProVe-lossless scores differ slightly from AR.Source: Table 5 · p. 15, Table 6 · p. 15
  • Table 5’s Avg is the unweighted mean of the six tasks.Source: Table 5 · p. 15

C Gating Rules Compared Offline

The offline comparison evaluates gating rules on tokens a sequential gate actually sees, measuring token-ranking quality and the accepted-token versus wrong-acceptance frontier. It finds that the verifier-informed product improves discrimination at deeper positions, while a learned scalar gate offers only a modest additional gain.

C Gating Rules Compared Offline

  • The verifier-informed product gains discrimination at deeper block positions, where accepted length is decided, while proposer confidence loses discrimination.The proposer confidence is computed under placeholder conditioning; the verifier sees proposed tokens before judging them.Source: C Gating Rules Compared Offline · p. 15
  • In the Figure 7 example, ProVe commits ten tokens, while MARS’s confidence rule would commit two.The eleventh proposed token is correct but receives verifier probability 0.12, so the cycle stops.Source: Figure 7 · p. 16
  • A logistic regression on recorded scalars accepts 2 to 4% more tokens at matched divergence, but the parameter-free product gate is retained.Reweighting the product factors moves the frontier by less than repeat-to-repeat noise.Source: C Gating Rules Compared Offline · p. 16
  • Table 7 compares gating rules on 26,407 blocks using AUC and accepted tokens per block at matched wrong-acceptance rates.The evaluated population is the one a sequential gate sees: a 3B verifier on GSM8K.Source: Table 7 · p. 16

D Measurement Validity

The measurement checks exactness, repeatability, uncertainty, verifier-pool behavior, and throughput under concurrency. Results show that observed ProVe-lossless deviations align with bf16 tie-breaking, while speedups vary with load and reported accuracy differences must be read against confidence intervals.

D Measurement Validity

  • ProVe-lossless tokens are non-argmax at 0.84% of positions, versus 0.85% for AR generations scored the same way; the reported gaps match bf16 tie-breaking.Across 300 GSM8K problems, 20 generations match token for token; 266 of 280 diverged pairs end at the same final answer.Source: D Measurement Validity · p. 16
  • Repeats of the same configuration on different nodes agree in accuracy to the item and throughput within 2%, except at the loosest 3B and 7B verifier thresholds.Source: D Measurement Validity · p. 16
  • Throughput is total generated tokens divided by total in-flight seconds, avoiding overweighting short generations dominated by prefill and the first cycle.Source: D Measurement Validity · p. 17
  • Accuracy differences within the reported 95% confidence half-widths are not interpreted as meaningful differences between decoding methods.For the six benchmarks, the half-widths range from ±0.9 to ±5.9 points; MATH-500’s is approximately ±2.4 points.Source: D Measurement Validity · p. 17
  • With a 7B verifier, an in-process pool fraction of 0.25 starves the pool and probes at 119 instead of 172 tok/s; the campaign uses 0.32.The proposer’s pool fraction is 0.55.Source: D Measurement Validity · p. 17
  • On GSM8K, ProVe-lossless’s relative throughput falls from 3.2× at C=1 to 1.6× at C=32; ProVe falls from 3.3× to 1.3×.Figure 8 plots total generated tokens per second and throughput relative to AR across concurrent requests; ProVe leads through C=4 and trails from C=8.Source: D Measurement Validity · p. 17, Figure 8 · p. 17

E Limitations

The paper’s results are bounded by block-specific proposer training, a research-oriented serving stack, greedy-only evaluation, and experiments on one model family at one proposer scale. The authors identify improved training and serving integration as potential routes to greater speedups.

E Limitations

  • The proposer is not a drop-in accelerator: it requires block-specific fine-tuning and uses one epoch each of ordinary AR and block training, versus five each in the original MARS recipe.The authors describe their proposer as substantially under-trained relative to that recipe.Source: E Limitations · p. 17
  • More training data, longer block training, or adding the block-proposal objective during pretraining could improve proposal quality and speedups.Source: E Limitations · pp. 17–18
  • The research-oriented serving stack adds scheduling, KV-cache, and verifier-call overheads that reduce realized speedup, particularly under concurrency.A more integrated propose–verify loop and optimized cache management are proposed as possible improvements.Source: E Limitations · p. 18
  • All reported experiments use greedy decoding; ProVe’s behavior and efficiency under sampling remain unstudied.Sampling for ProVe-lossless is described as a possible extension but has not been implemented or evaluated in the serving system.Source: E Limitations · p. 18
  • Experiments cover only the Qwen family and a 32B proposer, leaving gains across substantially different model families and scales open.The instruction proposer is evaluated on six benchmarks; the reasoning proposer is evaluated on GSM8K and MATH-500.Source: E Limitations · p. 18

F AI Use Statement

The authors used generative AI tools for several research and writing tasks, while excluding specified data and theory tasks. They reviewed AI-assisted content and retained responsibility for the final work.

F AI Use Statement

  • AI tools assisted with proof presentation, hypothesis refinement, methodology and experimental design feedback, implementation, and interpretation of results.They also helped draft manuscript portions, improve readability, and identify relevant literature.Source: F AI Use Statement · p. 18
  • The authors did not use AI to generate synthetic datasets, translate or reformat datasets, develop theoretical models or conceptual frameworks, or formulate or prove mathematical claims.Qualitative or thematic AI data analysis was not applicable.Source: F AI Use Statement · p. 18
  • Authors reviewed and verified AI-assisted content, evaluated AI-assisted implementation with manually written unit tests, and took responsibility for the final work.This responsibility covered text, claims, results, code, and other artifacts.Source: F AI Use Statement · p. 18
Loading 2610.27815v1…