PaperlayerView exact revision on arXiv

Source-linked AI summary

S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

Yuhang He, Hongyu Ge, Zihan Lin, Siyi Liu, Yongqi Zhang

arXiv:2610.13202v1cs.LGcs.CL

TL;DR

Long-context RLVR needs cheaper rollouts without changing the response groups used for policy optimization. S3R proposes tokens with sparse attention and uses full-attention verification with exact speculative sampling to preserve the dense response-group distribution. It reports higher end-to-end training throughput than AR on two Qwen3 models, with comparable task performance.

  • Problem

    Long-context RLVR rollouts incur substantial generation cost, while sparse-policy sampling can change response groups and group-relative updates depend on those groups.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Method

    S3R uses sparse self-drafting followed by full-attention verification and exact speculative sampling, sharing policy weights rather than training a separate drafter.

    Source: §1 · p. 2

  • Results

    S3R achieves 1.39× and 1.35× the end-to-end training throughput of AR on Qwen3-1.7B and Qwen3-8B, respectively, while maintaining stable training and comparable task performance.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Under the stated sampling conditions, sparse proposal budgets change decoding cost without changing the dense response-group distribution.

    Source: §3.3 · p. 4

    Self-sparse drafting retains projections, feed-forward computation, and dense verification, so its benefit depends on speculative-decoding costs and runtime state management preserving computational savings.

    Source: §5 · p. 5

Abstract

from arXiv · show

Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76\% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S$^3$R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S$^3$R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S$^3$R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S$^3$R achieves $1.39\times~1.35\times$ the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.

1 INTRODUCTION

Long-context RLVR rollouts are costly, and sparse sampling can change the response groups that determine group-relative updates. S3R confines sparsity to proposals and uses exact dense verification; experiments report higher throughput than autoregressive rollouts with comparable task performance.

1 INTRODUCTION

  • In GRPO, exact response-wise reweighting need not recover the expected dense-policy update because each response’s advantage depends on other group members’ rewards.The response-group distribution therefore matters to policy optimization.Source: §1 · p. 1
  • Attention accounts for 72–76% of autoregressive decoding kernel time as KV volume grows, making KV reads a major rollout cost.Sparse attention can reduce these reads, but direct sparse sampling changes the behavior policy and can destabilize training.Source: §1 · p. 1, Figure 1 · p. 2
  • S3R uses sparse proposals with full-attention verification and exact speculative sampling, sharing policy weights between drafting and verification while using KV-aware scheduling to limit cache thrashing.The scheduler aims to sustain rollout parallelism as dense verification retains the complete historical KV cache.Source: §1 · p. 2
  • 1.39× and 1.35×: S3R increases end-to-end training throughput over AR on Qwen3-1.7B and Qwen3-8B, respectively, while maintaining stable training and comparable task performance.The reported result is on mathematical RLVR workloads.Source: §1 · p. 2

2 RELATED WORK

Prior work reduces RL rollout costs through training orchestration, approximate rollouts, and speculative decoding, with different trade-offs in fidelity, proposal availability, and overhead. S3R applies sparse self-drafting with full-attention verification to preserve the dense rollout-policy distribution.

2.1 EFFICIENT REINFORCEMENT LEARNING FOR LLMS

  • RL systems improve GPU utilization through orchestration, overlap, and scheduling, while long responses still increase decoding latency and KV-cache pressure.These systems are complementary to reducing the cost of sampling each rollout.Source: §2.1 · pp. 2–3

2.2 SPARSE AND OTHER APPROXIMATE RL ROLLOUTS

  • Approximate rollouts reduce generation cost but differ from the dense policy; existing approaches use filtering, importance weighting, constrained schedules, low-precision methods, early exits, or proxy models.S3R instead uses self-sparse proposals and full-attention verification before reward and advantage computation.Source: §2.2 · p. 3

2.3 SPECULATIVE DECODING FOR INFERENCE AND RL ROLLOUTS

  • Speculative decoding can preserve the target distribution through exact rejection sampling, while retrieval, auxiliary proposers, and target-induced drafters differ in proposal source and maintenance costs.S3R uses sparse-attention self-drafting for group-based RLVR and supports repeated rollout–optimization transitions without separate drafter training.Source: §2.3 · p. 3

3 PROBLEM FORMULATION AND DESIGN PRINCIPLES

Sparse attention reduces KV reads but changes token probabilities, while GRPO updates depend on response groups rather than isolated responses. The design therefore puts sparsity in proposal generation and uses exact sampling to preserve dense-policy response groups.

3.1 SPARSITY-INDUCED MISMATCH IN RLVR

  • For each prompt, GRPO samples a group of complete responses from a frozen dense-policy snapshot, and their rewards determine group-relative advantages.Policy-gradient updates depend on the behavior policy that generated the responses.Source: §3.1 · p. 3
  • Sparse attention restricts KV reads without changing policy parameters, but using it as the behavior policy changes the token distribution and induces a sparsity mismatch.The sparse budget defines a sparse proposal policy from the frozen dense snapshot.Source: §3.1 · p. 3

3.2 IMPLICATIONS OF SPARSE ROLLOUTS FOR GRPO UPDATES

  • GRPO’s group-dependent advantages mean exact response-wise likelihood reweighting does not generally recover the expected update under dense response groups.A response’s weight corrects its own sampling mismatch, but its advantage still depends on rewards of other responses sampled from the sparse policy.Source: §3.2 · p. 4

3.3 PRESERVING DENSE RESPONSE GROUPS VIA EXACT SAMPLING

  • Exact speculative sampling accepts each draft token according to the target-to-proposal probability ratio and samples a residual correction token at the first rejection.The rejected candidate and its following suffix are discarded.Source: §3.3 · p. 4
  • With a fixed rollout snapshot, independent responses generated from sparse proposals and exact sampling have the same group distribution as dense autoregressive responses.This holds for any valid sparse budget under the stated sampling conditions.Source: §3.3 · p. 4
  • At fixed learner parameters, the training pipeline therefore retains the dense-rollout distribution and needs no additional sparsity-induced correction.The sparse budget can be selected for generation efficiency, and S3R applies sparsification to the dense snapshot without a separately trained drafter.Source: §3.3 · p. 4

4 S3R: SELF-SPARSE SPECULATIVE ROLLOUTS FOR REINFORCEMENT LEARNING

S3R couples sparse self-drafting with full-attention verification so that proposals reduce rollout cost while exact sampling preserves fidelity. Its drafting and verification use a shared policy snapshot and dense historical KV states.

4 S3R: SELF-SPARSE SPECULATIVE ROLLOUTS FOR REINFORCEMENT LEARNING

  • S3R combines self-drafting from sparse KV states with full-attention verification, using the shared policy rather than a separately trained proposer.The overview frames sparse drafting as reducing rollout cost and dense verification as preserving fidelity with KV-aware runtime support.Source: §4 · p. 4, Figure 2 · p. 5
  • Drafting and verification use a shared policy snapshot that is synchronized after each actor update and held fixed until requested responses are complete.Figure 2 and Algorithm 1 illustrate the connections and rollout-update loop.Source: §4 · p. 5
  • Sparse drafting begins with dense prefill and selects historical KV entries using attention scores from the previous verification pass.The configured sparse budget sets the target fraction of cached indices retained; the drafter records candidate tokens and their proposal distributions.Source: §4 · p. 5
  • The verifier processes draft tokens in one full-attention causal forward using dense historical KV, recomputes dense states, then applies exact sampling and refreshes the sparse context.Verification replaces provisional draft states with dense KV entries and removes unused candidate suffixes.Source: §4 · p. 5

5 EFFICIENCY ANALYSIS AND RUNTIME SUPPORT

The section quantifies when speculative drafting can beat autoregressive decoding and identifies KV-cache pressure as a runtime constraint. It pairs this cost analysis with adaptive admission and request-state management.

5 EFFICIENCY ANALYSIS AND RUNTIME SUPPORT

  • Acceleration requires η > 1, or τ > (kTq + TV)/Tp; higher acceptance helps only when it outweighs the added drafting cost.Source: §5 · pp. 5–6
  • Dense verification retains the full historical KV cache, so repeated eviction, recovery, and transfers can consume the latency margin.Source: §5 · p. 6
  • At (B, S) = (16, 4K), drafting and verification exceed the 7Tp AR reference even with full acceptance; break-even efficiency is about 3.8 output tokens per round.The estimate uses aggregate KV volume NKV = B · S ≈ 5.24 × 10^5 tokens.Source: Figure 3 · p. 6, §5 · p. 6
  • The KV-aware AIMD scheduler adjusts request admission using GPU KV utilization and restoration backlog, restoring paused requests before admitting new ones.Source: §5 · p. 6
  • Paused requests retain token and proposal state and finish under one frozen rollout snapshot, preserving their original group identities.Between RL steps, request and KV state are released for actor optimization, then inference state is rebuilt after policy synchronization.Source: §5 · p. 6

6 EXPERIMENTS

Across the evaluated Qwen3 models, S3R improves end-to-end training throughput while maintaining comparable task performance. The experiments also examine stability under sparse rollouts, rollout efficiency, and management of the full KV state needed for verification.

6.2 END-TO-END ACCELERATION WITH COMPARABLE TASK PERFORMANCE

  • S3R achieves 1.39× and 1.35× the end-to-end training throughput of AR on Qwen3-1.7B and Qwen3-8B, respectively, with comparable task performance.It has the highest end-to-end training throughput among the compared methods.Source: Table 1 · p. 7, §6.2 · p. 7

6.3 STABILIZING TRAINING UNDER AGGRESSIVE SPARSITY

  • With matched settings, S3R avoids the gradient spike and reward deterioration seen with direct sparse sampling plus mismatch correction; direct sparse rollouts remain relatively stable only at s = 0.9.At s = 0.05, S3R follows the Dense reward trajectory, while uncorrected direct sparse rollouts collapse early at s = 0.05 and are unstable at s = 0.7 and s = 0.8.Source: §6.3 · pp. 7–8, Figure 4 · p. 8

6.4 ANALYSIS OF RL ROLLOUT EFFICIENCY

  • On Qwen3-8B, S3R combines 78.2% median draft-token acceptance with the lowest median generation time, 0.0630 ms per processed token, among evaluated speculative methods.S3R maintains higher end-to-end training throughput than the evaluated baselines as training progresses and responses grow longer.Source: §6.4 · p. 8, Figure 5 · p. 8
  • As policy entropy rises, S3R’s overall acceptance decreases relatively 5%, while its block efficiency declines less than the frozen Auxiliary Proposer’s.Sharing the current rollout-policy weights avoids the frozen proposer’s parameter lag, although sparse approximation can still affect acceptance.Source: §6.4 · pp. 8–9
  • Raising s from 0.05 to 0.50 increases acceptance from 80.4% to 96.7% while reducing speedup from 1.370× to 1.058×; s = 0.05 is a practical balance in this workload.Reducing s to 0.03 lowers both acceptance and speedup; exact speculative sampling preserves the dense response-group distribution across budgets.Source: Figure 6 · p. 9, §6.4 · p. 9

6.5 MANAGING FULL-KV STATE UNDER MEMORY PRESSURE

  • At a concurrency ceiling of 256, offloading improves generation throughput by 2.0% over recomputation; AIMD adds 5.8% over Offload-256 and cuts transferred pages per 1,000 tokens from 18.0 to 1.3.Offload-128 and Offload-160 underperform Offload-256, while AIMD adapts admission to KV pressure and restoration demand to retain rollout parallelism.Source: Table 2 · p. 9, §6.5 · p. 9
  • Sparse self-drafting reduces KV reads, but faithful full-attention verification still requires the complete historical KV state, whose management becomes important as responses grow.A KV-aware scheduler limits repeated offloading and restoration while maintaining rollout parallelism.Source: §6.5 · p. 9

7 CONCLUSION AND FUTURE WORK

The conclusion presents S3R as a long-context RLVR rollout method that combines sparse self-drafting with exact speculative sampling. It reports higher end-to-end training throughput than AR on two Qwen3 model sizes and identifies several directions for future work.

7 CONCLUSION AND FUTURE WORK

  • With KV-aware runtime support, S3R achieves 1.39× and 1.35× the end-to-end training throughput of AR on Qwen3-1.7B and Qwen3-8B, respectively, while maintaining stable training and comparable task performance.Under a shared rollout snapshot, it preserves the dense response-group distribution without separate drafter training.Source: §7 · p. 9
  • Future work will explore adaptive sparse budgets and context selection, extensions to MoE and linear-attention models, and coordination of KV management with group completion and rollout–optimization transitions.The stated aim of coordinating KV management is to reduce overhead while preserving exact sampling.Source: §7 · p. 10

AI USE STATEMENT

The appendix analyzes why response-wise correction can fail under group-relative advantages, then details how S3R preserves dense response groups and manages sparse drafting and verification. It also records the authors’ use of generative AI tools.

AI USE STATEMENT

  • The authors used generative AI to find related work, draft and polish descriptions, and assist with mathematical derivations and checks; they reviewed the material and accept responsibility.The assisted derivations concerned response-wise importance correction, dense response-group preservation, and the drafting-cost trade-off.Source: AI USE STATEMENT · p. 10

A.1 TOKENWISE AND RESPONSE-WISE IMPORTANCE WEIGHTS

  • A single tokenwise sparsity ratio corrects the current token conditional but leaves the sparse prefix and continuation uncorrected.The appendix distinguishes this token-level ratio from complete-response importance weights and notes that group-relative advantages also depend on other sampled responses.Source: §A.1 · p. 17

A.2 LIMITS OF RESPONSE-WISE CORRECTION IN GRPO

  • Exact response-wise weights correct the focal response but leave comparison responses under the direct-sparse law, which can change the expected GRPO update.For larger groups, the other G −1 responses likewise retain the sparse law; a difference occurs when the expected contribution varies with comparison responses and the distribution shift does not cancel.Source: §A.1 · p. 17, §A.2 · p. 18

A.3 DENSE RESPONSE-GROUP PRESERVATION UNDER ADAPTIVE SPARSE PROPOSALS

  • Exact sampling accepts a proposal with probability min{1, p(v)/qs(v)} and draws a residual token on rejection, making the committed token follow the dense law.The identity holds conditional on preceding drafter states, including adaptive states; the same termination rules yield dense response distributions.Source: §A.3 · p. 19
  • S3R’s exact sampling preserves dense response-group distributions, so rewards and advantages are computed from groups with the dense-policy law.Independent responses retain their group identities and randomness; applying the same reward and training operations yields gradient estimates distributed as dense autoregressive rollouts.Source: §A.3 · p. 19

B.1 SPARSE-CONTEXT CONSTRUCTION AND VERIFICATION STATE

  • Drafting uses selected KV entries, while full-attention verification recomputes the proposed block using complete historical KV and replaces provisional draft states.The sparse selection is an index view into the complete cache; exact sampling discards KV entries for unaccepted suffix tokens.Source: §B.1 · p. 20

B.2 KV-AWARE AIMD SCHEDULER AND MIXED-STAGE EXECUTION

  • The KV-aware AIMD scheduler increases request concurrency under low KV utilization and backs off for high utilization or excessive restoration backlog.Paused requests are restored before new requests are admitted, subject to the concurrency limit and available capacity.Source: §B.2 · p. 20
  • Mixed execution packs prefill, draft, and verification requests into one model forward, using sparse attention for drafting and full attention for prefill and verification.Active request count can differ from the query count processed by the model.Source: §B.2 · p. 21

B.3 EXECUTION IN THE RL TRAINING LOOP

Rollout replicas use a synchronized actor snapshot for drafting and verification, return complete response groups for the actor update, and release rollout resources between phases.

B.3 EXECUTION IN THE RL TRAINING LOOP

  • At each RL step, rollout replicas synchronize to the actor snapshot, which supplies both drafting and verification; returned response groups retain prompt and group identities.After the actor update, rollout resources are released for the next generation phase.Source: §B.3 · p. 21
  • Algorithm 1 combines per-request drafting and verification with KV scheduling, mixed forwards, exact sampling, and subsequent reward and advantage computation.The procedure releases request and KV resources before actor optimization.Source: §B.3 · p. 21

C DETAILED EXPERIMENTAL SETUP

C.1 INFRASTRUCTURE, MODELS, AND DATA

The experimental setup fixes shared infrastructure, models, training and evaluation protocols, and throughput measurement conventions. It also specifies baseline and ablation configurations and frames checkpoint portability experiments.

C.1 INFRASTRUCTURE, MODELS, AND DATA

  • Main training experiments use Qwen3-1.7B and Qwen3-8B on DAPO-MATH-17k, with eight responses per prompt and no additional importance correction.The setup uses 256 prompts per batch, AdamW at 10^-5, and a 4K-token overlong-penalty window.Source: §C.1 · p. 21, §C.2 · p. 22

C.3 MEASUREMENT AND REPORTING

  • Evaluation reports AIME26 and ArXiv-Math pass@1 from 16 responses per problem, with each problem equally weighted within its benchmark.Reported throughput averages per-step tokens per second per GPU; the reporting window excludes the first warm-up step.Source: §C.3 · p. 23

C.4 BASELINE IMPLEMENTATIONS

  • The baselines include autoregressive decoding, n-gram lookup, history retrieval, and a frozen EAGLE3 auxiliary proposer.The auxiliary-proposer comparison measures acceleration without additional drafter training; the n-gram setup proposes at most five tokens per round.Source: §C.4 · pp. 23–24

C.5 ABLATION CONFIGURATIONS

  • The direct-sparse correction baseline applies sequence-level masking and tokenwise importance reweighting, while the KV-management ablation compares recomputation, fixed offload limits, and KV-aware AIMD.The KV-management comparison uses ten post-warm-up RL steps under the main Qwen3-1.7B configuration.Source: §C.5 · p. 24

D CHECKPOINT PORTABILITY AND ONLINE DRAFTER ADAPTATION

  • Checkpoint experiments address using speculative decoding on independently post-trained targets without additional drafter training and maintaining a drafter as the policy evolves.The study includes DAPO training checkpoints and a separately treated Nemotron-Cascade-8B target.Source: D CHECKPOINT PORTABILITY AND ONLINE… · p. 24

D.1 CHECKPOINT PORTABILITY WITHOUT AUXILIARY PROPOSER TRAINING

S3R retains generation speedup across target checkpoints without fitting a new drafter, while periodic auxiliary-drafter adaptation does not guarantee rising acceptance. Across model sizes, S3R tracks AR learning behavior, though acceptance is lower and declines more at later draft positions.

D.1 CHECKPOINT PORTABILITY WITHOUT AUXILIARY PROPOSER TRAINING

  • The checkpoint study compares five Qwen3-8B-family targets, including four DAPO–History Retrieval checkpoints and a separately treated Nemotron-Cascade-8B model.The targets are evaluated on 30 AIME24 problems with 16 responses per problem at temperature 0.6; reported speedups exclude RL optimization time.Source: §D.1 · pp. 24–25
  • Across five targets, S3R retains 76.32–78.78% draft-token acceptance and 1.698–1.799× generation speedup.On Nemotron, S3R achieves 76.74% acceptance and 1.740× speedup without fitting a new drafter; the fixed auxiliary proposer gives 1.136× with five drafts and 0.938× with eight.Source: §D.1 · p. 25, Figure 7 · p. 25

D.2 PERIODIC ONLINE DRAFTER ADAPTATION

  • From steps 61–75 to 76–82, mean acceptance length falls from 3.603 to 3.245 for DSpark and from 3.761 to 2.607 for DFlash; EAGLE3 changes from 2.075 to 2.124.These periodic adaptation configurations do not show a monotonic increase in proposal acceptance.Source: §D.2 · pp. 25–26, Figure 7 · p. 25

E.1 TRAINING DYNAMICS ACROSS MODEL SIZES

  • Across Qwen3-1.7B and Qwen3-8B, S3R follows AR with comparable reward trajectories despite response lengths decreasing in one model and increasing in the other.Qwen3-8B’s later reward trajectories remain close after some early differences.Source: §E.1 · p. 26, Figure 8 · p. 26

E.2 ACCEPTANCE ACROSS DRAFT POSITIONS

  • For Qwen3-8B at step 20, draft-token acceptance falls from 95.8% at position one to 67.9% at position six; position-six acceptance declines 6.6 points by step 100.Qwen3-1.7B shows the same ordering, with lower acceptance and larger training declines at later positions.Source: §E.2 · p. 26, Figure 9 · p. 27

E.3 QWEN3-1.7B THROUGHPUT AND TIMING–ACCEPTANCE RESULTS

This appendix examines when sparse self-drafting can improve rollout speed, tracing attention savings through drafting and verification costs to measured throughput. It relates the opportunity for sparse speculation to live KV volume, execution costs, and accepted output.

E.3 QWEN3-1.7B THROUGHPUT AND TIMING–ACCEPTANCE RESULTS

  • 1.392×: S3R achieved mean end-to-end throughput of 3,213.8 tokens/s/GPU versus 2,308.6 tokens/s/GPU for AR.Figure 10a shows the throughput trajectory during training.Source: Figure 10 · p. 27, §E.3 · p. 27
  • 76.3% median draft-token acceptance at 0.0209 ms per processed token compares with 29.1% at 0.0341 ms for Auxiliary Proposer and 36.6% at 0.0396 ms for History Retrieval.S3R's median generation-time metric is 38.9% lower than Auxiliary Proposer's and 47.3% lower than History Retrieval's.Source: Figure 10 · p. 27, §E.3 · p. 27

F.1 LATENCY DECOMPOSITION FOR SELF-SPARSE SPECULATIVE DECODING

  • Sparse self-drafting is favorable when attention savings outweigh retained projection work and dense verification costs; the modeled condition can fail at every acceptance rate.Larger batches can increase verification-to-AR latency, while longer contexts at fixed batch size push that ratio toward one.Source: §F.1 · p. 28

F.2 FROM ATTENTION SAVINGS TO ROLLOUT SPEEDUP

  • The profiling and throughput measurements trace whether attention savings persist after drafting and verification costs are included.The setup defines B as active batch per TP replica, S as per-request context length, and NKV = BS as aggregate live KV volume.Source: §F.2 · p. 28

F.2.1 LARGER LIVE KV VOLUMES MAKE ATTENTION THE BOTTLENECK

  • At B = 16, attention rises from 2.34 ms at S = 4K to 12.93 ms at S = 32K while FFN stays near 2.24 ms; at 512K, attention-to-FFN ratios are 4.92–5.88.Across the profiled range, aggregate live KV volume is a useful first-order indicator of the attention bottleneck.Source: §F.2.1 · p. 29, Figure 11 · p. 30

F.2.2 ATTENTION SAVINGS MUST COVER DRAFTING AND VERIFICATION

  • 38.3% attention-time reduction at 64K live KV tokens rises to 63.2–63.7% at 512K with s = 0.05, including sparse drafting and dense verification.Dense verification still requires the complete KV state. Under sequential stage accounting, break-even acceptance is 75.8–77.0% at 256K and 45.5–46.7% at 512K; costs exceed AR at 64K–128K even with full acceptance.Source: §F.2.2 · pp. 29–30, Table 6 · p. 30

F.2.3 THE SPARSE BUDGET TRADES ACCEPTED PROGRESS AGAINST EXECUTION COST

  • At 5% versus 50% draft budget, median speedup falls from 1.533× to 1.094× for 1.7B and from 1.370× to 1.058× for 8B despite higher acceptance.The 5–10% region balances accepted progress against execution cost, supports the 5% default without identifying a unique optimum, and preserves dense response groups under Proposition 2.Source: Figure 12 · p. 31, §F.2.3 · p. 31

F.3 INTERPRETING THE BUDGET–ACCEPTANCE SPEEDUP SURFACE

The budget–acceptance surface estimates speedup from accepted output relative to round cost, with drafting attention separated from other costs. Its calibration fits profiled execution settings, while the reported RL measurements use changing batches and KV offloading.

F.3 INTERPRETING THE BUDGET–ACCEPTANCE SPEEDUP SURFACE

  • The calibrated surface estimates speedup as accepted output per round divided by normalized round cost; c0 includes projection, dense-verification, and correction costs, while c1s is draft attention.Figure 13 plots this estimate over sparse budget and acceptance, with calibration and held-out observations.Source: Figure 13 · p. 32, §F.3 · p. 32
  • Held-out mean absolute percentage error falls from 84.61% to 0.80% for 1.7B and from 63.03% to 0.68% for 8B with the constant correction.The residual errors apply only to the profiled execution settings, and the correction cannot identify individual sources of error.Source: §F.3 · p. 33
  • 1.558× and 1.491× throughput speedups were measured for 1.7B and 8B RL steps, at 75.7% and 77.1% acceptance, respectively.These runs exclude optimization time and use changing batches, CPU KV offloading, and a vLLM AR baseline; local fitting accuracy does not establish RL-training prediction accuracy.Source: §F.3 · p. 33
Loading 2610.13202v1…