PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim, Coleman Hooper, Harman Singh, Amir Gholami, Hyung Il Koo, Wonjun Kang

arXiv:2606.18967v1cs.LG

TL;DR

RL rollouts are latency-bound, while evolving policies and shrinking batches make speculative decoding difficult to apply effectively. EfficientRollout uses a target-induced quantized drafter with system-aware toggling and adaptive draft lengths, reducing rollout latency by up to 19.6% and end-to-end latency by up to 12.7% over an accelerated AR rollout baseline.

  • Problem

    Accelerating on-policy RL rollouts remains challenging because policies evolve and shrinking batches create compute-bound phases where speculative decoding may be slower.

    Source: §1 · pp. 1–2

  • Method

    EfficientRollout induces a quantized drafter from the target policy and combines system-aware speculative-decoding toggling with adaptive draft lengths.

    Source: §1 · p. 2

  • Results

    Up to 19.6% lower rollout-generation latency and 12.7% lower end-to-end training-step latency were achieved over a standard accelerated AR rollout baseline.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EfficientRollout offers lower-latency RL rollouts without invasive changes to the RL pipeline.

    Source: §1 · p. 3, §6 · p. 10

    The framework focuses on chain verification, leaving tree verification as a possible extension.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) offers a natural way to address this bottleneck, as it is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.

1 Introduction

EfficientRollout addresses rollout decoding as a dominant latency bottleneck in on-policy RL by combining a target-induced self-drafter with controls for when and how much to speculate.

Introduction

  • SD drafts candidate tokens for parallel target verification, but speedup requires sufficient block efficiency and decoding outside compute-bound regimes.Block efficiency measures the average number of target-distributed tokens produced per draft-verify iteration.Source: §1 · p. 2
  • Evolving policies can stale fixed drafters, while large early active batches can make SD slower than AR before batches shrink during decoding.Post-training can sharpen the policy distribution, creating an opportunity for a lower-capacity drafter induced from the target to cover concentrated probability mass.Source: §1 · p. 2
  • History-based drafters can have limited block efficiency, while learned auxiliary drafters often require separate pretraining and continual adaptation [19] [35] [46] [8] [71].Rollout generation is a dominant RL latency bottleneck because long responses decode token by token, often outweighing more parallelizable policy updates [18] [37] [63] [45].Source: §1 · pp. 1–2
  • EfficientRollout quantizes the current target policy into a drafter and combines regime-aware SD toggling with adaptive draft lengths.The drafter is refreshed each training step without separate pretraining or online updates; SD is avoided during early compute-bound, large-batch phases, and draft length follows observed acceptance behavior.Source: §1 · p. 2, Figure 1 · p. 2
  • Up to 19.6% lower rollout latency and 12.7% lower end-to-end latency are reported over a standard accelerated AR rollout baseline.The paper also reports these reductions over prevalent RL training and serving stacks.Source: §1 · pp. 2–3

2 Related Work

RL rollouts are a major latency bottleneck, while speculative decoding (SD) can help in memory-bound regimes but may slow compute-bound decoding. Prior rollout methods trade off deployment simplicity, drafter quality, and adaptation overhead; EfficientRollout uses a quantized copy of the target policy as its drafter.

2.1 Reinforcement Learning for LLMs

  • On-policy RL uses the rollout policy being optimized, while long outputs make sequential autoregressive decoding a major latency bottleneck.RL with verifiable rewards is prominent in math and coding, and policy optimization advances have supported reasoning and agentic capabilities [5] [45] [47] [63]; rollout latency can dominate more parallelizable log-probability and policy-update stages [16].Source: §2.1 · p. 3

2.2 Speculative Decoding for LLM Inference

  • Speculative decoding drafts tokens sequentially and verifies them in parallel, preserving the target distribution through rejection sampling while helping memory-bound decoding.It can be slower than autoregressive decoding in compute-bound regimes; drafter types include independent models [6] [29], non-parametric methods [20] [40] [52], learned auxiliary modules [4] [31], and self-SD from layer skipping [69], sparse attention [65], or quantization [55].Source: §2.2 · p. 3

2.3 Speculative Decoding for Rollout in Reinforcement Learning

  • History-based, learned auxiliary, and target-induced drafters are the main SD approaches for RL rollouts, with EfficientRollout using the current target model’s quantized weights.History-based methods reuse prior rollouts but can have short accepted drafts, need warm-up data, and may use acceptance-boosting heuristics [19] [35] [46]. Auxiliary drafters can require dedicated pretraining or warm-up and online adaptation [8] [24] [71]; EfficientRollout stays synchronized without separate online adaptation. It focuses on lossless on-policy acceleration, excluding methods that change rollout distribution or termination, [10], [70] [75].Source: §2.3 · pp. 3–4

3 Preliminary

Efficient rollout acceleration depends on when speculative verification can exploit underused compute, while long-tail autoregressive generations dominate RL rollout latency. The section defines speculative-decoding speed factors and introduces roofline modeling to identify decoding regimes.

3.1 Speedup Factors in Speculative Decoding

  • SD speedup depends on block efficiency and drafting-plus-verification latency; at large compute-bound batches, costly verification can make SD slower than AR decoding.Block efficiency τ is the average number of accepted draft tokens plus one target-sampled token per block, and under i.i.d. acceptance τ depends on per-token acceptance rate α.Source: §3.1 · p. 4

3.2 Roofline Modeling for LLM Decoding Latency

  • Roofline modeling estimates workload latency from compute work and data movement, distinguishing effective compute capacity from effective memory bandwidth.For LLM decoding, data movement includes weight and KV-cache traffic, while compute includes dense projections and attention operations.Source: §3.2 · p. 4

3.3 RL Rollout Latency Bottlenecks from Long-Tail Decoding

  • Sequential rollout decoding accounts for nearly 70% of total latency on average, with a few long responses determining completion time as the active batch shrinks.The resulting underutilized compute provides runtime budget for speculative decoding to exploit.Source: §3.3 · p. 4

4 EfficientRollout: From RL Rollout Dynamics to System-Aware Self-SD

EfficientRollout addresses three RL rollout dynamics with an evolving-policy-aligned self-drafter, a system-aware SD toggle, and draft lengths that adapt to changing acceptance. Its design targets rollout-tail latency while avoiding speculation when decoding conditions make it unhelpful.

4.1 Weight-Quantized Drafter for Self-Speculative Decoding

  • The 4-bit weight-quantized self-drafter stays synchronized with the evolving target and reduces drafting cost for dense projections, which account for around 90% of rollout-tail decoding latency.The drafter is induced from the current target without separate pretraining or online adaptation; lightweight RTN quantization is applied to FFN and QKVO layers at each training step.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 Regime-Aware SD Toggle Policy via Roofline Modeling

  • The roofline-based toggle enables SD only when predicted speedup exceeds a safety margin, avoiding compute-saturated early batches and activating in the beneficial low-batch tail.Once activated, SD remains enabled through the rollout as the active batch monotonically shrinks; model–hardware calibration parameters are profiled once and reused across RL runs.Source: §4.2 · pp. 5–6

4.3 Adaptive Draft-Length Policy

  • Quantized-drafter block efficiency rises as training sharpens the target policy, with target-output entropy and token-level agreement showing a strong negative correlation (r = −0.99).The policy adjusts draft length using measured block efficiency, increasing it near the current ceiling or decreasing it when acceptance cannot amortize drafting, with changes gated by P consecutive steps.Source: Figure 2 · p. 5, §4.3 · p. 6

5 Experiments

EfficientRollout achieves the largest latency reduction among the evaluated methods on every model while preserving training dynamics. Experiments attribute its gains to system-aware SD toggling and draft lengths adapted to changing acceptance during training.

5.1 Setup

  • Experiments used Qwen2.5-{7B,14B} on SimpleRL-8k-hard and Llama3.1-8B on SimpleRL-8k-medium, with GRPO training for 100 steps on eight A100-80GB GPUs.Comparisons included veRL autoregressive inference, history-based Spec-RL, learned auxiliary EAGLE3, and always-on quantized self-SD.Source: §5.1 · pp. 6–7

5.2 End-to-End RL Training Acceleration

  • Up to 19.6% lower rollout-generation latency and 12.7% lower end-to-end training-step latency were achieved by EfficientRollout, with the largest latency reduction among methods for every evaluated model.Per-step quantization overhead was 1.3–2.6 s relative to rollout-time savings.Source: Table 2 · p. 8, §5.2 · p. 8
  • History-based drafting did not reduce rollout-generation latency, while learned auxiliary drafting had model-dependent gains and increased Llama3.1-8B step latency by 25.6%.It reduced Qwen2.5-14B step latency by 3.3%; always-on quantized self-SD also slowed Qwen2.5-14B, showing that the quantized drafter alone did not ensure robust speedups.Source: §5.2 · pp. 8–9
  • EfficientRollout closely tracked the veRL (AR) reward trajectory on Qwen2.5-7B, supporting preserved training dynamics during acceleration.The paper attributes distribution preservation to lossless speculative decoding and reports full reward and validation-accuracy trajectories in Sec. F.2.Source: Figure 5 · p. 9, §5.2 · p. 9
  • EfficientRollout maintained 95.8–98.2% per-token acceptance while using larger average draft lengths; learned auxiliary drafting reached only 54.3–60.3%.Quantized self-SD also had high acceptance, consistent with alignment between the target and quantized self-drafter.Source: §5.2 · p. 9

5.3 Further Analysis of Rollout Acceleration

  • Disabling SD during only the early 6–11% of decoding steps yielded consistently larger rollout-generation latency reductions than always-on SD, despite slightly lower block efficiency.The roofline-based toggle boundary separated SD-beneficial from SD-harmful batch-size and sequence-length regimes.Source: Figure 4 · p. 8, Figure 3 · p. 8, §5.3 · p. 9
  • On DAPO-Math-17K, auxiliary drafters achieved block efficiency τ = 1.2–2.4 over the first 30 steps, compared with τ = 3.6–3.9 for the quantized self-drafter.The paper suggests effective learned auxiliary drafting may require in-distribution rollouts, target-generated synthetic data, and potentially more aggressive online adaptation.Source: Figure 6 · p. 9, §5.3 · p. 10
  • Adaptive draft lengths reduced rollout-generation latency by 19.6%, versus 13.5% with fixed γ = 5 and 11.8% with fixed γ = 11.The policy adjusts γ within its preset range using block-efficiency feedback, using smaller drafts more effectively early and larger ones later.Source: §5.3 · pp. 9–10, Figure 5 · p. 9

6 Discussion and Future Directions · Appendix

EfficientRollout combines target-induced quantized self-speculation, dynamic speculation toggling, and adaptive draft budgets to reduce RL rollout latency without invasive pipeline changes. The discussion identifies extensions for verification, quantization, and long-context attention, while the appendix provides extended rollout-tail analyses.

6 Discussion and Future Directions

  • EfficientRollout combines a target-induced quantized drafter, dynamic SD toggling, and adaptive draft budgets to reduce rollout and end-to-end latency.It requires no separate drafter pretraining, warm-up, online adaptation, or invasive RL pipeline changes.Source: §6 · p. 10
  • The framework is designed for evolving policies, shrinking-batch dynamics, and rollout-tail latency.These conditions motivate its system-aware self-speculative decoding design.Source: §6 · p. 10
  • Chain verification is the practical focus under dynamic batch sizes, while tree verification is an orthogonal extension [31] [32].The authors suggest incorporating tree verification into the framework.Source: §6 · p. 10
  • Naive RTN may not produce strong early-stage drafters for every model family, motivating less lossy quantization that avoids excessive per-step overhead.Source: §6 · p. 10
  • When KV-cache loading dominates in long-context or dense-attention regimes [11], sparse-attention drafting may further reduce quantized self-SD cost.Source: §6 · p. 10

Appendix

  • The appendix presents extended rollout-tail analyses for Llama3.1-8B-Instruct and Qwen2.5-7B, including completion curves and step-time decompositions.The caption specifies a first-step-of-second-epoch completion curve for each model and a first-20-training-step decomposition for Qwen2.5-7B.Source: Figure 7 · p. 18

A Extended Analysis of Shrinking-Batch Dynamics … B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting

Rollout’s long-tailed shrinking-batch behavior makes the remaining decode steps a key acceleration opportunity. In these regimes, weight quantization targets the dominant decode cost and proves more practical than sparse-attention drafting.

A Extended Analysis of Shrinking-Batch Dynamics

  • For Qwen2.5-7B, rollout accounts for 58.0–71.4% of measured step time across the first 20 steps, averaging 64.3%.Rollout remains the largest component despite Qwen2.5-7B having shorter total step time than Llama3.1-8B-Instruct.Source: A Extended Analysis of Shrinking-Batch… · p. 18
  • Llama’s final request takes about 123 s after 99% finish within 30 s; Qwen’s rollout makespan is about 88 s despite 99% finishing within 30 s.The final few long generations dominate rollout completion, especially as the active batch shrinks.Source: A Extended Analysis of Shrinking-Batch… · p. 18

B.1 Detailed Decode-Time Decomposition in Rollout-Tail Regimes

  • The decode-time breakdown uses a roofline simulation of compute and memory-transfer costs for representative rollout-tail regimes, following prior work [26].The simulation assumes a single A100-80GiB SXM GPU, FP16 inference, no tensor parallelism, and full attention.Source: §B.1 · p. 18, Table 3 · p. 19
  • Across evaluated models, FFN and QKVO projections dominate shrinking-batch decode cost, while attention contributes comparatively little, favoring quantization over sparse-attention drafting.Table 3 breaks down simulated single-token decode time for Qwen2.5-7B/14B [62] and Llama3.1-8B-Instruct [17] in rollout-tail regimes.Source: §B.1 · p. 18, Table 3 · p. 19
  • Attention may again become significant in much longer rollouts, such as beyond 64k tokens, where KV-cache quantization could become useful.KV-cache quantization is not considered in this work.Source: §B.1 · p. 18

B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting

  • At γ = 5, a 4-bit RTN drafter reaches block efficiency τ = 5.2, versus roughly 3–4 for sparse-attention drafting under speedup-cheap sparsity budgets.The quantized drafter also synergizes with vLLM through optimized W4A16 Marlin kernels.Source: §B.2 · p. 19
  • Fine-grained token-wise sparsity [53] [65] is difficult in vLLM’s page-granular memory system [28], while page-level or sliding-window patterns noticeably reduce acceptance.The sparse-attention self-SD implementation follows prior work [65].Source: §B.2 · pp. 18–19

B.3 W4 versus W8: Latency–Acceptance Trade-off · B.4 Choosing RTN for Step-Wise Drafter Refresh

The appendix favors W4 for its lower draft cost despite W8’s stronger block efficiency, and finds step-wise RTN refresh sufficient on Qwen2.5-7B as AWQ’s early advantage fades.

B.3 W4 versus W8: Latency–Acceptance Trade-off

  • W4 reduces the draft-to-target latency ratio to 0.360, compared with 0.573 for W8, despite W8 achieving larger block efficiency across draft lengths.The latency difference matters because speculation must make drafting cheap enough to amortize target verification overhead.Source: §B.3 · p. 19
  • W8’s drafter weights require roughly twice W4’s memory—about 16.3 GiB for Qwen2.5-14B—and its more costly draft path may not yield end-to-end speedups under realistic overheads.During RL rollouts, drafter weights compete for GPU capacity with persistent training weights, rollout weights, and the vLLM KV cache.Source: §B.3 · p. 19
  • As target-policy entropy decreases, first-token acceptance of the RTN W4 drafter increases across models.Figure 9 presents this relationship across models.Source: Figure 9 · p. 20

B.4 Choosing RTN for Step-Wise Drafter Refresh

  • On Qwen2.5-7B, W4-AWQ has a small block-efficiency advantage over W4-RTN during the first few training steps, but the gap quickly disappears.RTN drafter quality improves and saturates over the first 50 steps; Figure 8 compares block efficiency τ for the two drafters.Source: §B.4 · p. 20, Figure 8 · p. 20
  • Because EfficientRollout refreshes the drafter every training step, AWQ’s calibration-data collection and more expensive quantization procedure can impose significant overhead.Less lossy quantization may still help models whose RTN-quantized drafter has low initial acceptance; more advanced step-wise quantization is left for future work.Source: §B.4 · p. 20
  • The implementation does not use BNB 4-bit quantization because its execution path is less optimized for production-grade vLLM rollout inference than the AWQ-compatible W4A16 Marlin kernels.BNB-style 4-bit quantization can provide more calibrated low-bit weights.Source: §B.4 · p. 20

C Policy Sharpening Improves Quantized Drafter Alignment … D.2 Roofline-Based SD Toggle Model and Calibration

RL training sharpens the target policy and improves quantized-drafter alignment. EfficientRollout’s shared-KV design and calibrated roofline model describe the system costs and execution conditions for self-speculative decoding.

C Policy Sharpening Improves Quantized Drafter Alignment

  • As entropy fell, first-token acceptance and block efficiency τ rose across three models, with Pearson correlations of −0.96 to −0.99 and −0.92 to −0.98, respectively.The results suggest that policy sharpening makes small quantization-induced perturbations less likely to change accepted draft tokens.Source: Figure 10 · p. 21, C Policy Sharpening Improves Quantized… · p. 21
  • Under fixed-length RTN W4 self-drafting, first-token acceptance measures local drafter–target alignment, while block efficiency τ measures target-distributed tokens per SD iteration.The analysis uses a quantized copy of the current target model at each training step, without regime-aware toggling or adaptive draft-length control.Source: C Policy Sharpening Improves Quantized… · pp. 20–21

D.1 KV-Cache Sharing between the Quantized Drafter and Target

  • The W4 drafter writes provisional KV states into the target’s shared cache; full-precision verification overwrites verified entries, and rejected-position and later states are discarded.Rejection sampling preserves the target-model sampling distribution, and each new iteration begins with a prefix whose KV states were generated by the target.Source: §D.1 · pp. 21–22
  • Shared KV storage avoids a separate drafter-cache memory cost, while quantizing the current target at each training step keeps the drafter synchronized with the evolving policy.The approach requires neither auxiliary drafter training nor online adaptation.Source: §D.1 · p. 22

D.2 Roofline-Based SD Toggle Model and Calibration

  • The W4 drafter is about 33–36% of original model size; C_dense and C_attn are architecture-derived, and effective KV traffic is κ_eff = ρκ κ_theoretical.κ_theoretical is determined by layer count, KV-head count, head dimension, and FP16 key/value storage; ρκ is typically below 1.Source: §D.2 · p. 22
  • The model’s η_D term captures W4A16 dequantization and kernel overhead, while residual target, drafter, and verification overheads are approximated as linear in batch size.These residual overheads were empirically dominated by batch-dependent effects.Source: §D.2 · p. 22
  • The current roofline formulation covers TP=1; extending it to TP>1 would require explicit communication-overhead terms such as all-reduce cost.The TP>1 extension is left for future work.Source: §D.2 · p. 22
  • Calibration measures effective compute throughput, sweeps target, verification, and W4-drafter latency, then fits shared and drafter/verification parameters in two stages.Fitted parameters are independent of draft length γ, allowing generalization to other draft lengths without recalibration; fitted effective bandwidth is typically 1560–1620 GB/s.Source: §D.2 · p. 22

D.3 Validation of the Roofline-Based SD Toggle Boundary · E Detailed Experimental Setup

The calibrated roofline model closely tracks the empirical SD-benefit boundary, supporting a conservative monotone toggle that enables SD only after predicted speedup clears a safety margin.

D.3 Validation of the Roofline-Based SD Toggle Boundary

  • The roofline-model validation measures target, quantized-drafter, and verification times across active batch size, sequence length, and draft length on one A100 GPU with TP=1.Empirical SD speedup is computed from these measured component timings.Source: §D.3 · p. 23
  • Empirical SD speedup is calculated as τTp(B,S) divided by γTq(B,S)+TV(B,S,γ).Source: §D.3 · p. 23
  • For validation, the empirical speedup calculation assumes τ=γ, corresponding to full draft acceptance, to isolate cost-model accuracy from block-efficiency variation.Source: §D.3 · p. 23
  • The predicted toggle boundary closely tracks the empirical beneficial/harmful frontier across models and draft lengths, with SD kept off unless predicted speedup clears a 5% margin.Several empirically beneficial points lie just above the boundary, reflecting the policy’s intentional conservatism.Source: §D.3 · p. 23
  • Because rollout batches shrink as surviving sequences grow, the decoding trajectory crosses the SD-beneficial region once, so SD is activated above the 1+ϵ threshold and remains enabled.The policy’s monotone behavior is motivated by batch size dominating boundary movement in the targeted regimes.Source: §D.3 · p. 23

E.1 Infrastructure, Datasets, and Reporting Window … F.4 Why History-Based Drafting Does Not Accelerate

Across the evaluated models, EfficientRollout reduced rollout time while tracking the autoregressive baseline’s training and validation quality. The merged sections also document the evaluation setup, baseline implementations, adaptive draft lengths, and why history-based drafting failed to accelerate rollouts.

E.1 Infrastructure, Datasets, and Reporting Window

  • Experiments used veRL 0.7.0 and vLLM 0.11.2 on one 8×A100 node, with SimpleRL math datasets selected by model family.Timing averages exclude the first training step and cover the following 100 steps; Qwen models used SimpleRL-8k-hard, while Llama3.1-8B-Instruct used SimpleRL-8k-medium.Source: §E.1 · p. 24

E.3 Training and Method Configuration

  • Training used 128 prompts per batch, eight rollouts per prompt, a maximum rollout length of 8,192 tokens, and temperature 1.0.The adaptive policy used Γ = {5, 7, 9, 11}, αup = 0.94, αdown = 0.85, and P = 2; γ = 3 was omitted because it was dominated by or comparable to γ = 5 in encountered regimes.Source: §E.3 · p. 24

E.4 Rollout-History-Based Drafting Baseline

  • Spec-RL reuses matching prefixes from earlier rollouts, but is a lossy baseline rather than strictly target-distribution-preserving speculative decoding.It lacks full vocabulary logits needed for exact rejection sampling after a rejected token and uses a lenience factor of e0.5 to increase prefix acceptance.Source: §E.4 · p. 24

F.1 Per-Step Rollout Generation Time

  • EfficientRollout had the lowest rollout-generation time throughout training on Qwen2.5-7B and Llama3.1-8B, and for most Qwen2.5-14B steps.Quantized self-SD could be slower when speculation stayed enabled, especially on Qwen2.5-14B; learned auxiliary drafting achieved reasonable speedup there. A small late-stage Qwen2.5-14B slowdown coincided with longer sampled responses, which the authors attribute to response-length stochasticity.Source: Figure 13 · p. 25, §F.1 · pp. 25–26

E.5 Learned Auxiliary Drafting Baseline; E.2 Metric Measurement Details

  • The learned auxiliary baseline used EAGLE3-style drafting in veRL+vLLM, with online drafter training and fixed γ = 3.The authors trained Qwen-compatible drafters on ShareGPT for 10 epochs, used a pretrained RedHatAI drafter for Llama, and updated the drafter inline after actor micro-batches. Preparation time includes direct drafter-update time; measurement conventions also account for method-specific setup overhead and average τ and γ over training steps.Source: §E.2 · p. 24, §E.5 · p. 25

F.2 Training Dynamics and Quality Preservation

  • Across Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B-Instruct, EfficientRollout closely followed veRL (AR) in training reward and validation accuracy.The authors report that the trajectories support preservation of training dynamics and downstream quality in the evaluated settings.Source: Figure 14 · p. 26, Figure 15 · p. 26, §F.2 · p. 26

F.3 Adaptive Draft-Length Schedules Across Models

  • The adaptive controller raised Qwen2.5-7B’s γ from 5 to 11 and Qwen2.5-14B’s from 5 to 9, while Llama3.1-8B-Instruct stayed at γ = 5.Increases followed block efficiency approaching the current draft-length ceiling; no downward adjustment was triggered in these runs.Source: §F.3 · p. 26, Figure 16 · p. 27

F.4 Why History-Based Drafting Does Not Accelerate

  • History-based drafting increased rollout-generation time by 19.4% on Qwen2.5-7B and 8.8% on Llama3.1-8B-Instruct, even in its active window.Spec-RL reused 30 tokens on average from a 660-token Qwen2.5-7B response (4.5% reuse); for Llama3.1-8B-Instruct, verification added 15.9 s per active step. The analysis covers the second epoch onward, when history is available.Source: Table 5 · p. 27, §F.4 · p. 27

F.5 Why Learned Auxiliary Drafting Remains Challenging

Learned auxiliary drafting remains difficult to make reliable for RL rollouts: pretrained drafters can have weak proposal quality, especially in long, high-temperature generations, and online adaptation does not always fix it.

F.5 Why Learned Auxiliary Drafting Remains Challenging

  • A randomly initialized drafter has block efficiency close to 1.0, while a ShareGPT-pretrained drafter starts at 1.4–2.0 on SimpleRL-math rollouts.In-distribution RL-drafter pretraining could improve the initial efficiency, but is not always available; the study therefore uses a general chat/text-pretrained drafter [32].Source: §F.5 · p. 27, Figure 17 · p. 28
  • For Qwen2.5 models, learned drafting gives limited acceleration because proposal quality is insufficient for high-temperature sampling (T = 1.0) and long reasoning generations.Activating SD during early large-batch phases can reduce or eliminate its benefit, making system-aware activation important.Source: §F.5 · pp. 27–28
  • Tq is around 0.8–0.9 ms across models and Llama3.1-8B-Instruct has TV /Tp = 1.45, only modestly above Qwen, so timing overhead alone does not explain its slowdown.Table 6 decomposes target, drafter, and verification timing.Source: Table 6 · p. 28, §F.5 · p. 28
  • In-distribution initialization or more aggressive online adaptation, such as multiple drafter-training steps per RL iteration or asynchronous adaptation, may improve block efficiency but adds training, scheduling, and configuration-search burden.The workload-matching requirement is analyzed further in Sec. F.6.Source: §F.5 · p. 28
  • EAGLE3 speedups reported for standard serving across Qwen and Llama models [32] are not directly comparable to rollouts with vLLM, T = 1.0 sampling, hard reasoning prompts, and long response budgets.The Llama3.1-8B-Instruct diagnosis used cold-start profiling over the first five RL steps under the rollout setting in Sec. 5.1.Source: §F.5 · p. 28
  • After five online-adaptation steps, Llama3.1-8B-Instruct remains near 1.2–1.3 block efficiency in 1k–8k-token bins, while Qwen models recover, leaving long-tail responses costly.With most Llama proposals rejected at deep positions, drafting and verification overhead can increase rollout latency.Source: §F.5 · p. 28

F.6 Auxiliary Drafters Aligned with RL Rollout Distributions Are Difficult to Obtain

Learned auxiliary drafters generally trail target-induced quantized self-drafters in block efficiency, and only selected configurations translate into rollout speedups. The results suggest that high efficiency requires alignment with long, high-temperature RL rollouts.

Depth-dependent block efficiency

  • Llama3.1-8B-Instruct block efficiency drops sharply after 512 generated tokens, while Qwen models degrade more gradually before adaptation and maintain or recover efficiency at deeper positions over the first five RL steps.Source: Table 7 · p. 29, Table 8 · p. 29

Block efficiency gap

  • Across 30 RL steps, quantized self-drafters sustain τ = 3.6–3.9 versus EAGLE3’s mostly τ = 1.2–2.4, while FastRL reaches τ = 2.2–2.6 but remains insufficient for consistent speedup.The best public EAGLE3 configuration, Qwen3-8B with the RedHatAI thinking drafter, reaches roughly τ = 2.2–2.4; online adaptation yields only modest improvement within the 30-step window.Source: §F.6 · p. 29, Figure 18 · p. 30

Generation-time consequence

  • Qwen3-8B with the RedHatAI thinking drafter reduces rollout-generation time by about 7.7%, while the FastRL Qwen2.5-7B drafter reduces it by 17.6% over steps 22–30.The other evaluated configurations are slower than the corresponding NeMo RL No-SD baseline for much of training, as low block efficiency fails to amortize drafting, verification, and online-update overhead.Source: §F.6 · p. 29, Figure 19 · p. 30

Output length and training distribution

  • Shorter output caps do not generally resolve weak block efficiency; training distribution appears more central, with generic chat/text data less aligned than synthetic reasoning or target-generated RL rollouts.The proposed specialized pipeline collects target-generated long rollouts and trains the auxiliary drafter on that workload data.Source: §F.6 · p. 30

Takeaway

  • A sufficiently aligned auxiliary drafter is not generally available from public checkpoints or generic recipes, and may require separate rollout collection, offline training, and more aggressive online adaptation.Target-induced self-drafting avoids external checkpoints and offline drafter pretraining while delivering high block efficiency from the beginning of RL training and practical rollout speedup.Source: §F.6 · p. 30

Related papers

Loading 2606.18967v1…