PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim, Coleman Hooper, Harman Singh, Amir Gholami, Hyung Il Koo, Wonjun Kang

arXiv:2606.18967v1cs.LG

TL;DR

RL rollout generation is a major latency bottleneck, and existing speculative decoding does not directly accommodate evolving policies and shrinking batch sizes. EfficientRollout uses target-induced self-speculation with system-aware toggling and adaptive draft lengths, reducing rollout latency by up to 19.6% and end-to-end latency by up to 12.7% over an accelerated AR baseline.

  • Problem

    Existing speculative decoding methods do not directly address RL rollouts because evolving policies stale fixed drafters and shrinking batches can make SD slower during compute-bound phases, while rollout latency dominates RL post-training.

  • Method

    EfficientRollout induces a weight-quantized drafter from the current target policy and combines system-aware SD toggling with adaptive draft-length control.

  • Results

    19.6% rollout and 12.7% end-to-end latency reductions are achieved over a standard accelerated AR rollout baseline.

  • Takeaways & Limitations

    The reported latency reductions require no separate drafter pretraining, warm-up, online adaptation, or invasive RL pipeline changes.

  • Takeaways & Limitations

    The framework focuses on chain verification, while naive RTN may not provide strong early-stage drafters for all model families.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) offers a natural way to address this bottleneck, as it is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.

1 Introduction

RL rollout generation is a dominant latency bottleneck, while evolving policies and shrinking active batches make fixed-model speculative decoding difficult to apply. EfficientRollout addresses these challenges with self-speculation, system-aware toggling, adaptive drafting, and reported latency reductions.

  • Motivation: RL post-training improves reasoning, math, and agentic capabilities, but rollout generation has become the RL pipeline’s dominant latency bottleneck.On-policy RL collects rollouts from the policy being optimized, improving training stability and model quality.
  • EfficientRollout: EfficientRollout coordinates target-induced self-drafting, dynamic SD toggling, and adaptive drafting budgets for system-aware RL rollout decoding.Its adaptive draft length uses observed block efficiency τ, while toggling responds to tail-heavy rollout dynamics.
  • Challenges: Evolving target policies make static drafters stale, while shrinking active batches shift decoding from compute-bound to regimes where speculative decoding can become beneficial.Large active batches can make speculative decoding slower than autoregressive decoding before shorter responses finish and the batch shrinks.
  • EfficientRollout: EfficientRollout induces a weight-quantized drafter from the current target policy at every training step, avoiding separate drafter pretraining while maintaining policy coupling.The framework is designed for on-policy RL rollouts and targets high block efficiency under an evolving policy.
  • Results: 19.6% and 12.7%: EfficientRollout reduces rollout and end-to-end latency, respectively, over a standard accelerated AR rollout baseline.The reported improvements preserve final model quality.

2 Related Work

Related work establishes RL, especially RLVR, as a major post-training paradigm for reasoning and agentic capabilities, while speculative decoding accelerates generation through draft-and-verify execution. Prior RL-specific approaches include history-based drafting and target-induced self-speculation; EfficientRollout uses a quantized target-model copy to remain synchronized with the evolving policy.

  • RL post-training: RL post-training has produced state-of-the-art reasoning and agentic capabilities, with RLVR prominent in math and coding tasks using rule-based verification.RLVR can obtain supervision from rule-based checkers or unit tests instead of learned reward models.
  • Speculative decoding: Speculative decoding drafts tokens sequentially and verifies them in parallel, preserving the target-model distribution through rejection sampling.Its benefits come from exploiting underutilized compute during memory-bound decoding.
  • Speculative decoding: In compute-bound regimes, speculative decoding can be slower than standard autoregressive decoding because parallel verification has little underutilized compute to exploit.This motivates system-aware use of speculation rather than unconditional activation.
  • RL-specific speculative decoding: History-based RL rollout drafting reuses prior-epoch rollouts as non-parametric proposals, whereas target-induced methods derive the drafter solely from the current target model.History-based methods are easy to deploy but tracking an evolving policy requires online adaptation, adding system complexity and management overhead.
  • RL-specific speculative decoding: EfficientRollout instantiates target-induced self-speculation with a weight-quantized target-model copy, remaining synchronized with the evolving policy without separate online adaptation.The method targets lossless on-policy RL acceleration while maintaining high block efficiency.

3 Preliminary

This section formalizes speculative decoding efficiency through accepted tokens and hardware-dependent latency ratios. It motivates system-aware regime detection by showing that rollout decoding dominates RL post-training latency and that speculation can underperform when verification is compute-bound.

  • Speculative-decoding efficiency: Speculative-decoding speedup depends on block efficiency τ and the latency ratios Tq/Tp and TV/Tp induced by batch size B and sequence length S.The drafting time is TSD(B, S, γ) = γTq(B, S) + TV(B, S, γ), while speedup scales with τ Tp(B, S) / TSD(B, S, γ).
  • Speculative-decoding efficiency: When decoding is compute-bound, often at large B, verification has little underutilized compute and can make speculative decoding slower than autoregressive decoding.This effect can be hidden by common B = 1 speculative-decoding benchmarks.
  • Hardware regime analysis: Roofline modeling identifies the hardware decoding regime by relating arithmetic work CFLOPs and data movement MBytes to effective compute capacity Feff and memory bandwidth BWeff.For LLM decoding, MBytes includes weight and KV-cache traffic, while CFLOPs includes QKVO, FFN, LM-head, and attention operations.
  • RL rollout bottleneck: Nearly 70% of total latency is attributed to sequential rollout decoding in RL post-training, making rollout generation the primary latency bottleneck.Long reasoning trajectories amplify this bottleneck because responses must be sampled token by token under autoregressive decoding.

4 EfficientRollout: From RL Rollout Dynamics to System-Aware Self-SD

EfficientRollout derives a self-speculative decoding pipeline from RL rollout dynamics: an evolving policy, rollout-tail latency, changing runtime regimes, and evolving drafter acceptance. It combines a weight-quantized drafter, a calibrated SD toggle policy, and adaptive draft lengths.

  • Self-speculative decoding: EfficientRollout uses a target-induced self-SD drafter that stays synchronized with the evolving policy without separate pretraining or online adaptation.The drafter is derived directly from the current target model, unlike auxiliary drafters that can become stale.
  • Rollout-tail latency decomposition: Around 90% of rollout-tail single-token latency comes from dense projections, exceeding attention time by more than 10×.Dense projections include QKVO, FFN, and LM-head projections, motivating a drafter that reduces their weight-loading cost.
  • Weight-quantized drafting: 4-bit weights (W4) are applied to FFN and QKVO layers with lightweight RTN quantization to reduce dense-projection weight-loading cost.Quantization is applied at the beginning of each training step as a practical latency–acceptance trade-off.
  • SD toggle policy: The SD toggle policy enables speculation only when predicted SD speedup exceeds a safety margin, targeting low-batch rollout-tail regimes.Early large-batch phases can make SD slower than AR decoding, while shrinking active batches make SD beneficial; once enabled, SD remains active through rollout end.
  • Adaptive draft length: r = −0.99 measures the strong negative correlation between target-output entropy and token-level agreement with the weight-quantized drafter.EfficientRollout adapts the next draft length using measured block efficiency τ, increasing or decreasing γ only after the relevant condition persists for P training steps.

5 Experiments

Experiments evaluate EfficientRollout across Qwen2.5 and Llama3.1 models in GRPO-based RL rollouts, comparing it with accelerated autoregressive and speculative-decoding baselines. EfficientRollout achieves substantial latency reductions while maintaining high acceptance, preserving training dynamics, and benefiting from regime-aware toggling and adaptive draft lengths.

  • Experimental setup: Experiments cover Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B under GRPO training with 8k maximum responses on eight A100-80GB GPUs.The Qwen2.5 models use SimpleRL-8k-hard, while Llama3.1-8B uses SimpleRL-8k-medium; training runs for 100 steps with rollout temperature 1.0.
  • Latency results: EfficientRollout reduces rollout-generation latency by up to 19.6% and end-to-end training-step latency by up to 12.7% versus veRL (AR).Per-step quantization adds 1.3–2.6 s, but remains modest relative to rollout-time savings; EfficientRollout has the largest latency reduction for every evaluated model.
  • Speculation quality: EfficientRollout attains 95.8–98.2% per-token acceptance while sustaining large block efficiency and increasing average draft length as training improves.These results indicate strong alignment between the evolving target model and its quantized self-drafter, while adaptive draft-length control exploits improved acceptance during training.
  • Training dynamics and quality preservation: EfficientRollout closely tracks the veRL (AR) reward trajectory, consistent with SD preserving the target distribution and training dynamics.This quality preservation is especially important because rollout-distribution shifts can affect RL training stability.
  • Regime-aware SD toggle policy: Regime-aware toggling improves rollout latency over always-on SD by disabling speculation during only the early 6–11% of decoding steps.The gain comes from avoiding verification in the initial large-batch regime, even though toggling produces slightly lower block efficiency.
  • Adaptive draft-length policy: Adaptive draft-length control achieves a 19.6% rollout-generation reduction by selecting smaller and larger draft lengths at different training stages.The controller raises draft length as block efficiency improves, avoiding overly large drafts early and exploiting longer drafts later.

6 Discussion and Future Directions … F Additional Evaluation Results

EfficientRollout is presented as a system-aware self-speculative decoding framework for RL rollouts, combining a target-induced quantized drafter with dynamic toggling and adaptive draft budgets. The paper also identifies future extensions and provides appendices covering system rationale, implementation, experimental setup, and additional evaluations.

  • 6 Discussion and Future Directions: EfficientRollout combines a target-induced quantized drafter, dynamic SD toggling, and adaptive draft budgets without separate drafter pretraining, warm-up, online adaptation, or invasive RL pipeline changes.The framework is designed around evolving policies, shrinking-batch dynamics, and rollout-tail latency.
  • 6 Discussion and Future Directions: Future work includes incorporating tree verification and developing less lossy quantization methods when naive RTN produces weak early-stage drafters.The paper focuses on chain verification because it is practical under dynamic batch sizes.
  • Appendix; B System Rationale for Weight-Quantized Self-Drafting in RL Rollouts: Appendix A analyzes shrinking-batch dynamics, while Appendix B examines decode-time decomposition, quantized versus sparse-attention drafting, W4 versus W8, and RTN-based drafter refresh.These topics are organized under the system rationale for weight-quantized self-drafting in RL rollouts.
  • C Policy Sharpening Improves Quantized Drafter Alignment; D Implementation and Calibration Details of EfficientRollout: Appendix C covers policy sharpening and quantized drafter alignment, while Appendix D details KV-cache sharing, roofline-based SD toggling, calibration, and boundary validation.The listed implementation topics concern EfficientRollout’s calibration and deployment details.
  • E Detailed Experimental Setup: Appendix E documents infrastructure, datasets, reporting windows, metric measurement, training configuration, rollout-history-based drafting, and learned auxiliary drafting baselines.These items comprise the paper’s detailed experimental setup.
  • F Additional Evaluation Results: Appendix F reports per-step rollout generation time, training dynamics and quality preservation, adaptive draft-length schedules, and analyses of history-based and learned auxiliary drafting challenges.It also examines why aligned auxiliary drafters are difficult to obtain for RL rollout distributions.
  • F Additional Evaluation Results: Figure 7 extends rollout-tail analysis with cumulative request-completion curves for Llama3.1-8B-Instruct and Qwen2.5-7B, plus Qwen2.5-7B step-time decomposition over its first 20 training steps.The Llama3.1 curve is shown at the first step of the second epoch, and the Qwen2.5-7B curve at the first step of the second epoch.

A Extended Analysis of Shrinking-Batch Dynamics … B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting

RL rollout time is dominated by shrinking-batch dynamics and long-tailed completions, while decode costs are concentrated in linear projections rather than attention. These findings motivate quantization-based self-speculative drafting over sparse-attention alternatives, whose implementation and acceptance trade-offs limit practical speedups.

  • A Extended Analysis of Shrinking-Batch Dynamics: Rollout accounts for 58.0–71.4% of Qwen2.5-7B’s four-phase step time across the first 20 training steps, averaging 64.3%.Rollout remains the largest step-time component despite Qwen2.5-7B having shorter total steps than Llama3.1-8B-Instruct.
  • A Extended Analysis of Shrinking-Batch Dynamics: About 50% of Llama3.1-8B-Instruct requests finish within 10 s, 90% within 16 s, and 99% within 30 s, while the final request takes about 123 s.A small number of long generations determine the rollout makespan; Qwen2.5-7B shows the same mechanism with a milder tail.
  • B System Rationale for Weight-Quantized Self-Drafting in RL Rollouts: Shrinking-batch rollout regimes make FFN and QKVO projections dominate single-token decode cost, whereas attention contributes only a relatively small fraction.The decomposition estimates each component latency as the maximum of compute time and memory-transfer time across representative rollout-tail regimes.
  • B.1 Detailed Decode-Time Decomposition in Rollout-Tail Regimes: Quantization-based self-SD directly reduces the dominant linear decode component, unlike sparse-attention drafting, which mainly targets the smaller attention component.The comparison supports the design choice of quantized self-SD across the evaluated models.
  • B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting: Sparse-attention self-SD typically requires token-wise sparsity updated dynamically at each draft step to achieve high acceptance.This fine-grained sparsity interacts poorly with serving-engine memory management.
  • B.1 Detailed Decode-Time Decomposition in Rollout-Tail Regimes: Table 3 evaluates single-token decode-time breakdowns for Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B-Instruct in RL rollout-tail regimes.The simulation assumes one A100-80GiB SXM GPU, FP16 inference, no tensor parallelism, and full attention; quantization targets weight-loading costs.
  • B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting: Table 4 reports W4–W8 latency–acceptance trade-offs using roofline predictions and measured block efficiency τ for γ ∈{3, 5, 7}.Measurements use Qwen2.5-7B-Instruct at batch size 1 and sequence length 2k, under memory-bound, zero-overhead assumptions.
  • B.2 Practical Trade-offs between Quantized and Sparse-Attention Self-Drafting: vLLM’s page-granular memory management makes token-wise sparse attention difficult to realize efficiently, while page-level or sliding-window patterns noticeably reduce acceptance.A sparse-attention self-SD implementation following prior work was evaluated under sparsity budgets intended to be cheap enough to yield speedup.

B.3 W4 versus W8: Latency–Acceptance Trade-off · B.4 Choosing RTN for Step-Wise Drafter Refresh

EfficientRollout favors W4-RTN over W8 because its substantially cheaper draft path and lower memory footprint better suit rollout latency, despite W8’s higher drafting quality. RTN is selected for step-wise refresh because AWQ offers only a small early advantage while adding repeated quantization overhead, and BNB is less optimized for the deployment path.

  • B.3 W4 versus W8: Latency–Acceptance Trade-off: W8 provides higher drafting quality and larger block efficiency across draft lengths than W4.W4 nevertheless remains reasonably accurate, making the comparison a latency–acceptance trade-off.
  • B.3 W4 versus W8: Latency–Acceptance Trade-off: 0.360 is W4’s draft-to-target latency ratio in the idealized memory-bound model, versus 0.573 for W8.The passage states that SD speedup depends on both accepted length and drafter cost.
  • B.3 W4 versus W8: Latency–Acceptance Trade-off: 16.3 GiB is the approximate memory required by the Qwen2.5-14B W8 drafter, roughly twice W4’s footprint.This adds pressure alongside persistent training weights, rollout weights, and vLLM KV-cache memory.
  • B.3 W4 versus W8: Latency–Acceptance Trade-off: As target-policy entropy decreases, first-token acceptance of the RTN W4 drafter increases across models.Figure 9 identifies policy sharpening as improving quantized-drafter alignment.
  • B.4 Choosing RTN for Step-Wise Drafter Refresh: W4-AWQ provides a small block-efficiency advantage over W4-RTN during the first few training steps on Qwen2.5-7B.Figure 8 compares block efficiency τ for the two W4 drafters over the first 50 training steps.
  • B.4 Choosing RTN for Step-Wise Drafter Refresh: AWQ requires calibration-data collection and a more expensive quantization procedure before each drafter refresh, unlike RTN.Because EfficientRollout refreshes the drafter every training step, this overhead can become significant without heavy optimization.
  • B.4 Choosing RTN for Step-Wise Drafter Refresh: EfficientRollout does not use BNB 4-bit quantization because its execution path is less optimized for production-grade vLLM rollout inference.The implementation instead uses AWQ-compatible W4A16 Marlin kernels.

C Policy Sharpening Improves Quantized Drafter Alignment … D.2 Roofline-Based SD Toggle Model and Calibration

RL training sharpens the target policy and improves alignment between the full-precision model and its quantized self-drafter. EfficientRollout combines shared-KV execution with a calibrated roofline-based SD toggle model whose parameters capture quantization, cache-traffic, and batch-dependent overheads.

  • C Policy Sharpening Improves Quantized Drafter Alignment: As policy entropy decreases, the RTN W4 drafter’s block efficiency τ increases across models.Figure 10 reports this relationship directly.
  • C Policy Sharpening Improves Quantized Drafter Alignment: RL training reduces policy entropy while increasing first-token acceptance across Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B-Instruct.The reported Pearson correlations range from −0.96 to −0.99 for entropy and first-token acceptance, and from −0.92 to −0.98 for entropy and block efficiency.
  • D.1 KV-Cache Sharing between the Quantized Drafter and Target: The quantized drafter shares the target model’s KV-cache storage, writes provisional drafted states, and lets parallel target verification replace them with clean states.Rejection sampling preserves the target-model sampling distribution, while rejected suffix states are discarded before the next iteration.
  • D.1 KV-Cache Sharing between the Quantized Drafter and Target: Reconstructing the W4 drafter from the current target model avoids a separate drafter KV cache and keeps it synchronized with the evolving RL policy.The design avoids auxiliary drafter training or online adaptation and reduces memory overhead for long maximum response lengths.
  • D.2 Roofline-Based SD Toggle Model and Calibration: The roofline model accounts for W4 drafter size, dense compute, effective KV-cache traffic, quantization overhead, and residual batch-dependent costs.Because only FFN and QKVO projection layers are quantized, WD/WT is about 33–36% rather than exactly 25%; ηD captures dequantization and kernel overhead.
  • D.2 Roofline-Based SD Toggle Model and Calibration: The current roofline formulation targets TP= 1, while TP> 1 would require explicit communication overhead such as an all-reduce cost ccomm.The TP> 1 extension is left to future work.
  • D.2 Roofline-Based SD Toggle Model and Calibration: Calibration measures hardware-specific compute throughput, sweeps decoding workloads across batch sizes, sequence lengths, and γ ∈{3, 7, 11, 15}, then fits model parameters.BWeff is fitted separately for each model because comparable standalone memory-bandwidth benchmarks are difficult; fitted values typically range from 1560–1620 GB/s.
  • D.2 Roofline-Based SD Toggle Model and Calibration: Figure 12 validates the roofline-based SD toggle policy by comparing predicted speedup against empirical SD-beneficial and SD-harmful points.The blue line marks SpeedupSD = 1 + ϵ.

D.3 Validation of the Roofline-Based SD Toggle Boundary · E Detailed Experimental Setup

EfficientRollout’s roofline-based SD toggle boundary is validated against measured W4 self-SD timings across batch sizes, sequence lengths, and draft lengths. The validated boundary supports a monotone policy that follows RL rollout states from dense batches toward shrinking-batch tails.

  • D.3 Validation of the Roofline-Based SD Toggle Boundary: The validation sweeps active batch size B, sequence length S, and draft length γ while measuring target decoding, quantized-drafter decoding, and target verification times on a single A100 GPU.Measurements use TP=1 and provide the empirical components for evaluating SD speedup.
  • D.3 Validation of the Roofline-Based SD Toggle Boundary: Empirical SD speedup is computed as Speedupemp(B, S, γ) = τTp(B, S) / (γTq(B, S) + TV(B, S, γ)).The notation follows Eq. (1).
  • D.3 Validation of the Roofline-Based SD Toggle Boundary: The validation uses the optimistic setting τ = γ, corresponding to full draft acceptance, to isolate cost-model accuracy from block-efficiency variation.This assumption removes acceptance variation from the boundary validation.
  • D.3 Validation of the Roofline-Based SD Toggle Boundary: The calibrated roofline model’s predicted boundary closely tracks the empirical beneficial/harmful frontier across models and draft lengths.Figure 12 marks the toggle boundary at SpeedupSD = 1 + ϵ with ϵ = 0.05; green circles indicate beneficial SD and red crosses harmful SD.
  • D.3 Validation of the Roofline-Based SD Toggle Boundary: During RL rollout, active batch size monotonically decreases as requests finish, while surviving sequences become longer, moving states from dense batches toward shrinking-batch tails.This creates a structured trajectory in the (S, B) plane.
  • D.3 Validation of the Roofline-Based SD Toggle Boundary: Because batch size dominates boundary movement, the rollout trajectory naturally crosses the SD-beneficial region once in the targeted regimes, motivating a monotone toggle policy.The observed state trajectory provides the rationale for activating speculation monotonically.

E.1 Infrastructure, Datasets, and Reporting Window … F.4 Why History-Based Drafting Does Not Accelerate

The paper evaluates EfficientRollout in a veRL+vLLM RL-training setup, comparing self-SD with history-based and learned auxiliary drafting. Results show faster rollouts without material quality loss, while adaptive drafting improves utilization and history reuse fails to amortize verification overhead.

  • E.1 Infrastructure, Datasets, and Reporting Window: Experiments use veRL v0.7.0, vLLM v0.11.2, Ray, FSDP, and 8 NVIDIA A100-SXM4-80GB GPUs with SimpleRL math datasets.SimpleRL-8k-hard is used for Qwen models, while the easier SimpleRL-8k-medium is used for Llama3.1.
  • E.2 Metric Measurement Details: Rollout, preparation, and step times are averaged across training steps, with EfficientRollout reporting training-step averages for τ and ¯γ and deriving α from measured τ.When γ varies, α is inverted within fixed-γ trajectory portions before averaging.
  • E.3 Training and Method Configuration: Training uses batch size 128, eight rollouts per prompt, an 8,192-token maximum, learning rate 5 × 10−7, and sampling temperature 1.0.The setup is purely on-policy because each mini-batch contains samples from the current policy.
  • E.4 Rollout-History-Based Drafting Baseline: Spec-RL reuses matching prefixes from previous-epoch rollouts, but its token-only history makes exact target-distribution-preserving rejection sampling difficult.The method is identified as a representative publicly available rollout-history-based baseline in the veRL+vLLM stack.
  • E.5 Learned Auxiliary Drafting Baseline: The learned auxiliary drafting comparison is implemented in the same veRL+vLLM stack, while compatible EAGLE3 drafters must exist or be pretrained for each target family.Online adaptation captures target hidden states during actor forward passes and trains the drafter after each micro-batch; decoding uses fixed γ = 3.
  • F.1 Per-Step Rollout Generation Time: EfficientRollout achieves the lowest rollout-generation time throughout training on Qwen2.5-7B and Llama3.1-8B, and for most Qwen2.5-14B steps.Quantized self-SD can become slower when enabled throughout rollout, particularly for Qwen2.5-14B because its larger drafter raises drafting cost.
  • F.2 Training Dynamics and Quality Preservation: EfficientRollout closely follows veRL (AR) in training reward and validation accuracy across Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B-Instruct.The reported trajectories support preserved training dynamics and downstream model quality in the evaluated settings.

F.5 Why Learned Auxiliary Drafting Remains Challenging

Learned auxiliary drafting remains difficult for RL rollouts because pretrained drafters have insufficient block efficiency under high-temperature, long reasoning generations, while improving them requires costly adaptation. On Llama3.1-8B-Instruct, slowdown is driven more by weak depth-dependent proposal quality than by drafter or verification timing.

  • Block efficiency: Pretraining and online adaptation are typically required to align learned auxiliary drafters with the target model and rollout distribution.A randomly initialized drafter has block efficiency close to 1.0, while general pretraining improves early behavior but leaves initial efficiency limited.
  • Qwen2.5 acceleration: Under T = 1.0 and long reasoning generation, pretrained drafters provide insufficient proposal quality for the Qwen2.5 math RL rollout distribution.Limited block efficiency restricts generation-time acceleration, and system-aware activation remains important because it can determine whether speculation benefits or hurts performance.
  • Adaptation burden: Higher block efficiency may require in-distribution initialization or more aggressive online adaptation, but these options add training, scheduling, and configuration-search burden.Suggested strategies include multiple auxiliary-drafter training steps per RL iteration or asynchronous adaptation.
  • Llama3.1-8B slowdown: Tq is around 0.8–0.9 ms across models, while Llama3.1-8B-Instruct has TV /Tp = 1.45; timing overhead alone does not explain its larger slowdown.The drafter cost is similar across models, and Llama’s verification-to-target ratio is only modestly higher than Qwen’s.
  • Llama3.1-8B slowdown: Before adaptation, Llama3.1-8B-Instruct has block efficiency close to one after 512 generated tokens and remains near 1.2–1.3 in the 1k–8k bins afterward.This persistent weakness indicates poor long-depth proposal quality, whereas Qwen models quickly recover relative to their pre-adaptation behavior.

F.6 Auxiliary Drafters Aligned with RL Rollout Distributions Are Difficult to Obtain

Learned auxiliary drafters are difficult to align with evolving RL rollout distributions: they achieve much lower block efficiency than target-induced quantized self-drafters, and only one evaluated configuration consistently reduces rollout-generation time. These results indicate that effective learned drafting requires a specialized alignment and training pipeline rather than a readily available public checkpoint.

  • Block efficiency gap: τ = 3.6–3.9 across the first 30 RL steps for quantized self-drafters, versus τ = 1.2–2.4 for evaluated EAGLE3 drafters under γ = 3.The best public configuration, Qwen3-8B with the RedHatAI thinking drafter, reaches roughly τ = 2.2–2.4, still below the ceiling of 4.
  • Generation-time consequence: 7.7% rollout-generation-time improvement is achieved by Qwen3-8B with the RedHatAI thinking drafter, the only evaluated EAGLE3 configuration consistently faster than the NeMo RL No-SD baseline.Other configurations are slower than No-SD because of insufficient block efficiency.
  • Output length and training distribution: Shorter output caps of 1k, 2k, and 4k leave most public or general-chat/text-pretrained drafters weak, indicating output length alone does not explain their low block efficiency.Several configurations are roughly length-flat across the ablation probes.
  • Takeaway: Effective auxiliary drafters require target-generated long-rollout collection, offline auxiliary-drafter training, and possibly more aggressive online adaptation.Such requirements raise the barrier to using learned auxiliary drafting as a drop-in rollout accelerator.
Loading 2606.18967v1…