PaperlayerView exact revision on arXiv

Source-linked AI summary

PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation

Alexandre Piché, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, Dzmitry Bahdanau

arXiv:2509.19128v2cs.LG

TL;DR

Scaling LLM reinforcement learning requires balancing accelerator utilization with data freshness, since common algorithms are designed for on- or near-on-policy data. PipelineRL runs generation and training concurrently with in-flight weight updates; in long-form reasoning experiments, it reached the same reward values about 2x faster than conventional RL.

  • Problem

    Scaling LLM RL is difficult because high generation throughput requires large batches, while using data from an older behavior policy can harm algorithms designed for on- or near-on-policy training.

    Source: §1 · pp. 1–2

  • Method

    PipelineRL runs asynchronous generation and training concurrently, updating generation weights in-flight so generation can continue after a brief pause.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    ∼2x faster sample throughput let PipelineRL reach the same reward values approximately ∼2x faster than Conventional RL with G = 32 optimizer steps on 128 GPUs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    In-flight updates help PipelineRL break the learning-speed ceiling of conventional two-stage RL, while recomputing the KV cache after updates can mitigate stale-cache risk but lowers throughput.

    Source: §7 · p. 10

    Throughput gains are limited when each prompt requires the same exact token count, and decrease with scarce or extensive compute resources.

    Source: §7 · p. 11

Abstract

from arXiv · show

Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Large Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high AI accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updates. This mechanism allows the LLM generation engine to receive updated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness of training data. Experiments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately $\sim 2x$ faster learning compared to conventional RL baselines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as a key contribution.

1 Introduction

Scaling LLM reinforcement learning is difficult because high accelerator throughput requires large generation batches, while training on data from an older behavior policy can harm algorithms designed for near-on-policy data. PipelineRL overlaps generation and training, updating weights in-flight, and reports roughly 2x faster learning than a conventional baseline on long-form reasoning tasks.

1 Introduction

  • High accelerator throughput requires large generation batches, but lagging behavior-policy data can harm RL algorithms designed for on- or near-on-policy training.The passage names REINFORCE, PPO, and GRPO among the affected algorithms.Source: §1 · pp. 1–2
  • PipelineRL overlaps asynchronous generation and training, briefly pausing generation to receive updated weights before continuing in-progress sequences.The authors say in-flight updates avoid waiting for the last sequence to finish, support high accelerator utilization at a constant generation batch size, and maximize policy adherence of recently generated tokens.Source: §1 · p. 2
  • ∼2x faster learning than the comparable conventional RL baseline was reported for long-form reasoning experiments on 4 DGX-H100 nodes.The authors also report highly on-policy training data and performance comparable to similarly trained models from the literature.Source: §1 · p. 2

2 Background

The background introduces policy-gradient RL for LLMs and importance sampling for differences between current and behavior policies, then describes alternating generation and training and practical generation-engine trade-offs.

2.1 Reinforcement Learning for Large Language Models

  • Policy-gradient methods train an LLM policy to maximize expected return, while importance sampling or trust-region constraints address differences between current and behavior policies.Normalized effective sample size (ESS) measures the equivalent number of current-policy samples represented by behavior-policy samples; values closer to 1 indicate more efficient sampling, and on-policy data has ESS 1.Source: §2.1 · pp. 2–3

2.2 Conventional RL

  • Conventional RL alternates sequence generation with policy training; when G > 1 optimizer steps are used, the current policy advances ahead of the behavior policy that generated the data.The paper calls the number of optimizer steps between those policies the lag.Source: §2.2 · p. 3

2.3 Efficient Sequence Generation with LLMs

  • Generation engines batch sequences and manage past keys and values in a KV cache, but very large batches can incur high per-sequence latency, memory pressure, or queue-management overhead.Figure 2 reports that short-prompt throughput increases up to batch size 256; as sequences finish and the number per GPU falls, generation time plateaus and throughput decreases.Source: §2.3 · p. 3, Figure 2 · p. 4

3 The learning speed ceiling of Conventional RL

Long-sequence RL can be slow because generation takes minutes, and conventional RL must balance sample throughput against learning effectiveness. Its on-policy setting preserves learning effectiveness but limits throughput, while increasing optimizer steps can make rollouts stale.

3 The learning speed ceiling of Conventional RL

  • Long-form reasoning generations can take several minutes, while high accelerator throughput requires large generation batches.Source: §3 · pp. 4–5
  • Pure on-policy Conventional RL maximizes learning effectiveness but processes at most B samples at a time, so adding accelerators yields diminishing throughput returns as B/N falls below the optimal range.Source: §3 · p. 4
  • As shorter sequences finish, fewer long generations remain, so keeping hardware busy requires each accelerator to generate many times more sequences than its optimal batch size.Source: §3 · p. 5
  • Using G>1 optimizer steps per RL step generates BG rollouts and can raise throughput, but high G eventually makes old-policy rollouts stale and lowers learning effectiveness.Source: §3 · p. 5

4 Pushing the learning speed ceiling with PipelineRL

PipelineRL overlaps training and generation, updating generation weights in flight to improve the throughput–effectiveness trade-off. The paper reports faster reward attainment than conventional baselines and a 1.5x simulation speedup at the same maximum lag.

4 Pushing the learning speed ceiling with PipelineRL

  • PipelineRL runs generation and training asynchronously in parallel and updates generation weights after every optimizer step in flight, without stopping sequence generation.Source: §4 · p. 5
  • Adding accelerators linearly increases PipelineRL sample throughput, while doubling them doubles early-token and average batch lag; its batches share a lag profile unlike Conventional RL’s progressively staler batches.Source: §4 · p. 6
  • At the same maximum lag gmax, PipelineRL can learn 1.5x faster than Conventional RL in simulation, with empirical gains potentially larger depending on update frequency without harming effectiveness.Source: §4 · p. 6
  • Training-accelerator count T trades off trainer speed and generation lag; the paper recommends avoiding extreme T settings that overburden generation or waste hardware.Source: §4 · p. 6
  • The implementation uses a three-stage actor–preprocessor–trainer pipeline with Redis streaming and supports modular integration of generation software through three HTTP endpoints.Source: §4 · pp. 6–7, Figure 4 · p. 7
  • Figure 5 reports PipelineRL attains the same average reward faster than conventional RL, matches sample efficiency of G=8 and G=16, and generates samples much faster.Source: Figure 5 · p. 7

5 Experiments

PipelineRL is evaluated on long-form mathematical reasoning, where long generations make on-policy learning important. It learns faster than Conventional RL while retaining comparable task performance and largely on-policy data, including when in-flight updates use a stale KV cache.

5 Experiments

  • Experiments train Qwen 2.5 base 7B on 17K OpenReasoner Zero math problems for 1000 optimizer steps, with batch size 1024 and importance-weighted REINFORCE clamped at 5.PipelineRL uses 48 GPUs for generation and 80 for training across 16 DGX-H100 nodes; the comparison simulates Conventional RL by buffering and shuffling BG samples before G optimizer steps.Source: §5 · p. 8
  • Approximately 2x faster reward matching than Conventional RL with G = 32 accompanies approximately 2x faster sample throughput.G = 32 was the largest Conventional RL setting with stable training; G = 64 diverged.Source: §5 · p. 8
  • Across compared methods, reward-versus-sample curves are indistinguishable until G = 32 becomes slower and unstable, likely from excessive off-policyness.This supports the claim that in-flight weight updates do not harm sequence generation.Source: §5 · pp. 8–9
  • PipelineRL matches or exceeds Open Reasoner Zero and SimpleRL Zero success rates on MATH500 and AIME2024.The comparison used batch size 4096.Source: §5 · p. 9, Table 1 · p. 9
  • Despite higher maximum lag than Conventional RL, PipelineRL maintains ESS similar to G = 8; ESS for G = 16 and especially G = 32 drops during training.The experiments focus on long-form mathematical reasoning, where generated sequences grow substantially and staying on-policy is important for effective learning.Source: §5 · pp. 7, 9, Figure 6 · p. 8

5.1 Impact of in-flight weight updates on on-policyness

  • Across checkpoints 0, 100, and 190, mixed-policy sequences closely align in KL divergence with on-policy sequences, while conventional off-policy sequences diverge more as lag increases.With maximum lag 32 and sequence length 2048, retaining stale KV cache causes only slightly higher divergence than recomputing it, supporting the more efficient cache-retention choice.Source: §5.1 · pp. 9–10, Figure 7 · p. 10

6 Related work

Prior asynchronous RL work primarily addressed video games, while later work explored asynchronous LLM RL and scalable implementations. PipelineRL is distinguished in this comparison by its in-flight weight updates and reported on-policyness relative to an earlier asynchronous LLM approach.

6 Related work

  • IMPALA decoupled acting from learning for GPU utilization, while SeedRL updated parameters during episodes; both lines of work focused on video games.IMPALA used truncated importance weights and held policy weights constant for each episode.Source: §6 · p. 10
  • Noukhovitch et al. (2024) synchronized generation and training across groups of G optimizer steps, yielding higher off-policyness than Conventional RL; DPO was reported to tolerate off-policyness better.The cited work explored asynchronous RL for LLMs.Source: §6 · p. 10
  • veRL uses a hybrid generation-training engine, OpenRLHF leaves training GPUs idle during generation and vice versa, and Magistral also introduced in-flight weight updates.The authors expect veRL throughput to be similar to their Conventional RL baseline.Source: §6 · p. 10

7 Conclusion and Discussion

In-flight weight updates help PipelineRL break the learning-speed ceiling of conventional two-stage RL, though its throughput advantage depends on the sequence-length and compute setting. The appendix estimates speedup at fixed maximum lag using accelerator, sequence-length, and utilization notation.

7 Conclusion and Discussion

  • In-flight weight updates help PipelineRL break the learning-speed ceiling of conventional two-stage RL.Recomputing the KV cache after each update can mitigate stale-cache risk at high GPU utilization, but lowers throughput.Source: §7 · p. 10
  • Future work should study when recent low-lag tokens help and where high lag on early tokens in long sequences hurts.Source: §7 · p. 11
  • PipelineRL’s throughput advantage decreases when prompts require the same token count, or when compute resources are scarce or extensive.In those settings, Conventional RL can keep a constant generation batch size, GPU utilization may already be high, or learning speed may be bounded by generation latency and environment feedback delay.Source: §7 · p. 11

A Analyical estimate of PipelineRL speedup for fixed max lag

  • The fixed-maximum-lag estimate uses N for accelerators, S = BG for sequences per Conventional RL step, L for maximum and average sequence lengths, K = SL for tokens per optimizer step, and U(h) for utilization.Source: A Analyical estimate of PipelineRL… · p. 13

A.1 Units

The appendix introduces flash as a time unit for comparing RL throughput independently of GPU model and LLM size. It then measures system throughput in tokens per flash and specifies an assumption for training time per token.

A.1 Units

  • A flash is the theoretically smallest amortized time for generating one token, so generating K tokens takes at least K flashes.At a typical generation utilization rate of ∼0.1, it takes 10K flashes; the treatment abstracts away variation in attention FLOPs for very long sequences.Source: §A.1 · p. 13
  • The appendix measures system throughput in tokens per flash and assumes PipelineRL and Conventional RL have similar amortized training time per token at scale.Both approaches can benefit from sequence packing.Source: §A.1 · p. 13
  • Figure 8 measures H100 utilization against theoretical bf16 FLOPS throughput using matrix multiplications, with padding up to h + 64 to improve speed.The passage reports utilization bumps when h is divisible by a higher power of 2, up to 128.Source: Figure 8 · p. 14

A.2 Conventional RL throughput

The appendix estimates Conventional RL throughput from generation and training time, then compares it with a PipelineRL pipeline whose throughput is set by its slowest stage. Its analysis highlights Conventional RL’s reduced efficiency as accelerator count grows at low per-GPU batch size.

A.2 Conventional RL throughput

  • Figure 9 plots PipelineRL and Conventional RL throughput against maximum lag gmax for N = 128 GPUs and batch size B = N.Source: Figure 9 · p. 15
  • At low per-GPU batch size, Conventional RL’s efficiency decreases as accelerator count N grows because hN(l)/U(hN(l)) decreases only slowly with N.The passage attributes this to x U(x) being nearly constant for small x on modern GPUs; the maximum token lag in the described setup is S − 1.Source: §A.2 · p. 15

A.3 PipelineRL throughput

  • PipelineRL throughput is determined by its slowest pipeline stage, and its maximum lag depends on generation length and batch configuration.The analysis searches (H, I) configurations for maximum throughput subject to a maximum-lag constraint; the supplied passages give the constraint only in fragments.Source: §A.3 · p. 15

A.4 A PipelineRL speedup case study

In the case study, PipelineRL’s estimated speedup depends on maximum lag and batch size. Its higher generation efficiency with fewer GPUs accounts for the reported throughput advantage.

A.4 A PipelineRL speedup case study

  • 1.57x faster at gmax ∼133: PipelineRL’s estimated throughput exceeds Conventional RL’s in the H100 case study.The estimate assumes a uniform sequence-length distribution from 1 to maximum length L, H100 GPUs, N = 128, and B = 128; the passage cautions that this lag may be too high for many practical setups.Source: §A.4 · p. 16
  • At B = 2048, the same sequences per GPU correspond to gmax ∼8, described as a practical 16x lower lag.Source: §A.4 · p. 16
  • 44 generation GPUs produce 16.9 tokens per flash, compared with 18.3 tokens per flash from 128 GPUs in Conventional RL.The passage identifies the more efficient generation on fewer GPUs as the root cause of PipelineRL’s speedup.Source: §A.4 · p. 16

B Additional Results

Figure 10 reports that the G=64 setting diverges.

B Additional Results

Related papers

Loading 2509.19128v2…