Source-linked AI summary
Laminar: A Scalable Asynchronous RL Post-Training Framework
Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, Haibin Lin, Xin Liu, Chuan Wu
TL;DR
Long-tail trajectory generation and global weight synchronization constrain the scalability of RL post-training. Laminar decouples components to enable trajectory-level asynchrony, relay-based weight updates, and dynamic repacking. It achieves up to 5.48× speedup over state-of-the-art systems and reduces model convergence time.
Problem
Long-tail generation underuses GPUs, while global synchronization imposes rigid updates that limit asynchronous RL systems on skewed workloads.
Source: §1 · p. 1, §2.3 · p. 4Method
Laminar uses a fully decoupled architecture with relay workers and dynamic repacking to let trajectories progress asynchronously.
Source: §1 · p. 2, §10 · p. 13Results
Laminar achieves up to 5.48× speedup over state-of-the-art systems and reduces model convergence time.
Source: §10 · p. 13Takeaways & Limitations
Decoupling isolates component failures, so one rollout machine failure does not halt training and can be recovered from swiftly.
Source: §1 · p. 2Partial rollout synchronization repeatedly rebuilds the KVCache for interrupted trajectories, and inconsistent policy versions within a response can harm convergence.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
Reinforcement learning (RL) post-training for Large Language Models (LLMs) is now scaling to large clusters and running for extended durations to enhance model reasoning performance. However, the scalability of existing RL frameworks is limited, as extreme long-tail skewness in RL trajectory generation causes severe GPU underutilization. Current asynchronous RL systems attempt to mitigate this, but they rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule. This global synchronization is ill-suited for the highly skewed and evolving distribution of trajectory generation latency in RL training, crippling training efficiency. Our key insight is that efficient scaling requires breaking this lockstep through trajectory-level asynchrony, which generates and consumes each trajectory independently. We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture. First, we replace global updates with a tier of relay workers acting as a distributed parameter service. This enables asynchronous and fine-grained weight synchronization, allowing rollouts to pull the latest weight anytime without stalling the actor's training loop. Second, a dynamic repack mechanism consolidates long-tail trajectories onto a few dedicated rollouts, maximizing generation throughput. The fully decoupled design also isolates failures, ensuring robustness for long-running jobs. Our evaluation on a 1024-GPU cluster shows that Laminar achieves up to 5.48$\times$ training throughput speedup over state-of-the-art systems, while reducing model convergence time.
1 Introduction
Laminar targets long-tail generation and rigid global synchronization, which leave GPUs underused and constrain RL post-training scalability. It enables trajectory-level asynchrony through decoupled components, relay workers, and dynamic repacking.
1 Introduction
- Generation accounts for up to 83.1% of reasoning-task execution time, while long-tail trajectories leave cluster GPUs underutilized.The 99th-percentile output length in mathematics and coding can exceed the median by an order of magnitude.Source: §1 · p. 1
- Existing asynchronous systems use global weight synchronization, and larger staleness bounds may absorb skew but negatively affect model convergence.A one-step staleness pipeline cannot effectively hide long-tail generation latency, leaving the optimal bound difficult to tune.Source: §1 · pp. 1–2
- Laminar makes trajectories independently generated and consumed at their own pace, breaking rigid data and parameter dependencies across the system.Its fully decoupled architecture isolates component failures so a failed rollout machine does not halt training.Source: §1 · p. 2
- Laminar reports up to 5.48× throughput speedup over state-of-the-art RL systems at scales up to 1024 GPUs.The evaluation also reports reduced model convergence time.Source: §1 · p. 2
- Relay workers provide asynchronous, fine-grained weight synchronization, while dynamic repacking concentrates long-tail trajectory generation on fewer rollout replicas.Rollouts can retrieve new parameters anytime without stalling actor training; repacking monitors rollout idleness using KVCache-based metrics.Source: §1 · p. 2
2 Background and Motivation
LLM post-training alternates trajectory generation and model training, but skewed and changing generation costs leave existing systems waiting on long trajectories. Laminar’s motivation is to replace rigid synchronization with trajectory-level asynchrony, while addressing weight-transfer, rollout-utilization, and reliability challenges.
2.1 RL for LLM Post-Training
- RL post-training generates and scores trajectories, stores them as experiences, then samples batches to update the actor; some algorithms estimate advantages from multiple trajectories per prompt.The generation stage can include environment interactions, and training may use a critic or simplify advantage computation without one.Source: §2.1 · pp. 2–3
2.2 Recent Trends in RL Post-Training
- Generation can consume up to 83.1% of reasoning-task execution time, as highly skewed trajectory lengths and unpredictable environment latency leave GPUs underutilized.In some tasks, the 99th-percentile trajectory length is ten times the median.Source: §2.2 · p. 3
2.3 Limitations of existing asynchronous RL systems
- Synchronous systems struggle with skew, while asynchronous systems still couple progress to full batches and global weight updates, leaving training to wait for slow trajectories.Inter-stage fusion offers limited benefit because experience preparation is only 7.3% of total RL iteration time.Source: §2.3 · pp. 3–4, §2.2 · p. 3
- Global synchronization creates pipeline bubbles that added rollout GPUs cannot remove, while partial rollouts incur KVCache rebuilding and can slow convergence with mixed policy versions.A larger staleness bound may hide generation latency but can harm convergence; a smaller bound reduces policy divergence but masks latency less effectively.Source: §2.3 · p. 4
2.4 Opportunity and Challenges
- Trajectory-level asynchrony lets each trajectory finish on a consistent model version and enter the experience buffer independently, so long trajectories need not stall shorter ones.Staleness then follows each trajectory’s generation latency rather than a static bound.Source: §2.4 · p. 4
- Asynchronous weight updates must avoid GPU-memory buffers and communication contention, which can reduce training batches or KVCache capacity or reintroduce blocking waits.The proposed relay tier aims to minimize GPU idle time without extra GPU memory consumption on actors or rollouts.Source: §2.4 · pp. 4–5
- Repacking incomplete trajectories from memory-bound, small-batch rollouts onto a few destinations can free replicas to fetch fresh weights; KVCache utilization guides when to consolidate.The source notes that small decode batches can have nearly the same latency as larger ones.Source: §2.4 · p. 5
- Component isolation supports rapid recovery from rollout failures, avoiding the loss of in-progress work that can take hours to regenerate on complex tasks.The design aims to recover components independently during long-running training.Source: §2.4 · p. 5
3 Fully Decoupled Architecture Design
Laminar decouples trajectory production, actor training, and weight synchronization so they can proceed independently. Relay workers support asynchronous weight pulls, repacking addresses rollout imbalance, and component-level recovery supports long-running jobs.
3 Fully Decoupled Architecture Design
- Laminar fully decouples data and parameter dependencies between actor and rollouts, enabling independent trajectory generation, training, and weight synchronization while isolating faults.Each trajectory can finish at its own pace while training uses diverse trajectories under minimal staleness.Source: §3 · p. 5
3.1 System Components
- Rollout, data, relay, and trainer modules divide generation, trajectory storage, weight distribution, and model updates across the system.The rollout manager monitors workload and health; data storage includes prompt, partial-response, and experience pools; relays host weights in CPU memory.Source: §3.1 · pp. 5–6
3.2 Training Workflow
- Rollouts stream in-progress trajectories to a partial-response pool and completed ones to an experience buffer, while the trainer samples completed trajectories in parallel.This separates data production from consumption.Source: §3.2 · p. 6
- After an update, the trainer sends weights to the master relay and resumes training; relays broadcast in the background, and rollouts can pull weights on demand.The rollout manager repacks trajectories from underutilized rollouts onto fewer replicas, freeing others to fetch newer weights.Source: §3.2 · p. 6
3.3 Fault Tolerance and Recovery
- Fault isolation allows rollouts to be added or removed without halting training, while recovery preserves in-progress trajectory states and redirects interrupted work to healthy rollouts.Relay failures trigger broadcast-chain rebuilding within seconds without disrupting generation; trainer failures use checkpoint recovery while rollouts continue generating.Source: §3.3 · pp. 6–7
4 Asynchronous Weight Synchronization Using Relay Workers
Laminar uses relay workers and RDMA to decouple rollout weight access from actor training. Its broadcast chain is rebuilt after failures so healthy rollouts can continue generating.
4.1 Design Considerations
- Relay workers in rollout-machine CPU memory use local host memory and RDMA to avoid storage serialization, I/O, and contention during weight synchronization.The design avoids dedicating separate GPUs as relays and uses host memory that remains largely underutilized.Source: §4.1 · p. 7
4.2 Hierarchical Relays and Their Workflow
- A master relay receives actor weights, reshards them, and pipelines chunks through per-machine relays while rollouts fetch updates from their colocated relay at any time.Broadcasting a 72B model to 127 relays takes less than 1.6 seconds, according to the passage.Source: §4.2 · p. 7
4.3 Fault-Tolerant Relay Broadcast
- After a rollout-machine failure, the scheduler rebuilds the broadcast chain in less than one second; healthy rollout generation continues during repair.If the master relay fails, the manager selects a replacement and notifies the trainer of its IP address.Source: §4.3 · pp. 7–8, Figure 7 · p. 7
5 Bubble Elimination in Long-tail Trajectory Generation
Laminar repacks unfinished long-tail trajectories from underutilized rollouts onto fewer destinations within the same weight-version group. KVCache utilization and trajectory-count limits guide source selection and placement.
5.1 Workflow of the Repack Mechanism
- The rollout manager groups rollouts by weight version, identifies long-tail sources, plans consolidation, and transfers their unfinished trajectories to destinations.Repacking is checked periodically and can also start after a global-batch weight update to free rollouts for generation with the latest version.Source: §5.1 · p. 8
5.2 Online Trajecgtory Repacking
- A rollout becomes a repacking candidate as KVCache use falls below its threshold with fewer than B trajectories remaining, avoiding workload-specific threshold tuning.KVCache use stays near its peak while waiting trajectories fill freed space, then falls when no waiting trajectories remain and active trajectories finish.Source: §5.2 · pp. 8–9, Figure 9 · p. 8
- The Best-Fit heuristic matches source rollouts to destinations that remain within KVCache and batch-size limits, favoring dense packing.The algorithm runs in O(|S|^2), where |S| is the number of candidate rollouts.Source: §5.2 · p. 9
- Trajectory repacking increases generation throughput while promoting the generation of on-policy data.The mechanism consolidates long-tail work onto destination rollouts and releases source rollouts.Source: §5.2 · p. 9
6 Trajectory-level Asynchrony Analysis
Laminar’s trajectory-level asynchrony gives each trajectory an inherent staleness determined by generation latency and trainer update speed. The system does not configure a staleness bound, and observed staleness typically remains under 3.
6 Trajectory-level Asynchrony Analysis
- Inherent staleness is the difference between the actor’s current model version and the version used to generate a completed trajectory.It depends on trajectory generation latency and trainer model-update speed.Source: §6 · p. 9
- Trajectory staleness in Laminar remains typically under 3 as rollouts update weights after completing or releasing a batch.The design pursues minimal staleness by eliminating rollout idle time and updating each rollout as soon as possible.Source: §6 · pp. 9–10
- Experience-sampling strategies are orthogonal to this work; using massive generated experiences effectively remains future work.The passage gives priority-based sampling as one possible strategy.Source: §6 · p. 10
7 Implementation
Laminar is implemented in about 11,000 lines of Python, building its trainer and rollout modules on verl and separating them across GPU sets.
7 Implementation
- The framework comprises about 11,000 lines of Python, including 3,800 lines for trainer and rollout modules built on verl.The trainer and rollout modules run on different sets of GPUs.Source: §7 · p. 10
- A partial response pool stores ongoing trajectory tokens and statistics to support fast recovery and online analysis.Inter-module data transmission uses Ray remote procedure calls.Source: §7 · p. 10
- Relay workers use pinned shared CPU memory for PCIe weight transfers, while UCX supports chain-based broadcast with heartbeat monitoring and dynamic rebuilding.The rollout manager supports repacking and is implemented in 1,600 lines of code.Source: §7 · p. 10
8 Evaluation
Laminar is evaluated on a 1024-GPU cluster against synchronous and asynchronous RL frameworks for math reasoning and multi-turn tool calling. Its throughput, convergence, synchronization overhead, repacking, and recovery are measured across the supplied experiments.
8 Evaluation
- The evaluation uses a 128-machine, 1024-GPU cluster and compares Laminar with synchronous, one-step-staleness, stream-generation, and partial-rollout frameworks on math and tool-calling tasks.The experiments use Qwen2.5 models at 7B, 32B, and 72B; throughput is averaged over five iterations after a ten-iteration warm-up.Source: §8 · p. 10
8.1 End-to-End Training Throughput
- Laminar averages 2.56× speedup over verl on math and 2.62× across baselines on tool calling, with gains across model and cluster scales.The throughput comparisons are shown for the math and tool-calling tasks in Figures 11 and 12.Source: §8.1 · p. 11, Figure 11 · p. 11, Figure 12 · p. 11
- At the largest cluster scales, Laminar averages 3.34× speedup across baselines and model sizes; baseline strong-scaling efficiency reaches only 33.6% on math and 12.9% on tool calling.Laminar's strong-scaling efficiency is 53.7% on math (up to 68.2% on 32B) and 46.5% on tool calling.Source: §8.1 · p. 11
8.2 Training Convergence
- Laminar converges about 1.77× faster for 7B and 1.59× faster for 32B than on-policy verl on math reasoning.Figure 13 plots reward against wall-clock time; Laminar's maximum staleness is 4 in these comparisons.Source: §8.2 · p. 11, Figure 13 · p. 11
8.3 Weight Synchronization Overhead
- Laminar reduces average and best-case rollout weight-waiting time by up to 37% and 47%, while actor stalls are 0.64 seconds for 32B and 1.40 seconds for 72B.Figure 14 compares rollout waiting time; the actor transfers weights to one master relay, keeping its communication overhead constant as rollout replicas increase.Source: §8.3 · p. 12, Figure 14 · p. 12
8.4 Repack Efficiency
- Repacking raises generation throughput by 26%, increases average KVCache utilization by 14.8%, and adds 0.69 seconds of overhead without increasing trajectory-generation latency.Figure 16 reports generation throughput, and Table 1 reports rollout statistics with and without repacking.Source: §8.4 · p. 12, Figure 16 · p. 12, Table 1 · p. 12
8.5 Fault Tolerance Analysis
- After a rollout machine containing two replicas is killed, generation throughput drops immediately, but training throughput is unaffected or drops slightly; the system recovers in about 252 seconds.After recovery, generation and training throughput return to their original levels (Figure 15).Source: §8.5 · p. 12, Figure 15 · p. 12
9 Related Work
Prior LLM RL systems include synchronous and asynchronous approaches, but asynchronous frameworks still rely on global synchronization or partial rollouts. The paper situates Laminar alongside small-scale asynchronous deep RL and fault-tolerant training infrastructure.
RL frameworks for LLMs
- Prior asynchronous LLM RL systems retain global weight synchronization [17] [18] [19] [21] [22], while partial rollouts can rebuild KVCache and mix policy versions within a trajectory [7] [8] [18] [19].Laminar's trajectory-level asynchrony instead generates each trajectory with a single, consistent policy version.Source: §9 · pp. 12–13
Asynchronous deep RL systems
- Asynchronous deep RL systems target small-scale DNNs rather than modern LLM generation and training patterns, while fully asynchronous stale-parameter updates can destabilize training [97].The paper cites AlphaStar and OpenAI Five [66] as examples of large-scale deep RL systems.Source: §9 · p. 13
Fault-tolerant training
- Prior fault-tolerant training infrastructure can complement Laminar's heartbeat-based failover and relay-worker rerouting to improve observability and recovery.The related work also notes that checkpointing optimizations can further mitigate stalls and support flexible transfer [59] [60].Source: §9 · p. 13
10 Conclusion
Laminar combines trajectory-level asynchrony, relay-based asynchronous weight synchronization, and dynamic repacking in a decoupled RL post-training framework. The paper reports evaluation up to 1024 GPUs, including throughput speedups and reduced convergence time.
10 Conclusion
- Laminar's decoupled architecture and dynamic repacking target scalable, robust RL post-training, and the paper reports up to 5.48× speedup over state-of-the-art systems with reduced convergence time.The conclusion describes independently paced trajectory generation and consumption, asynchronous relay synchronization, and component failure isolation.Source: §10 · p. 13
A.1 Response Length Distribution of Each Model
- Figure 17 lists response-length distributions for the model checkpoints used in the throughput experiments on DAPO-Math-17k.The appendix identifies the distributions as covering the checkpoints used in those experiments.Source: §A.1 · p. 18, Figure 17 · p. 18
A.2 Hyperparameter Details
- The appendix specifies throughput and convergence experiment models, GPU allocations, sampling and concurrency settings, parallelism configurations, and other hyperparameters.The throughput experiments cover math and tool calling; convergence experiments use Qwen2.5-Math-7B and Qwen2.5-32B.Source: §A.2 · p. 18
B Detailed Experiment Analysis
Laminar's throughput advantage is modest at small cluster sizes and more pronounced at large scales, where it averages 3.34× speedup across baselines and model sizes.
Performance in small cluster scales
- At no more than 64 GPUs, Laminar averages 1.41× speedup across baselines and model sizes; model-parallel resource needs constrain placement choices and throughput balancing.In these limited configurations, the throughput improvement mainly comes from eliminating long-tail bubbles.Source: B Detailed Experiment Analysis · p. 18
Performance in large-scale clusters
- At the largest cluster scales, Laminar averages 3.34× speedup across baselines and model sizes, while global synchronization and partial-rollout KVCache recomputation constrain competing systems.The paper attributes Laminar's scale-up performance to eliminating long-tail bubbles and keeping relay and repack overhead minimal.Source: B Detailed Experiment Analysis · p. 18
- As cluster size increases, repacking becomes increasingly effective because Laminar has more rollout replicas to manage.Source: B Detailed Experiment Analysis · p. 18
C Discussion
The discussion flags unresolved experience-sampling and partial-rollout stability issues, then develops a chain-based pipelined broadcast model for distributing large model weights.
C Discussion
- Throughput experiments specify GPU allocations, with colocated verl alternating training and rollout, while asynchronous convergence runs use mini-batches of 2048.Source: Table 2 · p. 19, Table 3 · p. 20
- Partial rollout can be combined with Laminar, but mid-generation updates create mixed-version trajectories that distort output-length distributions and may hinder convergence in valueless RL.The expected number of weight versions in a trajectory grows with sequence length and actor-update frequency.Source: C Discussion · pp. 18–19
- Laminar’s generation can outpace trainer consumption, making effective experience sampling necessary to avoid learning from low-utility experiences.Experience sampling remains underexplored in large-scale LLM post-training.Source: C Discussion · pp. 19–20
D Theoretical Analysis of Chain-Based Pipelined Broadcast
- The chain-based broadcast analysis aims to show that large-message latency is dominated by a term independent of node count.The stated goal is to establish scalability for distributing large model weights.Source: D Theoretical Analysis of Chain-Based… · p. 20
D.1 Communication Model
- The model chains p−1 relays after a master, parameterizes adjacent links by startup and per-byte costs, and splits M-byte weights into k equal chunks.Source: §D.1 · p. 20
D.2 Latency Derivation
- Broadcast latency combines the first chunk’s traversal of p−1 hops with the final relay’s receipt of the remaining k−1 chunks, one chunk interval at a time.Source: §D.2 · p. 20
D.3 Scalability Analysis
The scalability analysis minimizes broadcast latency over the chunk count and argues that, for large model weights, a node-independent bandwidth term dominates.
D.3 Scalability Analysis
- The analysis minimizes T(p,k) by setting its derivative with respect to k to zero, then substitutes k* to obtain minimum broadcast time for given p and M.Source: §D.3 · p. 21
- For 140 GB 70B BF16 weights, the node-independent bandwidth term dominates; startup grows linearly and the pipeline term as O(√p), but both contribute much less under realistic parameters.The analysis gives 2000 × 5μs = 10ms as an example of the startup term’s contribution.Source: §D.3 · p. 21