PaperlayerView exact revision on arXiv

Source-linked AI summary

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

Bo Wu, Sid Wang, Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang, Yundi Qian, Beibei Zhu, Rui Hou

arXiv:2505.24034v2cs.LGcs.AI

TL;DR

Large-scale LLM reinforcement learning is challenging because of high latency and memory demands. LlamaRL is a distributed asynchronous framework for training across model and GPU-cluster scales. It reports a 10.7x throughput increase over state-of-the-art systems on 405B models, with efficiency advantage growing as model scale increases.

  • Problem

    High latency and memory demands make it challenging to develop efficient RL frameworks that reliably manage policy models with hundreds to thousands of billions of parameters.

    Source: §1 · p. 1

  • Method

    LlamaRL is a fully distributed asynchronous RL framework that combines scalable training with co-located model offloading and distributed GPU-native weight synchronization.

    Source: §1 · p. 1, §4.1 · p. 6, §5.2 · p. 10

  • Results

    10.7x increase in RL training throughput over state-of-the-art systems on a 405B-parameter model, with the efficiency advantage growing as model scale increases.

    Source: §1 · p. 1, §9 · p. 20

  • Takeaways & Limitations

    LlamaRL is presented as a practical path for RL training as model sizes and training demands continue to grow, from small H100 clusters to thousands or more GPUs.

    Source: §9 · p. 20

    Asynchronous learning trains on samples generated by a previous policy iteration, although off-policy corrections make its impact negligible compared with fully on-policy training.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, because of the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL, a fully distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs with various model sizes (8B, 70B, and 405B parameters) on GPU clusters ranging from a handful to thousands of devices. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL's efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically during the Llama 3 post-training, by leveraging best practices such as colocated model offloading, asynchronous off-policy training, and distributed direct memory access for weight synchronization, LlamaRL achieves significant efficiency gains -- up to 10.7x speed-up compared to DeepSpeed-Chat-like systems on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework's suitability for future large-scale RL training.

1 Introduction

LlamaRL targets the latency, memory, and utilization challenges of RL training at large LLM scales. It combines a scalable architecture and distributed training strategies, with reported efficiency gains that grow with model scale.

1.1 Challenges of Scaling RL for LLM

  • RL frameworks must support varied algorithms and model flows, from PPO's four-model pipeline to critic-free and multi-objective methods.As algorithms involve different numbers of models and data flows, the paper identifies flexibility in scheduling those flows as a framework challenge.Source: §1.1 · p. 2
  • Scaling RL to larger models is difficult because multiple actors with different sizes must fit memory and computation constraints across large GPU clusters.The cited Llama 3.1 405B PPO example requires at least 512 H100 GPUs, TP size 32, FSDP shard size 16, and an 8B reward model.Source: §1.1 · p. 2
  • Variable response lengths and slow data flow create idle GPU bubbles that worsen with larger models and can waste resources.The passage notes that high GPU utilization can shorten fine-tuning wall-clock time and let users train with fewer GPU resources.Source: §1.1 · p. 2

1.2 Contributions

  • LlamaRL uses a streamlined single-controller architecture built on native PyTorch, designed to scale across thousands of GPUs and support diverse RL algorithms.The paper presents its modular structure and intuitive control logic as facilitating adaptation and extension.Source: §1.2 · p. 2
  • 10.7x efficiency improvement over systems such as DeepSpeed-Chat is reported for a 405B-parameter model, with the advantage growing as model scale increases.The introduction also describes support for models from 8B to 405B parameters and clusters from a handful to thousands of H100 GPUs.Source: §1.2 · p. 2
  • Generation is offloaded from the training cluster to address its memory demands and execution-time contribution, enabling fine-grained parallelism and quantization.The inference policy and training policy run in different processing groups; rule-based scorers remain with the training policy.Source: §1.2 · p. 3
  • Training and generation run asynchronously in parallel to reduce computational bubbles, while AIPO uses off-policy importance weighting to mitigate instability from delayed samples.The described delay ranges from 1 to n steps.Source: §1.2 · p. 3
  • DDMA uses direct GPU-to-GPU zero-copy transfers over NVLink for weight synchronization, with linear scalability and approximately 2-second updates for terabyte-scale models across thousands of GPUs.The paper also presents a formal proof that asynchronous RL strictly speeds up over its synchronous counterpart.Source: §1.2 · p. 3

2 Background

The background describes LLMs and RL as a setting in which training systems handle multiple models and data dependencies. Its simplified online-RL example illustrates one such workflow.

Large language models (LLM)

  • The passage characterizes LLMs as generative models that, after large-scale pre-training and relatively lightweight fine-tuning, can perform tasks including summarization, coding, and translation.It also describes specialized capabilities beyond those tasks.Source: §2 · p. 3

Scaling laws

  • Scaling-law research describes LLM performance as reasonably predictable from model size, data size, and compute, while RL on verifiable tasks is described as scaling inference-time compute.The passage connects these developments to the need for more efficient reinforcement-learning systems.Source: §2 · p. 3

Training algorithms and systems for RL

  • PPO typically uses policy, critic, reference-policy, and reward models across response generation, data preparation, and actor-and-critic updates.The passage notes that critics may be omitted and reward models replaced by rule-based scorers depending on the application.Source: §2 · p. 4
  • Figure 1's simplified online-RL flow uses rule-based scorers, estimates a baseline from group scores, and has separate FSDP training and CUDA Graph inference policy instances.The example omits a reference model and learned critic for simplicity.Source: Figure 1 · p. 4

3 Related Work

Related work covers optimization and placement strategies for RL systems, along with scaling and asynchronous-training challenges. The paper frames weight synchronization and off-policyness as relevant concerns for large-scale LLM training.

Different Optimizations for Inference and Training

  • RL frameworks commonly use distinct inference and training engines because the two phases have different constraints and benefit from different optimization techniques.The passage describes this as a common framework design choice.Source: §3 · p. 4

Co-located Model Placement

  • Co-location shares limited device memory among models and can bottleneck throughput, while distributed placement assigns models to separate device groups.DeepSpeed-Chat (Yao et al., 2023) and HybridFlow (Sheng et al., 2024) are described as optimizing co-located policy models; NeMo-Aligner (Shen et al., 2024) co-locates models with the same architecture but different weights.Source: §3 · pp. 4–5

Distributed Model Placement

  • OpenRLHF (Hu et al., 2024) measured 111 seconds for 70B weight communication and estimated over 900 seconds for 405B, attributing the bottleneck to weight reloading rather than inter-node GPU bandwidth.The passage describes the increase as faster than linear with model size.Source: §3 · p. 5

Asynchronous RL and Off-policyness

  • Asynchronous learner-actor execution can improve throughput, but off-policyness may be more severe for policy-based RL used in LLM post-training.The passage contrasts this concern with value-based RL examples including AlphaZero (Silver et al., 2017a), AlphaGoZero, and OpenGo (Tian et al., 2019).Source: §3 · p. 5

Scaling up

  • Earlier asynchronous LLM training was reported for 2.8B and 8B models on multiple GPUs within a single host, while Kimi Team (2025) used partial rollouts with synchronous, sequential execution.The latter segments long responses across iterations while keeping trainer and generator co-located.Source: §3 · p. 6

4 High Level Strategies

LlamaRL separates generation from training, runs the groups asynchronously, and supports distinct parallelism and precision choices. These strategies address memory constraints, sequential idle time, and imperfectly matched group execution times.

4.1 Co-located model offloading

  • LlamaRL offloads the memory-bound generation phase, which contributes substantially to execution time, to a separate processing group from the FSDP training policy.Rule-based scorers remain allocated with the training policy.Source: §4.1 · p. 6
  • Distributed Direct Memory Access enables second-scale weight communication between processing groups, while fully distributed placement provides flexibility for asynchronous RL.The passage contrasts DDMA with partial co-location such as Hybrid Engine, which can save weight communication overhead.Source: §4.1 · p. 6

4.2 Asynchronous off-policy RL

  • Asynchronous off-policy learning unblocks RL workflow components that otherwise execute sequentially, addressing low GPU memory utilization and poor scalability.Figure 2 contrasts synchronous on-policy execution with asynchronous off-policy execution in which generator and trainer run in parallel.Source: §4.2 · p. 6, Figure 2 · p. 7
  • Asynchronous training inevitably uses samples generated by a previous policy iteration, but the reported impact with off-policy corrections is negligible relative to fully on-policy training.The figure caption also states that the asynchronous design accelerates training without compromising model quality.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Fine-grained parallelisms and quantization

  • LlamaRL decouples data, model, and pipeline parallelism across processing groups and supports inference-side FP8 or FP4 quantization.The passage says these choices can align group throughput, reduce inference inter-node communication, tune execution-time alignment, and enable smaller inference model-parallel size for faster generation.Source: §4.3 · pp. 7–8

5 System Design

LlamaRL organizes distributed RL training around modular executors and communication channels, coordinated by a single controller. Its DDMA weight synchronization and offloaded-component example extend this design to large-scale training.

5.1 High Level Architecture

  • A single controller launches and coordinates executors, which communicate prompts, trajectories, rewards, and updated weights through directed channels.Executors can run asynchronously within an iteration, each performing its own computations without waiting for the others to finish.Source: §5.1 · p. 8, §5.1.3 · p. 9, §5.1.2 · p. 9

5.1.1 Executor

  • Executors encapsulate distributed models on predefined GPU groups, and isolating RL stages makes components easier to scale or swap without monolithic code changes.The base executor supports initialization, RL steps, checkpointing, and exposing outputs or model states to other executors.Source: §5.1.1 · p. 9

5.1.2 Communication Channel

  • Communication channels define executor interactions and also carry model-weight updates, a central challenge in large-scale RL.Figure 3 depicts two executors connected by a communication channel and managed by one controller.Source: Figure 3 · p. 8, §5.1.2 · p. 9

5.1.3 Controller

  • Because each executor is an autonomous SPMD process, the controller remains a concise event loop for coordinating training.The controller sets executor steps, communicates across channels, triggers executor steps, and coordinates checkpointing.Source: §5.1.3 · p. 9

5.2 Distributed Direct Memory Access (DDMA) Weights Update

  • DDMA distributes weight shards across GPUs and enables linear scaling to thousands of GPUs, with terabyte-scale model updates taking about 2 seconds.It uses the same tensor and parallel groups as training and avoids a single-node memory bottleneck.Source: §5.2 · p. 10
  • DDMA uses direct GPU-memory transfers over NVLink and InfiniBand to bypass CPU memory and reduce weight-update latency.The paper notes that data movement can inflate step times by hundreds of seconds at large scales.Source: §5.2 · pp. 10–11

5.3 Example: RL with Offloaded Reward and Generator

  • The offloaded example assigns the generator and reward model separate GPU groups, while channels pass weights, generated sequences, and scored sequences between components.The modular design allows alternative reward models, additional models, or a different RL optimization algorithm without altering the rest of the pipeline.Source: §5.3 · p. 12

6 LlamaRL Off-policy Learning

LlamaRL uses importance sampling to address the off-policy discrepancy between asynchronously generated samples and the learner policy. Its AIPO algorithm clips the ratio, trading bias against variance; the authors report that tested clipping values help relieve training instability.

6 LlamaRL Off-policy Learning

  • AIPO weights the learner's policy-gradient update by the importance-sampling ratio between learner and actor token probabilities.The actor generates tokens and their probabilities, which are communicated to the trainer for the learner update.Source: §6 · p. 12
  • The per-token advantage is reward minus baseline, and one described baseline averages rewards across n generations sampled from a single prompt.The paper says this constant sequence-level baseline has proved reasonably effective for a number of applications.Source: §6 · p. 12
  • AIPO clips the importance-sampling ratio at a fixed constant ρ, trading lower bias for higher variance as ρ increases.The paper names this overall approach Asynchronous Importance weighted Policy Optimization.Source: §6 · p. 13
  • ρ values from 2 to 10 seem to work generally well and help relieve training instability in the authors' large-scale experiments.Instability can appear as sudden or gradual drops in training performance and may arise from interactions among asynchronous training and other factors.Source: §6 · p. 13
  • The authors report that single-sided clipping works better in their setting than PPO's double-sided clipping, which they relate to the difference between trust-region and asynchronous-training formulations.They also compare the correction with GRPO clipping.Source: §6 · p. 13

7 Theoretical justification of LlamaRL Speedup

The paper proves that, under its stated memory and batch-scaling assumptions, LlamaRL can achieve a strictly shorter RL step time than any synchronous configuration. The proof attributes the advantage to asynchronous execution and independently optimized trainer and generator memory constraints.

7 Theoretical justification of LlamaRL Speedup

  • Asynchronous training separates the trainer and generator memory constraints, allowing their parallelism and microbatch sizes to be optimized independently.The proof identifies freed GPU memory and reduced model parallelism as another source of efficiency gains.Source: §7 · pp. 13, 17
  • The result requires per-sample training and generation time to decrease monotonically with batch size, but assumes no specific analytical form for processing time.Figure 5 reports empirical support for decreasing per-sample time as microbatch size or decoding concurrency increases.Source: §7 · pp. 14–15, Figure 5 · p. 15
  • Under the stated setup and batch-size assumption, some memory-feasible LlamaRL configuration has a strictly shorter step time than the minimum possible synchronous baseline.The proof assumes negligible inter-device communication overhead and that the listed components dominate CUDA memory consumption.Source: §7 · p. 14
  • The synchronous step time sums generation and training time, whereas asynchronous step time is determined by the slower component.The analysis formulates each framework's step time and minimizes it subject to GPU-memory constraints.Source: §7 · pp. 15–16
  • The proof omits quantization and communication savings from reduced model parallelism, which the authors say would further strengthen its efficiency conclusion.These are conservative omissions, not required assumptions for the theorem as stated.Source: §7 · p. 17

8 Experiments

The experiments compare LlamaRL with a synchronous baseline across three model sizes and evaluate both RL-step efficiency and training quality. LlamaRL's speedup grows with model size, while its 8B model performs on par with the baseline on MATH and GSM8K.

8.1 Experiment setup

  • Unless stated otherwise, models are trained on MATH (Hendrycks et al., 2021) and evaluated on its full test split, GSM8K (Cobbe et al., 2021), and MATH-500, a 500-problem held-out subset.For fair comparison, LlamaRL and the synchronous baseline use identical optimizations, RL algorithms, datasets, training hyperparameters, evaluation protocols, and number of H100 GPUs.Source: §8.1 · pp. 17–18

8.2 Efficiency comparison

  • 2.52×, 3.98×, and 10.7× are LlamaRL's RL-step speedups over the synchronous baseline on 8B, 70B, and 405B models under equal GPU budgets.Table 3 reports the speed comparison; the experiments use 256 GPUs for 8B and 70B, and 1024 GPUs for 405B.Source: Table 3 · p. 18, §8.2 · p. 18
  • LlamaRL's relative speedup grows with model size; the authors identify flexible parallelism as the primary source of improvement.At 405B, the generator uses mp_size 8 while the bf16 trainer uses mp_size 16; the synchronous baseline requires one shared mp_size.Source: §8.2 · pp. 18–19

8 Experiments

  • With more sophisticated data mixtures, 8B and 70B training can be sporadically unstable without importance-sampling corrections; corrections stabilize training across model scales.The ablation examines the effect of off-policy corrections in asynchronous training.Source: Figure 8 · p. 19

8.3 Quality comparison to on-policy learning

  • On MATH test and GSM8K, the 8B LlamaRL model performs on par with the synchronous baseline, while MATH-500 results improve consistently during training.The authors note that the MATH-500 gain may reflect statistical variation and leave its interpretation for future work.Source: Figure 6 · p. 19, §8.3 · p. 20

9 Conclusion

LlamaRL combines asynchronous, distributed RL training with efficiency techniques for large language models. The conclusion reports theoretical and empirical speed-ups, scalability across GPU-cluster sizes, and future directions.

9 Conclusion

  • The framework combines asynchronous training and parallelization strategies with mixed precision, while DDMA supports efficient weight updates among actors.Source: §9 · p. 20
  • LlamaRL scales from small H100 clusters to thousands or more H100 GPUs as model sizes and training demands grow.Source: §9 · p. 20
  • 10.7× higher RL training throughput on 405B models, alongside a proof of strictly lower step time than synchronous RL under the same resource constraints.The experiments also indicate off-policy corrections preserve training stability and final performance.Source: §9 · p. 20
  • The authors identify advanced off-policy learning, multi-task objectives, and broader modalities as future directions.Source: §9 · p. 20

A A Comparative Discussion with PPO and GRPO

The discussion distinguishes PPO’s trust-region motivation from LlamaRL’s asynchronous off-policy setting. It compares their clipping choices and notes GRPO’s removal of critic learning.

A A Comparative Discussion with PPO and GRPO

B Lemmas for Theoretical Justification for LlamaRL Speed-up

The appendix presents supporting lemmas for the speed-up theorem, using minimization properties and related arguments. Some proof details are omitted as similar to earlier arguments.

B Lemmas for Theoretical Justification for LlamaRL Speed-up

Loading 2505.24034v2…