PaperlayerView exact revision on arXiv

Source-linked AI summary

RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation

Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, Zixiao Huang, Mingjie Wei, Yuqing Xie, Ke Yang, Bo Dai, Zhexuan Xu, Jiakun Du, Xiangyuan Wang, Xu Fu, Letong Shi, Zhihao Liu, Kang Chen, Weilin Liu, Gang Liu, Boxun Li, Jianlei Yang, Zhi Yang, Guohao Dai, Yu Wang

arXiv:2509.15965v2cs.LGcs.AIcs.DC

TL;DR

RL workflows need flexible orchestration to balance intuitive programming with efficient execution. RLinf decouples workflow logic from execution through M2Flow and supporting mechanisms, improving throughput across reasoning and embodied RL.

  • Problem

    RL workflows need flexible orchestration to align execution with component and workflow characteristics while preserving intuitive programming.

    Source: §2.3 · p. 4

  • Method

    RLinf uses M2Flow to decouple RL workflow programming from execution, combining adaptive communication, context switching, elastic pipelining, and profiling-guided scheduling.

    Source: §1 · pp. 1–2

  • Results

    RLinf improves end-to-end throughput by up to 1.7× in reasoning RL and 2.43× in embodied RL compared with state-of-the-art systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose M2Flow as a blueprint for future AI runtimes that orchestrate heterogeneous components under one unified execution framework.

    Source: §7 · p. 13

Abstract

from arXiv · show

Reinforcement learning (RL) has demonstrated immense potential in advancing artificial general intelligence, agentic intelligence, and embodied intelligence. However, the inherent heterogeneity and dynamicity of RL workflows often lead to low hardware utilization and slow training on existing systems. In this paper, we present RLinf, a high-performance RL training system based on our key observation that the major roadblock to efficient RL training lies in system flexibility. To maximize flexibility and efficiency, RLinf is built atop a novel RL system design paradigm called macro-to-micro flow transformation (M2Flow), which automatically breaks down high-level, easy-to-compose RL workflows at both the temporal and spatial dimensions, and recomposes them into optimized execution flows. Supported by RLinf worker's adaptive communication capability, we devise context switching and elastic pipelining to realize M2Flow transformation, and a profiling-guided scheduling policy to generate optimal execution plans. Extensive evaluations on both reasoning RL and embodied RL tasks demonstrate that RLinf consistently outperforms state-of-the-art systems, achieving $1.07\times-2.43\times$ speedup in end-to-end training throughput.

1 Introduction

RL training systems must handle heterogeneous, dynamic workflows, yet existing execution modes and manual scheduling struggle to fit their demands. RLinf separates workflow logic from execution planning and reports higher throughput across reasoning and embodied RL tasks.

1 Introduction

  • RL workflows combine components with different memory, dynamicity, parallelization, and compute needs, making efficient execution challenging.For example, generation lengths vary, while embodied simulation uses CPU physics and GPU rendering.Source: §1 · p. 1
  • Existing systems’ single execution modes miss workflow diversity, while flexible hybrid scheduling is difficult to support and often requires manual tuning.Collocation can leave accelerators idle due to variable generation lengths; disaggregated pipelining can create memory and computation imbalance.Source: §1 · p. 1
  • M2Flow separates logical workflow programming from physical execution planning, allowing RLinf to transform coarse-grained workflows into hardware- and workload-tailored execution plans.The system explores temporal multiplexing, spatial pipelining, and hybrid scheduling without changing program semantics.Source: §1 · pp. 1–2
  • RLinf combines flexible worker placement and adaptive communication with elastic pipelining, context switching, and profiling-guided scheduling.Together, these mechanisms support pipeline-granularity tuning, temporal accelerator multiplexing, and execution-mode selection.Source: §1 · p. 2
  • Up to 1.7× in reasoning RL and 2.43× in embodied RL: RLinf improves end-to-end throughput over state-of-the-art systems.The evaluations span diverse models and scales.Source: §1 · p. 2

2 Background and Motivation

LLM-era RL workflows span diverse algorithms and environments, but heterogeneous resource demands, dynamic rollouts, and complex dependencies make efficient execution difficult. The section motivates flexible orchestration that bridges coarse collocation with fine-grained pipelining.

2.1 RL Workflows in LLM Era

  • LLM-era RL varies across algorithms and applications: GRPO uses one LLM, RLHF/PPO uses four, and embodied or Deep Research workflows add simulator or search interactions.Embodied trajectories and Deep Research rollout results feed training; Deep Research inference follows the GRPO workflow.Source: §2.1 · pp. 2–3
  • RL components differ in memory, compute, accelerator, and parallelism needs, while fine-grained or cyclic data flows and weight-update barriers complicate coordination.Training needs more GPU memory than inference, and generation can underutilize GPU cores because it is memory-bandwidth bottlenecked.Source: §2.1 · p. 3

2.2 Inefficiencies in Diverse RL Workflows

  • <5% of responses remain unfinished, yet a few long-tail responses stall generation and leave GPUs idle; scaling out worsens idle time.This is reported for a 7B math reasoning RL experiment on 8 nodes with 8 H100 GPUs each.Source: §2.2 · p. 3
  • Pipelining lets inference and training begin from partial samples, mitigating long-tail waits but introducing a wait for the first generated batch.Neither collocated execution nor pipelining is universally optimal.Source: §2.2 · p. 3
  • Embodied RL profiles favor disaggregated simulator and generation GPUs during rollout, then CPU offloading and GPU-sharing for training.This hybrid mode combines pipelining and collocation to balance efficiency.Source: §2.2 · pp. 3–4

2.3 Flexibility as a Key to Efficiency

  • Choosing orchestration is difficult because RL components and dependencies vary, and no clear guidelines identify the best execution mode.The proposed design goal is to retain an intuitive logical workflow while allowing flexible execution modes.Source: §2.2 · p. 4, §2.3 · p. 4

3 RLinf Design

RLinf separates the logical programming of RL workflows from their physical execution, then uses flexible worker orchestration and profiling-guided scheduling to select execution flows. Its design combines spatial pipelining, temporal context switching, and adaptive communication.

3.1 Overview

  • M2Flow separates procedural workflow logic from physical scheduling, allowing RLinf to transform high-level worker interactions into execution plans across spatial and temporal dimensions.Workers communicate through the workflow while the controller orchestrates their placement and execution.Source: §3.1 · p. 4

3.2 Workflow Construction Interface

  • The interface separates worker programs defining component logic from a runner that launches workers and orchestrates their interactions.WorkerGroup calls are asynchronous, with wait primitives providing synchronization barriers; data channels decouple control and data flows.Source: §3.2 · p. 5

3.3 M2Flow Transformation

  • Elastic pipelining varies data granularity so downstream workers can begin on smaller batches, while device-lock context switching lets workers share accelerators sequentially.Dependency-aware lock priority lets children acquire resources after parents enqueue data and release the lock.Source: §3.3 · p. 6
  • Temporal scheduling runs workers sequentially on shared accelerators, spatial scheduling pipelines workers on separate devices, and hybrid scheduling combines both modes.Temporal scheduling can suffer GPU idleness from long-tail response lengths; spatial pipelining requires resource balancing so worker execution times align.Source: §3.3 · pp. 6–7

3.4 Scheduling Policy

  • The scheduler profiles component time and memory, then recursively compares temporal and spatial plans for workflow-graph partitions to select device allocations and processing granularities.Cycle-collapsed nodes are evenly partitioned across GPUs to avoid exhaustive partition enumeration while retaining near-optimal performance.Source: §3.4 · p. 7

3.5 Adaptive Communication

  • Placement-aware communication selects suitable backends and supports structured Python payloads; a load-balancing data channel decouples producer-consumer control and data flows.The channel supports CPU and GPU data and can offload GPU data to CPU to reduce GPU memory consumption.Source: §3.5 · p. 8

4 Implementation

RLinf combines reusable workers and broad algorithm and model support with flexible device allocation and profiling utilities. Workflow runners can remain concise while being scheduled across temporal and spatial execution modes.

Implementation

  • A typical LLM reasoning RL workflow runner is under 100 lines of code and needs no code changes to run with temporal or spatial scheduling.RLinf is implemented in 20K lines of Python, including 5K lines for core worker, controller, and scheduler components.Source: §4 · p. 8
  • RLinf supports reasoning, agentic, and embodied RL, with PPO, GRPO, DAPO, and REINFORCE++ algorithms and language, multimodal, and embodied models.It also supports some off-policy asynchronous algorithm versions.Source: §4 · pp. 8–9
  • RLinf uses Ray for cluster management but allocates workers to specified global device IDs, including non-accelerator devices such as robot arms.This allows workers to use any device or devices across cluster nodes.Source: §4 · p. 9
  • Worker-group timers measure remotely invoked functions and reduce process-level values with a specified method; custom timers support finer-grained profiling.Timer values can be retrieved through the asynchronous handle returned by the worker-group function.Source: §4 · p. 9

5 Evaluation

Across reasoning and embodied RL, RLinf outperforms evaluated alternatives, while the best execution mode varies by workload; its scheduling policy also selects execution modes quickly.

5 Evaluation

  • RLinf beats veRL and Slime by 1.07×∼1.70× on math-reasoning settings and outperforms embodied RL baselines across LIBERO and ManiSkill.On LIBERO, RLinf is 1.05×∼2.43× faster than SimpleVLA-RL; on ManiSkill, its hybrid mode yields up to 1.87× improvement over other strategies.Source: §5 · p. 9

5.1 End-to-End Experiments

  • End-to-end tests span reasoning and embodied workloads, with reasoning comparisons using matched rollout and training systems and throughput averaged over 10 iterations.The reasoning setup uses Qwen2.5 models from 1.5B to 32B and Qwen3-30B-A3B, with veRL v0.5 and Slime v0.1 as baselines; tests ran on a 32-node, 256-H100 cluster.Source: §5.1.1 · p. 9, §5.1 · p. 9, §5.1.2 · p. 10

5.1.1 Reasoning RL Training

  • RLinf-Temporal beats veRL by 1.10× to 1.58× on Qwen2.5 GRPO, while spatial mode underperforms veRL by 44.3–68.6% for 7B.Temporal-mode gains are attributed to larger KV-cache allocations and reduced synchronization overhead; spatial mode slows rollout and training by splitting GPUs among stages.Source: §5.1.1 · pp. 9–10, Figure 8 · p. 9
  • For Qwen2.5 PPO, RLinf-Spatial is 38.7–60.7% faster than veRL on 7B and 27.2–56.5% faster on 14B.On 1.5B, it beats veRL by 69.6%, 35.0%, and 35.6% on 16, 32, and 64 GPUs; spatial mode also beats temporal mode across the reported PPO sizes.Source: Figure 10 · p. 10, §5.1.1 · p. 10
  • On Qwen3-30B-A3B GRPO, RLinf-Spatial beats Slime-Colocate by 31.2% and 7.2% on 32 and 64 GPUs, respectively.At 128 GPUs, RLinf-Temporal is 3.7% faster than Slime-Colocate, while spatial mode underperforms because training continues for 80 seconds after rollout finishes.Source: §5.1.1 · p. 10, Figure 12 · p. 11, Figure 13 · p. 11

5.1.2 Embodied RL Training

  • On ManiSkill, RLinf-Hybrid gains 52.2–69.1% over temporal and 60.7–87.2% over spatial; on LIBERO, RLinf-Temporal is 37.8%, 42.6%, and 143.4% faster than SimpleVLA-RL.Hybrid performs worse than temporal on CPU-intensive LIBERO, whereas ManiSkill’s GPU-memory-limited environment scaling favors dedicating GPUs to environment execution.Source: §5.1.2 · pp. 10–11, Figure 14 · p. 11

5.2 Effectiveness of Search Policy

  • The search policy’s throughput estimates have under 2% error for temporal mode and under 5% for pipelined spatial mode, without changing execution-mode rankings.For three-node workflows, search remains under five seconds across 8–1024 GPUs; reported mode-selection times range from 7×10−4 to 5.98s on clusters of 8 to 1024 GPUs.Source: §5 · p. 9, §5.2 · pp. 11–12, Figure 16 · p. 11

5.3 Model Performance

  • RLinf-trained models achieve strong reasoning and embodied task results: the 1.5B model beats baselines on three benchmarks, and trained OpenVLA models improve task success.The 1.5B reasoning model improves by up to 20 points over its base model; OpenVLA exceeds RL4VLA on ManiSkill, and RL-trained OpenVLA-OFT improves on LIBERO.Source: Table 1 · p. 12, §5.3 · p. 12, Table 2 · p. 12

6 Related Works

Related work spans RL training frameworks, distributed LLM training systems, and dataflow systems; RLinf emphasizes flexible placement and coordination for dynamic RL workflows.

RL Training Frameworks

  • RL frameworks commonly colocate phases on shared GPUs or separate components across devices, while RLinf enables more flexible component-to-device placement.Examples include DeepSpeed Chat [51] and veRL [44] for task-colocated execution, and NeMo-Aligner [43] and OpenRLHF [14] for task-separated execution.Source: §6 · p. 12

Distributed Training Systems for LLM

  • Distributed LLM systems advance scaling and memory efficiency, but RL adds dynamic data generation and asynchronous updates that complicate system design.Megatron-LM [45] combines tensor, pipeline, and data parallelism; DeepSpeed uses ZeRO techniques [36] [37] [40] to shard model state.Source: §6 · p. 12

Dataflow System

  • Traditional dataflow systems rely on static task graphs and centralized scheduling, whereas RL pipelines need flexible coordination for dynamic graphs and asynchronous components.Ray [28] is described as better suited to this coordination through actor-based execution and decentralized control.Source: §6 · p. 12

7 Conclusion

The conclusion presents RLinf’s decoupling of workflow logic from execution as a route to efficiency and programmability for diverse RL workloads, and as a possible blueprint beyond RL.

7 Conclusion

  • RLinf addresses diverse and dynamic RL workflows that rigid execution models do not accommodate well.The conclusion frames RL as increasingly important to LLM progress relative to pretraining.Source: §7 · p. 13
  • Macro-to-micro transformation decouples workflow logic from execution and is presented as unlocking efficiency and programmability.Source: §7 · p. 13
  • The authors propose the approach as a blueprint for unified AI runtimes that orchestrate training, inference, simulation, and reasoning.They describe RLinf as an early step toward an operating system for AI workloads.Source: §7 · p. 13

Related papers

Loading 2509.15965v2…