PaperlayerView exact revision on arXiv

Source-linked AI summary

Nereus: Adaptive Parallelism for LLM Post-Training

Songlin Jiang, Tuo Shi, Sitong Zhang, Zeke Wang, Mario Di Francesco, Bo Zhao

arXiv:2609.34645v1cs.DCcs.AI

TL;DR

RL post-training jobs coordinate models across stages, but drift can make fixed plans inefficient or infeasible, and adapting them requires decisions about payback, reusable state, and GPU-transfer ordering. Nereus combines cost-aware global planning with Elastic Model Units and a transition DAG. It reports throughput gains over OpenRLHF and Verl, alongside lower step latency in a real-data trace and low transition overhead in a large run.

  • Problem

    Resource and workload drift can make RL post-training plans inefficient or infeasible, while coupled models complicate adaptation decisions about transition cost, state reuse, and GPU transfers.

    Source: §1 · p. 1

  • Method

    Nereus selects memory-feasible global plans with an online-calibrated cost model, represents model-stage replica state as Elastic Model Units, and orders transitions in a global DAG.

    Source: §1 · p. 2

  • Results

    Nereus improves end-to-end 8B PPO throughput by 2.14–7.27× over OpenRLHF and 1.10–1.47× over Verl across evaluated clusters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In a real-data trace, online TP/PP adaptation lowers average step latency by 27.7% versus the initial fixed TP/PP layout with DP scaling, while six transitions in a 1,000-step run reaching 1,024 GPUs consume 0.079% of run time.

    Source: §1 · p. 2

    Verl was evaluated only on NVIDIA GPUs because it did not support the AMD MI250X setup at the time of writing.

    Source: §7.2.1 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job's distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14--7.27$\times$ over OpenRLHF and by 1.10--1.47$\times$ over Verl across diverse clusters.

1 Introduction

RL post-training coordinates multiple models across stages, but resource and workload drift can make a fixed execution plan inefficient or infeasible. Nereus addresses when to adapt, what state to reuse, and how to coordinate transitions, with reported throughput and latency gains.

1 Introduction

  • RL post-training coordinates an actor with critic, reward, and reference models across generation, inference, and training, while each model-stage receives a GPU and parallelism assignment.An execution plan also determines state placement, communication, and division of work.Source: §1 · p. 1
  • Resource volatility, sequence growth, memory pressure, and shifting stage bottlenecks can make a suitable plan slow or infeasible during a run.The passages report a 16× increase in generated sequence length within 1,000 steps and describe the resulting memory and bottleneck shifts.Source: §1 · p. 1
  • Existing systems leave open how to choose a worthwhile global adaptation, reuse shared logical state despite changed sharding, and order cross-stage GPU transfers safely.Coarse state units can transfer redundant state, while fine-grained units impose complex shard-level constraints.Source: §1 · pp. 1–2
  • Nereus combines a cost-aware controller, dependency-aligned Elastic Model Units, and a global transition DAG to adapt plans without reconstructing the full job state.The DAG orders transformations and GPU transfers, including resource dependencies caused by transient GPU overlap.Source: §1 · p. 2, Figure 1 · p. 2
  • Nereus improves end-to-end 8B PPO throughput by 2.14–7.27× over OpenRLHF and by 1.10–1.47× over Verl across evaluated clusters.In a real-data trace, online TP/PP adaptation reduces average step latency by 27.7% versus the initial fixed TP/PP layout with DP scaling; six transitions in a 1,000-step run reaching 1,024 GPUs consume 0.079% of total run time.Source: §1 · p. 2

2 Design Overview

Nereus separates target-plan selection and transition admission from transition execution. It represents distributed state in model-stage units and orders their transformations and transfers globally.

2 Design Overview

  • The controller monitors runtime conditions, selects the memory-feasible plan with the lowest predicted steady-state step latency, and admits a transition when savings repay its estimated cost.Admission also checks feasibility; the transition engine estimates the one-time cost.Source: §2 · p. 3
  • Each Elastic Model Unit contains one model-stage replica's tightly coupled TP/PP layout and state, while its number of replicas determines the DP degree.The representation supports reuse of GPU-resident state across plan changes.Source: §2 · p. 3
  • The transition engine converts plan differences into Split, Merge, Extend, and Destroy primitives, orders their dependencies in a global DAG, and executes the DAG after admission.The overview illustrates sequence-length-driven resharding and replica changes from a GPU-count change.Source: §2 · p. 3

3 Background and Design Space

RL post-training couples model-stages through shared resource assignments and workflow dependencies, so changes in resources, workload, or hardware efficiency can alter the best global execution plan. Online adaptation must jointly decide when to change plans, what state to reuse, and how to execute the change.

3.1 RL Post-Training Execution Model

  • PPO coordinates actor, reward, critic, and reference models across generation, inference, and training, with distinct model-stage compute and memory needs.DP runs replicas, TP partitions tensors within layers, and PP partitions layers into pipeline stages; changing a model-stage's layout affects state resharding and overall bottlenecks.Source: §3.1 · p. 3
  • A valid transition must respect transient GPU ownership because, without free GPUs, one model-stage may need to release GPUs before another expands.A global transition DAG encodes these cross-stage resource dependencies.Source: §3.1 · p. 3

3.2 Sources of Drift

  • Resource availability, sequence length, and achieved hardware efficiency can each change which plan is best or feasible during a run.AWS traces showed more than ten availability changes for a 16-GPU job over ten hours; for Llama-3.1-8B, sequence length grew from 500 to 8,000 tokens within 1,000 steps, changing the fastest feasible training layout at 16 GPUs.Source: §3.2 · pp. 3–4
  • Drift affects model-stages differently, requiring whole-job adaptation that accounts for different resharding directions and available memory headroom.Longer sequences prompt Merge in generation and inference and Split in training; training can lower TP through Split only when it has memory headroom.Source: §3.2 · p. 4

3.3 Design Space for Online Adaptation

  • Online adaptation links global feasibility and payback, state-boundary selection, and transition execution into three coupled design decisions.Nereus addresses them with a cost-aware policy, dependency-aligned state abstraction, and safe concurrent transition orchestration.Source: §3.3 · p. 4

4 Cost-Aware Adaptation Policy

Nereus monitors drift, searches for a memory- and resource-feasible global plan, then admits transitions according to urgency or predicted payback. Its cost model is calibrated against the running job and estimates both step latency and transition cost.

4 Cost-Aware Adaptation Policy

  • The controller replans on resource or memory events and threshold crossings, admitting an executable transition when the current plan is infeasible or savings repay transition cost.The opportunistic admission rule compares predicted per-step latency savings against transition cost and the time until the triggering signal is expected to cross its threshold again.Source: §4 · p. 4, §4.3 · p. 6

4.1 Monitor Phase: Detecting Runtime Drift

  • The monitor tracks sequence length, GPU availability, peak memory, and achieved compute and communication efficiency to detect drift and calibrate the cost model.GPU-pool changes or predicted memory violations trigger replanning; drifting signals trigger when they exceed relative thresholds from their reference values.Source: §4.1 · p. 4

4.2 Replan Phase: Selecting a Target Plan

  • The planner selects the lowest-predicted-latency global plan subject to per-GPU memory capacity and stage-wise GPU-pool constraints.Its cost model estimates computation, pipeline bubbles, collectives, and DP synchronization, then uses the critical path of the RL workflow for end-to-end step latency.Source: §4.2 · p. 5
  • Online calibration refines estimates of operation-class efficiency from measured execution, while plan search filters infeasible or dominated candidates before selecting a global plan.Peak-memory estimates cover parameters, gradients, optimizer state, activations, KV cache, co-resident model-stages, and workspaces; feasibility checks use 10% headroom.Source: §4.2 · p. 5

4.3 Act Phase: Admitting a Transition

  • Nereus checks whether the target plan and transition DAG can fit retained state, workspaces, intermediate units, co-resident state, and collective buffers before moving state.If the current plan is invalidated, the urgent path waives profitability and may use smaller batches or recomputation, falling back to checkpoint/restart if no candidate passes.Source: §4.3 · p. 5
  • For opportunistic transitions, admission requires predicted per-step savings to repay the full transition cost within a confidence-adjusted estimate of time before conditions change again.The policy uses the full-job savings and complete transition cost; the confidence margin reserves time for uncertainty.Source: §4.3 · p. 6

5 Dependency-Aligned State Abstraction (EMU)

Nereus represents each model-stage replica as an Elastic Model Unit (EMU), keeping tightly coupled TP/PP state inside the unit while exposing DP replicas for adaptation. Four primitives transform EMUs to reshard state or scale resources.

5 Dependency-Aligned State Abstraction (EMU)

  • An EMU is one model-stage replica containing its TP/PP layout, GPU set, parameter shards, and optimizer shards; a model-stage’s DP degree is its number of active EMUs.The planner reasons about whole units while primitives operate on the underlying shards.Source: §5 · p. 6

5.1 Why the Model-Stage Boundary

  • Keeping TP/PP inside an EMU preserves tightly synchronized layouts, while leaving DP outside allows replicas to be created, removed, or reassigned independently.TP/PP ranks exchange activations and gradients; DP replicas synchronize periodically.Source: §5.1 · p. 6
  • When sequence length grows and GPUs become available, actor generation can increase TP for memory while critic training increases DP to use the added GPUs.In the example, the actor changes from (TP=1, PP=4, DP=2) to (TP=2, PP=4, DP=2), while the critic changes to (TP=1, PP=4, DP=4).Source: §5.1 · p. 7

5.2 Core EMU Primitives for Online Adaptation

  • Split and Merge reshard state between TP/PP layouts, while Extend and Destroy create or remove DP replicas; together, the four primitives transform EMU collections.Split partitions one unit, Merge fuses replicas, Extend copies a unit to new GPUs, and Destroy removes a redundant copy.Source: §5.2 · p. 7
  • Compositions of the primitives cover the plan space, which uses homogeneous TP/PP layouts within each model-stage and searches power-of-two TP/PP degrees.Compositions may require intermediate memory and a schedule that retains each model-stage’s logical state.Source: §5.2 · p. 7

6 Safe Concurrent Transition Orchestration

Nereus translates each model-stage’s plan change into a local transition DAG, then adds cross-stage dependencies when GPU overlap blocks resource acquisition. The global DAG orders safe concurrent execution while preserving logical training state.

6 Safe Concurrent Transition Orchestration

  • Nereus creates one local transition DAG per model-stage, then adds cross-model-stage dependency edges when transient GPU overlap blocks an acquisition.The global DAG’s critical-path cost is used for transition admission and estimated from bytes, link bandwidth, and communicator setup.Source: §6 · p. 7, Figure 6 · p. 8

6.1 Decomposition and DAG Construction

  • Each local DAG follows Split, Destroy, Extend, then Merge, using the largest common intermediate TP/PP layout before reaching the target layout and DP degree.A unit with no other replica is destroyed only after Extend copies it.Source: §6.1 · p. 8
  • Global ordering prioritizes resource release, breaks ties by generation → inference → training, and avoids circular waits by excluding destroys that depend on a blocked acquisition.If no suitable Destroy can release the needed GPUs, the target plan fails the feasibility check.Source: §6.1 · p. 8

6.2 DAG Execution and Safety

  • Ready primitives run concurrently when predecessors and resource requirements are satisfied, with GPU collectives and memory lifetimes coordinated by the DAG.The per-GPU feasibility check ensures retained state, intermediate units, and collective buffers fit during each operation.Source: §6.2 · p. 8
  • At safe boundaries, Split and Merge preserve logical state, while Extend and Destroy copy or remove redundant replicas without losing the retained state.DP scaling adjusts per-replica batch size and gradient accumulation to preserve the global batch size and update rule.Source: §6.2 · p. 9

7 Evaluation

The evaluation measures Nereus against synchronous and asynchronous RL systems across GPU clusters, workloads, and execution modes. It tests end-to-end performance, cost-aware adaptation, transition orchestration, and training behavior.

7.1 Experimental Setup

  • Comparisons use matched models, prompts, sampling parameters, response limits, and GPU budgets, across three clusters and synchronous and asynchronous baselines.The evaluation also tests Qwen3-14B, Qwen3-32B, and Llama-3.3-70B, plus ReMax and GRPO; most latency and throughput values average 50 steps after warm-up.Source: §7.1 · p. 9, §7 · p. 9

7.2 End-to-End Performance

  • End-to-end 8B PPO throughput improves 2.14–7.27× over OpenRLHF and 1.10–1.47× over Verl across clusters and scales.Nereus also outperforms Verl for model sizes from 8B to 70B on Cluster #3.Source: Figure 7 · p. 9, §7.2.1 · p. 10, §7.2.2 · p. 10

7.3 Cost-Aware Adaptation Policy

  • On 72 out-of-sample plans, the latency model has 4.96% mean absolute percentage error; target plans match the empirical optimum in 61.1% of settings and stay within 5% in all settings.The model’s maximum error is 14.61%; replanning overhead ranges from 0.17 ms at 32 GPUs to 338 ms at 1,024 GPUs.Source: §7.3.1 · p. 10, §7.3.2 · p. 10

7.3.1 Online Cost-Model Calibration and Overhead.

  • On the first held-out drift trace, Nereus lowers average step latency 27.7% versus fixed TP/PP, 8.3% versus DynaRL-style admission, and 2.5% versus 3-step adaptation.Across three held-out traces, it averages 858.7 s/step versus 873.8 s for 3-step adaptation and 928.3 s for DynaRL-style admission.Source: §7.3.3 · pp. 10–11, Figure 11 · p. 11

7.4 EMUs and Transition Orchestration

  • Six TP/PP transitions in a 1,000-step PPO run reaching 1,024 GPUs consume 0.079% of total runtime; for a 70B model on 64 GPUs, resharding costs fall 99.1% versus MCP and 96.7% versus Tenplex.The 8B resharding comparison averages Split/Merge transitions over the same feasible TP/PP layouts at each GPU count.Source: §7.4.1 · p. 11, §7.4.2 · p. 11, Figure 12 · p. 11
  • At 1,024 GPUs, transition-DAG planning takes 1.22 ms versus SCIP’s 120.10 s; resource-dependency coordination succeeds in all trials versus 34–62% for independent local DAGs.Nereus’s transition-completion-time gap to SCIP stays within 6.9% at every scale.Source: §7.4.3 · pp. 11–12, Figure 13 · p. 11, §7.4.4 · p. 12

7.5.1 Generality Across Algorithms and Execution

  • On Cluster #3, Nereus outperforms Verl at every GPU count, reducing latency by up to 13.3% for ReMax and 15.8% for GRPO, and by up to 37.3% versus Laminar in asynchronous RL.The asynchronous setting adapts at weight-synchronization boundaries.Source: §7.5.1 · p. 12, Figure 14 · p. 12
  • Over 50 steps, Nereus’s reward and PPO KL estimate track Verl’s closely, while the run including four transitions completes 13.9% sooner.At step 50, reward is 0.9199 versus 0.9160, and mean absolute PPO KL estimate is 4.9 × 10^-5 versus 7.4 × 10^-5.Source: §7.5.2 · p. 12, Figure 15 · p. 12

8 Related Work

Nereus adapts RL post-training across steps and changing GPU pools, coordinating state and transitions across model-stages. Its role complements transfer engines, which provide data-movement paths.

8 Related Work

  • Nereus adapts across steps and pool changes with payback-based admission and coordinated transitions, unlike DynaRL’s fixed-pool, within-step reallocations.StreamRL adds generation replicas online but restarts to reconfigure trainers.Source: §8 · p. 12
  • Prior state-management systems reuse GPU-resident state for single models, while Nereus coordinates state per model-stage replica across stages sharing GPUs.Nereus’s model-stage boundary distinguishes its multi-model, multi-stage coordination.Source: §8 · pp. 12–13
  • Fabric-lib provides RDMA point-to-point transfer paths; Nereus complements it by deciding which state moves and in what order.The cited transfer uses include KV cache transfer, MoE dispatch, and RL weight synchronization.Source: §8 · p. 13

9 Conclusion

Nereus aligns state boundaries with job dependencies to adapt execution plans. It reports higher 8B PPO throughput than OpenRLHF and Verl, alongside lower trace latency and low transition overhead.

9 Conclusion

  • 2.14–7.27× higher 8B PPO throughput over OpenRLHF and 1.10–1.47× over Verl across evaluated clusters.The gains also extend to ReMax, GRPO, and asynchronous RL.Source: §9 · p. 13
  • 27.7% lower average step latency than the initial fixed TP/PP layout in a real-data trace; selected plans stay within 5% of the empirical optimum in all 18 measured settings.The trace comparison uses the initial fixed TP/PP layout as its baseline.Source: §9 · p. 13
  • Across three held-out traces, Nereus averages 858.7 s/step versus 928.3 s for DynaRL-style admission.Source: §9 · p. 13
  • Six transitions consume 0.079% of run time in a 1,000-step run reaching 1,024 GPUs.Source: §9 · p. 13
  • Scaling is 3.8–16.2× faster than Oobleck and Tenplex, and coordinated transitions succeed in all overlap trials versus 34–62% for DynaRL’s per-component migration.Source: §9 · p. 13
  • The model-stage boundary and modular runtime provide a path toward context/expert parallelism, autoscaling external tool services, and additional RL frameworks.Source: §9 · p. 13

Related papers

Loading 2609.34645v1…