ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

1 INTRODUCTION

• A typical PPO-based RLHF iteration uses four independently parameterized LLMs for generation, inference, and training tasks with data dependencies.The Actor generates responses; Reward, Reference, and Critic evaluate them, and Actor and Critic update parameters using the results.Source: §1 · p. 1
• Fixed cluster-wide parallelization can cause synchronization and communication overhead, while per-LLM allocations can leave GPUs under-utilized because tasks depend on one another.Different computational tasks can benefit from different parallelization strategies.Source: §1 · p. 2
• ReaL addresses fixed resource allocation by dynamically reallocating model parameters and tailoring parallelization to individual model function calls.The approach executes calls concurrently with a smaller parallelization degree.Source: §1 · pp. 2–3
• ReaL searches for execution plans that specify resource allocation and parallelization for each model function call, then deploys the selected plan through parallelization and parameter redistribution.The generator uses MCMC sampling and a lightweight profiling-assisted runtime estimator.Source: §1 · pp. 2–3
• ReaL’s searched execution plans surpass Megatron-based heuristic plans by 54% on average and up to 81% with a longer context length, while achieving up to 3.58× speedup over baseline systems.Source: §1 · p. 3

2.1 Introduction to RLHF

• An RLHF iteration uses six model function calls across four LLMs; generation, inference, and training have distinct computation patterns.Generation includes prefill and repeated decoding, inference is a forward pass, and training includes forward, backward, and parameter update steps.Source: §2.1 · p. 3
• PPO training can update parameters after each minibatch, unlike gradient accumulation, which performs one update across minibatches.Source: §2.1 · p. 3

2.2 Parallelization of Large Language Models

• Data parallelism replicates the model across data partitions, while tensor parallelism splits parameters and incurs per-layer all-reduce communication.Pipeline parallelism divides adjacent layers into stages; Megatron-LM combines data, tensor, and pipeline parallelism as 3D parallelism.Source: §2.2 · p. 3

• ReaL assigns suitable GPU resources and parallelization to each model function call, enabling distinct calls to run concurrently and targeting lower communication overhead.The design also targets improved GPU utilization while staying within device memory limits.Source: §3 · p. 3, Figure 4 · p. 4
• ReaL parses RLHF into a model-function-call dataflow graph, searches for a fast execution plan, then runs that plan in a worker-based runtime.Source: §3 · p. 4
• The plan generator uses MCMC and a profiling-assisted estimator; a centralized master resolves task dependencies and dispatches requests to model workers.After the search time limit, the fastest plan found is passed to the runtime engine.Source: §3 · p. 4

• The RLHF workflow is represented as a dataflow graph whose nodes are model function calls and whose edges encode data or parameter-version dependencies.The graph spans training iterations, allowing potentially overlapping computations without mutual dependencies.Source: §4 · p. 4
• An execution plan assigns each function call a device mesh and parallelization strategy, with extra graph nodes representing transfers, parameter redistribution, and offloading.Source: §4 · p. 4
• The search space restricts device meshes to full hosts or consecutive host portions and considers 3D-parallelism degrees and microbatch counts.Microbatches provide an option to avoid out-of-memory issues with large batches and context lengths.Source: §4 · pp. 4–5
• The formulation can represent RLHF algorithms whose workflows decompose into function calls and dataflow graphs, including DPO, ReMax, and GRPO.Source: §4 · p. 5

5.1 Estimation

• Profiling takes minutes, while evaluating a candidate plan with ReaL’s estimator takes hundreds of microseconds.The estimator profiles individual layers and uses their computation patterns to estimate model-function-call costs.Source: §5.1 · p. 5
• For function-call nodes, ReaL profiles forward, backward, and communication costs at power-of-two input sizes, interpolating for other sizes.Data and parameter transfer costs are simulated from data size and bandwidth rather than measured with a real NCCL operation.Source: §5.1 · p. 5
• The time estimator accounts for concurrent nodes on disjoint device meshes and prevents simultaneous execution on overlapping meshes.Its scheduling simulation derives the plan’s total time from node start and end times.Source: §5.1 · p. 5
• A plan is executable only when its peak runtime memory fits device limits; ReaL estimates static and active memory to find the peak.Source: §5.1 · p. 5

5.2 Execution Plan Search

• With over 500 options per function call in an (8, 8) cluster and over 10^16 plans overall, ReaL uses MCMC rather than brute-force enumeration.It samples plans using a cost distribution that penalizes out-of-memory plans and selects the plan with the lowest estimated TimeCost encountered.Source: §5.2 · pp. 5–6

Workers

• Model workers keep function-call data on their GPUs while a CPU master coordinates ready calls and communicates data locations to initiate transfers.Workers poll requests for local LLM handles and execute queued requests sequentially.Source: §6 · p. 6

Redistributing Parameters

• Parameter redistribution remaps layers between 3D parallelization strategies, using asynchronous host-device copies and device-device communication.For device-device remapping, pipeline-stage pairs exchange common-layer parameters; destination GPUs are assigned the lowest-cost source, whose parameters are broadcast in parallel.Source: §6 · p. 6

Data Transfer Among Function Calls

• Function calls produce disjoint data partitions along DP and replicate data along TP, so the runtime reuses the broadcast-based parameter-transfer algorithm with TP-DP dimensions reversed.Source: §6 · p. 6
• The system does not target an optimal communication algorithm for this data-transfer scenario when its cost is minor relative to other RLHF workloads.The authors refer to Figure 11 for that cost comparison.Source: §6 · p. 6

Applicability

• ReaL applies to GPT-like LLM workflows composed of training, inference, or generation function calls.Source: §7 · p. 6

Advantages

• ReaL’s method is orthogonal to techniques that accelerate individual function calls or fuse calls, which can be integrated for better performance.The paper names PagedAttention and RLHFuse as examples.Source: §7 · p. 6
• ReaL can also significantly accelerate RLHF algorithms beyond PPO, including DPO.Source: §7 · p. 6

Limitations

• An unstable cluster or a dynamic workflow, such as one with highly variable generation lengths, can violate the cost-estimation assumption.Source: §7 · p. 6

8.1 Comparison with Baselines

• REAL reaches up to 3.58× higher throughput than baseline systems; searched plans beat REAL-Heuristic by 54% on average and 81% at 8192-token context.The comparison covers throughput at different context lengths.Source: Figure 8 · p. 8, §8.1 · p. 8

8.2 Breakdown Analysis

• Similar-sized Actor and Critic training can overlap on separate devices, while a 70B Actor and 7B Critic are trained sequentially because their computational imbalance makes concurrent execution inefficient.The breakdown also reports incremental gains as generation, training, and inference optimizations are applied.Source: §8.2 · p. 8
• REAL reduces memory I/O and communication overhead by limiting redundant kernels and excessive TP or PP, and by running function calls concurrently on separate device subsets.For decoding, it prioritizes TP over PP and maximizes DP within memory constraints; training uses larger PP with many micro-batches to reduce TP communication while limiting bubble time.Source: §8.2 · pp. 8–9, Figure 10 · p. 8, Figure 11 · p. 9
• Profiling takes less than 4 minutes and reuses layer and bandwidth statistics across experiments within a model family; estimator errors stay below 25% while preserving execution-plan ordering.The estimator’s comparison is between estimated and real execution costs.Source: §8.2 · p. 9, Figure 12 · p. 9
• The search finds plans with significant throughput improvements within 150 seconds across tested configurations, and pruning lets MCMC find a fast plan within 5 minutes at 1024 GPUs.The pruning ablation uses a single-threaded implementation; multi-core parallelization could accelerate it further.Source: §8.2 · pp. 9–10, Figure 14 · p. 9
• In a 7B+7B, 8-GPU setting, MCMC reaches more than 95% of brute-force best performance in 5 seconds and finds optimal plans within 10 minutes.The comparison is limited to the stated model and GPU setting.Source: Figure 15 · p. 9, §8.2 · p. 10

8.3 RLHF Algorithms Beyond PPO

• Across DPO, GRPO, and ReMax on 16 nodes with a 70B Actor and 7B Critic, REAL improves throughput over REAL-Heuristic by 87% on average.ReMax benefits from concurrent generation calls, while GRPO’s 8× larger batch makes its workload more compute-bounded and its gains more modest.Source: §8.3 · p. 10, Figure 16 · p. 10

8.4 Strong Scaling Trend

• Strong scaling is sub-linear for 7B actors and super-linear for 34B actors; the authors recommend treating static memory utilization below 60% as a signal of diminishing returns from added GPUs.They attribute diminishing scaling returns to autoregressive generation’s sequential processing and iterative KV-cache loading, and suggest finding each configuration’s transition from super-linear to sub-linear scaling.Source: §8.4 · pp. 10–11, Figure 17 · p. 10

9.1 Systems for Training and Serving LLMs

• Distributed training and serving systems address individual workloads, while combining dependent training, inference, and generation workloads in RLHF remains challenging; prior RLHF systems often rely on inflexible hand-crafted strategies.The paper describes concurrent systems as conceptually similar, but says they overlook parameter reallocation.Source: §9.1 · p. 11

9.2 GPU Memory Management for Distributed Training

• Memory-management methods generally reduce runtime memory rather than improve throughput; parameter synchronization can underutilize GPUs, while ReaL generalizes HybridEngine beyond the Actor on one device mesh.ReaL’s parameter reallocation applies to any models in the algorithm, whether their devices are disjoint or overlapping.Source: §9.2 · p. 11

9.3 Automatic Parallelization of DL Models

• Prior automatic-parallelization systems search tensor-operator graphs, but the paper argues this misses runtime parameter reallocation and makes RLHF’s four-model search prohibitively expensive; ReaL searches at model-function-call granularity.The paper says this formulation both addresses parameter reallocation and explores a smaller solution space.Source: §9.3 · p. 11

10 CONCLUSION

• The authors present ReaL as the first system to automatically find and execute a fast RLHF execution plan with parameter reallocation.They say it may encourage development of novel LLM algorithms and broaden access to RLHF training.Source: §10 · p. 11

A EXPERIMENT DETAILS

• Experiments use a global batch of 512, context length 2048, and maximum prompt length 1024; random data with maximum prompt lengths and generation lengths supports fair comparisons.For weak scaling, model and batch sizes increase with device count; longer-context experiments keep the global token count fixed.Source: A EXPERIMENT DETAILS · p. 17

B THE API OF REAL

• Users define the algorithm’s dataflow graph with ModelFunctionCallDef objects and wrap experiment configuration in the auto decorator to start plan search.The plan is turned into a worker scheduling configuration; users may provide distinct interface implementations for different workflows.Source: B THE API OF REAL · p. 17, Figure 18 · p. 17

C SIMULATION ALGORITHM

• The appendix states that the simulation algorithm is presented in Algorithm 1.Source: C SIMULATION ALGORITHM · p. 17

D BASELINES

• The experiments compare ReaL with DeepSpeedChat, OpenRLHF, NeMoAligner, and concurrent system veRL; ReaL uses its own generation backend and model and pipeline parallelization.Table 7 lists baseline versions and backends; ReaL also uses Megatron-LM tensor parallelization and optimizer implementation.Source: D BASELINES · pp. 17, 19, Table 7 · p. 20
• DeepSpeedChat uses tensor parallelism for generation, ZeRO-3 data parallelism for training and inference, and HybridEngine to reshard parameters between actor training and generation.It supports sequential execution of model function calls.Source: D BASELINES · p. 17
• OpenRLHF separates actor/reference, critic/reward, and generation GPU groups; actor and critic training can overlap, but generation cannot overlap with training.The text attributes significant GPU idle time to the generation-training dependency.Source: D BASELINES · pp. 17, 19
• Table 6 reports reduced end-to-end time for ReaL through faster model calls and concurrent execution of independent computations.The table covers two common representative cases; the supplied text gives no wall-time values.Source: Table 6 · p. 19
• NeMoAligner places actor training and generation on the same GPU group and pipelines micro-batches to reduce GPU idle time.It uses TRT-LLM for generation and Megatron-LM for training.Source: D BASELINES · p. 19
• veRL supports both model colocation and split placement, with multiple generation and training backends for different parallelization strategies.Its listed options include SGLang or vLLM for generation and Megatron-LM or PyTorch FSDP for training.Source: D BASELINES · p. 19