Performance Foundations of Parallel & Distributed Reasoning Language Models

1 INTRODUCTION

• RLM training has enormous computational costs, with o3 scaling train-time RL and inference-time reasoning compute by an additional order of magnitude while still benefiting from further scaling.Frontier training runs may already cost tens to hundreds of millions of US dollars in compute and could exceed $1B by 2027.Source: §1 · p. 1
• PPO and GRPO are central RLM training frameworks: GRPO reduces memory pressure by removing the critic, whereas PPO retains value-based variance reduction and token-level credit assignment.PPO-style pipelines maintain actor, reward, critic, and reference models across Generation, Assessment, and Training stages; PPO has also been used in reasoning-focused systems such as VAPO [268].Source: §1 · p. 1
• RLM pipelines couple multiple large models and heterogeneous stages, so effective training requires joint reasoning about model placement, inter-model scheduling, and workload characteristics rather than reusing supervised-training parallelization naively.This combination of scale and multi-model dependencies makes RLMs fundamentally a parallel and distributed systems problem.Source: §1 · p. 1
• The paper’s taxonomy covers intra-model data, tensor, pipeline, sequence, context, and expert parallelism, plus inter-model placement, stage fusion, hybrid configurations, and asynchronous execution.These strategies target component concurrency, stage overlap, reduced redundant execution, and bounded-staleness decoupling of rollout generation from learning.Source: §1 · p. 2
• The study analyzes general-purpose and specialized RLM frameworks to catalogue supported parallelism techniques, identify systematic gaps, and guide framework selection and future optimizations.The comparison includes systems such as TRL, OpenRLHF, HybridFlow, NeMo RL, ReaL, RLHFuse, and AReaL.Source: §1 · p. 2

1.1 Complementary Analyses & Related Work

• Unlike prior surveys and analyses focused on general RL, reasoning scaling, resource efficiency, or individual pipeline components, this work provides an in-depth account of the performance foundations enabling parallel and distributed RLM scaling.Its formal work-depth analysis defines work as total operations, depth as the longest sequential execution chain, and memory as parameters, gradients, optimizer state, and stored activations.Source: §1.1 · p. 2, §1.2 · p. 3

2.1 RL Post-Training Pipeline: Overview

• The RL post-training loop repeatedly generates responses, assesses them, and trains the policy using feedback until convergence or a preset number of iterations.Assessment and Training may invoke reward, critic, and reference models, while each updated policy initializes the next cycle.Source: §2.1 · p. 3

2.2 Terminology: RLVR, RLHF, RLAIF & Others

• RL-style reasoning post-training encompasses RLVR, RLHF, RLAIF, d-RLAIF, learned-reward methods, and offline preference optimization because they share parts of the same staged dataflow.Online methods execute the full loop, whereas DPO removes online generation and explicit reward assessment but retains policy/reference comparison and training.Source: §2.2 · p. 3
• Strict RLVR uses automatic verifier rewards for checks such as exact answers, unit tests, compilation, and theorem proving, whereas RLAIF variants use AI-generated judgments.Learned reward models support reasoning but can be more compute-heavy than lightweight exact-match RLVR, especially with process-level feedback.Source: §2.2 · p. 3
• Offline DPO-style reasoning methods retain policy/reference forward passes and policy backpropagation while optimizing static preference pairs without online rollouts or explicit reward-model inference.Reasoning adaptations include iterative reasoning preference optimization and step-wise DPO [134].Source: §2.2 · p. 3

2.3 Auxiliary Models in the RL-LLM Pipeline

• The pipeline combines a policy with auxiliary reward, critic, and reference models that have distinct execution and optimization roles despite often sharing LLM-based architectures.These models are conceptually distinct but practically similar in architecture.Source: §2.3 · p. 4

2.3.1 Reward Model

• Reward models assign scalar quality scores to complete responses in a frozen, typically non-autoregressive pass, while process models provide intermediate feedback at substantially higher cost.Process-based rewards may require multiple forward passes or Monte Carlo Tree Search and are difficult to train [26].Source: §2.3.1 · p. 4

2.3.2 Critic Model

• The critic estimates expected future rewards for every token position, combines with observed rewards to compute advantages, and is updated as the policy changes.Unlike the fixed reward model, the trainable critic tracks the return function induced by the evolving policy through value regression.Source: §2.3.2 · p. 4

2.3.3 Reference Model

• The reference model is a frozen policy baseline, typically copied from the post-SFT policy before reinforcement learning, that stabilizes optimization.Source: §2.3.3 · p. 4

• Reasoning-oriented post-training pipelines extend beyond next-token prediction by using feedback within a shared systems abstraction.Source: Classes of schemes · p. 5
• Offline preference optimization trains on preference pairs, whereas online reinforcement learning generates rewards during optimization.Source: Classes of schemes · p. 5
• Reinforcement Learning with Verifiable Rewards uses an automatic verifier to produce rewards, while related approaches may rely on a learned reward model or critic.Source: Classes of schemes · p. 5

2.4 Auxiliary Models vs. Algorithmic RL Frameworks

• PPO uses actor, critic, reward, and reference models; GRPO removes the critic, while DPO uses only policy and reference models with offline preference training.These model combinations determine computational cost, parallelism potential, and system-level design.Source: §2.4 · p. 5

2.4.1 Variants and Emerging Frameworks

• Online methods are dominated by rollout generation, critic-free methods remove Vψ-dependent costs, and offline preference methods trade exploration for cheaper batched training.PPO variants retain actor–critic computation, GRPO variants retain critic-free online computation, and DPO variants preserve an offline teacher-forced profile.Source: §2.4.1 · pp. 5–6

2.5 Computational Analysis of RL-LLM

• PPO is the most expensive online method; GRPO removes critic work and memory but retains autoregressive generation depth, while DPO removes online generation and explicit reward or critic evaluation.DPO therefore has an inner-loop depth closer to standard SFT, but it cannot directly explore behavior outside its fixed preference dataset.Source: §2.5 · p. 6

2.6 Key Insights & Takeaways

• PPO and GRPO retain an irreducible token-by-token dependency within each rollout, making efficient decoding kernels central to reducing latency and increasing generation throughput.Inter-stage fusion can overlap assessment with ongoing generation, trading additional memory for reduced pipeline idle time.Source: §2.6 · p. 6
• Frozen reward and reference models can be replicated, quantized, offloaded, or independently served, while process rewards increase assessment cost relative to scalar outcome rewards.Trainable policy and critic models instead dominate memory through activations, gradients, and optimizer state.Source: §2.6 · pp. 6, 8

3 INTRA-MODEL PARALLELISM

• Intra-model parallelism combines data, tensor, sequence/context, pipeline, and expert parallelism with memory optimizations to balance footprint, communication, and utilization.Hybrid schemes such as ZeRO-style sharded data parallelism plus tensor parallelism support these tradeoffs [219].Source: §3 · p. 8

3.1 Data Parallelism

• Naive data parallelism becomes memory-limited for very large models because each device stores full parameters, gradients, and optimizer state, while all-reduce overhead increases with device count or limited bandwidth.ZeRO/FSDP partially shard parameters, gradients, and optimizer state, while communication overlap, gradient bucketing, and gradient accumulation provide communication optimizations.Source: §3.1 · p. 8

3.1.1 Implications for RL-LLM Pipelines

• RL-LLM pipelines typically use DP plus ZeRO for trainable components and pure DP for forward-only components.Forward-only actor, reward, and reference models can use replicated batched inference, whereas actor and critic training benefit from ZeRO-2/3 or FSDP.Source: §3.1.1 · p. 9

3.2 Tensor Parallelism

• Tensor parallelism shards layer dimensions across devices and reconciles partial results through all-reduce or all-gather, commonly partitioning Transformer FFNs and attention heads.Combined with data and sometimes pipeline parallelism, TP forms higher-dimensional configurations for training trillion-parameter models [219].Source: §3.2 · p. 9

• For frozen models, data parallelism becomes batched inference without gradient All-Reduce, enabling high throughput during Generation & Assessment.Source: Data parallelism (DP) · p. 12
• Replicated weights and optimizer states create severe memory bottlenecks for large models, mitigated by combining data parallelism with ZeRO and FSDP state sharding.Trainable RLM pipelines typically use DP + ZeRO during the Training stage to optimize memory.Source: Data parallelism (DP) · p. 12
• Training All-Reduce can bottleneck execution, while gradient bucketing overlaps backward computation with communication; inference requires no communication.Source: Data parallelism (DP) · p. 12
• Data parallelism replicates weights, shards the global input batch, and synchronizes gradients across devices with All-Reduce after the backward pass.Source: Data parallelism (DP) · p. 12

• Mode-aware pipeline parallelism uses tailored configurations for heterogeneous components.Source: Pipeline parallelism (PP) · p. 12
• Pipeline parallelism distributes parameters, gradients, and optimizer states across stages, reducing per-device memory pressure so deep models fit in aggregate HBM.Source: Pipeline parallelism (PP) · p. 12
• Complete-sequence assessment and training benefit from standard micro-batched pipeline schedules, while sequential stage dependencies can leave hardware idle.Source: Pipeline parallelism (PP) · p. 12
• Adjacent-stage point-to-point transfers send activations and gradients without requiring global collectives.Source: Pipeline parallelism (PP) · p. 12
• Generation retains the O(T) critical path because tokens traverse stages sequentially, and incomplete sequences cause bubbles under naive schedules.Schedules such as 1F1B, Zero Bubble, and DualPipe interleave forward and backward passes to improve hardware utilization.Source: Pipeline parallelism (PP) · p. 12

• Partial results are aggregated during the forward pass, requiring four All-Reduce collectives per forward-backward pass.High-performance runtimes overlap these communication steps with chunked matrix computations using custom kernels.Source: Tensor parallelism (TP) · p. 12
• Frequent collectives constrain TP groups to high-bandwidth intra-node interconnects and can severely bottleneck throughput during inference.Consequently, TP should be avoided in Generation & Assessment stages whenever memory permits.Source: Tensor parallelism (TP) · p. 12
• TP shards weight matrices along hidden dimensions across MHA and FFN layers, reducing each rank’s memory footprint in proportion to TP size.This includes column-wise splitting of QKV, gate, and up-projection matrices and row-wise splitting of output projections.Source: Tensor parallelism (TP) · p. 12

Expert parallelism (EP)

• Expert parallelism splits weights row-wise across ranks.Source: Expert parallelism (EP) · p. 12
• During training, gradients inflate memory by 3x, making tensor sharding necessary.Source: Expert parallelism (EP) · p. 12
• Figure 6 presents intra-model parallelism details for data, pipeline, tensor, and expert parallelism across each RLM model execution.Source: Figure 6 · p. 12

Sequence parallelism (SP)

• Sequence parallelism reduces activation memory linearly with the tensor-parallel group size by sharding norm and dropout activations while retaining tensor-parallelized MHA and FFN operations.TP+SP replaces all-reduces with all-gather and reduce-scatter pairs, yielding zero net communication overhead in the stated model.Source: Sequence parallelism (SP) · p. 13

3.3.1 Implications for RL-LLM Pipelines

• SP reduces per-GPU memory for long policy and critic updates, enabling longer reasoning traces or larger batches under fixed memory budgets.Its benefits are concentrated in training stages where activations must be stored or recomputed for every token.Source: Sequence parallelism (SP) · p. 13, §3.3.1 · p. 14

3.4 Context Parallelism

• Context parallelism extends sequence partitioning through attention and FFN blocks, enabling training and inference with contexts of hundreds of thousands or millions of tokens through cross-device KV exchange.Ring Attention pipelines key/value communication and blockwise attention to reduce latency and memory pressure.Source: §3.4 · p. 14, §3.4.1 · p. 14

3.4.1 Implications for RL-LLM Pipelines

• Context parallelism reduces per-device context memory but does not eliminate sequential online generation, so rollout latency still requires complementary communication, batching, or overlap techniques.It is therefore primarily a feasibility and long-context scaling mechanism for long-CoT, tool, verifier, and interaction-history contexts.Source: §3.4.1 · p. 14, §3.2.1 · p. 14

3.5.1 Implications for RL-LLM Pipelines

• Pipeline parallelism fits deep models by partitioning layers across stages, but Generation retains an O(T) sequential critical path and can suffer pipeline bubbles, whereas Assessment and Training benefit more directly from micro-batched schedules.Mode-aware configurations and inter-batch pipelining can overlap generation, assessment, and training, while asynchronous schedules may introduce destabilizing weight staleness.Source: §3.5.1 · p. 15

3.6.1 Implications for RL-LLM Pipelines

• Expert parallelism distributes MoE experts across devices, reducing per-token compute and enabling large capacity, but its gains can be erased by all-to-all communication, routing imbalance, and stragglers.RL post-training makes routing harder because evolving token distributions can overload experts, motivating dynamic or bias-based balancing mechanisms.Source: §3.6 · p. 15, §3.6.1 · p. 15

3.7 Complexity Analysis

• The work–depth–memory analysis assigns distinct roles to parallelism: DP favors throughput when models fit per device, TP and PP support large or low-latency invocations, CP targets long contexts, and 3D/5D sharding is most important during training.The framework reports both per-rank and global costs while abstracting away communication, synchronization, scheduling, utilization, and load imbalance.Source: §3.7 · p. 16

3.7.1 ZeRO & FSDP

• ZeRO-1, ZeRO-2, and ZeRO-3/FSDP progressively shard optimizer states, gradients, and parameters, reducing the standard data-parallel model-state footprint from approximately 4P to approximately 4P/Pz elements.The expressions are P + P + 2P/Pz for ZeRO-1, P + 3P/Pz for ZeRO-2, and approximately 4P/Pz for ZeRO-3/FSDP, excluding transient all-gather buffers.Source: §3.7.1 · p. 16

3.7.2 Activation Checkpointing

• Activation checkpointing replaces MAct with κ_ckptMAct, 0 < κ_ckpt < 1, while increasing training work and depth through recomputation during backpropagation.The trade-off is most relevant to actor and critic training on long reasoning trajectories.Source: §3.7.2 · p. 16

3.8 Key Insights & Takeaways

• Stage-aware hybridization is essential because generation is autoregressive and latency-sensitive, assessment is forward-only and throughput-oriented, and training is memory-intensive.These differing bottlenecks motivate specialized parallelism rather than one uniform intra-model strategy.Source: §3.8 · p. 16
• Data parallelism increases prompt, completion, or preference-pair throughput but does not shorten the token-by-token critical path of an individual rollout or model invocation.Each replica still executes the full autoregressive chain D_gen(π_θ; S, T) = O(TD_f(π_θ; S + T)).Source: §3.8 · pp. 16, 18
• Sequence and context parallelism partition long sequences to address activation and attention memory bottlenecks from reasoning traces, tool histories, retrieved documents, and multi-turn contexts.They are especially useful for long-context assessment and training, and for generation when rollout context exceeds single-device memory.Source: §3.8 · p. 18
• Frozen reward and reference models often favor replicated batched inference, whereas trainable actors and critics benefit from optimizer-state sharding, tensor or pipeline parallelism, and 3D/5D combinations.PPO is the most demanding case because it may train both π_θ and V_ψ, while GRPO and DPO train only π_θ.Source: §3.8 · p. 18
• Expert parallelism can raise reasoning capacity with limited per-token compute, but routing all-to-all communication and policy-driven load imbalance require routing- and communication-aware design.Dense attention remains on the critical path, so expert parallelism does not eliminate all system bottlenecks.Source: §3.8 · p. 18
• Practical systems should specialize layouts by model and stage: generation prioritizes KV-cache memory and rollout latency, assessment favors cheap replicated inference, and training may require 3D parallelism.This specialization motivates inter-model placement, stage fusion, hybrid execution, and asynchronous strategies.Source: §3.8 · p. 18

4.1 Model Structure Configurations

• Model structure trades memory and work against optimization coupling: shared actor–critic saves backbone, optimizer, gradient, and activation resources, whereas independent models avoid gradient interference and permit critic downsizing.Critic-free methods such as GRPO remove the value model and use group-relative advantages, changing both the algorithm and systems profile [107,213].Source: §4.1 · p. 18, Model placement strategies · p. 20

4.3 Stage Fusion

• RLHFuse combines inter-stage and intra-stage fusion, improving throughput by up to 3.7× by mitigating long-tail generation skew and pipeline bubbles.Fusion streams completed samples into assessment and interleaves training microbatches, reducing depth and idle time without reducing total work.Source: §4.3 · p. 21

4.4 Hybrid Parallelism and Adaptive Scheduling

• Hybrid execution assigns different layouts to generation and training, but it is profitable only when specialized-layout gains exceed resharding, communication, memory, and orchestration costs.Examples include ReaL parameter reallocation, HybridFlow/verl’s 3D-HybridEngine, and NeMo RL’s distinct pipeline- and tensor-parallel modes.Source: §4.4 · p. 22

4.5 Asynchronous Execution

• Asynchronous execution overlaps iterations through bounded-stale parameter reads, improving utilization but introducing policy drift and potentially requiring a persistent inference-side shadow copy.AReaL uses staleness-aware PPO and separates behavior-policy correction from the proximal trust-region center.Source: §4.5 · p. 22

4.6 Complexity Analysis

• The strongest combined configuration overlaps disaggregated placement, stage fusion, bounded asynchrony, and intra-model sharding, but cannot remove the autoregressive decode chain in actor generation.Reducing that bottleneck requires inference parallelism, faster decoding kernels, batching, KV-cache management, shorter trajectories, or algorithmic changes.Source: §4.6 · p. 24, §4.7 · p. 24

4.7 Key Insights & Takeaways

• Most inter-model techniques reduce depth or peak memory rather than total work; shared actor–critic also reduces work and trainable model-state memory by eliminating a separate critic backbone.Disaggregation, fusion, hybrid execution, and asynchrony primarily replace sequential depth sums with maxima or overlap, whereas shared actor–critic uses one backbone.Source: §4.7 · p. 24, §4.6 · p. 24

4.2 Model Placement Strategies

• Disaggregation reduces co-resident memory and enables concurrency and heterogeneous hardware, but requires fusion, streaming, or asynchrony to reduce depth beyond strict Generation → Assessment → Training barriers.Disaggregated execution introduces explicit communication of rollouts, scores, values, rewards, and sometimes parameters.Source: §4.2 · p. 21, §4.7 · p. 24

4.7 Key Insights & Takeaways

• Asynchrony can improve hardware utilization and reduce steady-state depth from a sum to a max through bounded staleness, but it increases memory demands and may cause policy drift.Relevant controls include staleness, KL drift between behavior and current policies, and PPO clip fraction, alongside throughput and memory.Source: §4.7 · p. 24
• Stage- and role-aware designs combine shared or critic-free structures, disaggregated placement, stage fusion, hybrid layouts, and bounded asynchrony according to each component’s bottlenecks.Actor generation, reward/reference assessment, critic inference, actor training, and critic training require different execution strategies; bounded asynchrony is appropriate when the learning rule tolerates stale data.Source: §4.7 · p. 24
• Shared actor–critic architectures are particularly effective when device groups and memory are scarce and inter-model concurrency is limited.They share a single backbone between the policy and value functions, reducing resource requirements under constrained conditions.Source: §4.7 · p. 24

5 PARALLELISM-FOCUSED SPECIFICATIONS

• Unified formulations of PPO, GRPO, and DPO expose parallelism opportunities across rollout generation, online training, and preference optimization.PPO and GRPO share a common rollout-generation routine; DPO provides the offline counterpart.Source: §5 · p. 24
• Independent prompts, candidates, rollouts, and preference pairs naturally use data parallelism, while dependency-free model invocations may run concurrently across device groups.Inter-model concurrency applies to fork–join regions without data dependencies, followed by reductions for losses, statistics, or gradients.Source: §5 · pp. 24–25
• Tensor and pipeline parallelism shard actor, critic, reward, or reference models for capacity or single-invocation latency, but collectives and pipeline bubbles can reduce throughput.These strategies are conditional when a model already fits on one device.Source: §5 · p. 25
• Context parallelism supports long-context forward and training invocations, including autoregressive decoding, when rollout KV or attention memory exceeds single-device capacity.Sequence parallelism instead targets trainable teacher-forced forward/backward paths by reducing stored activation memory, not ordinary forward-only assessment or decoding.Source: §5 · p. 25
• Expert parallelism applies only to MoE actor, critic, reward, or reference models, reducing per-device expert storage and compute while adding routing and all-to-all communication.If actor and critic share a backbone, their invocations are one shared-model invocation with two heads; DPO reference values may be precomputed and stored.Source: §5 · pp. 24–25

6.2 Training & Inference Frameworks for RLMs

• Rollout and generation often dominate wall-clock time for long-output reasoning, so high-throughput frameworks maximize inference throughput or overlap generation with evaluation and training through stage fusion or asynchronous execution.Table 11 compares public frameworks under the paper’s intra-model and inter-model systems taxonomy.Source: §6.2 · p. 26, Table 11 · p. 28

6.1 Reasoning Language Models

• RLMs explicitly target reasoning trajectories, long-CoT, verifiable rewards, search, tools, agentic environments, or controllable test-time computation, unlike general aligned LLMs focused primarily on instruction following, safety, preference alignment, and assistant quality.The comparison also notes that hybrid fast/thinking modes increasingly coexist within routed or unified model families.Source: §6.1 · pp. 25–26, Table 10 · p. 27

6.2 Training & Inference Frameworks for RLMs

• Frameworks span library-centric stacks with limited inter-model scheduling, orchestrator-centric multi-role dataflows, and scheduling-centric systems that improve long-tailed rollout utilization through fusion, streaming, disaggregation, load balancing, or bounded-stale execution.These optimizations increase complexity through queues, weight-version management, and convergence validation.Source: §6.2 · pp. 26, 28
• Practical choices depend on workload scale: use library-centric stacks for small experiments, FSDP/ZeRO or Megatron for memory-bounded runs, disaggregated rollout engines when generation dominates, and async or streaming systems for frontier-scale reasoning RL.The broader design problem is placement and coordination of heterogeneous roles, weight synchronization or reshaping, optimizer-state memory, communication, KV-cache pressure, and variable rollout lengths.Source: §6.2 · p. 28

7 RESEARCH OPPORTUNITIES

• Resource-aware test-time compute could optimize expected utility under token, latency, memory, tool-call, and energy budgets using online predictors and scheduling.Reasoning budget could become a first-class scheduling variable informed by task difficulty, verifier value, branch utility, and stopping time.Source: §7 · p. 28
• Nonlinear reasoning structures, including trees, graphs, and hypergraphs, could be integrated into RL execution pipelines to improve search and aggregation efficiently [26] [259].Prior work also explores distilling aggregation into model weights during fine-tuning [8] [150] [151] [275].Source: §7 · pp. 28–29
• Automated planners could select model-invocation parallelization and derive reasoning topologies and schedules from hardware, task, uncertainty, verifier, and performance conditions.Open decisions include branch counts, pruning, verifier calls, thought merging, and mapping the resulting DAG to device groups [26].Source: §7 · p. 29, others · p. 29

others.

• RLVR research should develop reliable process-level and structural rewards beyond final-answer checks, while optimizing caching, batching, placement, and overlap for their many assessment nodes [69] [165] [241].Weak or spurious rewards can still improve reasoning, and outcomes depend on model family [212].Source: others · p. 29
• Graph analytics, GNNs, and graph representation learning could supervise reasoning, score branches, construct process rewards, and predict tool calls without bottlenecking Generation–Assessment–Training or MCTS loops.Conversely, integrating RLMs into analytics frameworks and databases offers opportunities across document, tabular, and video processing.Source: others · p. 29
• Asynchronous RL systems should characterize when bounded staleness preserves rollout usefulness, how to correct policy drift, and how throughput trades against reasoning quality [111] [145] [216] [281].Tools, retrieval, simulators, verifiers, and agent calls could be represented as operators alongside generation, reward evaluation, and training [44] [93].Source: others · p. 29
• RLM performance research could exploit emerging hardware and model-driven algorithm discovery for kernels, collectives, placement heuristics, and routing policies [182].Relevant platforms include processing-in-memory, SmartNICs, serverless systems, next-generation interconnects, and chiplets.Source: others · p. 29
• Evaluation pipelines should standardize reasoning budgets, sampling, rollout length, staleness, subsystem metrics, and reproducibility, while shared-weight snapshots could reduce synchronization costs in distributed RLHF.Versioned parameter slots with buffering and publication fences can preserve snapshot consistency while reducing bandwidth pressure and synchronization stalls.Source: others · pp. 29–30

8 CONCLUSION

• RLM training combines algorithmic alignment with major parallel and distributed systems challenges, motivating unified work–depth–memory analysis and optimization of computation, memory, placement, fusion, and execution bottlenecks.The analysis covers autoregressive rollouts, auxiliary-model assessment, actor/critic updates, long-context memory, model-state sharding, and bounded-stale execution.Source: §8 · p. 30

APPENDIX A RLM PIPELINE: FUNCTIONAL DESCRIPTION

• The RLM pipeline comprises Generation, Assessment, and Training stages that transform batches through sampled outputs, learning signals, and updated model parameters.The iterative loop feeds the updated policy into the next Generation stage until convergence or a preset iteration limit.Source: Appendix A · p. 42, §A.1 · p. 42, §A.2 · p. 42

A.1 Generation Stage

• Generation samples K candidate completions per prompt from the behavior policy, while PPO-like methods may also record per-token actor log-probabilities for Training.The rollout batch contains the generated token sequences and associated sampling information.Source: §A.1 · p. 42

A.2 Assessment Stage

• Assessment applies auxiliary models to rollouts to produce sequence-level rewards, token-level advantages, and reference log-probabilities, with GRPO computing group-normalized advantages without a critic pass.PPO-like methods typically use a reward model, reference policy, and critic to construct these learning signals.Source: §A.2 · p. 42

A.3 Training Stage

• Training updates PPO actors and critics, only the actor in GRPO, and only the policy from preference pairs in DPO, producing the next policy and any applicable critic parameters.PPO uses policy-ratio terms for the actor and a value-regression loss for the critic, whereas DPO uses teacher-forced likelihoods.Source: §A.3 · p. 42

APPENDIX B DETAILS OF MATHEMATICAL DERIVATIONS

• The derivations model forward and backward costs, critical-path depth, parameter counts, KV-cache memory, optimizer state, and training activations for the policy, reference, reward, and value models.Backward cost is approximated as about twice forward cost, while backward depth adds O(log(B(S + T))) for batch-wide weight-gradient accumulation.Source: §B.1 · p. 43, §B.1.2 · p. 43, §B.1.4 · p. 44

B.1.3 Autoregressive generation: prefill and decode

• Autoregressive generation must be decomposed into prompt prefill and sequential decode, with total cost Cgen(πθ; S, T) = Cprefill(πθ; S) + Cdec(πθ; S, T).The dominant sequential dependence comes from the T decode steps, while prefill initializes the KV cache from the prompt.Source: §B.1.3 · pp. 43–44, §B.1.4.4 · p. 44

B.1.5.7 Training

• KV-cache memory decreases proportionally when fewer than BK rollouts are generated concurrently, trading reduced peak memory for reduced concurrency.The training activation model assumes no activation checkpointing and suppresses constant-factor temporary-buffer storage.Source: §B.1.5 · p. 45, §B.1.5.7 · p. 45
• Frozen forward-only models can reuse intermediate layer buffers across layers because backpropagation does not require retaining them.Source: §B.1.5.7 · p. 45
• The leading-order inference buffer is derived for Q sequences without fully materializing the attention matrix, excluding model parameters and autoregressive KV-cache storage.Source: §B.1.5.7 · p. 45

B.2.1 Online Frameworks

• Online PPO and GRPO repeatedly execute generation, assessment, and training, with autoregressive generation forming the primary wall-clock bottleneck.Assessment and training use teacher-forced, non-autoregressive passes; PPO updates both policy and critic, whereas GRPO updates only the policy.Source: §B.2.1 · p. 45, §B.2.1.2 · p. 45

B.2.2 Offline Frameworks

• Offline DPO removes fresh rollout generation from the inner loop, leaving batched forward and backward passes over fixed preference pairs.Its critical path is a single batched forward/backward evaluation rather than an autoregressive decode chain.Source: §B.2.2 · p. 46

B.3.1 Baseline Cost Model

• The intra-model baseline expresses dense Transformer training in terms of layer count, hidden and FFN dimensions, vocabulary, sequence length, batch size, work, reduction depth, and memory.The explicit depth model includes forward and backward layer dependencies plus a global batch-token parameter-gradient reduction; memory includes model state and stored activations.Source: §B.3.1 · p. 46

B.3.2 Data Parallelism

• Data parallelism reduces per-rank work and activation memory but replicates model state and preserves the global batch-gradient dependency.Each rank processes B/Pd sequences, while the globally aggregated gradient still depends on all B(S + T) positions.Source: §B.3.2 · p. 47

B.3.3 Pipeline Parallelism

• Pipeline parallelism partitions layers and divides layer-dependent per-stage memory, while preserving global work and model-state memory up to boundary-stage effects.Realized runtime additionally depends on microbatch schedules, fill/drain bubbles, and whether single-microbatch latency or steady-state throughput is measured.Source: §B.3.3 · p. 47

B.3.4 Tensor Parallelism

• Tensor parallelism shards hidden-dimensional computation, model-state memory, and activation memory by Pt, reducing per-device pressure and hidden-dimensional reductions.The model ignores communication buffers and collective overheads, so layerwise communication is an unmodeled cost.Source: §B.3.4 · p. 47

B.3.5 Context Parallelism

• Context parallelism partitions sequence-local work and activations but retains the global exact-attention reduction over S + T rather than replacing it with (S + T)/Pc.It is therefore primarily a sequence-work, memory, and long-context feasibility technique; distributed-attention communication dependencies are outside the FLOP-based depth abstraction.Source: §B.3.5 · pp. 47–48

B.3.6 Expert Parallelism

• Under isolated expert parallelism, non-expert computation is replicated while expert FFN parameters and routed computation are partitioned, so EP does not remove the dense Transformer critical path.The isolated analysis assumes EP is the only partitioning mechanism and excludes additional DP, TP, PP, or CP partitioning outside expert FFNs.Source: §B.3.6 · p. 48

B.3.7 3D Parallelism

• 3D parallelism reduces per-device memory by partitioning batch, layers, and hidden computation, but realized runtime depends on communication, pipeline scheduling, and microbatching.Source: §B.3.7 · p. 48

B.3.8 5D Parallelism

• 5D parallelism uses source-token distribution with expert placement, so shared dense arithmetic is partitioned rather than replicated across expert ranks despite replicated shared state.The model assumes balanced routing and excludes expert dispatch/combine and distributed-attention communication from the arithmetic-depth abstraction.Source: §B.3.8 · p. 49

B.4 Results for Inter-Model Parallelism

• The inter-model analysis derives PPO-style generation, assessment, and training work, depth, and memory from forward, backward, and autoregressive generation primitives.It excludes communication, kernel efficiency, and scheduling overheads from the work–depth abstraction, representing some effects through memory buffers.Source: §B.4 · p. 49, §B.4.1 · p. 50

B.4.2 Baseline Co-Located Execution / B.4.3 Shared Actor–Critic

• Separate actor–critic PPO executes all stages sequentially on one device group, while shared actor–critic is the only listed configuration that reduces global work.The shared model uses one Transformer backbone for policy and value estimation; its benefit is largest when critic and actor sizes are comparable.Source: §B.4.3 · p. 50, §B.4.2 · p. 50

B.4.4 Disaggregated Placement

• Disaggregated placement preserves total work but enables concurrent independent branches and lowers peak memory by separating actor, reference, reward, and critic device groups.It does not remove the Generation → Assessment → Training dependency; it only exposes concurrency where the dataflow graph permits it.Source: §B.4.4 · p. 50

B.4.5 Hybrid Execution and Resharding

• Hybrid execution selects inference- and training-oriented layouts by stage without changing FLOP work, but resharding and orchestration costs must be outweighed by stage-specific gains.Source: §B.4.5 · p. 51

B.4.6 Stage Fusion

• Stage fusion preserves work and can reduce depth through ideal overlap, although streaming generations and assessment increases memory because multiple stage fragments remain live.Source: §B.4.6 · p. 51

B.4.7 Asynchronous Execution

• Bounded asynchrony overlaps iterations to reduce depth but requires parameter snapshots, potentially adding roughly one extra lower-precision or sharded model copy per asynchronously consumed trainable model.Source: §B.4.7 · p. 51

B.4.8 Combined Configuration

• The combined configuration composes stage-specific intra-model sharding, disaggregated model placement, stage fusion, and bounded asynchrony.Its analysis also considers shared actor–critic backbones when changing the model structure.Source: §B.4.8 · p. 51
• The combined analysis distinguishes actor generation depth under inference-oriented sharding from the depth of the sharded training layout and retains a conservative assessment tail.These quantities are represented by G, Dshard, and T in the supplied analysis.Source: §B.4.8 · p. 51
• The strategy achieves the smallest idealized depth and peak per-device-group memory in Table 9.This result holds only under sufficient hardware, careful scheduling, and acceptable staleness.Source: §B.4.8 · p. 51