PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth

arXiv:2606.30634v1cs.LG

TL;DR

Asynchronous pipeline parallelism removes pipeline bubbles but introduces gradient staleness, and PipeDream-2BW reduces this challenge to a fixed one-step delay. This paper shows that robust optimizers such as Muon, combined with lightweight correction, can closely match synchronous training, including at 10B-parameter scale.

  • Problem

    Asynchronous pipeline parallelism avoids pipeline bubbles but introduces gradient staleness, motivating evidence on whether fixed one-step delay remains practical as pipeline depth increases.

  • Method

    The paper evaluates optimizers under fixed one-step delay, compares constant- and variable-delay schedules, and develops an optimizer-agnostic Error Feedback-inspired correction.

  • Results

    Async PP with Error Feedback matches the synchronous final validation loss for a 10B-parameter MoE model using the same hyperparameters.

  • Takeaways & Limitations

    One-step delay is not a fundamental barrier to large-scale asynchronous pipeline-parallel training with suitable optimizers and lightweight correction.

  • Takeaways & Limitations

    The study lacks a complete mechanistic explanation for momentum’s robustness, restricts hyperparameter grids to the 135M model, and evaluates WPipe-style schedules only in limited appendix experiments.

Abstract

from arXiv · show

Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parallelism eliminates these bubbles, maximizing throughput at the cost of gradient staleness. Among asynchronous schedules, PipeDream-2BW is particularly appealing: unlike the original PipeDream schedule, it ensures a constant one-step gradient delay regardless of pipeline depth. However, its adoption remains limited due to the common belief that optimizing under staleness is fundamentally unstable. In this work, we challenge this assumption, demonstrating that degradation under one-step delay depends strongly on optimizer choice rather than being an intrinsic limitation. We provide the first comprehensive empirical analysis showing that while AdamW, the predominant optimizer at the time when PipeDream-2BW was introduced, indeed suffers from severe degradation, recent methods like Muon exhibit strong robustness under a one-step delay. We introduce an optimizer-agnostic Error Feedback-inspired correction to further mitigate delay effects. We provide supporting theoretical analysis demonstrating convergence for Muon with and without this correction. Extensive evaluation on models up to 10B parameters confirms that our strategies bridge the performance gap with synchronous training, highlighting the practical potential of asynchronous pipeline parallelism at scale.

1. Introduction

Asynchronous pipeline parallelism avoids the bubbles of synchronous training but introduces gradient staleness, whose impact depends strongly on optimizer and delay structure. The paper shows that constant one-step delay, Muon, and Error Feedback can make Async PP competitive with synchronous training, including at 10B-parameter scale.

  • Motivation: Synchronous pipeline parallelism leaves GPUs idle in unavoidable pipeline bubbles, whereas asynchronous execution avoids synchronization to improve utilization.The supplied passage identifies bubbles as empty schedule slots caused by synchronous parameter updates.
  • Motivation: Variable gradient delays in the original PipeDream schedule cause severe convergence degradation as pipeline depth increases.Ajanthan et al. report validation loss increasing by more than 0.2 versus synchronous training, according to the passage.
  • Motivation: PipeDream-2BW provides uniform staleness of 1 across pipeline stages by updating once every M backward passes, reducing the optimization challenge to constant delay.The passage presents PipeDream-2BW as a natural candidate for Async PP because its delay remains independent of pipeline size.
  • Optimizer robustness: AdamW suffers substantial quality loss under staleness, while Muon maintains competitive synchronous performance and a small sync-async gap under default hyperparameters.The paper’s empirical analysis links optimizer choice to degradation under delayed gradients.
  • Staleness mitigation: An optimizer-agnostic Error Feedback correction consistently narrows the synchronous-asynchronous gap and further improves Muon’s already small gap.The correction is inspired by Error Feedback and is evaluated as a staleness-mitigation strategy.
  • Large-scale validation: With Muon, Async PP, and Error Feedback, a 10B-parameter MoE trained on 200B tokens achieved a final loss identical to the synchronous baseline using the exact same hyperparameters.The paper describes this as the first successful demonstration of Async PP at this scale without quality degradation, to the authors’ knowledge.

2. One-Step Delayed Optimization

This section defines one-step delayed optimization using PipeDream-2BW and shows that delay robustness depends strongly on optimizer dynamics: AdamW degrades severely, while Muon remains comparatively robust.

  • Delayed-update abstraction: PipeDream-2BW updates stage parameters after a full minibatch of M micro-batches, avoiding the variable gradient delays and multiple stashed versions required by PipeDream.PipeDream updates each stage immediately after a local backward pass, producing delay levels that vary across stages.
  • Experimental setup: Delayed runs are compared with synchronous baselines using identical hyperparameters, isolating the effect of one-step staleness.The experiments use 135M and 360M models trained on FineWeb-Edu with a fixed global batch size.
  • Optimizer comparison: Muon achieves a much smaller final sync-async gap than AdamW, showing that one-step staleness is not uniformly harmful and depends strongly on optimizer dynamics.Both optimizers train competitively synchronously, but their delayed variants behave very differently.
  • Optimizer comparison: AdamW suffers severe quality degradation under one-step delay (> 0.2), while synchronous-start stabilization improves it but does not eliminate the problem.The delayed AdamW trajectory diverges early from the synchronous trajectory.

3. Staleness Mitigation

Section 3 evaluates schedule-, gradient-, and optimizer-update-level approaches to mitigate one-step staleness. Error Feedback is the most consistent strategy, while synchronous start is sensitive and other simple interventions provide limited or destabilizing benefits.

  • Synchronous Start: Synchronous start recovers around 20–30% of the remaining sync-async gap for many optimizers but can become highly unstable at the transition, especially for SOAP and Adan with larger β2.It also temporarily reintroduces pipeline bubbles and requires supporting both execution modes, reducing Async PP’s practical throughput benefit.
  • Synchronous Cooldown: Synchronous cooldown yields only marginal improvements for Muon and AdamW, so making only the final training phase synchronous does not readily remove the residual loss gap.
  • DC-ASGD / Taylor-based Delay Compensation: Taylor-based DC-ASGD correction is extremely small at typical LLM scales and destabilizes training only after substantial scaling, despite sweeping λ from 10^4 to 10^8.Gradients are typically around 10^-5, while parameter updates are around 10^-3 because they scale with the learning rate.
  • Error Feedback: Error Feedback recovers roughly 50–70% of delayed-training degradation for Muon, AdaMuon, SOAP, and NorMuon, and 85–90% of the gap for MARS and AdamW in 360M experiments.It is not universally beneficial, slightly degrading Adan and NAdam, but provides a more consistent benefit than the baseline strategies.
  • Error Feedback: The update-level Error Feedback correction stores one additional model-sized buffer, adding only a small constant memory overhead, whereas its raw-gradient variant diverges.The correction applies at the optimizer-update level and is later used with Muon for large-scale validation, with and without Error Feedback.

4. Theoretical analysis

This section analyzes Muon-style Linear Minimization Oracle updates under fixed gradient staleness, where theoretical guarantees remain limited. It formulates Muon’s optimization problem for a bounded-below, differentiable objective and states assumptions for the analysis.

  • Theoretical analysis: The analysis studies Muon-style Linear Minimization Oracle updates under gradient staleness, addressing limited guarantees for fixed-delay LMO methods.The paper positions this as, to the authors’ knowledge, the first convergence analysis of this setting, though the supplied passage is truncated.
  • Theoretical analysis: The paper theoretically formulates Muon’s optimization problem following the setting of Kovalev (2025).This formulation provides the starting point for the subsequent convergence analysis.
  • Theoretical analysis: The objective f(·): Rm×n →R is assumed to be bounded from below and differentiable.These properties are stated as part of the theoretical setup.

1. Stochastic gradient estimator. Eξ[∇f(X; ξ)] = ∇f(X), Eξ[∥∇f(X; ξ) −∇f(X)∥2

The section analyzes delayed Muon under stochastic-gradient assumptions of bounded variance, smoothness, and star convexity. Its convergence guarantee differs from synchronous Muon through an additional delay-dependent noise term that is generally small under the prescribed stepsize.

  • Assumptions: These stochastic-gradient assumptions are widely used in analyses of stochastic optimization algorithms.The passage attributes this observation to Gower et al. (2019), Horváth et al. (2023), and Kovalev (2025).
  • Delayed Muon: The section formulates Muon with gradient delay in Algorithm 2 and states a convergence theorem for its weight-decay variant.The theorem assumes Assumption 4.1 and initializes M0 = G(X0), with iterations using weight decay λ > 0.
  • Proof strategy: The delayed-Muon theorem follows as a corollary of a general convergence guarantee for delayed LMO algorithms.The proof uses R(X) ≡0, the operator norm and dual nuclear norm, and ρ = min(m, n).
  • Comparison with synchronous training: The delayed setup replaces the synchronous noise bound √µρσ with √2µρ2σ2 + 8(Lη)2, adding a delay-dependent term.Under η = O(...), the passage states that the additional term caused by delayed gradients is generally small.

5. Large Scale Experiments

Large-scale experiments show that Muon-based Async PP remains competitive with synchronous training across 2B- and 10B-parameter MoE models. Error Feedback eliminates the remaining loss gap at 10B scale, demonstrating practical quality preservation under asynchronous training.

  • Experimental Setup: Muon is evaluated on 2B- and 10B-parameter MoE models to test whether Async PP can match synchronous training quality at realistic scale.The 2B model uses 500M active parameters, while the 10B model is trained for 200B tokens.
  • Scaling with Training Horizon: The 2B MoE synchronous and asynchronous scaling curves remain nearly parallel from 50B to 200B tokens, showing no growth in the sync-async gap with longer training.Error Feedback consistently reduces the remaining gap across training horizons.
  • 10B-Parameter MoE: At 10B scale, standard Async PP incurs only a small final loss gap versus synchronous training, with losses of 1.911 vs. 1.906.The comparison uses a 10B-parameter MoE model trained for 200B tokens.
  • 10B-Parameter MoE: Error Feedback closes the 10B-scale gap entirely, matching the synchronous final loss of 1.906 with the exact same hyperparameters.This result establishes quality parity between Async PP with Error Feedback and the synchronous baseline.
  • Practical Implications: These experiments provide the first successful demonstration of Async PP at this scale without quality degradation, supporting its practical potential for large-scale LLM pre-training.The motivation is that asynchronous execution can avoid pipeline bubbles and their wasted accelerator time.

6. Comparison with PipeDream

Comparison experiments show that Muon and SOAP are more robust than Nadam under the original PipeDream schedule, but degradation increases with pipeline depth. Robust optimizers and Error Feedback help shallow pipelines, whereas PipeDream-2BW’s constant-delay guarantee remains important for scalable asynchronous pipeline parallelism.

  • Comparison with PipeDream: On the 135M model, original PipeDream progressively degrades relative to corresponding PipeDream-2BW runs as P increases.PipeDream-2BW maintains a constant one-step delay independent of pipeline depth, while original PipeDream has variable delay.
  • Experimental setup: The original PipeDream schedule was evaluated at P ∈ {4, 8, 16} stages with Muon, SOAP, and Nadam using Section 2’s best hyperparameters.The effective batch size per update was set to Bsync/P because the schedule applies an optimizer step after every backward pass.
  • Optimizer robustness: Muon and SOAP are generally more robust than Nadam, especially as the number of pipeline stages increases.This extends the optimizer robustness trends observed earlier to the original PipeDream schedule.
  • Optimizer robustness: For Muon, increasing momentum from µ = 0.95 to µ = 0.99 improves performance at every pipeline depth, with or without Error Feedback.Error Feedback also provides small but consistent improvements for Muon and SOAP.
  • Scaling with pipeline depth: At P = 16, even Muon with µ = 0.99 and Error Feedback loses more than 0.03 relative to its synchronous baseline.All methods degrade substantially as the number of stages increases, so these techniques do not remove the original schedule’s scaling issue.
  • Implications: Robust optimizers can partially compensate for original PipeDream at shallow depths, but scalable asynchronous pipeline parallelism benefits substantially from PipeDream-2BW’s constant-delay guarantee.At larger depths, Muon and Error Feedback remain insufficient to prevent substantial degradation.

7. Related work

Related work spans asynchronous pipeline schedules, optimizer benchmarking, Error Feedback, and optimization with delayed gradients. This work positions itself at their intersection by benchmarking optimizers under asynchronous delay and studying Error Feedback-inspired mitigation.

  • Asynchronous Pipeline Parallelism: PipeDream established asynchronous pipeline parallelism with weight stashing, while PipeMare, Spec-Train, XPipe, and PipeOptim removed stashing to prioritize memory efficiency.PipeDream provided consistent weights for forward and backward passes but incurred variable gradient staleness.
  • Optimizer Benchmarking: Recent studies benchmark LLM optimizers for convergence, performance, and quantization interactions; this work benchmarks optimizers under asynchronous gradient delay.The cited studies include Semenov et al. (2025), Wen et al. (2025), and Vlassis et al. (2025).
  • Error Feedback: Error Feedback originated for quantization errors, expanded into gradient compression, and has also been studied with Muon and gradient delays.The passage cites Seide et al. (2014), compression studies by Stich et al. (2018), Alistarh et al. (2018), and Karimireddy et al. (2019), plus later Muon and delay work.
  • Optimization with Delayed Gradients: Delayed-gradient optimization has established theoretical foundations and includes distributed systems work such as Pipe-SGD and mitigation-related work such as SAPipe.Pipe-SGD pipelines AllReduce-based data-parallel training and provides convergence guarantees for convex and strongly convex objectives.

8. Discussion and Limitations … A.7. Empirical Noise Level Estimation

The paper concludes that one-step delay is not a fundamental barrier for asynchronous pipeline-parallel training, while identifying optimizer dynamics, hyperparameter sensitivity, and stochastic variation as important limitations and considerations. Across additional experiments, Error Feedback and robust optimizers closely approach synchronous performance, including at 10B scale.

  • 8. Discussion and Limitations: One-step delay is not an unavoidable limitation: PipeDream-2BW with Muon or lightweight update-level correction can closely match synchronous baselines.The paper lacks a complete mechanistic explanation for momentum-based robustness, restricts batch-size and learning-rate grids to 135M, and studies WPipe only in limited appendix experiments.
  • A. Hyperparameter Sensitivity Results and Other Additional Experiments; A.1. One-Dimensional Hyperparameter Sweeps: On the 135M model, primary momentum decay is the most consistent robustness-control hyperparameter, while other sweeps show optimizer-dependent effects.The appendix separates one-dimensional sweeps from batch-size and learning-rate interactions, using matched hyperparameters for synchronous and delayed runs.
  • A.1. One-Dimensional Hyperparameter Sweeps: Smaller learning rates and longer warmups mildly reduce the sync-async gap, whereas large learning rates worsen delayed AdamW training and warmup effects are more visible for AdamW.The one-step delay is enabled from the first training step; gradient clipping has little systematic effect, and Muon, NorMuon, and SOAP remain relatively stable across tested schedulers.
  • A.1. One-Dimensional Hyperparameter Sweeps: Weight decay, second-moment decay, and optimizer-specific settings affect robustness in optimizer-dependent ways, including divergence for small weight decay in NorMuon and AdamW.NorMuon benefits substantially from Nesterov-style momentum, while SOAP becomes mildly more sensitive when preconditioner updates are less frequent.
  • A.2. Batch Size Impact: Decreasing batch size substantially reduces the sync-async gap, but very small batches worsen synchronous quality; increasing batch size can make the gap exceed 0.1.The batch-size sweep jointly varies batch size and peak learning rate because optimal learning rate depends on batch size.
  • A.3. DC-ASGD Delay Compensation; A.4. Error Feedback Coefficient Ablation: None of the tested DC-ASGD coefficients improves over standard delayed Muon training, while Error Feedback performs best at optimizer-dependent strengths and λ = 0 is consistently worse.NorMuon’s best Error-Feedback coefficient is around λ = 0.75; AdamW, Muon, and SOAP favor roughly λ = 1.5–2.0, but λ = 1 remains the default for simplicity.
  • A.5. Learning Rate Robustness for the 2B Model; A.6. 10B Benchmarking Results: Additional 2B learning-rate runs evaluate stability across training horizons, while 10B benchmarks show Async + EF closely matches synchronous downstream performance.The Async + EF setup exactly matches the synchronous validation loss in the reported 10B MoE evaluation.
  • A.7. Empirical Noise Level Estimation: The 135M Muon final-loss standard deviation is approximately 10−3, remains below 2 × 10−3 across synchronous optimizers, and stays well below the 0.01 constraint with Error Feedback.Error Feedback adds variance, but it remains small for robust optimizers and more pronounced for unstable ones.

A.8. AdamW Ablations … B.2. WPipe Scheduling Scheme

The ablations identify AdamW’s first-moment dynamics as central to one-step-delay degradation, while synchronous cooldown and gradient-level correction provide limited or unstable remedies. Related analyses connect SAPipe weight prediction to Error Feedback and show WPipe can preserve optimizer-level benefits under partial delay.

  • A.8. AdamW Ablations: AdamW’s stale-update cosine similarity and relative error are not worse than Muon’s, so update discrepancy alone does not explain its delayed-training degradation.The comparison uses fresh updates from corresponding non-delayed runs on the 135M model.
  • A.8. AdamW Ablations: Keeping the language-model head synchronous provides little improvement: AdamW remains far worse than its synchronous baseline, with final loss above 3.0.This result indicates the instability is not localized to the final language-model head.
  • A.8. AdamW Ablations: Delaying only AdamW’s first-moment update produces loss almost identical to fully delayed AdamW, supporting first-moment dynamics as central to one-step-delay robustness.The second-moment update remains synchronous in this ablation.
  • A.8. AdamW Ablations: Synchronous AdamW with β1 = 0.99 reaches final loss 2.939 versus 2.877 for standard β1 = 0.9, showing that larger β1 eventually destabilizes or degrades AdamW.Increasing β1 improves delayed robustness only up to a point, because very large values can be harmful even synchronously.
  • A.9. Ablation on Synchronous Cooldown: Switching from asynchronous to synchronous training near the end yields only marginal improvements for the 135M-model cooldown ablation.The study evaluates both Muon and AdamW under cutoffs relative to warmup duration W.
  • A.10. Gradient-Based Error Feedback: Gradient-level Error Feedback is unstable and diverges, so the main experiments use update-level correction instead.The gradient-level variant applies correction to raw gradients before the optimizer.
  • A.11. Effect of β2 on the Synchronous-Start Loss Spike: The train-loss spike is notably larger for β2 = 0.999 than for β2 = 0.95.Figure 20 examines the relationship between β2 and the synchronous-start loss spike.
  • B. Additional Related Work: SAPipe and WPipe: SAPipe and this work both involve one-step delayed gradients, but SAPipe hides data-parallel communication while this work studies pipeline-induced staleness and modern optimizer robustness.SAPipe-WP’s latest-synchronized-gradient variant uses the same information available in one-step delayed asynchronous pipeline parallelism; its update displacement is closely related to Error Feedback.

C. Delayed Stochastic Non-Euclidean Trust-Region Theory … D.2. Proof of Lemma D.2

The paper develops a delayed stochastic non-Euclidean trust-region method with momentum under unbiased bounded-variance gradients, smoothness, and norm equivalence, then proves delay-aware convergence bounds. Its proof decomposes momentum tracking error into stochastic-noise and drift terms, showing that delay enlarges the noise bound relative to synchronous optimization.

  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: The general problem uses a differentiable lower-bounded objective f and a proper convex regularizer R on a finite-dimensional inner-product space.The formulation follows the stated general optimization setting.
  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: The analysis assumes an unbiased bounded-variance stochastic gradient estimator, Lipschitz gradients, and norm equivalence with constant ρ > 0.These are Assumptions C.1–C.3.
  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: Algorithm 3 applies momentum to a gradient evaluated at the delayed iterate x_prev(k), with either a Standard Async update or an Error-Feedback update.The delay is defined as prev(k) = k − τ for arbitrary τ > 0.
  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: Under Assumptions C.1–C.3, Theorem C.4 establishes an inequality for Algorithm 3, extending the convergence framework to arbitrary gradient delays τ ≥ 1.The proof explicitly captures how staleness propagates through momentum accumulation.
  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: With star convexity and suitable η, β conditions, Theorem C.6 extends the delayed convergence result to Algorithm 3 with weight decay β > 0.The result combines the delay-aware framework with weight-decay analysis.
  • C. Delayed Stochastic Non-Euclidean Trust-Region Theory: Compared with synchronous optimization, delay enlarges the noise bound because the estimation now includes gradient-delay effects in addition to stochastic-oracle noise.This is identified as the main change in the delayed estimation.
  • D.1. Proof of Theorem C.4: The proof of Theorem C.4 retains the descent lemma in the delayed setting and introduces a key lemma bounding the momentum’s tracking error.The delay affects the momentum terms, motivating Lemma D.2.
  • D.2. Proof of Lemma D.2: The proof of Lemma D.2 decomposes delayed-gradient error into stochastic-noise and drift components, using conditional unbiasedness for noise cancellation and smoothness with the delay bound for drift.The derivation also uses triangle inequality, norm compatibility, Jensen’s inequality, and the iterate-step constraint ∥x_i+1 − x_i∥ ≤ η.

D.3. Proof of Error-Feedback convergence … E. Experimental Setup

The paper states convergence results for the Error-Feedback delayed LMO algorithm and develops supporting inequalities for its proof, including the weight-decay case. Experiments use SmolLM2 decoder-only models on Fineweb-Edu under a 20:1 Chinchilla token-to-parameter ratio.

  • D.3. Proof of Error-Feedback convergence: Theorem D.3 states inequalities satisfied by Algorithm 3 under Assumptions C.1–C.3, with x0 ∈ domR, R ≡ 0, and m0 = g(x0, ξ0).
  • D.3. Proof of Error-Feedback convergence: Lemma D.4 provides an inequality for the Error-Feedback iterations of Algorithm 3 under Assumption C.2 with R ≡ 0.
  • D.3.1. PROOF OF LEMMA D.4: The Lemma D.4 proof upper-bounds F(xk+1) using Assumption C.2, algebraic addition and subtraction, and the dual-norm definition.
  • D.3.1. PROOF OF LEMMA D.4: The proof then applies the Error-Feedback update, R ≡ 0, Cauchy–Schwarz, and an estimate of ∥xk+1 − xk∥ to continue the inequality derivation.
  • D.4. Proof of Theorem C.6: Lemma D.5 establishes bounds ∥x − (1 − β)xk∥ ≤ η, ∥x − xk∥ ≤ 2η, ∥x − xk+1∥ ≤ 2η, and ∥xk+1 − xk∥ ≤ 2η.
  • D.4. Proof of Theorem C.6: Lemma D.6 gives an inequality for Algorithm 3 with Weight Decay under Assumptions C.1–C.3, and the proof of Theorem C.6 follows a similar argument.
  • E. Experimental Setup: Experiments train decoder-only SmolLM2 models with 135M and 360M parameters on the Fineweb-Edu dataset for standard next-token prediction.
  • E. Experimental Setup: Unless otherwise stated, training follows a Chinchilla compute-optimal token-to-parameter ratio of 20:1.

E.1. Hyperparameters and Training Details · E.2. Model architectures · F. Memory and Runtime Overhead

The study systematically tunes optimizers and batch sizes, then evaluates dense SmolLM-2 baselines alongside sparse 2B and 10B MoE architectures. The models span 135M to 10B total parameters, with the MoE designs emphasizing sparse active computation.

  • E.1. Hyperparameters and Training Details: AdamW tuning used logarithmic learning-rate grids and four weight-decay values, with 0.1 fixed after consistently producing optimal results.The tuning targeted 135M and 360M SmolLM-2 models.
  • E.1. Hyperparameters and Training Details: Other optimizers were tuned near optimal AdamW learning rates, while Lion used weight decay 0.5 and learning rate approximately 5e-4.Lion was treated separately because it operates on a distinct scale.
  • E.1. Hyperparameters and Training Details: Global batch sizes were set to 256 for the 135M model and 512 for the 360M model using scaling-law predictions and approximately 700-token average sequences.The configured context length was 1024 tokens, but FineWeb padding reduced the average sequence length.
  • E.1. Hyperparameters and Training Details: The larger 2B and 10B MoE runs used token batch sizes from 1M to 4M, a regime that theoretically disadvantages Async PP by reducing optimization steps.The 2B model used 1M, 1.5M, and 2.25M batches for 50B, 100B, and 200B tokens, while the 10B model used 4M for 200B tokens.
  • E.2. Model architectures: Experiments used two dense SmolLM-2 models, 135M and 360M, and two custom sparse MoE models with 2B and 10B total parameters.These four architectures provide coverage from small dense baselines to large sparse models.
  • E.2. Model architectures: The SmolLM-2 dense baselines use Llama-style architectures with Grouped Query Attention, RMSNorm, SwiGLU, 1,024-token contexts, and a 49,152-token vocabulary.Both models share these architectural and tokenizer settings.
  • E.2. Model architectures: The custom MoE models use a 128k vocabulary and support 8,192-token contexts, with the 10B design maintaining approximately 0.65B active parameters during inference.The 10B model combines Full Attention and linear attention layers, 512 experts, and top-10 gating.

F.1. Memory overhead · F.2. Runtime overhead

PipeDream-2BW and Error Feedback add only a small, shard-local memory cost in realistic large-scale training. Async PP eliminates schedule-level pipeline bubbles, offering runtime advantages over synchronous schedules, though end-to-end gains depend on implementation and hardware.

  • F.1. Memory overhead: PipeDream-2BW and Error Feedback each add one parameter-sized state, but this applies to each GPU’s local model shard rather than the full model.The added states are respectively an extra parameter version and an Error Feedback residual buffer.
  • F.1. Memory overhead: On 80GB GPUs, the additional PipeDream-2BW or Error Feedback state is not prohibitive for DeepSeek-V3.This conclusion follows from the shard-local storage pattern and the limited per-stage allocation described for the 681B-parameter model.
  • F.1. Memory overhead: DeepSeek-V3 assigns at most four hidden layers per pipeline stage, with each GPU storing only four routed experts per MoE layer under 64-way expert parallelism.The model uses 2048 GPUs, 16-way pipeline parallelism, and 64-way expert parallelism; the remaining data-parallel degree is 2.
  • F.1. Memory overhead: 3.16B resident parameters per GPU precede FSDP sharding for LLaMA 3 405B with 8-way tensor and 16-way pipeline parallelism.With an effective FSDP sharding factor of about 128, one additional FP32 sharded state is described as negligible.
  • F.1. Memory overhead: 800 MB is the approximate cost per GPU of one additional FP32 parameter-sized state in the 10B-parameter MoE experiment.Each GPU stores at most about 200M master-weight parameters; total Async PP memory cost with Error Feedback was below 1.5GB per GPU, less than 2% of an 80GB GPU.
  • F.2. Runtime overhead: Async PP has no pipeline bubbles in the schedule-level model, with ρasync = 0; synchronous slowdown relative to the async ideal is represented by 1 + ρ.The bubble-to-compute ratio uses pipeline depth, micro-batch count, forward time, backward time, weight-gradient time, and overlapped forward/backward time.
  • F.2. Runtime overhead: 1.938 is the reported slowdown for synchronous 1F1B at both B = 2F and B = 3F, while Async PP / PipeDream-2BW is 1.000.For B = 2F, ZB1P and DualPipe are 1.313 and 1.292; for B = 3F, they are 1.469 and 1.438.
  • F.2. Runtime overhead: The bubble analysis estimates schedule-level runtime benefits, not end-to-end wall-clock performance, which depends on implementation, communication overlap, and hardware.Under standard bubble models, synchronous PP can still incur substantial overhead at practical micro-batch counts, whereas Async PP removes the bubble term entirely.
Loading 2606.30634v1…