PaperlayerView exact revision on arXiv

Source-linked AI summary

TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication

Weicheng Xue, Bingqiang Wang, Li Yuan, Huihui Zhou, Yonghong Tian

arXiv:2608.27874v1cs.DC

TL;DR

MoE training needs costly all-to-all communication, while hierarchical dispatch trades reduced slow-fabric traffic for extra collective and arrival-chain costs. TerraceMoE builds a gated communication-call cost model for that trade and finds breakeven sensitivity to implementation overhead, without licensing deployment or training-throughput predictions.

  • Problem

    MoE expert parallelism makes all-to-all communication central to scaling, but a quoted hierarchy ratio omits provisioning, fixed costs, and implementation terms needed to screen dispatch restructuring.

  • Method

    TerraceMoE composes primitive communication costs into a ratio-based screening model and binds its output to validation gates that remove capabilities when they fail.

  • Results

    3.98 is the corrected effective breakeven hierarchy ratio for the measured PyTorch chain, versus 1.49 for a hypothetical fused target and 1.10 at zero implementation overhead.

  • Takeaways & Limitations

    Hierarchical dispatch can be screened at the communication-call level when target constants are measured at workload message sizes and communication-level gates pass.

  • Takeaways & Limitations

    The step-level gate fails, so the paper makes no step-time or training-throughput prediction; measured machines do not reach the hierarchical regime.

Abstract

from arXiv · show

Hierarchical two-hop dispatch can reduce slow-fabric traffic in expert-parallel Mixture-of-Experts training, but it adds a second collective and an arrival-side operator chain. We present a cost model for screening that trade at the communication-call level, bounded by validation gates that withdraw a capability in code when they fail rather than reporting a caveat. At a reference geometry with 16 groups of 8 ranks, $q=3$, $H=2048$, and 4096 tokens per rank, the corrected effective breakeven hierarchy ratio is 3.98 for the measured PyTorch arrival chain, 1.49 for a hypothetical fused target, and 1.10 at zero implementation overhead. These are ratio-only sensitivity results, not deployment predictions: platform A measures 1.03, platform B has no separated fast/slow measurement, and neither machine measured here reaches the hierarchical regime. Four communication-level corpora pass their gates; a drift probe and the step-level gate fail. The latter failure is enforced in code, so we make no training-throughput prediction. The enabling routing constraint fixes per-token fan-out and per-selected-group quota, while aggregate per-peer counts remain data-dependent. Its measured validation-loss cost is small but nonzero (+0.0034 nats); downstream equivalence is reported with incomplete estimator provenance and is therefore not independently reconstructible from the artifact. Code, calibration constants and the validation gates are at https://github.com/weich97/TerraceMoE-simulator.

1 Introduction

MoE all-to-all communication becomes costly as experts span slower interconnect domains, motivating hierarchical dispatch and a communication-level cost model. The model screens this trade with code-enforced validation gates, but its results remain bounded by measured coverage and do not license step-level predictions.

  • Motivation: MoE sparsity shifts the scaling burden from arithmetic to communication because every routed token must reach and return from its expert device.All-to-all communication can be the largest single component of step time in production MoE training.
  • Motivation: Hierarchical dispatch sends one copy per target group across the slow fabric, then scatters within each group over faster links.This deduplicates slow-side traffic but adds a second collective and an arrival-side unpacking chain.
  • Research question: The paper asks whether primitive microbenchmarks can predict dispatch restructuring for unbuilt machines and geometries with an actionable negative verdict.The motivation is both compositional prediction and accountability when rejected designs are never built.
  • Contribution: The ratio-only screening model is bounded by validation gates whose failures withdraw capabilities in code rather than add caveats.Two gates fail, including the step-level gate, so no step-level prediction appears.
  • Contribution: The routing constraint bounds per-token fan-out and fixes per-selected-group quota, while its validation-loss effect is small but nonzero.Downstream, load, and step-time evidence retain the limitations stated for the constraint.
  • Scope: 1.03 is the only measured hierarchy ratio, while platform B lacks separated fast and slow measurements; higher-ratio claims are model extrapolations.Neither measured machine supplies an observation of a hierarchical cluster.

2 Problem Formulation

The formulation separates fast and slow communication levels, collective fixed costs, and arrival-chain overhead in comparing one-hop with two-hop dispatch. It expresses the decision through a breakeven hierarchy ratio while emphasizing that the ratio alone is insufficient.

  • Baseline dispatch: One-hop dispatch emits Tk payload rows per rank per MoE layer, with each row H elements wide, across the full expert-parallel group.The standard expert-parallel formulation uses degree EP and E evenly distributed experts.
  • Machine model: The machine model distinguishes fast intra-group links, slow inter-group links, and the full fabric, tracking both bandwidth and collective issuance cost.The hierarchy ratio βfast/βslow is measured at workload message sizes, while fixed collective costs do not shrink with bytes.
  • Hierarchical dispatch: Two-hop dispatch sends one representative copy per target group, then scatters rows within the group, reducing slow-side payload while adding fast-side rows and one collective.The slow-side payload changes from q = k/M rows per token and target group to one row.
  • Routing constraint: Two-hop dispatch requires routing constraints that limit the groups a token can reach and impose exactly k/M selected experts within each chosen group.These constraints make the hierarchical communication pattern expressible.
  • Decision metric: G compares one-hop with two-hop cost, with G > 1 favoring restructuring; communication-level G concerns dispatch calls, whereas step-level G concerns whole training steps.The paper keeps these levels separate because only the communication-level comparison is assembled from collective costs.
  • Breakeven analysis: The breakeven ratio rbe is a workload threshold, but fixed costs and arrival-chain implementation can move it without changing the machine’s hierarchy ratio.Consequently, machines sharing a ratio can disagree, and one machine can change verdict without changing its fabric.

3 Experimental Design

The experimental design specifies machines, corpora, estimands, validation gates, and reporting rules, distinguishing measured validation from extrapolation and post hoc analysis. Gates are bound to model capabilities, while protocol amendments and registration status are disclosed.

  • Machines: Two machines instantiate the model, but only machine A supplies an observed hierarchy ratio; machine B tests model-form usability without isolating its fast/slow hierarchy.Machine B fits a single asymptotic bandwidth across levels, leaving the level split and implementation-side constants as assumptions.
  • Measurement protocol: Every target is reported as a pooled median over repeated runs because measured drift can exceed the effects resolved by the validation gates.A 2048-token two-hop benchmark produced 0.648, 0.772, and 0.711 ms, with 19% spread about the median; two end-to-end geometries showed 0.6% to 0.8% spread.
  • Corpora: Six scored corpora cover conjunction-scored C2–C4 and separately scored C5, with G1 varying micro-batch size, node count, and cross-group fan-out.G1 uses one calibration geometry and six holdouts; C4 carries the out-of-sample prediction claim for the x1/2 parameter.
  • Falsification conditions: Each gate specifies an estimand, threshold, registration status, and capability removed upon failure, rather than treating failure as an unbound caveat.The applicability criterion was added post hoc after seven G1 geometries, and its refutation rule is stated explicitly.
  • Estimands: Tier-1 and Tier-1b score collective-time relative error, while Tier-2 scores step-time-ratio error and sign agreement on defined holdouts.The sixth G1 geometry is excluded from sign counting because its measured ratio, 0.9935, lies within the run-to-run spread of unity.
  • Reporting policy and protocol amendments: Four protocol changes followed observed instrument problems, weakening claims or loosening bounds; an outcome-favoring alternative reading of α was rejected.The amendments affected downstream equivalence, the x1/2 bound, exploratory overlap families, and the Q1 validation-loss instrument.

4 The Cost Model

The cost model prices one-hop and two-hop collectives using measured fixed costs, payload-dependent bandwidth, and the arrival-side overhead of hierarchical dispatch. Its validation and fitting protocol addresses parameter identifiability, non-pipelined calls, balanced-load assumptions, and estimator-dependent model selection.

  • Strategy costs: Two-hop cost adds a fast-level collective, split-size retrieval, and an arrival chain to the slow-level collective.The arrival chain expands received pairs into the intra-group scatter plan and is measured at 0.0875 µs per row.
  • Strategy costs: The model assumes uniform destination mass when converting total send volume into per-peer message size.Aggregate routing traffic can still be skewed; the released code exposes ρmax as a separate sensitivity, so the base model is not a tail-latency guarantee.
  • Fixed-cost decomposition: Two back-to-back collectives are priced serially because per-call cost stops falling at N ≈16 and remains flat through N = 1024.This behavior indicates that the collectives do not pipeline under the measured scan.
  • Parameter identification: The fixed cost α(w) is measured directly, while only the bandwidth pair is fitted because α and x1/2 are not jointly identified by size sweeps.On C3, the protocol resolves x1/2 as 54 KiB rather than the 320 KiB returned by a free fit.
  • Model form selection: The saturating bandwidth form with α pinned achieves 8.1% median and 12.5% worst-case error, whereas free fitting gives 17.9% median and 101.6% worst-case.Tier-1 evaluates the model on the corpus used for form selection, so it is a fit-quality check rather than independent evidence; C4 is untouched by selection.

5 Validation Results

Validation supports communication-call modeling within measured coverage, but failures in independent drift and step-level checks withdraw broader predictive capabilities. Extrapolation beyond 128 ranks is withdrawn because the only covering corpus is poorly fitted and yields treatment-dependent trend directions.

  • Tier-1 and Tier-1b: Four corpus outcomes pass their gates, while two fail under their registered estimands, thresholds, and statuses.Tier-1 passes, but its targets are pooled medians and its calibration corpus is self-referential; C4 provides the sole out-of-sample prediction test.
  • Tier-1 and Tier-1b: 15.1% median error and −9.1% bias make C5 miss Tier-1b on the world-8 sweep.The discrepancy changes sign across message sizes, and alternative α readings would change the gate outcome, so the authors retain the registered result.
  • Tier-2: end-to-end step time: Tier-2 fails because summed phase timings and step-level deltas disagree by roughly a factor of five under different communication-compute overlap.Single-parameter overlap families were exploratory, none passed, and the tier remains locked because no instrument resolves exposure at the required granularity.
  • Tier-2: end-to-end step time: No family places a single point inside the ±0.035 step-time tolerance band required for four of six holdouts.The closest family has MAE 0.060 and the all-signs-correct family has MAE 0.045 against a 0.025 threshold.
  • Calibration uncertainty: At hierarchy ratio 8, the least favourable propagated-calibration draw remains 1.41, but ratio 1.03 has a zero-overhead p95 of 1.03 and receives no robust sign claim.The uncertainty analysis uses 400 Monte Carlo draws over measured constant spreads, including the bootstrap interval of x1/2.
  • Calibration uncertainty: Beyond 128 ranks, four defensible α treatments span 1.37–2.94 at 512 ranks and reverse the trend direction, so the claim is withdrawn.Through 128 ranks, treatments agree to the digit; the only corpus covering larger worlds is also the model’s worst fit.

6 Applicability

The model’s defensible output is a communication-call sensitivity study, not a deployment prediction. Its applicability depends on measured fast/slow collectives, implementation-specific arrival costs, hidden width, and validation gates.

  • Applicability boundary: A target deployment requires its own fast/slow collective measurements and a passing communication-level gate; nominal link ratios alone do not license prediction.The model’s capability is withdrawn when a gate fails rather than retained with a caveat.
  • Breakeven sensitivity: 3.98 is the effective breakeven hierarchy ratio at the reference geometry with the measured PyTorch arrival chain.The corresponding thresholds are 1.49 for a hypothetical fused target and 1.10 at zero implementation overhead.
  • Hidden-width sensitivity: 2.90 and 2.40 are the measured-chain thresholds at H = 4096 and H = 8192, respectively, because payload bytes grow faster than the measured chain.At the reference width H = 2048, the threshold is 3.98; the chain share of two-hop falls from 46% to 27% over H = 2048 to 8192.
  • Breakeven sensitivity: The arrival chain dominates this local sensitivity calculation: removing it shifts breakeven from 3.98 to 1.10, whereas one extra host exposure shifts it to 4.15.The measured host-exposure overhead is 39% to 59% of a call’s cost.
  • Applicability boundary: Only platform A supplies a measured hierarchy ratio, 1.03; platform B lacks separated fast/slow measurements, so higher-ratio points are synthetic scenarios.The table holds platform A’s calibrated constants fixed and varies only slow-side bandwidth; these are not commercial-platform predictions.

7 Cost of the Enabling Routing Constraint

The enabling routing constraint makes two-hop dispatch expressible by limiting group fan-out and fixing selected-group quota, but it does not make aggregate peer messages constant. Its validation-loss cost is small but nonzero, while downstream and step-time evidence remains qualified.

  • Routing constraints: The group limit bounds each token’s group fan-out, while the per-group quota fixes expert rows contributed to each selected group.Aggregate per-peer counts remain data-dependent, so counts exchange, padding, or capacity-limited transport may still be required.
  • Validation loss: +0.0034 nats is the conjunction’s measured validation-loss cost, equal to 3.4% of the prespecified 0.1-nat no-loss margin.All 24 paired seed deltas are positive, so the effect is resolved and small rather than absent.
  • Validation loss: +0.00063 at 20k and +0.00180 at 30k are the quota costs relative to the group-limited arm.Quota-only and conjunction arms are not nested, so their differing losses do not show that adding the group limit is free.
  • Downstream outcomes: Downstream equivalence is reported for HellaSwag at +0.158 points and LAMBADA at −0.084 points under a ±1.0-point band.The underlying per-seed/checkpoint results and estimator provenance are incomplete, so the tests are not independently reconstructible.
  • Load and step time: Expert load directions favor balance, but the load and step-time axes have no noninferiority margin and therefore establish less than the quality axis.Entropy is no lower and coefficient of variation no higher under constrained modes; step-time ratios across two runs average 0.9976.
  • Scope: The quality experiment uses one geometry, four seeds, a 13.14B-parameter model, and validation loss on a holdout shard rather than a deployed endpoint.Adopters must revalidate quality at their own geometry and scale.

8 End-to-End Test of the Criterion on Seven Geometries

The seven-geometry end-to-end test broadly agrees with the communication-call criterion on machine A, where the measured hierarchy ratio is below every effective breakeven. One geometry is a measured miss, and several points have limited replication.

  • Criterion test: 1.03 is machine A’s hierarchy ratio, below every effective breakeven, so the criterion predicts that two-hop loses there on bytes.The fixed-cost condition clears, but only by about a tenth of a millisecond per call pair.
  • Criterion test: Five of six scorable geometries agree with the predicted direction; the base tier is a measured miss and the seventh lies within the run-to-run spread of unity.The base tier measures above 1 rather than being a rounding artifact, while the seventh is excluded from the sign count.
  • Experimental design: The experiment swaps only the all-to-all implementation while holding cards per group, hidden width, sequence length, batch, precision, and model depth fixed.Each arm uses the median of ten steady-state steps after discarding the first 300.
  • Experimental design: The seven geometries vary scale, micro-batch tokens, and routing load, but no axis isolates a single factor.Global batch remains fixed while expert parallelism changes with node count, altering micro-batches and all-to-all calls per step.
  • Model validation: The adopted model does not reproduce the full ordering across axes: Tier-2 predicts a ratio above 1 at all seven geometries and misses all six holdouts’ tolerance bands.The recovered directional overlap families were enumerated after Tier-2 failed and none was adopted.
  • Replication: Four of the seven geometries were run once, although three lose by margins many times larger than the token-axis spread.The base tier has seven runs, and two other geometries have three runs each.

9 Limits of the Model

The model’s scope is limited by non-square compute modeling, uniform fast-domain assumptions, unresolved compute/communication overlap, and capability gates that remove unsupported predictions.

  • Compute model: A square-curve indexing choice can vary by up to 2.25× across smallest-dimension, geometric-mean, and row-count rules for non-square expert matmuls.The paper uses the geometric mean after direct measurements show the smallest-dimension rule is not appropriate.
  • Compute model: The earlier claim that narrow experts are intrinsically less efficient is withdrawn; the surviving conclusion concerns small work per call, not expert width alone.A 1536×2048×5504 expert FFN measures 34.6 GFLOP and fills the machine better than square-1024 work.
  • Compute model: The geometric-mean rule is within 6.4%, but the residual bias remains uncorrected because eight points on one machine are insufficient for a correction fit.Slightly underpredicting compute safely understates compute’s share when communication is used as an upper bound.
  • Topology assumption: The hierarchy-ratio model assumes one uniform fast-side bandwidth for every accelerator pair inside a domain.This assumption holds on the listed regular domains but fails on heterogeneous fast-side graphs.
  • Topology assumption: For a non-uniform fast domain such as Frontier, the model declines to report a ratio rather than produce a confidently wrong answer.Machines with heterogeneous fast-side graphs require a different model.
  • Capability boundaries: Step-level prediction, statements above world 128, and stronger arrival-chain interpretations were removed by failed gates; compute/communication overlap also remains unresolved.The communication share is therefore treated as an upper bound under no-overlap assumptions.

10 Related Work

TerraceMoE situates its communication-level cost model among hierarchical collective methods, MoE dispatch optimizations, and distributed-training simulators. Its distinguishing contribution is capability-bounded validation rather than a single aggregate accuracy claim.

  • MoE systems research has addressed expert-parallel all-to-all through adaptive parallelism, pipelining, contention management, expert movement, and combined systems.
  • Hierarchical MoE dispatch follows earlier node-local gather, inter-node exchange, and node-local scatter designs, including node-aware allreduce and node-limited routing.
  • The model uses bandwidth-dominated communication costs and omits separate overhead and latency terms as an explicit modelling choice.
  • Compute-side modelling uses a measured compute ceiling indexed by problem size rather than the original roofline model’s operational-intensity axis.
  • Unlike prior simulators that typically report a single aggregate error figure, TerraceMoE adds a different reporting contract centered on validation gates.

11 Conclusion

The paper concludes that hierarchical dispatch can be screened at the communication-call level when target-specific constants and communication gates are available. The failed step-level gate limits the present evidence to communication calls rather than step time or training throughput.

  • Communication-call screening is supported when target constants are measured at workload message sizes and communication-level gates pass.
  • The failed step-level gate prevents the present data from supporting step-time or training-throughput predictions.
  • 3.98, 1.49, and 1.10 are effective breakevens for the measured PyTorch chain, hypothetical fused target, and zero implementation overhead, respectively.These values apply to the reference geometry and are ratio-only sensitivity results.
  • Only machine A has a measured hierarchy ratio, 1.03; machine B’s ratio is unresolved, so higher ratios in the applicability table are synthetic sensitivities.
  • A measured machine in the hierarchical regime, a communication-level holdout, and an overlap-aware step instrument are identified as needed measurements.

Reproducibility

The reproducibility artifact exposes calibrated constants, executable gates, and figure-generation procedures, while withholding raw sweeps and complete downstream estimator inputs. This supports rerunning procedures but not full independent reconstruction of every reported gate outcome.

  • Five target parameters must be measured: α, βfast, βslow, x1/2, and the implementation-specific arrival-chain constant cchain.Only x1/2 requires a wide size sweep; the other four come from single microbenchmarks.
  • The artifact is executable but not a full measurement reproduction package because raw sweeps and complete downstream estimator inputs are unpublished.
  • Every figure is generated by the same script that produces the repository copy, with model figures computed live and measurement values carried inline.
  • The repository contains the cost model, calibration constants, validation gates, figure generation, applicability profile, and capability-enforcing checks.
  • Table 4 distinguishes machine properties, physics, and implementation properties to determine which constants transfer and which require remeasurement.
  • Table 5 records gate outcomes with corpus labels, estimands, thresholds, registration status, and the capability removed by each failure.
  • Table 6 evaluates six single-parameter overlap families on six holdouts using a Tier-2 MAE threshold of ≤0.025.The table distinguishes M0 from the Table 5 Tier-2 row, which reports MAE 0.135.
  • Table 7 reports ratio-only sensitivity as one-hop over two-hop communication-call time, with values above 1 favoring two-hop in the model.
Loading 2608.27874v1…