Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

1. Introduction

• Expert parallelism distributes MoE computation across devices, but its All-to-All communication can contribute approximately 50% of total execution time and impede scaling.Existing hierarchical All-to-All and pipelining mitigate communication delays only partially because communication and computation remain sequentially dependent.Source: §1 · p. 1
• ScMoE routes preceding-layer representations through a top-1 MoE and processes current-layer representations with a shared expert, combining their outputs in the current layer.This shortcut-connected design decouples communication from the conventional sequential ordering of MoE operations.Source: §1 · p. 1
• All-to-All communication accounts for 60% of total time on 8×A30 GPUs, 15% on 8×A800 GPUs, and approaches 50% across multiple nodes.The passage attributes the lower single-node share on A800 to its 6× higher GPU-to-GPU NVLink bandwidth, while inter-node Ethernet keeps communication costly.Source: Figure 1 · p. 2
• Adaptive scheduling can double overlap duration versus pipelining and fully overlap communication when its duration does not exceed computation.The strategy dynamically schedules operators using actual performance metrics and is described as device-agnostic and compatible with current optimizations.Source: §1 · p. 2
• 1.49× and 1.14× training speedups, and 1.82× and 1.21× inference speedups, are reported for ScMoE versus standard top-2 MoE on 8×A30-PCIe and 8×A800-NVLink, respectively.The configurations represent high and low communication-overhead scenarios, respectively.Source: §1 · p. 2
• Experiments and analyses report comparable or better model quality than existing methods on vision and language downstream tasks.The paper also reports theoretical analysis and discussion of the shortcut connection.Source: §1 · p. 2

2.1. Sparsely-Gated Mixture of Experts

• MoE layers comprise multiple MLP experts and a trainable gating network that selects a subset of experts for each iteration.Source: §2.1 · p. 2
• Larger top-k values move MoE closer to a dense architecture, generally improving prediction accuracy while increasing computational overhead.Source: §2.1 · p. 3
• The prevailing top-2 design intersperses MoE blocks with conventional MLP blocks, while MoE modules may also appear in every block or every four blocks.The passage cites prior work for these placement patterns: [Lepikhin et al., 2021; Du et al., 2022; Shen et al., 2023; Hwang et al., 2023; Lieber et al., 2025] and [Jiang et al., 2024; Dai et al., 2024] for every-block placement, and [Zoph et al., 2022; Xue et al., 2024] for every-four-block placement.Source: §2.1 · p. 3
• Shared-expert MoE combines a fixed dense MLP processing all tokens with the top-1 gating expert, reducing dynamic expert selection and communication volume while activating the same number of experts as standard MoE.The architecture was originally proposed by DeepSpeed-MoE [Rajbhandari et al., 2022]; later studies report model quality on par with standard MoE.Source: §2.1 · p. 3

2.2. Expert Parallelism

• Expert parallelism assigns unique experts to devices and routes tokens through gate routing, input encoding, All-to-All dispatch, expert computation, All-to-All combine, and output decoding.Input encoding aggregates token data into a contiguous layout before dispatch, and output decoding reverses the process after combine.Source: §2.2 · p. 3

3.1. Architectural Design

• ScMoE sends preceding-layer representations to a top-1 MoE through a shortcut while a shared expert processes current-layer representations; their outputs are combined.The two operations are conducted independently, facilitating communication-computation overlap.Source: §3.1 · p. 4
• Pos-1, Pos-2, and Pos-3 connect to the preceding layer’s output, intermediate representation, and input, with overlap windows of TAtten + TSE, TAtten + TSE + TMLP, and 2TAtten + TSE + TMLP, respectively.The shared expert processes current-layer representations in each configuration.Source: §3.1 · p. 4
• Pos-2 achieved the highest accuracy in both vision and language experiments while retaining substantial overlap duration.The authors favor Pos-2 for practical development.Source: §3.1 · p. 4
• With an MoE module in every Transformer block, Pos-1 fully uses the available overlap window; less frequent MoE placement extends that window and may increase acceleration.The architecture supports varying MoE placement frequencies.Source: §3.1 · p. 4

3.2. Overlapping Strategy for Expert Parallelism

• Adaptive scheduling places gate routing and encoding as early as viable, defers decoding, and selects among four expert-computation locations based on operator timing.Execution times vary with model and hardware configuration.Source: §3.2 · p. 5
• ScMoE reduces total time by combining shorter communication with overlap across TAtten + TSE + TMLP; communication can be fully hidden when it fits the window, with pipelining available when it exceeds that window.Unlike pipeline parallelism, the strategy can overlap initial and terminal data transmissions.Source: §3.2 · p. 6

4.1. Experimental Setup

• Experiments cover SwinV2-MoE, GPT2-MoE, GPT3-MoE, and LLaMA2-MoE on 8×A30-PCIe, 8×A800-NVLink, and two-node 16×A800-NVLink; shared-expert MoE and ScMoE use one shared and one gate-routed expert to match top-2 computation.Source: §4.1 · p. 6, §4.2 · p. 6

4.2.1. VISION MODEL

• In 8×A30-PCIe, ScMoE (Pos-2) matches standard top-2 MoE at 79.3% accuracy and improves training speed by 30% and inference speed by 40%.Source: §4.2.1 · p. 6, Table 1 · p. 6

4.2.2. LANGUAGE MODEL

• ScMoE models score 38.69 in GPT2-MoE and 38.96 in LLaMA2-MoE; versus standard top-2 MoE, GPT2-MoE training and inference speed improve by 11% and 15%, while LLaMA2-MoE accelerates by 1.14× and 1.21×.Source: §4.2.2 · p. 6, Table 2 · p. 7

4.2.3. ANALYSIS OF OVERHEAD AND ACCELERATION

• In 8×A30-PCIe, ScMoE overlaps 70% of communication and improves speed by 27% over shared-expert MoE, 42% over pipelined top-2 MoE, and 15% over pipelined top-1 MoE.In 8×A800-NVLink, it fully overlaps communication despite minimal communication overhead.Source: §4.2.3 · p. 7, Figure 8 · p. 7
• Across two-node 16×A800-NVLink, ScMoE fully overlaps communication and improves speed by 24% over shared-expert MoE and 43% over pipelined standard top-2 MoE.The inter-node Ethernet yields more communication overhead than single-node 8×A800-NVLink.Source: §4.2.3 · p. 7, Figure 8 · p. 7
• ScMoE accelerates standard top-2 MoE and outperforms top-1 MoE when communication exceeds approximately 20% of total MoE time; it can fully overlap communication when that share is at most an estimated 50%.Source: §4.2.3 · p. 7

5.1.1. ANALYSIS OF GATING BEHAVIORS

• Repeated top-1 expert selection for current- and preceding-layer representations peaks at 98%, while a distinct-expert constraint makes DGMoE’s model quality comparable to standard top-2 MoE.The repeated-selection percentage initially rises and later declines; the reported correlation with representation distance is discussed as a possible explanation.Source: §5.1.1 · pp. 7–8, Figure 9 · p. 8

5.1.2. ANALYSIS OF SIMILARITY IN REPRESENTATIONS

• Adjacent Transformer-block representations have cosine similarity close to 1.0, and ScMoE routes preceding-layer representations while a shared expert processes current-layer representations.Similar observations in LLaMA2-MoE and OLMoE are cited as further evidence of generalizability; the discussion also provides a theoretical gradient-propagation analysis.Source: §5.1.2 · p. 8, Figure 10 · p. 8

5.2. Configuration of ScMoE Architecture

• CG-1 has the lowest final validation loss among the tested combination methods, while Pos-2 has the lowest among tested positions and outperforms Pos-2 (L0 Pos-1).The position comparison uses CG-1; all three combination methods use Pos-2. The vision experiments also report highest accuracy for Pos-2.Source: §5.2 · p. 9
• Pos-2 (L0 Pos-1) places the first MoE module at Pos-1 and the remaining modules at Pos-2, but yields slightly higher loss than Pos-2 alone.The discussion says ScMoE quality is not necessarily improved by connecting representations that are more or less similar.Source: §5.2 · p. 9

5.3. Optimization for Memory-Limited Inference

• Up to 60% lower peak GPU memory and up to 75% lower expert-migration costs are reported for overlapping determinate migration in memory-limited inference.The strategy builds on ScMoE’s advance of expert selection to the preceding layer.Source: §5.3 · p. 9

• ScMoE and a communication-overlapping parallel strategy address the dependency between communication and computation, with reported efficiency gains and model quality maintained or exceeded relative to existing methods.The conclusion also notes theoretical analysis and discussion of ScMoE.Source: §6 · p. 9

• The gradient analysis is presented for ScMoE (Pos-2), with the authors stating that its principles and derivations extend to other shortcut-connected MoE architectures.The derivation builds on Equations 7 to 10 and applies the chain rule to the loss E.Source: §A.1 · p. 14
• The analysis treats each Block-MoE and Block-MLP pair as one entity and recursively derives the uppermost sub-layer output from the sub-layer input-output relation.The prose identifies x_l as the input and x_l+1 as the output of the l-th sub-layer.Source: §A.1 · p. 14
• The additive error-gradient component provides direct information propagation to any sub-layer and has a number of product elements independent of network depth.The authors argue this makes vanishing or exploding gradients less likely as depth increases, supporting stable training and sustained performance.Source: §A.1 · p. 14

• DGMoE independently applies dual top-1 gating to preceding- and current-layer representations and prevents duplicate expert activation by switching to the second-highest-ranked expert for the current layer.If the current layer selects the same expert as the preceding layer, the method activates the second-highest-ranked expert from the top-2 selection.Source: §A.2 · p. 14
• DGMoE achieves accuracy comparable to standard top-2 MoE across vision and language tasks, while ScMoE performs more like shared-expert MoE.Source: §A.2 · p. 14

A.3. Shortcut-connected MoE for Optimizing Memory-Limited Inference

• CPU offloading addresses inference memory limits, but transferring selected experts after selection blocks computation until the transfer finishes.For autoregressive generation, only the two activated top-2 experts for each token need to move from CPU to GPU.Source: §A.3 · p. 15
• Pre-gated MoE predicts expert selection from preceding layers to preload experts, but speculative migration can be inaccurate and reduce inference accuracy.Speculative methods overlap migration with preceding-module computation while retaining standard MoE representation flow.Source: §A.3 · p. 15
• ScMoE uses the gate-routed expert on preceding-layer representations, enabling early determinate migration while maintaining pre-trained model logic.Source: §A.3 · p. 15
• Existing expert migration methods cannot overlap expert-parallel All-to-All communication because representation-flow dependencies remain coupled.Source: §A.3 · p. 15

A.3.1. EXPERT OFFLOADING STRATEGY

• Nonexpert and shared-expert modules stay on the GPU, while gate-routed experts are offloaded to CPU memory.Source: §A.3.1 · p. 15
• After preceding-layer Attention, the gate selects an expert and triggers asynchronous migration, which overlaps transfer with computation without speculation.Source: §A.3.1 · p. 15
• Figure 12 contrasts speculative and determinate migration; dashed red marks expert selection and parameter transfer, while solid lines mark representation flow.Source: Figure 12 · p. 15

A.3.2. EVALUATION

• Offloading cuts peak GPU memory by 50% for GPT2-MoE-Medium and 60% for GPT3-MoE-XL; asynchronous migration reduces added costs by 75% and 25%, respectively.Blocking migration adds 80% overhead for GPT2-MoE-Medium and 240% for GPT3-MoE-XL; migration costs make up a larger latency share in the XL model because decoding is memory-bound.Source: §A.3.2 · pp. 15–16

A.4. Analysis of More Activated Experts

• ScMoE-2 has validation perplexity 16.27 versus ScMoE’s 16.46, and uses 95% and 93% of standard top-2 MoE’s time in training and inference.At equal computational volume, ScMoE architectures are reported to exceed standard top-3 MoE in model quality and efficiency.Source: §A.4 · p. 16, Table 4 · p. 16

A.5. Coefficient Gating Network in Vision Task

• In the vision experiments, coefficient gating improves performance; without it, shared-expert MoE quality falls from standard top-2 MoE to standard top-1 MoE.Source: §A.5 · p. 16, Table 5 · p. 16

A.6. Evaluation Across Different Model Sizes

• Tables 6 and 7 report analogous model quality across model sizes for the evaluated MoE architectures.Source: §A.6 · p. 16, Table 6 · p. 16, Table 7 · p. 17

A.7. Share MoE Across Multiple Layers via Shortcut Connections

• DGMoE-Share reduces parameters from 157M to 124M at the same expert-computation volume as standard top-1 MoE, with 78.45% accuracy and a 0.5% decrement.The authors state that training-hyperparameter optimization for shortcut-connected MoE needs further investigation.Source: §A.7 · pp. 16–17

A.8. Experimental Details

• Experiments span 8×A30-PCIe, 8×A800-NVLink, and two-node 16×A800-NVLink setups, with SwinV2-MoE vision and GPT/LLaMA language models.Vision modifications were applied to stage 3; efficiency results averaged samples across periods, with hyperparameters and random seeds held consistent for comparisons.Source: §A.8 · p. 17

A.9. Additional Examples of Intermediate Representations Similarities

• Figures 14 and 15 provide additional intermediate-representation similarity examples for LLaMA2 and OLMoE.Source: Figure 14 · p. 18, Figure 15 · p. 18