PaperlayerView exact revision on arXiv

Source-linked AI summary

Janus: Disaggregating Attention and Experts for Scalable MoE Inference

Zhexiang Zhang, Ye Wang, Yumiao Zhao, Jiayu Xiao, Qianjing Yang, Xiangyu Wang, Jingzhe Jiang, Qizhen Weng, Ruichuan Chen, Shaohuai Shi, Yin Chen, Minchen Yu

arXiv:2512.13525v4cs.DC

TL;DR

Large MoE models must meet token-level latency SLOs despite memory-heavy experts, heterogeneous layer demands, and dynamic workloads. JANUS disaggregates attention and MoE, balances expert activations, and jointly scales resources and placement. It improves per-GPU throughput by up to 4.7× over state-of-the-art systems while satisfying SLOs.

  • Problem

    Serving large MoE models under dynamic demand requires meeting token-level latency SLOs while minimizing GPU over-provisioning, but monolithic systems force attention and MoE layers with different demands to share resources.

    Source: §1 · p. 1

  • Method

    JANUS disaggregates attention and MoE onto separate GPU pools and combines adaptive two-phase communication, activated-expert-balanced scheduling, and fine-grained SLO-aware scaling with expert placement.

    Source: §1 · p. 2, §3.4 · p. 7, §3.5 · p. 8

  • Results

    JANUS improves per-GPU throughput by up to 4.7× over state-of-the-art systems while satisfying TPOT SLOs.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Under real-world inference traces, JANUS adjusts attention-side and MoE-side configurations at fine granularity, reducing resource cost by about 40% compared with baselines.

    Source: §1 · p. 2

    Existing disaggregated systems generally do not balance activated-expert counts, and their elasticity remains limited under dynamic workloads.

    Source: §2.3 · pp. 4–5

Abstract

from arXiv · show

Serving large Mixture-of-Experts (MoE) models is challenging because of their large memory footprints, heterogeneous resource demands, and highly dynamic inference workloads. Most existing MoE inference systems deploy the entire model as a monolithic unit, forcing attention and MoE layers to share the same resource configuration despite their different scaling behaviors and resource bottlenecks. Such coarse-grained provisioning leads to resource inefficiency and suboptimal performance. We present JANUS, a scalable and resource-efficient MoE inference system built around three key principles. First, JANUS disaggregates attention and MoE layers onto separate GPU worker pools, enabling independent resource provisioning for the two layer types, and uses an adaptive two-phase communication mechanism for low-latency data exchange. Second, because MoE-layer execution is often memory-bound and highly sensitive to activated-expert imbalance, JANUS introduces a lightweight, microsecond-scale activation scheduler that balances per-layer activated experts across MoE instances to reduce inference latency. Third, JANUS employs a fine-grained, SLO-aware resource scaling scheme that jointly selects attention resources, MoE resources, and expert placement to minimize GPU cost under token-level SLOs. Evaluation shows that JANUS improves per-GPU throughput by up to 4.7x over state-of-the-art MoE inference baselines while satisfying token-level latency SLOs.

1 Introduction

Large MoE models make it difficult to meet token-level latency SLOs without costly over-provisioning, while monolithic serving forces attention and MoE layers with different demands to share resources. JANUS separates and independently scales those layers, balancing expert activation and optimizing communication and placement; evaluation reports higher per-GPU throughput while meeting SLOs.

1 Introduction

  • Dynamic demand and resident expert parameters make it costly to meet token-level latency SLOs without over-provisioning GPUs.Most experts must remain in GPU memory even though each token activates only a small subset.Source: §1 · p. 1
  • Monolithic serving forces attention and MoE layers with distinct resource demands to share configurations, while existing disaggregated systems often retain static expert placement.This coarse-grained provisioning can lead to over-provisioning or degraded performance.Source: §1 · pp. 1–2
  • JANUS separates attention and MoE layers onto distinct GPU worker pools so each can be provisioned and scaled independently.It also aims to balance activated-expert load and minimize resource costs under SLO constraints.Source: §1 · p. 2
  • JANUS uses adaptive two-phase transmission to replace O(m × n) small cross-node transfers with fewer, larger data transfers.It first aggregates intermediate data from intra-node instances, then transfers it in bulk to destination nodes.Source: §1 · p. 2
  • JANUS balances layer-wise expert activations across MoE instances with a fast heuristic, while keeping scheduling and synchronization overhead at the microsecond level.The scheduler runs as a GPU kernel without CPU–GPU synchronization or cross-GPU coordination.Source: §1 · p. 2
  • JANUS jointly scales attention and MoE instances and optimizes expert placement to meet token-level SLOs with minimal GPU resources.Evaluation reports up to 4.7× and 3.3× higher per-GPU throughput than state-of-the-art monolithic and disaggregated systems, respectively, and about 40% lower resource cost than baselines under real-world traces.Source: §1 · p. 2

2 Background and Motivation

The background motivates separate resource provisioning because attention and MoE layers scale differently, MoE latency tracks distinct activated experts, and online demand is bursty. It also explains why existing monolithic and disaggregated systems leave gaps in expert balancing and fine-grained elasticity.

2.1 MoE Inference

  • MoE model memory footprints are dominated by expert parameters, while static provisioning for fluctuating demand risks SLO violations or idle GPUs.The paper focuses on decode-phase MoE inference, where per-token latency accumulates across generated sequences.Source: §2.1 · pp. 2–3

2.2 Characteristics and Requirements of MoE Inference

  • Attention and MoE layers scale differently with parallelism, so a single shared parallelism degree is inefficient.Figure 1 normalizes each panel to latency at the smallest parallelism degree and includes ideal linear scaling for comparison.Source: Figure 1 · p. 3, §2.2 · p. 3
  • MoE latency rises approximately linearly with the number of distinct activated experts, while batch-size increases and uniform versus skewed activations have marginal or nearly identical effects when all experts activate.These measurements use a 32-expert DeepSeek-V2 layer and support balancing distinct activated-expert counts across GPUs.Source: Figure 2 · p. 3, Figure 3 · p. 4, §2.2 · p. 4
  • Online request arrivals are highly bursty, with peaks reaching approximately 7.5× the trace-wide mean.The trace covers one week of production LLM serving.Source: Figure 4 · p. 4

2.3 Limitations of Existing Solutions

  • Monolithic systems co-locate attention and MoE, couple their memory budgets, and scale by reconfiguring or replicating full model instances.They also lack layer-wise expert activation scheduling and fine-grained elasticity.Source: §2.3 · pp. 4–5
  • Disaggregated systems separate attention and MoE execution but generally do not balance activated-expert counts or provide fine-grained elasticity for changing workloads.MegaScale-Infer tunes the attention-to-MoE resource ratio, but its restricted configuration space can produce suboptimal performance or SLO violations.Source: §2.3 · p. 5

3 JANUS Design

JANUS disaggregates attention and MoE layers into independently managed GPU sub-clusters, then addresses communication, activation balancing, and SLO-aware scaling challenges. Its design combines adaptive communication, activation scheduling, and joint resource and expert-placement decisions.

3.1 Design Principles and Challenges

  • Disaggregation creates three challenges: cross-cluster activation transfers, balancing activated experts with microsecond-level scheduling overhead, and dynamically finding low-cost configurations that meet token-level SLOs.The resource-efficiency objective is per-GPU throughput; the resource decision also includes expert-replica placement.Source: §3.1 · p. 5

3.2 Design Overview

  • JANUS uses separate attention and MoE GPU sub-clusters, independently managed and scaled, with low-latency transfers between them.On the MoE side, instances run a lightweight GPU-kernel scheduler and collect activation statistics for periodic resource and placement adjustments.Source: §3.2 · pp. 5–6

3.3 Adaptive Two-Phase Communication

  • JANUS performs gating on the MoE side, aggregates activations within source nodes, then adaptively sends aggregated payloads directly or through a designated MoE node.The strawman direct approach creates O(m×n) point-to-point transfers; the adaptive scheme selects its transfer regime based on resource configuration and traffic load.Source: §3.3 · p. 6

3.4 Activated-Expert-Balanced Scheduling

  • AEBS assigns replicated activated experts to the least-loaded eligible MoE instances, greedily reducing the maximum activated-expert count with negligible computational overhead.The GPU-kernel scheduler runs independently on each instance with identical inputs, avoiding cross-instance synchronization while preserving the same assignment.Source: §3.4 · p. 7

3.5 Fine-Grained Scaling under SLOs

  • JANUS jointly selects attention and MoE instance counts under token-level SLOs, choosing the feasible configuration with the fewest GPUs and placing replicas to limit co-activation pressure.It estimates activated-expert counts from recent activation traces and solves for steady-state batch size; scaling is reapplied as demand changes.Source: §3.5 · pp. 8–9

4 Implementation

JANUS extends SGLang to support disaggregated MoE inference, reusing its attention-side serving components and implementing communication and scheduling for the separated pools.

4 Implementation

  • JANUS is implemented on SGLang with about 4K lines of Python and 300 lines of CUDA/C++ code.It reuses SGLang request batching, dispatching, and KV-cache management on the attention side.Source: §4 · p. 9
  • Cross-sub-cluster transfers use NVSHMEM and GPUDirect RDMA, while intra-node NVLink collectives use NCCL.JANUS uses one-sided NVSHMEM primitives and packs lightweight metadata into the signal value.Source: §4 · p. 9
  • JANUS overlaps shared-expert execution with attention-to-MoE data transfer and runs AEBS as a GPU kernel on each MoE instance.CPU-side metadata unpacking occurs only at the first MoE layer, with the result reused at subsequent layers.Source: §4 · p. 9

5 Evaluation

JANUS is evaluated against monolithic and disaggregated MoE inference systems on representative models and workloads. It meets evaluated TPOT SLOs while improving throughput and reducing resource use, and its communication, scheduling, and scaling mechanisms are examined separately.

5.1 Experimental Setup

  • JANUS was evaluated on up to four H100 nodes using ShareGPT and BurstGPT workloads, against monolithic SGLang and disaggregated MegaScale-Infer and xDeepServe baselines.The setup includes DeepSeek-V2, Qwen3-MoE, and scaled DeepSeek-style variants.Source: §5.1 · pp. 9–10

5.2 End-to-End Performance

  • 4.7×, 2.2×, and 3.3× higher per-GPU throughput than SGLang, MegaScale-Infer, and xDeepServe, respectively, while JANUS met target SLOs across evaluated models and batches.The comparison covers DeepSeek-V2 and Qwen3-MoE at the reported SLOs and batch sizes.Source: Figure 8 · p. 10, §5.2 · p. 10
  • At B=256, relaxing the SLO shifted JANUS from 5A10E to 2A6E and raised TPG from roughly 170 to 240 tok/s/GPU; 4A6E reached about 340 tok/s/GPU at the most relaxed SLO.At B=64, 1A6E satisfied all evaluated SLOs at about 99 tok/s/GPU; the strictest SLO was infeasible at B=512.Source: Figure 9 · p. 10, §5.2 · pp. 10–11
  • On Scaled-DS-2, E16 reduced TPOT by 41–50%; on a 24-hour trace, JANUS used 7–56 GPUs and cut GPU-hours 39% versus SGLang and 16% versus MegaScale-Infer.The trace-driven simulation maintained target latency requirements; the Scaled-DS-2 result reflects restored expert redundancy at 16 MoE instances.Source: Figure 10 · p. 11, Figure 11 · p. 11, §5.2 · p. 11

5.3 Microbenchmarks

  • 2PC+EGate improved throughput 4–34% over 2PC+AGate, and adding AEBS increased throughput a further 11–15%.Without intra-node aggregation, 1PC+EGate raised TPOT to 185 ms and 350 ms at batch sizes 256 and 512, respectively.Source: Figure 12 · p. 11, §5.3 · p. 11
  • AEBS reduced maximum activated-expert counts versus EPLB across batch sizes, with larger reductions at 16 than 8 MoE instances and greater latency gains at larger scales.E16 substantially reduced MoE-layer latency at larger batch sizes, while EPLB remained close to baseline latency.Source: Figure 13 · p. 11, Figure 14 · p. 12, §5.3 · p. 12
  • AEBS scheduling overhead stayed below 20 µs at small batch sizes and below 90 µs even at batch size 4096.Its cost gradually plateaus as most experts become activated, and increasing the MoE sub-cluster from 8 to 16 GPUs adds only small overhead.Source: Figure 15 · p. 12, §5.3 · p. 12
  • JANUS selected 1A6E, 2A6E, and 4A6E configurations that met SLOs with 7–10 GPUs, illustrating the value of searching asymmetric attention/MoE allocations.The search-space figure plots TPG against total GPU count and marks SLO-feasible configurations.Source: Figure 16 · p. 12, §5.3 · p. 12

6 Discussion and Other Related Work

JANUS can map attention and MoE instances to separate hardware pools, including heterogeneous hardware environments. The discussion cautions that pipelining may offer little latency benefit at typical online batch sizes while adding overhead and interference.

Heterogeneous hardware

  • JANUS can map attention and MoE instances to separate hardware pools in heterogeneous systems, including compute-focused GPUs and bandwidth-focused MoE accelerators.The passage cites Vera Rubin and LPX as an example platform pairing.Source: §6 · p. 12

Pipelining across attention and MoE

  • For typical online batch sizes often below 100, splitting batches into micro-batches provides little per-micro-batch latency benefit.The passage reports measurements for this setting.Source: §6 · p. 12
  • Micro-batch pipelining adds synchronization overhead, implementation complexity, and resource interference; effective pipelining requires careful coordination and complements JANUS.Source: §6 · p. 12

7 Conclusion

JANUS disaggregates attention and MoE layers into separate GPU sub-clusters and scales them independently. Its mechanisms support low-latency, resource-efficient inference under dynamic workloads, with reported SLO attainment and throughput gains.

7 Conclusion

  • JANUS separates attention and MoE layers into distinct GPU sub-clusters and scales them independently.Source: §7 · p. 12
  • Adaptive two-phase communication, activated-expert-balanced scheduling, and fine-grained resource scaling are JANUS’s mechanisms for MoE inference under dynamic workloads.Source: §7 · p. 12
  • JANUS satisfies TPOT SLOs and improves per-GPU throughput by up to 4.7× over state-of-the-art systems.Source: §7 · p. 12

A Theoretical Bound for amax

The appendix derives a conservative upper bound on maximum activated experts per instance and validates it against ShareGPT estimates. Scheduling leverage is concentrated at B∈[10,100], where modest placement shifts can affect TPOT SLOs.

A Theoretical Bound for amax

  • The bound models B-token expert hits as balls into bins and assumes replicated-expert activations all land on the analyzed instance, independent of scheduler routing.For each expert, the hit probability is 1−(1−pe)^B; instance load is bounded by the number of hosted experts hit.Source: A Theoretical Bound for amax · p. 16
  • Under uniform activation, expected load grows with B toward C; negative association and a Bernstein-type tail bound yield an instance-wise bound with +1 slack for replication overflow.The bound considers the maximum expected instance load across ne instances.Source: A Theoretical Bound for amax · p. 16
  • The bound never under-predicts: in saturation (B ≥64), its gap is within one or two experts, while at small B it remains below ∼2×.The one-sided error means the scaling solver errs toward over-provisioning, the safe direction under SLO constraints.Source: A Theoretical Bound for amax · p. 16
  • Although the adversarial bound is conservative, its values stay within or below the observed 8–22 activated-expert range; the Monte Carlo estimate absorbs placement and AEBS effects.The bound does not drive the solver into regions that contradict measured behavior.Source: A Theoretical Bound for amax · p. 16
  • At B∈[10,100], each 4× increase in B raises amax by 4–7 experts (ne=6: 8.46→15.66 from B=16→64), at 30–60% of C=27.Per-instance batch sizes in online decode traces fall in this high-leverage window.Source: A Theoretical Bound for amax · p. 16

B Activation-Aware Replica Placement

JANUS assigns redundant expert replicas according to activation pressure, then places them to minimize the maximum co-activation load across instances. A greedy heuristic approximates this min–max objective while supporting periodic online reconfiguration.

B Activation-Aware Replica Placement

  • After seating one replica of each logical expert, JANUS assigns remaining slots iteratively to the expert with the highest per-replica activation load, equalizing pressure across replicas.It uses activation counts from a sliding window; hot experts gain replicas while cold experts can remain singletons.Source: B Activation-Aware Replica Placement · p. 17
  • JANUS minimizes the maximum per-instance co-activation load, estimated from recent traces; colocating experts with high co-activation raises concurrent activations and MoE latency.The placement uses replica assignments and per-instance capacity.Source: B Activation-Aware Replica Placement · p. 17
  • The min–max placement problem is NP-hard; JANUS greedily places replicas at the least co-activation penalty and uses bounded swaps when no feasible slot remains.The heuristic processes replicas in descending load order and is efficient enough for periodic online reconfiguration.Source: B Activation-Aware Replica Placement · p. 17

Related papers

Loading 2512.13525v4…