Source-linked AI summary
AFORE: Attention-FFN Disaggregation with Overlapped Reconfiguration of Experts
Wenshuang Li, Youhe Jiang, You Peng, Jiawei Jiang, Binhang Yuan
TL;DR
Input-dependent expert routing and dynamic workloads make MoE serving difficult, while AFD can turn expert imbalance into a pipeline bottleneck. AFORE schedules expert reconfiguration using upcoming microbatch demand and overlaps migration with prior work. Across four dynamic workloads, it improves throughput and P95 inter-token latency over the strongest competing baseline.
Problem
Input-dependent expert activation and changing workloads make MoE serving costly, and AFD can make skewed experts bottleneck the independent FFN stage.
Source: §1 · pp. 1–2Method
AFORE uses microbatch-level demand and migration-aware scheduling to adapt expert placement, overlapping migration with computation of preceding in-flight microbatches.
Source: §1 · p. 3Results
AFORE improves output throughput by 10.1–17.6% and reduces P95 inter-token latency by 7.1–9.5% versus the strongest competing baseline across four dynamic workloads.
Source: §1 · p. 4Takeaways & Limitations
Migration profiling shows that AFD pipeline overlap can fully hide expert-migration latency.
Source: §1 · p. 4
Abstract
from arXiv · showhide
Efficient serving of Mixture-of-Experts (MoE) models is challenging due to large expert parameters, input-dependent expert activation, and dynamic workloads. Expert parallelism distributes expert computation across GPUs, while attention-FFN disaggregation (AFD) separates attention and feed-forward computation into independent worker pools. However, we observe that a naive AFD implementation could make expert load imbalance more harmful: once FFN computation becomes an independent pipeline stage, overloaded experts directly slow the FFN stage and degrade end-to-end serving performance. To solve this problem, we present AFORE, a timely expert reconfiguration system for AFD-based MoE serving. AFORE exploits two architectural properties of AFD. First, AFD exposes the expert-token distribution of upcoming microbatches before they reach FFN execution, enabling placement decisions based on near-future demand instead of stale historical profiles. Second, AFD creates a pipeline window in which expert migration for a target microbatch can be overlapped with the computation of preceding in-flight microbatches. AFORE formulates expert reconfiguration as a microbatch-aware scheduling problem and uses a migration-aware scheduler to decide when and which experts to migrate. AFORE further implements lightweight demand prefetching and NVLink-based GPU-GPU expert migration to reduce reconfiguration overhead. Evaluation on a 110B-parameter MoE model across four dynamic workloads shows that AFORE improves output throughput by 10.1-17.6% and reduces P95 inter-token latency by 7.1-9.5% compared with the strongest competing baseline. Compared with static placement, AFORE improves throughput by 29.8% on average and reduces P95 inter-token latency by 18.2% on average. Migration profiling further shows that AFD pipeline overlap can fully hide expert-migration latency.
1 Introduction
Input-dependent expert routing creates load imbalance, and AFD can make it a pipeline bottleneck. AFORE uses upcoming microbatch demand and pipeline overlap to reconfigure experts; evaluation reports higher throughput and lower P95 latency than competing approaches.
1 Introduction
- AFD makes expert imbalance a distinct FFN-stage bottleneck that can create pipeline backpressure and degrade end-to-end serving performance.Existing reconfiguration also incurs scheduling, migration, metadata-update, and coordination overhead that can offset load-balancing benefits when exposed on the critical path.Source: §1 · pp. 2–3
- Input-dependent routing makes expert demand vary across requests, layers, and generation phases, so static or historical placement can miss upcoming demand.In AFD, skewed routing reduces normalized throughput by 20.1%, compared with 11.9% under monolithic attention-FFN execution.Source: §1 · pp. 2–3, Figure 2 · p. 3
- AFD exposes a target microbatch’s expert-token distribution before FFN execution, letting placement decisions use near-future rather than stale demand.The pipeline window also lets migration overlap with preceding in-flight microbatch computation, reducing migration overhead exposed on the critical path.Source: §1 · p. 3
- AFORE formulates reconfiguration as microbatch-aware scheduling, evaluating placement changes and migrating only when expected FFN-bottleneck reduction outweighs exposed migration cost.Its runtime prefetches demand, updates placement at microbatch boundaries, and uses NVLink-based GPU-GPU expert transfer.Source: §1 · pp. 3–4
- AFORE improves output throughput by 10.1–17.6% and reduces P95 inter-token latency by 7.1–9.5% versus the strongest competing baseline across four dynamic workloads.On static placement, it improves throughput by 29.8% on average and reduces P95 inter-token latency by 18.2% on average; migration profiling shows AFD overlap can fully hide migration latency.Source: §1 · p. 4
2 Background and Related Work
Prior work scales MoE execution through expert parallelism, disaggregates attention and FFN computation, and addresses expert-load imbalance through routing, dispatch, statistics, or placement. AFORE focuses on timely serving-time placement adaptation under AFD and schedules reconfiguration to overlap migration with ongoing work.
2 Background and Related Work
- Expert-parallel performance depends on expert placement because uneven activation can leave some GPU workers as stragglers.FastMoE and Tutel optimize distributed expert execution and token dispatch, while DeepSpeed-MoE supports large-scale MoE training and inference.Source: §2 · p. 4
- AFD separates attention and feed-forward computation into independently provisioned worker pools, making FFN-side load balance relevant to pipeline service rate.Figure 3 tracks P95 inter-token latency and output throughput across static placement and three progressively enabled reconfiguration approaches.Source: §2 · p. 4, Figure 3 · p. 5
- Prior imbalance approaches include load-balancing losses, dispatch optimization, routing statistics, and expert placement or prediction methods.These generally rely on training-time regularization, static placement, or historical feedback, whereas AFORE uses timely workload information from the AFD microbatch pipeline.Source: §2 · pp. 4–5
- Expert reconfiguration adapts placement, replication, or resources to runtime load, but its overhead can offset load-balancing benefits.AFORE treats reconfiguration as scheduling and overlaps migration with preceding in-flight microbatch computation.Source: §2 · p. 5
3 Case Study and Motivation
The case study tests whether dynamic expert reconfiguration, microbatch-level demand prefetching, and migration overlap improve AFD-based MoE serving. Together, these mechanisms reduce imbalance and exposed migration wait while improving throughput and P95 inter-token latency.
Case study results
- 30.3% higher throughput and 17.8% lower P95 inter-token latency over Static with the complete design; mean exposed migration wait falls to 0.000 ms.Approach 3 also improves throughput by 16.4% and reduces P95 inter-token latency by 11.8% compared with Approach 2.Source: Figure 4 · p. 6, §3 · p. 6
- Static placement leaves mean token-assignment EP load imbalance at 1.70×, while even history-based reconfiguration reduces FFN-stage stragglers.The case study evaluates a dynamic serving workload on a 110B-parameter MoE model with eight FFN workers.Source: §3 · p. 6
- 22.2% reduction in mean token-assignment EP load imbalance (1.42× to 1.10×) with prefetch versus Approach 1, alongside 7.1% higher throughput and 3.0% lower P95 inter-token latency.Using the target microbatch’s actual demand better matches placement to upcoming FFN demand than relying on historical demand.Source: §3 · p. 6
- Overlapping migration with preceding in-flight FFN computation reduces exposed migration wait by 100% versus Approach 2.AFD’s pipeline window allows migration to proceed after target demand is known but before that target executes on the FFN side.Source: §3 · pp. 6–7
Implications for system design
- AFORE combines microbatch-level demand prefetching, migration-aware placement scheduling, and fast migration within the AFD pipeline window.The design implications are that placement must adapt to changing demand and that overlap can reduce exposed migration overhead.Source: §3 · p. 7
4 AFORE Overview
AFORE adds an expert-reconfiguration layer to an AFD-based MoE runtime. It observes upcoming demand, evaluates placement changes, and migrates experts while preceding microbatches continue executing.
AFORE Overview
- AFORE’s three components—demand prefetcher, placement scheduler, and placement migrator—observe demand, decide whether to adjust placement, and carry out migration.The overview presents these components as a layer between attention and FFN workers.Source: §4 · p. 7
- The demand prefetcher extracts a target microbatch’s expert-token distribution on the attention side before it reaches FFN execution.The scheduler combines prefetched demand with current placement and FFN worker status to estimate imbalance and choose a target placement when useful.Source: §4 · pp. 7–8
- The placement migrator runs selected migrations while preceding in-flight microbatches compute on FFN workers.After migration completes, the new placement is committed at a microbatch boundary for the target or subsequent microbatches.Source: §4 · p. 8
5 Scheduling Framework
AFORE makes expert placement decisions for each target microbatch using its upcoming demand and the migration overhead exposed on the FFN critical path. Its scheduler first exploits existing replicas, then considers new ones when their net gain is positive.
5.1 Problem Formulation
- At each scheduling epoch, AFORE observes the target microbatch’s expert demand and chooses whether to retain the current placement or migrate replicas to relieve the FFN bottleneck.The objective accounts for migration overhead as well as end-to-end AFD performance.Source: §5.1 · p. 8
- Grouped-GEMM tile load, rather than token load alone, captures worker work because assignments from different experts cannot share a tile.The scheduler models tile load using the kernel’s M-dimension tiles.Source: §5.1 · p. 9
- AFORE selects placements by weighing FFN-time savings against exposed migration cost, counting only migration beyond the available overlap window.The scheduler installs a placement change only when its net gain is positive.Source: §5.1 · pp. 9–10
5.2 Scheduling Algorithm
- AFORE uses a three-phase greedy search: initialize assignments, rebalance across existing replicas, then explore adding replicas.The search exploits the current placement before incurring new weight-copy costs.Source: §5.2 · p. 10
- New-replica candidates must satisfy slot and copy-budget constraints; the scheduler saves the best positive-gain plan and otherwise retains the preceding phase’s plan.Slots in use or reserved by preceding microbatches cannot be rebound.Source: §5.2 · p. 11
6 AFORE Design and Implementation
AFORE uses AFD’s early visibility into target-microbatch demand and its pipeline window to plan expert placement and overlap replica migration with preceding computation.
6.1 Demand Prefetching and Plan Delivery
- Attention workers derive target-microbatch routing demand, which AFORE sends to the scheduler before the full execution input is assembled.The IDs and weights are reused for FFN execution, avoiding a second gate and top-k computation.Source: §6.1 · p. 12
- The CPU scheduler aggregates worker demand, prepares a target-tagged plan, and publishes it once through a shared-memory ring for all FFN workers.A queue slot is reused only after every FFN worker has copied its plan.Source: §6.1 · p. 12, Figure 6 · p. 12
6.2 Placement Migrator and Plan Execution
- Workers read and validate the next plan, then submit missing GPU-to-GPU copies on a dedicated migration stream while the current microbatch continues executing.Copy destinations exclude slots still used by the in-flight microbatch; a completion event tracks replica readiness.Source: §6.2 · p. 13, Figure 7 · p. 13
- Before using the next plan, workers check replica-copy completion and wait only for unfinished copies, then partition token–expert pairs according to the plan.Workers share gathered routing results to select disjoint intervals, and an all-reduce combines partial outputs before StepMesh returns slices to attention.Source: §6.2 · pp. 13–14
7 System Evaluation
AFORE is evaluated on a 110B-parameter MoE model across four dynamic workloads, with throughput, tail latency, scheduler quality, ablations, and migration overlap examined.
7.1 Experimental Setup
- The evaluation uses GLM-4.5-Air, a 110B-parameter MoE model, and four replayed dynamic workloads spanning conversation, documents, coding, and mathematical reasoning.The reported serving metrics are output throughput and P95 inter-token latency.Source: §7.1 · p. 14
7.2 End-to-end Performance (RQ1)
- Across four workloads, AFORE improves output throughput by 10.1–17.6% and reduces P95 ITL by 7.1–9.5% versus each workload’s strongest competing baseline.AFORE has the highest throughput and lowest P95 ITL on all four workloads.Source: Figure 8 · p. 15, §7.2 · p. 15
7.3 Scheduler Overhead and Solution Quality (RQ2)
- Scheduler median latency is 8.23 µs at EP=2 and 245.81 µs at EP=128, remaining below one millisecond across tested EP degrees.The scheduler is evaluated at EP degrees 2, 4, 8, 16, 32, 64, and 128.Source: Figure 9 · p. 16, §7.3 · p. 16
- AFORE achieves the optimal maximum worker tile load on 99.5% of 1,400 targets; at EP degrees 8–128 it is optimal on all 200 targets per degree.At EP=2 and EP=4, the mean optimality gap is at most 0.04%, with a 3.60% worst case across those settings.Source: Table 2 · p. 16, §7.3 · p. 16
7.4 Component-wise Ablation (RQ3)
- Disabling prefetching cuts throughput 12.4–17.9% and raises P95 ITL 6.3–10.5%; disabling overlap cuts throughput 15.2–21.3% and raises P95 ITL 12.2–16.0%.The two mechanisms play complementary roles: prefetching improves placement selection, while overlap reduces installation cost.Source: §7.4 · pp. 16–17
- Against Static, full AFORE improves throughput by 26.0–34.1% (29.8% on average) and reduces P95 ITL by 16.7–19.6% (18.2% on average).The figure reports these measures across four workloads.Source: §7.4 · p. 16, Figure 10 · p. 17
7.5 Migration Hiding Effectiveness (RQ4)
- Mean raw migration cost is 0.453–0.495 ms versus a 2.008–2.119 ms mean lookahead window; all 324 sampled targets have zero exposed migration cost.The measurements indicate that pipeline overlap fully hides migration time in these samples.Source: §7.5 · p. 17, Figure 11 · p. 18
- Median time remaining between copy completion and first use ranges from 1.459–1.668 ms across workloads, with a minimum of 0.447 ms.The measurement takes the minimum remaining time across participating ranks for each target.Source: §7.5 · p. 17, Figure 11 · p. 18
8 Conclusion
AFORE uses upcoming microbatch demand and migration overlap to reconfigure experts in AFD-based MoE serving. It improves throughput and P95 inter-token latency against the strongest baseline across diverse workloads.
8 Conclusion
- AFORE addresses FFN-side load imbalance in AFD by using upcoming microbatch demand and migrating experts while preceding microbatches execute.It formulates reconfiguration as microbatch-aware scheduling and uses a migration-aware cost model with greedy load redistribution.Source: §8 · p. 18
- Against the strongest baseline, AFORE improves output throughput by 10.1–17.6% and reduces P95 inter-token latency by 7.1–9.5%.The evaluation reports gains across diverse workloads.Source: §8 · p. 18
- The scheduler remains below one millisecond even at high expert-parallelism degrees, supporting microbatch-level operation.The conclusion also reports that ablation isolates benefits from true target-microbatch demand and early copy submission.Source: §8 · p. 18