SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading

Motivation

• MoE’s sparse scaling offers an offloading opportunity, but load-on-demand stalls expose expensive memory I/O and eager offloading achieves only about 2% of FullLoad performance.The goal is to load activated experts in time while keeping only a subset of model weights in device memory.Source: §1 · p. 1

Predictive memory management

• Joint prefetch scheduling selects experts under deadline constraints to maximize expected hits and improve bandwidth efficiency.The formulation treats prediction confidence as an objective and deadlines as constraints on bandwidth contention.Source: §1 · pp. 2–3
• Forecast-driven caching uses predicted future activations for probabilistic Belady eviction, prioritizing experts by likely near-term reuse.Rolling the historical sequence forward supplies forecasts for cache eviction rather than relying only on coarse historical statistics.Source: §1 · pp. 2–3
• SeqMoE recasts expert activation prediction as sequence modeling, forecasting multi-step, multi-layer activations to provide a long, reliable window for downstream decisions.The approach captures routing dependencies across the full layer stack and generation trajectory.Source: §1 · p. 2

Graph-compatible runtime

• A graph-compatible runtime coordinates predictive memory management with inference using compute-transparent expert placement and synchronization-free orchestration.These principles address dynamic placement and host-device synchronization that conflict with graph execution.Source: §1 · pp. 2–3

Results

• At 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance.At that residency, the strongest baseline averages an 88.50% hit rate, while FreeToken reaches 51.50% of full-load performance.Source: §1 · p. 3

2.1 MoE’s Structural Advantages for Offloading

• Only activated experts need to reside in device memory at each inference step, so timely loading can in principle approach FullLoad performance with a smaller memory footprint.MoE layers dominate model parameters while only a small fraction of experts are activated per step.Source: §2.1 · p. 3

2.2 Expert Activation Prediction

• Existing activation predictors cannot simultaneously provide high accuracy and sufficient lookahead, leaving downstream scheduling with a narrow, unreliable optimization window.Cross-layer methods are limited to one to three layers, while first-order cross-step statistics are too coarse for precise scheduling.Source: §2.2 · pp. 3–4

2.3 Sequence Modeling

• Sequence models fit expert activations arranged across generation steps; Mamba combines a fixed-size historical state with long-range modeling and efficient parallel training.The paper adopts Mamba as the backbone of its sequence-to-sequence expert activation predictor.Source: §2.3 · p. 4

2.4 Graph-based Execution Mechanisms

• Graph execution reduces host-side dispatch overhead, but dynamic expert placement and CPU-GPU synchronization conflict with its fixed structures and replay requirements.The runtime must coordinate prediction, prefetching, and caching with inference under graph execution constraints.Source: §2.4 · p. 4

3.1 Expert Activation Sequence Modeling

• Shuffled-sequence training fails to converge, indicating that expert-activation predictability stems from sequence-level dependence.The study encodes activations across the full layer stack at each generation step and arranges them in generation order.Source: §3.1 · p. 4
• On GPT-OSS-120B with e=128 and k=4, both reported measures exceed 90% and substantially outperform existing cross-step prediction methods, demonstrating the necessity of explicitly modeling sequential dependence.Source: §3.1 · p. 5

3.2 Joint Structure in Prefetch Scheduling

• Confidence estimates expected hit contribution, while deadlines couple transfers through capacity; weighted matroid greedy optimally solves the feasible task-selection problem.The formulation treats bandwidth scarcity as constrained task selection rather than ranking prefetch tasks independently.Source: §3.2 · p. 5

3.3 Future-Aware Cache Eviction

• Future-aware policies outperform history-based eviction, and Belady-F8 closely approaches Belady-8 using predicted activations.LRU and LFU perform similarly; Belady uses the complete future sequence, while Belady-8 limits oracle knowledge to eight steps.Source: §3.3 · p. 5

3.4 End-to-End Graph Execution

• With all weights resident to simulate a 100% expert hit rate, eager execution and synchronization substantially degrade inference performance.The comparison contrasts eager execution and synchronization with fused MoE kernels and CUDA Graphs.Source: §3.4 · p. 5

• Each Transformer block enters the offloading pipeline after attention and routing, where prior prefetches land before missing experts are loaded.The pipeline updates cache slot states when prefetch tasks land.Source: §4 · p. 5
• At a trigger layer, the predictor forecasts multi-step activations; first-step probabilities generate prefetch tasks scheduled with matroid greedy.Subsequent-step probabilities update the probabilistic Belady policy for future-aware eviction.Source: §4 · pp. 5–6
• Compute-transparent expert placement lets execution leave the offloading pipeline and resume fused MoE computation, coordinated by four runtime orchestration disciplines.Source: §4 · p. 6

5.1 Expert Activation Collection

• A two-pass trace-collection process gathers token IDs under continuous batching, then packs traces into a prefill pass to dump expert activations.The workloads include mathematics, code, text comprehension, and multi-turn conversations.Source: §5.1 · p. 6

5.2 Model Architecture

• Preprocessing normalizes active-expert scores and merges layer and expert dimensions; including more layers improves accuracy but shortens the prefetch overlap window.Normalization preserves relative preference among activated experts for scheduling confidence while suppressing noise from inactive experts.Source: §5.2 · p. 6

5.3 Prediction Execution

• Recursive rollout repeatedly feeds predictions back as input to forecast multiple future steps, while restoring recurrent state so rollout does not affect the next one-step prediction.Recursive prediction also runs at every generation step to update the cache policy.Source: §5.3 · p. 7

5.4 Training

• Training combines ranking and KL-divergence losses, then uses scheduled sampling after teacher forcing to expose the predictor to its own outputs.The ranking loss targets ordering around the Top-k decision boundary, while scheduled sampling aims to improve robustness to recursive error accumulation.Source: §5.4 · p. 7

6.1 Prefetch Task Generation

• The predictor’s expert activation probabilities generate up to k prefetch tasks for experts with high predicted activation.Each task’s confidence estimates its expected contribution to expert hits, while its deadline marks when the expert is needed for computation.Source: §6.1 · p. 7
• Each task remains eligible during an overlap window of L−(l′−1−p), when its transfer can run alongside inference.Once selected, the task transfers the expert from next-level memory to device memory.Source: §6.1 · p. 7

6.2 Prefetch Task Selection

• The scheduler builds the highest-confidence feasible task set under deadline constraints, then uses Earliest Deadline First to choose the next transfer.It updates transfer budgets with an exponential moving average, removes expired tasks, and drops the lowest-confidence task when the retained set exceeds deadline capacity.Source: §6.2 · pp. 7–8

7.1 Probabilistic Belady Policy

• Probabilistic Belady eviction uses recursively predicted activation probabilities over the next c steps to prioritize experts expected to be reused sooner.The policy extends Belady’s preference for earlier future reuse to probabilistic activations; a discount factor gives nearer predictions greater weight.Source: §7.1 · p. 8

7.2 Slot State Transition

• Cache slots are Empty, Used, Fetch, or Compute; Fetch and Compute slots are protected from eviction.Used slots hold resident experts eligible for eviction, while Empty slots are free for allocation.Source: §7.2 · p. 8
• Prefetch creation reserves an Empty or Used slot as Fetch, and a successful transfer makes it Compute; before each layer runs, its Compute slots become Used.On-demand loads move allocated slots directly to Compute, and allocation prefers Empty slots before evicting the lowest-CacheScore Used expert.Source: §7.2 · p. 8

8.1 Compute-Transparent Expert Placement

• Sharing cache slots across MoE layers restores a fixed base address for fused kernels and decouples the offloading pipeline from model computation.The shared cache also permits flexible scheduling and avoids per-layer slot limits during prefill, while measured kernel latency remains unchanged despite num_slots exceeding num_experts.Source: §8.1 · p. 8

8.2 Offloading Orchestration Discipline

• Four orchestration disciplines keep prediction, cache management, miss loading, and prefetch communication free of host-device synchronization in the execution path.Prediction and cache management run on-device; the device initiates miss loading, while CPU prefetch scheduling and host-initiated DMA remain asynchronous.Source: §8.2 · pp. 8–9

8.3 Implementation

• The implementation captures the main thread’s CUDA operations in a CUDA Graph while a CPU worker runs the prefetcher asynchronously.Mapped host memory and a custom kernel support graph-capturable on-demand loads without host synchronization.Source: §8.3 · p. 9

8.4 Prefill Offloading

• For prefill, SeqMoE prioritizes offline-profiled frequent experts and reserves e slots for evicting and loading missed experts.Graph capture is omitted during prefill because sequence lengths vary and computation is substantial.Source: §8.4 · p. 9

9.1 Experimental Methodology

• The evaluation covers four MoE models, five workload datasets, three GPU platforms, and cache capacities from 15% to 45% (15% to 40% for QW36).It reports tokens per second, time to first token, expert hit rate, and prediction recall; inference evaluation uses 600 test traces.Source: §9.1 · pp. 9–10, §9.2 · p. 10

9.2 End-to-End Performance

• At 45% capacity (40% for QW36), SeqMoE reaches 84.82%, 75.69%, 79.20%, and 76.65% of FullLoad performance on QW3, QW36, GPT, and DSV4, respectively.FreeToken reaches 55.15%, 57.60%, and 45.45% on QW36, GPT, and DSV4; SeqMoE matches FreeToken at 25% capacity versus 40–45%.Source: Figure 6 · p. 10, §9.2 · p. 10

9.3 Expert Hit Rates

• At cache capacities of 15%, 25%, 35%, 40%, and 45%, average hit rates are 84.56%, 91.72%, 95.09%, 96.11%, and 96.97% for SeqMoE, versus 60.02%, 74.24%, 84.49%, 86.37%, and 88.50% for LRU.The evaluation also reports larger performance gains as concurrent request counts increase and lower prefill latency through flexible prefetching and cache-state transitions.Source: §9.2 · p. 10, Figure 7 · p. 11, §9.3 · p. 11

9.4 Prediction Accuracy

• Seq-8’s average recall drops only 3.58 percentage points from Seq-1, and Seq-L1 remains comparable to Seq-1 on sequences longer than 4,096 tokens.Top k+3 selection yields recalls of 90.90%, 88.39%, 90.76%, and 85.81% on QW3, QW36, GPT, and DSV4, respectively.Source: Figure 8 · p. 11, §9.4 · p. 11

9.5 Offloading Overhead

• At 45% cache capacity (40% for QW36), on-demand loading remains the largest inference-time component despite an average miss rate of 3.25%.Predictor inference accounts for 3.46% of execution time on average, and CUDA management kernels account for 2.05%.Source: §9.5 · pp. 11–12, Figure 9 · p. 12

• SeqMoE substantially outperforms existing offloading systems and approaches full-load performance.Source: §10 · p. 12
• Sequence-modeling forecasts guide prefetch scheduling and cache eviction to improve expert hit rates and reduce on-demand stalls.Source: §10 · p. 12
• Compute-transparent expert placement and synchronization-free orchestration reconcile dynamic offloading with graph execution and enable end-to-end graph capture.Source: §10 · p. 12