PaperlayerView exact revision on arXiv

Source-linked AI summary

RapidMoE: Exploiting Cross-Asymmetry via Adaptive Residual Offloading for Large-Scale MoE Inference

Wenxun Wang, Likai Ma, Zongle Huang, Chen Tang, Yongpan Liu

arXiv:2610.01265v1cs.DC

TL;DR

Large MoEs strain GPU memory, while existing hybrid offloading faces PCIe and CPU-computation bottlenecks. RapidMoE combines adaptive expert arbitration with residual splitting to align routed workloads with CPU and GPU strengths. It reports up to 3.5× speedup over SOTA systems and 43 tokens/s decode and 150 tokens/s prefill for DeepSeek-R1-671B on 2×A800-80G GPUs.

  • Problem

    Large MoE parameter counts exceed GPU VRAM, while existing hybrid systems encounter PCIe loading bottlenecks or rely heavily on CPU computation.

    Source: §1 · p. 1

  • Method

    RapidMoE uses UMIA to adaptively identify critical experts and RESplit to separate CPU-side residual weights from GPU-side quantized weights.

    Source: §1 · p. 2

  • Results

    Up to 3.5× speedup over SOTA systems is reported, with 43 tokens/s decode and 150 tokens/s prefill for DeepSeek-R1-671B on 2×A800-80G GPUs.

    Source: §1 · p. 3, §9 · p. 14

  • Takeaways & Limitations

    RapidMoE is presented as practical for large-model inference in memory-constrained deployment environments.

    Source: §9 · p. 14

    RapidMoE is best suited to low-concurrency, memory-constrained deployment; its latency model should be recalibrated for each platform, and achievable gains remain hardware-dependent.

    Source: §7 · p. 14

Abstract

from arXiv · show

The widespread adoption of Mixture-of-Experts (MoE) has created a growing need for deployment on heterogeneous platforms. However, it exposes a fundamental mismatch between the algorithmic demands of large-scale MoE and the disparate characteristics of hardware.Existing CPU-GPU hybrid inference systems fail to resolve this as they either encounter PCIe bandwidth bottlenecks when loading experts to GPUs, or rely heavily on CPU computation. Consequently, this leads to low resource utilization and inevitable violations of fixed latency budgets as parameters scale. In this paper, we identify and exploit Cross-Asymmetry--a structural alignment between the algorithmic workload skew of MoE routing and the physical disparity of heterogeneous hardware. To this end, we introduce RapidMoE, a residual offloading system for efficient large-scale MoE inference. We propose how RapidMoE leverages a residual-split framework to enable offloading paradigm shift from expert-level to bit-level, which unfolds across three key dimensions: (1) data representation, enabling compact and decoupled storage; (2) routing strategy, partitioning computation into dual paths aligned with hardware capabilities; (3) execution parallelism, scheduling a balanced storage-compute workload across devices. We further employ a novel Unified Multi-Level Importance Arbitration to adaptively adjust the critical expert set at runtime, ensuring the accuracy-latency Pareto frontier. These innovations exploit inherent cross-asymmetry, fundamentally breaking the algorithm-hardware misalignment. Experimental results show that RapidMoE achieves up to 3.5x speedup in decoding and 2.1x speedup in prefill compared to state-of-the-art (SOTA) offloading systems.

1 Introduction

Large MoEs exceed GPU memory, making heterogeneous workstations a potential private, cost-efficient deployment option, but existing CPU-GPU offloading binds computation to weight location and struggles with fixed latency budgets. RapidMoE aligns skewed expert importance with hardware differences through runtime arbitration and bit-level residual splitting.

1 Introduction

  • Existing hybrid systems face scaling offload overhead and fixed latency budgets while leaving GPUs underutilized as they wait for CPU computation or I/O.Expert weights must be computed where they reside, coupling storage and computation despite the hardware mismatch.Source: §1 · pp. 1–2
  • A few critical experts need high precision but use few FLOPs, while non-critical experts tolerate low precision and dominate computation.This workload skew aligns with CPUs' storage capacity and GPUs' throughput.Source: §1 · p. 2
  • Runtime-varying expert importance, redundant dual-precision storage, and CPU stragglers make expert-wise offloading difficult to schedule efficiently.The challenges span dynamic accuracy-latency arbitration, storage, and device utilization.Source: §1 · p. 2
  • RapidMoE combines UMIA with RESplit: it adaptively identifies critical experts, stores CPU-side residuals separately from GPU-side quantized weights, and splits routing into critical and non-critical paths.RESplit's execution parallelism assigns lightweight quantized weights to GPUs and storage-intensive residuals to CPUs.Source: §1 · pp. 2–3
  • Up to 3.5× speedup over SOTA systems is reported for RapidMoE, evaluated on heterogeneous platforms with DeepSeek-671B and Qwen3-235B.The listed system innovations include cross-asymmetry formalization, UMIA, and RESplit methods across routing, representation, and parallelism.Source: §1 · p. 3

2 Background

MoE routes each token to a sparse subset of experts, while reasoning models can extend decoding and large model parameter counts strain GPU memory. Hybrid CPU-GPU systems address capacity through expert offloading, but face PCIe latency or underused GPUs.

2.1 Mixture of Experts and Reasoning Models

  • MoE uses a gate to route each token to its top-k experts, and reasoning models such as DeepSeek-R1 activate 37B of 671B parameters while extending decoding.DeepSeek-R1 requires an average of 9363 output tokens on MATH-500; the passage attributes end-to-end latency challenges to extended decoding time.Source: §2.1 · p. 3

2.2 CPU/GPU Hybrid Inference for MoEs

  • GPU-centric offloading loads missing experts over PCIe, while CPU-centric systems shift computation to CPUs; both approaches face latency or resource-utilization drawbacks.Transferring eight experts for one DeepSeek-V3 layer over PCIe 4.0 takes 12.3 ms, over 7× the 1.64 ms budget for a 100 ms TPOT target; CPU-centric systems can leave GPU utilization at 10–20%.Source: §2.2 · pp. 3–4

3 Motivation

MoE routing varies in importance across experts, layers, and inference phases, while CPU-GPU hardware offers contrasting storage and compute strengths. The section motivates preserving low-precision information and aligning workload granularity with those hardware differences.

3 Motivation

  • Cross-asymmetry frames the motivation: skewed expert workloads can align with heterogeneous hardware's contrasting memory capacity and compute throughput.The section uses this alignment to motivate workload partitioning.Source: §3 · p. 4

3.1 Uncovering Algorithmic Asymmetry

  • Importance diverges locally, spatially, and temporally: top experts matter more than low-scoring ones, shallow layers are more sensitive, and prefilling is more accuracy-sensitive than decoding.Omitting the two lowest-scoring experts raises perplexity by 0.12, whereas dropping the top two degrades outputs by 93.28; reducing top-k in shallow layers and during prefilling causes greater quality loss.Source: §3.1 · p. 4

3.2 Leveraging Algorithmic Asymmetry

  • Preserving non-critical experts in low precision restores most quality lost by skipping them, motivating split routing of critical top-r and remaining top-(k-r) experts.Skipping non-critical experts even at k=4 causes severe performance loss; preserving partial information supports robustness.Source: §3.2 · p. 4

3.3 Aligning Cross-Asymmetry

  • Critical experts need high precision but few FLOPs, whereas non-critical experts tolerate low precision and dominate computation, matching CPUs' capacity and GPUs' throughput.Expert-level scheduling also requires dual weight copies as importance changes and can leave GPU bubbles even under an ideal mapping; the section consequently motivates fine-grained RESplit scheduling.Source: §3.3 · p. 5

4 RapidMoE System

RapidMoE co-designs split routing, UMIA, residual representation, and CPU-GPU execution to align MoE workload skew with heterogeneous hardware. It adapts critical-set size while partitioning computation between low-precision weights and residuals.

4.1 Design Overview

  • RapidMoE combines split expert routing with runtime UMIA and RESplit, placing low-precision WQ on GPUs and residual WR on CPUs for concurrent paths.Source: §4.1 · pp. 5–6

4.2 Split Expert Routing Mechanism

  • At r*∈[1,2], CPU and GPU execution times align; beyond it the system is CPU-bound, and synchronization contributes about 20% of decode latency.Source: §4.2 · p. 6

4.3 Unified Multi-Level Importance Arbitration

  • UMIA combines router scores with layer- and phase-wise modulation, then uses latency-constrained frontier search to choose critical-set sizes balancing KL loss and execution latency.Source: §4.3 · p. 7

4.4 Residual-Split Data Representation

  • RESplit stores low-precision WQ plus quantized residual WR, reconstructing high-precision weights without dual copies and enabling GPU WQ/CPU WR computation.Source: §4.4 · p. 8

4.5 Residual-Split Expert Parallelism

  • GPUs begin critical gate/up projections, then process non-critical experts while CPUs compute critical residuals and produce outputs, synchronizing at operator boundaries.Source: §4.5 · pp. 8–9
  • GPU utilization rises from 33% to 77% at r=2 and from 69% to 100% at r=1 versus the naive scheduling pipeline.Source: §4.5 · p. 9

5 Implementation Details

RapidMoE uses a hybrid implementation with specialized CPU and GPU kernels, a 4.25-bit residual-split quantization configuration, and deployment adaptations for limited VRAM.

5 Implementation Details

  • RapidMoE uses specialized C++ CPU expert-residual kernels and CUDA GPU kernels, with CUDA graph capture for MoE operations.Source: §5 · p. 9
  • IQ1_M_R4 WQ plus Q2_K_R4 WR totals 4.25 bits and achieves accuracy comparable to 4.5-bit Q4_K and AWQ.Source: §5 · p. 9
  • When VRAM cannot hold WQ, RapidMoE falls back from cached GPU execution to a CPU-centric path; model scale or WQ quantization can also tailor VRAM use.Source: §5 · p. 9

6 Evaluation

RapidMoE is evaluated for accuracy, efficiency, and deployment trade-offs across large MoE models, hardware configurations, and workloads. It outperforms the compared systems in decoding and prefilling while retaining accuracy and adapting to memory constraints.

6.1 Evaluation Setup

  • Evaluation tests DeepSeek-V3-671B, DeepSeek-R1-671B, and Qwen3-235B-A22B on A800-80G and RTX 4090 platforms, measuring prefill/decode throughput (tokens/s) and serving TTFT/TPOT, including p50 and p99 latencies.Source: §6.1 · p. 10

6.3 End-to-end Efficiency

  • 3.5×, 2.8×, and 2.1× averaged decoding speedups over KTransformers for DeepSeek-R1, DeepSeek-V3, and Qwen3, respectively, on P1.On P2, RapidMoE achieves up to 2.9× speedup with a different CPU-GPU setting.Source: Figure 10 · p. 11, §6.3.1 · p. 11

6.2 End-to-end Accuracy

  • RapidMoE matches or exceeds Q4 accuracy, with changes of -0.1% in general knowledge, +0.7% in mathematical reasoning, and +0.1% in code generation.Accuracy remains stable with token-averaged routing at batch size 4 and across calibration datasets; normalized β varies 4.1–7.2%, while Θ and α vary by at most 5 of 100 grid positions.Source: Table 5 · p. 10, §6.2 · p. 11, Table 6 · p. 11

6.3 End-to-end Efficiency

  • 138.9 tokens/s on DeepSeek-V3 and an average 2× higher throughput than baselines in prefilling; at 8K, 12K, and 16K inputs, it averages 1,905 tokens/s.For long contexts, layer-wise overlap transfers next-layer weights during current-layer computation; under C11/C12, it hides most of the 136 ms transfer cost with 116 ms of per-layer computation at 8K.Source: Figure 11 · p. 11, §6.3.2 · pp. 11–12
  • At concurrency 4 and 0.2 requests/s, RapidMoE meets the 100ms TPOT and 5s TTFT SLOs for most requests; at comparable 149 GB HBM, it uses 240 GB DRAM versus 385 GB for KTransformers and 471 GB for MoE-APEX.The DRAM reduction is attributed to RESplit’s compact representation, which stores Q2 Gate/Up residuals and Q4 Down weights on CPUs.Source: Table 9 · p. 12, §6.3.3 · p. 12, Table 10 · p. 12

6.4 Ablation Study

  • Adding split routing to KTransformers yields 1.2×–1.3× speedup, dynamic arbitration adds about 1.4×, and adding RESplit representation and parallelism yields a final speedup of up to 1.39×.The reported sequence is +S, then +S+U, then +S+U+R.Source: §6.4.1 · p. 12, Figure 12 · p. 12
  • UMIA’s runtime adaptation advances the accuracy-throughput frontier over fixed-size routing: skipping causes accuracy collapse, while preserving non-critical experts in low precision mitigates it.Its critical-set size varies by layer, phase, and iteration; single-modulator variants degrade accuracy, while full UMIA recovers accuracy with competitive throughput.Source: §6.4.2 · p. 13, Figure 13 · p. 13, Figure 14 · p. 13
  • RapidMoE sustains competitive throughput when only 25% of low-precision experts are cached, though its performance sweet spot is caching all of W_Q; MoE operations remain the latency bottleneck in both stages.The 25% cache footprint is 37 GB; the 100% footprint is 149 GB. Batch inference during decoding reduces the non-MoE share of overall latency.Source: §6.4.3 · p. 13, §6.4.4 · p. 13

7 Discussion

RapidMoE exposes a memory-accuracy-latency trade-off and targets memory-constrained, low-concurrency deployment. Its latency model and achievable gains depend on platform-specific CPU and GPU characteristics.

7 Discussion

  • RESplit sets a memory-accuracy trade-off through W_Q precision, while CPU residual computation recovers errors without additional VRAM; UMIA then adapts the critical set to balance accuracy and latency.The two mechanisms address complementary dimensions of the deployment trade-off.Source: §7 · p. 13
  • RapidMoE is best suited to low-concurrency, memory-constrained deployment: larger batches improve GPU utilization but benefit CPU-bound residual computation less, while higher concurrency raises TTFT and TPOT.The system was validated on PCIe-connected Intel CPU–NVIDIA GPU systems; supporting other platforms requires backend-specific adaptation.Source: §7 · p. 14
  • The asymmetric latency model and achievable gains are hardware-dependent, so R(r) should be recalibrated for each platform as CPU and GPU performance shifts their balance.Faster CPU execution shifts the transition toward larger r; faster GPU execution shifts it toward smaller r.Source: §7 · p. 14

8 Related Work

Related MoE offloading systems trade expert-loading latency against CPU computation and GPU utilization. RapidMoE’s related-work context also includes model-compression methods and its choice of GGUF.

8 Related Work

  • GPU-centric offloading systems use prefetching and caching but remain limited by expert-loading latency on the critical path; CPU-centric systems address that latency concern but can under-utilize GPUs and bottleneck on CPU throughput.The cited approaches include expert prefetching, cache replacement, and pipelining on the GPU-centric side, and optimized CPU kernels for unavailable experts on the CPU-centric side.Source: §8 · p. 14
  • MoE compression includes pruning and quantization; RapidMoE builds on GGUF for cross-platform compatibility, contrasting with AWQ and GPTQ’s limited CPU support.The passage notes integration of quantization methods such as AWQ, GPTQ, and GGUF into mainstream inference engines.Source: §8 · p. 14

9 Conclusion

The conclusion presents RapidMoE as an adaptive expert-residual offloading system for large-scale MoE inference on heterogeneous platforms. It reports DeepSeek-R1-671B throughput results on two A800-80G GPUs.

9 Conclusion

  • RapidMoE is an adaptive expert-residual offloading system for large-scale MoE inference on heterogeneous platforms.Source: §9 · p. 14
  • The system combines cross-asymmetry exploitation with a residual-split framework and runtime arbitration.The conclusion describes these as supporting resource utilization, latency, and throughput.Source: §9 · p. 14
  • On 2×A800-80G GPUs, RapidMoE achieves 43 tokens/s in decode and 150 tokens/s in prefill for DeepSeek-R1-671B.Source: §9 · p. 14

Related papers

Loading 2610.01265v1…