Fast Inference of Mixture-of-Experts Language Models with Offloading

1 Introduction

• Open-access LLMs often require multiple high-end GPUs for basic inference, prompting the use of compression or offloading to RAM or SSD.The paper gives examples of 16-bit models requiring approximately 360GB for Falcon-180B and 140GB for LLaMA-2 70B.Source: §1 · p. 1
• MoE blocks activate only a subset of experts per input, but MoE models can still be large and existing efficient-inference techniques may perform suboptimally on them.The authors describe MoE models as both more efficient than dense counterparts and among the largest available LLMs.Source: §1 · p. 1
• The approach exploits adjacent-token expert reuse and early-layer expert prediction in an MoE-specific offloading strategy, combining mixed quantization to run Mixtral-8x7B-Instruct interactively at 2-3 tokens per second on a T4, RTX 3060, and RTX 3080 Mobile.Source: §1 · p. 2

2.1 Mixture-of-Experts

• MoE models use a gating function to select experts, typically in feedforward blocks, and can reduce compute or inference costs while requiring more parameters than dense models of equivalent perplexity.Mixtral-8x7B is described as near state-of-the-art but still requiring a high-end GPU accelerator.Source: §2.1 · p. 2

2.2 Post-training Quantization of LLMs

• Quantization and related compression methods reduce model size, but very large transformers pose challenges such as emergent outliers, and MoE-specific quantization work is emerging.The passage describes around 4 bits per parameter as generally optimal for most LLMs, while more extreme compression is typically inferior to a smaller model quantized near 4 bits.Source: §2.2 · pp. 2–3

2.3 Inference with Parameter Offloading

• Parameter offloading loads weights from cheaper memory when needed and can pre-dispatch the next layer, but interactive autoregressive inference is slower because it processes few tokens and waits for weights.Large-batch workloads can amortize each layer load across many tokens, unlike interactive generation.Source: §2.3 · p. 3

2.4 Hardware Setup

• The target setting assumes enough system RAM for model parameters, an 11–16GB GPU, and 8–16GB/s host-to-device communication, while experts do not fit in GPU memory even after quantization.Generative inference also needs memory for activations and past attention keys and values.Source: §2.4 · p. 3

3.1 Expert Locality and LRU caching

• Experts can be reused across short sequences of 2–4 tokens or recur after gaps, motivating an LRU cache that retains recently used experts within the limited GPU memory.The cache keeps the same number of experts per MoE layer; even this simple policy is reported to speed inference.Source: §3.1 · p. 4

3.2 Speculative Expert Loading

• MoE expert choices are not known until the prior layer finishes, preventing the dense-model schedule of preloading the next layer during current-layer computation.The proposed speculative loading guesses likely experts using the next layer’s gate on previous-layer hidden states, leveraging residual connections; reliable prefetching without changing the architecture would require retraining.Source: §3.2 · pp. 4–5

3.3 System Design & Implementation Details

• The system combines caching and prefetching with mixed MoE quantization, using lower bitwidth for experts while keeping non-expert layers at 4-bit.The authors chose HQQ for convenience and report that sub-1-bit compression caused too much perplexity loss for Mixtral-8x7B.Source: §3.3 · p. 5
• For limited host RAM, experts are split between host and GPU memory, and loading an expert into the GPU cache sends the least recently used on-device expert back to RAM.Expert parameters use contiguous buffers and four shared temporary buffers for asynchronous copies and prefetching.Source: §3.3 · p. 5

4.1 Expert LRU Cache and Speculative Loading

• The evaluation measures expert recall for LRU caching across cache sizes and speculative-loading recall when experts are preloaded 1, 2, or 10 layers ahead.Speculative recall measures how often the actual next experts were already loaded; a recall of 1.0 means both active experts were prefetched.Source: §4.1 · p. 6, Figure 2 · p. 6

4.3 Practical offloading performance

• 2–4 tokens per second with the full algorithm across evaluated setups, and all schemes significantly outperform naive offloading that loads the entire MoE layer.Table 2 reports inference speed for Mixtral-8x7B in low-tier setups.Source: Table 2 · p. 7, §4.3 · p. 7

4.2 Mixed MoE Quantization

• Quantizing experts to 2 or 3 bits while keeping attention layers at 16 or 4 bits offers favorable performance–size trade-offs for the target hardware constraints.Table 1 summarizes perplexity and model size across quantization configurations.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Practical offloading performance

• The practical evaluation uses Mixtral-8x7B-Instruct with 4-bit HQQ attention layers and 2- or 3-bit experts across four GPU configurations.The setups include a free-tier Colab T4, RTX 3080 Mobile, RTX 3060, and A100; 3-bit evaluation uses an Azure T4 because free-tier Colab lacked enough RAM.Source: §4.3 · p. 7

• The paper proposes a MoE-centric approach to offloading for accelerating MoE language models on consumer hardware with limited GPU memory.Source: §5 · p. 7
• The proposed strategies produce a significant increase in generation speed over naive approaches on consumer-grade hardware, including free-tier Google Colab.Source: §5 · p. 8
• The method is presented as a practical solution for inference on resource-constricted hardware, with broader access for research and development as its stated consequence.Source: §5 · p. 8