Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

1 Introduction

• Existing intra-token prefetching offers only a microsecond-scale window, insufficient to overlap millisecond-scale PCIe transfers for gigabyte-sized experts.Input-dependent, sequential routing constrains prediction of future expert use.Source: §1 · pp. 1–2
• A quantized version of the target MoE model provides high-fidelity expert predictions without separate draft-model training, enabling transfers to be scheduled in advance.Its shared routing architecture and parameter lineage support expert-activation predictions, while draft-then-verify preserves the output distribution.Source: §1 · p. 2
• Applying speculative decoding to MoE offloading also risks expert explosion, PCIe saturation, and synchronization bubbles, requiring coordinated system design.Prior approaches described here either introduce quality-latency tradeoffs or require structurally compatible draft–target pairs.Source: §1 · p. 2
• Specter combines a quantized draft model’s multi-step lookahead with expert prefetching and parallel target verification to accelerate offloaded MoE inference.The system targets memory-constrained inference on a single GPU with model weights partially offloaded to host memory.Source: §1 · p. 2
• 1.50× average TPOT speedup, up to 1.88×, over SpecMoEOff under single-request inference.The system is evaluated across three representative MoE architectures under strict hardware constraints.Source: §1 · p. 2

2.1 MoE Offloading Demands Multi-step Lookahead

• Existing intra-token prefetching provides only microseconds to hide millisecond PCIe transfers, while extending lookahead without verification risks routing errors and degraded generation quality.Speculative decoding instead verifies draft tokens in parallel and guarantees lossless generation quality through its accept/reject mechanism.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.2 Speculative Decoding Provides Multi-Step Lookahead

• An INT4 quantized version of the target MoE avoids separate draft-model training and achieved a 90.8% mean top-4 expert-ID match rate against the FP16 Qwen1.5-MoE target.Figure 2 evaluates match rates across 1,000 requests and 24 MoE layers; the three reported distributions peak above 90%.Source: §2.2 · p. 3, Figure 2 · p. 3
• A quantized draft and target share routing architecture and parameter lineage, providing high-fidelity predictions that let the system schedule PCIe transfers in advance.The paper describes this predictability as converting data-dependent routing into a deterministic I/O prefetching and scheduling problem.Source: §2.2 · p. 3

2.3 Speculative Offloading Needs System Efforts

• Speculative offloading also contends with redundant draft weights and KV caches, fragmented INT4 draft kernels, and serial execution that leaves computation, transfers, and verification poorly overlapped.These bottlenecks motivate co-design of model-state management, I/O scheduling, and kernel execution.Source: §2.3 · p. 4
• Increasing speculation depth K from 1 to 16 raises DeepSeekV2-Lite per-layer expert demand from 6.0 to 31.7, a 5.3× surge that can trigger critical-path PCIe thrashing when the working set exceeds cache.The Figure 4 measurement describes demand covering nearly half of the model’s 64 experts.Source: §2.3 · p. 4, Figure 4 · p. 4

3.1 Design Principles and Architectural Overview

• Specter uses multi-step draft lookahead to overlap expert transfers with computation, while parallel target verification improves decoding efficiency.Its design integrates prediction-driven data orchestration with tightly coordinated execution.Source: §3.1 · pp. 4–5

3.2 Speculative Inference Controller

• The controller shares non-expert parameters, keeps routing-sensitive components and shared experts in FP16, and quantizes remaining expert MLP parameters to INT4.It also selects speculation depth through offline profiling and maintains target-consistent token and KV state across iterations.Source: §3.2.1 · p. 5, §3.2.2 · p. 6, §3.2.3 · p. 6

3.3 Predictive I/O Orchestrator

• The PIO aggregates draft expert activations in a GPU-resident ELB, ranks layer experts for prefetch, and selects draft chunks to overlap computation with estimated H2D transfer time.Its phases are Cold Start, Prefetch Overlap, and Final Cleanup.Source: §3.3.1 · p. 6, §3.3.2 · p. 7

3.3.3 Verification Expert Cache Residency Manager.

• During verification, the residency manager prioritizes cache-hit experts for computation while loading cache-miss experts asynchronously, protecting in-use slots from premature replacement.The rolling load-compute process continues until the layer’s target-selected experts have processed their assigned activations.Source: §3.3.3 · p. 7

3.4 Streamlined Execution Engine

• The execution engine combines fused multi-expert draft kernels, cache-aware expert-wise verification, and parallelism optimization to reduce execution fragmentation and pipeline stalls.Fusion gathers tokens by expert into a single GPU launch; verification batches tokens routed to the same expert and overlaps resident-expert computation with transfers for missing experts.Source: §3.4 · p. 8, §3.4.1 · p. 8, §3.4.2 · p. 8
• Expert-wise reordering changes the schedule of independent MoE FFN operations while preserving target routing decisions and strict mathematical equivalence.Attention retains the standard causal mask, and synchronization ensures all tokens finish before the next layer.Source: §3.4.2 · p. 8

4.1 Experimental Setup

• The evaluation uses one A100 GPU, three representative MoE models, and five datasets selected to cover diverse domains and routing patterns.The setup also tests representative GPU-memory budgets; workload analysis reports distinct expert working sets and routing distributions across datasets.Source: §4.1.1 · p. 9, §4.1.2 · p. 9

4.2 End-to-End Performance

• 1.50× average TPOT speedup (up to 1.88×) over SpecMoEOff, with Specter achieving the lowest average TPOT across evaluated models and memory configurations.The low-memory results show the same overall trend, with Specter achieving the lowest TPOT on most datasets and in per-model averages.Source: Figure 14 · p. 10, §4.2.1 · p. 10
• From B=1 to B=32, Specter throughput improves 9.41× on DeepSeekV2-Lite and 3.54× on Qwen1.5-MoE, while TPOT reaches 1.65× and 7.37× their single-request values, respectively.The accompanying working-set analysis links larger batches to more distinct experts during verification and increased target computation and H2D pressure.Source: Figure 15 · p. 11, §4.2.2 · p. 11, Figure 16 · p. 11

4.3 Memory Saving Analysis

• For Qwen1.5-MoE at 12K context, shared model components reduce GPU memory use to 7.68 GB, 43% below a naive implementation with independent models.The sharing strategy includes experts, non-expert layers, and KV cache; target-validated KV entries extend the shared prefix cache after verification.Source: Table 4 · p. 12, §4.3 · p. 12

4.4 Impact of Prefetch Strategy

• On DeepSeekV2-Lite, Specter records cache hit rates of 99.85%, 98.62%, and 96.25% under 16, 24, and 32 GB budgets, respectively.Both single-prefetch policies outperform reactive LRU, while LRU-S becomes competitive at 32 GB through additional cache capacity.Source: Figure 17 · p. 12, §4.4 · p. 12

4.5 System Component Analysis

• Across DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE, Specter overlaps 55.5%, 60.5%, and 22.4% of total loading time, respectively.On DeepSeekV2-Lite, disabling PIO raises TPOT from 69.2 to 83.5 ms/token; additionally removing SIC raises it to 94.4 ms/token, and removing SEE as well reaches 130.4 ms/token.Source: Table 5 · p. 12, §4.5 · p. 12

4.6 Routing-Prediction Sensitivity and Robustness

• As prefetch accuracy falls from 1.000 to 0.410, TPOT rises from 63.0 to 84.2 ms/token, a 1.34× slowdown.Recall remains 0.930–0.961 across prompt domains, context lengths, and decoding strategies; target verification preserves the output distribution despite routing mispredictions.Source: §4.6 · pp. 12–13, Table 6 · p. 13

5.1 Efficient MoE Inference Systems

• MoE inference systems address memory and communication costs through offloading, expert placement, caching, and prefetching, but overlap remains limited when demand emerges during target execution.The cited systems span general-purpose serving frameworks and specialized designs that exploit routing locality or predicted demand.Source: §5.1 · p. 13

5.2 Speculative Decoding for LLMs

• Speculative-decoding research improves draft training, draft trees, verification, runtime efficiency, and speculation-depth selection; MoE variants explore hardware scheduling, speculative I/O, and approximate expert execution.Specter uses a quantized draft’s multi-token routing lookahead for prefetching while standard target verification preserves the output distribution.Source: §5.2 · p. 13

5.3 Quantization for LLM Inference

• Low-bit quantization methods reduce LLM memory and computation costs, while aggressive quantization can degrade generation quality.Marlin accelerates inference through efficient low-bit GEMMs.Source: §5.3 · p. 13

6 Conclusion

• Specter predicts expert routing across future tokens with a quantized draft, then coordinates prefetching, verification-time residency, expert-wise execution, and fused draft computation to overlap PCIe transfers with computation.The system improves decoding performance without changing the target model’s output distribution.Source: §6 · p. 13

A.1 Abstract

• The artifact provides installation instructions and scripts for decode latency, baseline comparisons, speculative depth, input length, and system measurements.Source: §A.1 · p. 16
• DeepSeek-V2-Lite checkpoints support high- and low-memory experiments with Specter, SpecMoEOff, and Mixtral-Offloading.Source: §A.1 · p. 16
• Qwen1.5-MoE and Phi-3.5-MoE are also supported with separately prepared checkpoints.Source: §A.1 · p. 16

• The artifact implements speculative expert offloading for MoE inference.Source: §A.2 · p. 16
• The reference setup uses one A100 GPU with 40 GB memory and a host with 256 GB memory.Source: §A.2 · p. 16
• The workflow requires approximately 200 GB of disk space and about one hour of preparation, excluding downloads.Source: §A.2 · p. 16
• The artifact is publicly available and has an archived DOI.The archive is identified as https://doi.org/10.5281/zenodo.22937478.Source: §A.2 · p. 17

• The archived source corresponds to Git commit c0325dc and is obtained with five specified files placed in a downloads directory.The README provides complete download commands, and the resource check validates checkpoint files and reads five nonempty inputs from each dataset.Source: §A.3.1 · p. 17
• The reference experiments use one A100 40 GB GPU at batch size one, with 256 GB host memory for offloaded experts.NUMA binding is optional.Source: §A.3.2 · p. 17
• The release requires CUDA Toolkit 12.8, a CUDA driver, a C++ compiler, Python 3.8 or newer for downloading, and pinned supplied software builds.The supplied wheels include GPTQModel 4.0.0.dev0, Transformers 4.53.3, FlashAttention 2.8.1, and vLLM 0.9.2; matching upstream version numbers are insufficient.Source: §A.3.3 · p. 17
• The reference main run takes about 20 minutes, while each depth or prefix sweep takes about 20–30 minutes including model loading.Runtime depends on the host and storage system.Source: §A.3.4 · p. 17

• The setup installs uv without sudo and places caches and temporary build files on the work filesystem.Source: §A.4 · p. 17
• The installation script runs with a copied configuration, and the environment, caches, and results remain outside the source tree.The installer pins flashinfer-python and applies build constraints including torch==2.7.0.Source: §A.4 · p. 17
• After selecting the A100 and CUDA 12.8 toolkit, users validate resources, build kernels, and run tests before continuing.The resource check validates checkpoint files and reads five nonempty inputs from each dataset.Source: §A.4 · p. 17

A.5 Alternate Setup: Existing Environment

• An existing setup can reuse its installed environment, configuration, datasets, and DeepSeek checkpoints after updating the source files to this release.The original main, depth, and prefix commands retain their workloads and parameters.Source: §A.5 · p. 17

A.6 Accessing DeepSeek-V2-Lite

• The release provides an FP16 target checkpoint and a GPTQ INT4 draft with group size 128, plus five datasets.The download helper verifies and reconstructs split archives and resumes interrupted downloads; no local quantization or GitHub login is required.Source: §A.6 · pp. 17–18

A.7 Experiment workflow

• The workflow runs main, depth, and prefix experiments; `--experiment all` runs them sequentially, `--memory` accepts high, low, or both, and Specter can be compared with both baselines in both memory settings.Source: §A.7 · p. 18

A.8 Evaluation and expected results

• The reference high-memory main run achieves approximately 70 ms/token.Comparison outputs report per-dataset and aggregate TPOT, speedup, output agreement, and GPU memory; speedup is baseline mean decode TPOT divided by Specter mean decode TPOT.Source: §A.8 · p. 18
• Users can generate plots and source tables from result directories, and the comparison command writes separate high- and low-memory outputs.The comparison command includes per-method records under each memory setting.Source: §A.8 · p. 18

A.9 Experiment customization

• Main and baseline comparisons use five inputs from each of five datasets, three repeats, a 16-token prefix cap, and 128 output tokens.Depth and prefix sweeps instead use three GSM8K inputs, with depth values from 2 to 32 and prefix caps of 8, 16, and 32 tokens.Source: §A.9 · p. 18