EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs

1 Introduction

• Prefill-only mobile services derive useful outputs from prefill-stage logits without autoregressive decoding, including scam-call detection and intent mapping.On-device inference keeps data local and can operate under poor network conditions.Source: §1 · p. 1
• MoE prefill couples runtime-dependent expert execution, which conflicts with fixed NPU graphs, with request-level activation of nearly the entire expert pool.Existing CPU/GPU offloading reduces expert residency but handles execution off the NPU; pruning and expert substitution trade accuracy for efficiency.Source: §1 · pp. 1–2
• EStream shares one topology-invariant expert graph, binds routes and parameters at runtime, and streams expert groups through a bounded NPU-addressable arena.The UFS–NPU pipeline overlaps loading with computation and is automatically configured; execution remains on the NPU without model modification.Source: §1 · p. 2
• EStream is evaluated on a Snapdragon 8 Elite Gen 5 smartphone across 18 settings for three MoEs and prompts up to 4,096 tokens.Additional experiments use Qwen3-30B-A3B and Mixtral-8×7B; the implementation uses weight-only Q4 quantization, floating-point activations, and no approximation-based sparsity.Source: §1 · p. 2
• 2.25–27.57× pure-prefill TTFT speedup and 6.45–12.29× less physical memory than the fastest completed non-offloading baseline at each point.These results cover 18 primary model–length settings.Source: §1 · p. 2

2.1 NPUs on Mobile SoCs

• At least 64-route experts account for 94.8% of CPU expert projection time during 1K-token OLMoE prefill; at that route count, the NPU is 4.4× faster than the CPU and 4.6× faster than the GPU.The mobile HTP combines scalar threads, HVX, HMX, DMA, and software-managed VTCM to process different tiles concurrently.Source: §2.1 · p. 2

2.2 Obstacles to On-Device MoE Prefill

• At 512 tokens, evaluated models activate 80.1–98.1% of layer–expert pairs, rising to 92.4–100% at 4K; expert parameters make up 91.4–94.7% of their weight payloads.Dynamic expert sets and routed-token counts conflict with fixed graph shapes and parameter bindings; demand-paged mmap can accumulate touched pages and trigger reclamation and refaults.Source: §2.2 · p. 3

2.3 Opportunities and Challenges

• Compatible experts share the same computation topology, and immutable expert weights need not be physically co-resident if a bounded arena is safely reused after each group’s final use.Because compute grows with prompt length while expert I/O approaches the pool’s fixed size once routing densifies, longer prompts offer more computation behind which UFS reads can be hidden.Source: §2.3 · p. 3

3.1 System Overview

• EStream prepares reusable non-expert and expert graphs, streams expert groups into a bounded arena, and overlaps loading with NPU execution.Offline preparation also converts expert weights to the NPU-consumable layout and profiles device service times to configure group size, queue depth, and resident banks.Source: §3.1 · pp. 3–4

3.2 Intra-NPU MoE Execution

• One graph serves shape-compatible experts by binding routes and parameter addresses at invocation time; a fused operator pipelines expert stages within the NPU.The graph keeps its topology and tensor bases fixed while runtime descriptors and parameter bindings change.Source: §3.2 · pp. 4–5

3.3 Expert Virtualization with UFS Streaming

• A fixed-address arena reuses slots for streamed expert groups, bounding residency; offline tile-major packing lets loaded weights enter designated banks without runtime reformatting.Grouping amortizes transfer and dispatch overhead, while larger groups can fetch inactive experts and leave fewer groups for pipeline overlap.Source: §3.3 · p. 6

3.4 Pipelining UFS Streaming and NPU Execution

• Starting prefetch during non-FFN computation reduces the 1,024-token OLMoE bubble rate from 34.41% to 9.05%.Prefetching after routing yields 21.36%.Source: §3.4 · pp. 6–7, Figure 6 · p. 7
• The pipeline fixes queue depth at 2, then jointly selects group size and bank count under the memory budget using brief profiles and max-plus evaluation.The emitted configuration is stored with the prepacked model and reused across requests.Source: §3.4 · p. 7

Runtime and HTP operators

• The HTP backend implements operators for embeddings, attention components, routing, and the fused expert path through weighted scatter-add.The host maps reusable graph-template buffers to cDSP and submits descriptors through DSPQueue.Source: §4 · p. 7

Numerical representation

• Expert weights and large non-expert matrices use weight-only GGUF Q4_0, while selected output-head and MLA matrices retain Q6_K or FP16 representations.Activations remain FP32, and KV caches use FP16.Source: §4 · pp. 7–8

Application workloads

• Figure 7 presents mean request latency–memory tradeoffs across seven workloads, with models by row, datasets by column, and logarithmic axes.The lower-left region represents better tradeoffs.Source: Figure 7 · p. 8

Model structures

• Table 1 summarizes model structures and deployed payloads, with Layers denoting total blocks and E denoting experts per MoE block.Source: Table 1 · p. 8

Expert loading and synchronization

• Loader threads place offline-packed weights directly into free UDMABUF-backed banks; synchronization prevents execution on partially loaded weights or premature bank reuse.The executor waits for a completed binding and releases a bank only after the NPU completion fence.Source: §4 · p. 8

Evaluation setup

• Table 2 covers datasets, metrics, and mean prompt lengths.Source: Table 2 · p. 8

5.1 Experiment Settings

• Phone experiments cover OLMoE-1B-7B, LFM2.5-8B-A1B, and DeepSeek-V2-Lite, three models no larger than 16B; expert and large non-expert matrices use GGUF Q4_0, while activations remain FP32.Measurements use a OnePlus PLK110 running Android 16 with 15.1 GiB of memory, a Snapdragon 8 Elite Gen 5 platform, and UFS 4.1 storage.Source: §5.1 · p. 8

5.2 Performance Evaluation

• On longer application workloads, EStream is 3.79–5.75× faster and uses 5.91–12.87× less memory; short requests can still favor optimized CPU runtimes.It is within 8% of the fastest baseline on 66-token ToxicChat and 1.03–1.21× faster on 80-token HellaSwag.Source: §5.2 · p. 9
• 2.25–27.57× faster TTFT and 6.45–12.29× lower physical memory than the fastest completed non-offloading baseline across 18 model–length settings.At 4K tokens, TTFT is 2.12 s for OLMoE, 2.14 s for LFM2.5, and 9.53 s for DeepSeek-V2-Lite.Source: §5.2 · p. 9, Figure 8 · p. 9
• Qwen3-30B-A3B sustains 211.7–599.1 tokens/s within 1.46–1.87 GiB, while Mixtral-8×7B sustains 144.5–301.5 tokens/s within 2.77–4.11 GiB.The evaluated models contain 30.5B and 46.7B parameters, respectively.Source: Table 3 · p. 9, §5.2 · p. 10

5.3 Energy Efficiency

• EStream is the most energy-efficient runtime in every completed model–length setting, improving efficiency by 1.19–8.41× over the strongest baseline.Across 21 settings, the geometric mean improvement is 4.08×.Source: §5.3 · p. 10, Figure 9 · p. 10

5.4 Model Accuracy

• Quality changes are generally small against the Host reference, although Qwen3 has a 5.51% relative PPL increase and DeepSeek has a 3.00-point ARC-C exception.Across three model families, the MoE deployments outperform comparable active-parameter-scale dense checkpoints on 18 of 21 reported metrics.Source: §5.4 · p. 10, Table 4 · p. 10

5.5 Pipeline Configuration and Auto-Tuning

• The auto-configurator matches 9 of 12 constrained minima, with zero median and 3.45% worst-case regret, and selects configurations 34.2–76.7× faster than exhaustive sweeps.Selected points satisfy measured budgets; new configurations can be selected from reusable profiles in 0.34–2.79 ms.Source: §5.5 · p. 11, Figure 10 · p. 11

5.6 System Ablation

• EStream is 1.83–30.14× faster than CPU-resident execution; expert-group pipelining is 1.18–1.73× faster than naive streaming for 2.7–14.1% more memory.At 4K, streaming adds 4.3–23.0% latency versus resident NPU execution while using 3.73–5.41× less memory; EStream beats the Layer-B=2 schedule in 15 of 18 cases.Source: §5.6 · pp. 11–12, Figure 11 · p. 11

Related Work

• Prior parameter-offloading systems target dense models, decode locality, CPU/GPU execution, or altered precision; none supports full-NPU MoE prefill acceleration on mobile devices.PowerInfer-2 overlaps UFS reads with NPU-centric prefill but uses precompiled graph variants and lacks a public mobile implementation.Source: §6 · p. 12
• Mobile prefill systems use computation reuse, specialization, or heterogeneous execution, while NPUMoE assumes resident weights and leaves routing and aggregation on the CPU.Consequently, NPUMoE cannot bind runtime route extents and streamed expert addresses within reusable NPU execution.Source: §6 · p. 12
• Programmable Hexagon systems expose execution below graph operators, but the cited work does not support dynamic MoE prefill or efficient MoE memory management.The passage names FastRPC-connected operators, Hexagon-MLIR, and llada.cpp as examples.Source: §6 · p. 12

More aggressive quantization

• EStream currently uses weight-only Q4_0 without activation quantization; lower or mixed precision could reduce UFS traffic but may increase NPU dequantization cost.The discussion calls for jointly optimizing quality, I/O latency, and dequantization cost.Source: §7 · p. 12

MoE decode

• EStream targets prefill-only workloads, not decode, where one-token expert inputs offer little matrix work to saturate HMX or hide storage access.A decode extension would need a separate policy for expert retention, route prediction, and potentially batching or speculative verification.Source: §7 · p. 12

Portability across NPUs

• The design can port beyond Qualcomm if a platform exposes programmable NPU vector and matrix computation plus asynchronous data movement.Heterogeneous expert dimensions or expert-specific operators may require multiple shared graph templates.Source: §7 · p. 12

• EStream separates reusable expert computation topology from runtime route and parameter bindings, streams weights through a bounded NPU-addressable arena, and overlaps UFS loading with full-NPU prefill.Source: §8 · pp. 12–13
• Across structurally distinct MoEs and 256–4,096-token inputs, EStream consistently achieves a better latency–memory operating point on a commercial Snapdragon smartphone.Source: §8 · p. 13
• Against resident CPU and QNN context-switching baselines, EStream accelerates prefill by up to 30.14× and 21.02×, respectively.Source: §8 · p. 13
• Compared with an I/O-free resident reference, EStream uses 3.73–6.36× less peak physical memory while incurring 4.3–23.0% higher latency at 4K.Source: §8 · p. 13
• EStream has the highest energy efficiency in every completed model–length setting, and the results show deployable MoE capacity need not be limited by resident expert memory.Source: §8 · p. 13