PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices

Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu

arXiv:2308.14352v2cs.LGcs.AIcs.CL

TL;DR

EdgeMoE tackles the memory and runtime costs of deploying large MoE LLMs on edge devices. It stores non-expert weights in memory, fetches activated experts from external storage, and reduces I/O through expert-wise bitwidth adaptation and predictive preloading. Experiments report memory savings, inference speedups, and real-time inference with tolerable accuracy loss.

  • Problem

    Large MoE LLMs remain difficult to deploy on edge devices because expert weights are too large for device memory and repeated autoregressive swapping incurs substantial I/O overhead.

    Source: B. On-device Sparse LLM Inference · pp. 3–4

  • Method

    EdgeMoE stores non-expert weights in memory, loads activated expert weights from external storage, adapts expert bitwidths, and preloads predicted experts alongside computation.

    Source: §I · pp. 1–2

  • Results

    EdgeMoE achieves 1.19×–2.77× inference speedup over memory-optimized baselines and enables >10B-sized LLM inference with accuracy loss ≤2%.

    Source: §I · p. 2

  • Takeaways & Limitations

    EdgeMoE enables real-time MoE LLM inference on edge CPU and GPU platforms while maintaining tolerable accuracy loss.

    Source: §VI · p. 13

Abstract

from arXiv · show

Large language models (LLMs) such as GPTs and Mixtral-8x7B have revolutionized machine intelligence due to their exceptional abilities in generic ML tasks. Transiting LLMs from datacenters to edge devices brings benefits like better privacy and availability, but is challenged by their massive parameter size and thus unbearable runtime costs. To this end, we present EdgeMoE, an on-device inference engine for mixture-of-expert (MoE) LLMs -- a popular form of sparse LLM that scales its parameter size with almost constant computing complexity. EdgeMoE achieves both memory- and compute-efficiency by partitioning the model into the storage hierarchy: non-expert weights are held in device memory; while expert weights are held on external storage and fetched to memory only when activated. This design is motivated by a key observation that expert weights are bulky but infrequently used due to sparse activation. To further reduce the expert I/O swapping overhead, EdgeMoE incorporates two novel techniques: (1) expert-wise bitwidth adaptation that reduces the expert sizes with tolerable accuracy loss; (2) expert preloading that predicts the activated experts ahead of time and preloads it with the compute-I/O pipeline. On popular MoE LLMs and edge devices, EdgeMoE showcase significant memory savings and speedup over competitive baselines. The code is available at https://github.com/UbiquitousLearning/mllm.

I. INTRODUCTION

EdgeMoE addresses the memory and I/O challenges of running sparse MoE LLMs on edge devices by storing hot non-expert weights in memory and fetching cold experts on demand. Expert-wise bitwidth adaptation and predictive preloading further reduce expert-loading overhead, yielding substantial memory savings and inference speedups.

Motivation

  • MoE LLMs reduce computing complexity through sparse expert activation, but their large expert weights and repeated swapping still make edge inference costly.Experts can dominate memory while contributing relatively little computation, and autoregressive decoding amplifies I/O overhead.Source: §I · p. 1

EdgeMoE Design

  • EdgeMoE keeps frequently used non-expert hot weights in device memory and treats remaining memory as a buffer for cold expert weights stored externally.Only activated experts are loaded from storage on demand, allowing the model’s expert count to scale beyond device memory.Source: §I · pp. 1–2

Expert-wise Bitwidth Adaptation

  • Expert-wise bitwidth adaptation progressively quantizes robust experts until a user-specified accuracy-loss threshold is met, minimizing model size and loading time.The selection jointly considers quantization robustness and the inference-speed benefit of lower bitwidths.Source: §I · p. 2

In-memory Expert Management

  • Predictive expert preloading estimates the next layer’s activation from prior-layer activations and overlaps loading with computation.An eviction policy combines activation frequency with experts’ positions in execution, while activation paths are highly skewed despite balanced single-expert frequencies.Source: §I · p. 2

Results

  • 1.19×–2.77× inference speedup over dynamically loading expert and STI baselines, while enabling >10B-sized LLM inference with accuracy loss ≤2%.Across 7 MoE LLMs and Raspberry Pi 4B and Jetson TX2, EdgeMoE also reduces memory footprint by 1.05×–1.18× versus holding the whole model in memory.Source: §I · p. 2

Contributions

  • The paper combines expert-centric storage, bitwidth adaptation, and in-memory expert management into an on-device MoE inference engine evaluated through extensive experiments.The contributions include preliminary edge-device analysis, the EdgeMoE engine, two I/O-reduction techniques, and empirical validation.Source: §I · p. 2

II. PILOT EXPERIMENTS AND ANALYSIS

MoE models scale parameters with nearly constant computation by activating only a few experts per token, but their expert weights create severe memory and I/O costs on edge devices. Pilot measurements motivate storing non-experts in memory, swapping experts selectively, and exploiting skewed activation paths for preloading.

A. A Primer on LLM with Mixture-of-Experts

  • MoE layers activate only a fixed small number of experts per token, enabling sublinearly increased computing complexity as model parameters scale.Routers typically select one or two experts from each layer.Source: A. A Primer on LLM with… · p. 2
  • Scaling experts improves model capacity but increases peak memory almost linearly, making sparse MoE models difficult to fit on edge devices.The pilot describes Raspberry Pi 4B as unable to hold variants larger than the smallest eight-expert Switch Transformer.Source: Figure 2 · p. 3, B. On-device Sparse LLM Inference · p. 3

B. On-device Sparse LLM Inference

  • Layer-by-layer swapping can make I/O loading 30.9× slower than computation, while expert swapping raises per-token decoding time by 3.2×–3.9× on Jetson TX2 and 3.3×–3.8× on Raspberry Pi 4B.The measured latency penalty is relative to an oracle with infinite memory and no expert I/O.Source: B. On-device Sparse LLM Inference · pp. 3–4, Figure 3 · p. 3
  • Expert weights account for 86.5% of ST-base-16 memory but only 26.4% of computation, motivating memory-resident non-experts and on-demand expert swapping.Under this strategy, memory can approach all non-expert weights plus one expert, while I/O is reduced to one expert per layer.Source: B. On-device Sparse LLM Inference · p. 4
  • Expert-level activation dependencies prevent straightforward compute/I/O overlap, and balanced within-layer activation makes frequency-based expert caching ineffective.The next expert is known only after the prior layer computes, while experts in the same layer have similar activation chances.Source: B. On-device Sparse LLM Inference · p. 4
  • The 20% most frequent expert activation paths cover more than 99% of token cases, revealing cross-layer skew that can support predictive preloading.This power-law behavior was observed on two MoE models and two datasets.Source: B. On-device Sparse LLM Inference · p. 4, Figure 4 · p. 4

III. ED G EMOE DESIGN

EdgeMoE is an on-device MoE inference engine that keeps non-expert weights in memory while managing compressed experts between memory and storage. It combines offline expert-specific quantization with runtime expert prediction and preloading, targeting decoder-dominated single-request inference under constrained memory.

A. Overview

  • EdgeMoE targets fast edge inference by exploiting the fact that decoder execution can dominate MoE end-to-end latency by up to 93%.The framework supports interactive tasks such as text generation and summarization.Source: A. Overview · p. 4
  • EdgeMoE preprocesses experts with different bitwidths according to their accuracy sensitivity and a user-specified tolerable accuracy loss.The resulting model may assign different bitwidths even to experts within the same transformer layer.Source: A. Overview · pp. 4–5, Figure 6 · p. 5
  • At runtime, EdgeMoE predicts upcoming experts from activation statistics, preloads them from disk before routing, and overlaps fetching with current-layer computation.It dynamically manages experts in an expert buffer and evicts them when the buffer is full.Source: A. Overview · pp. 4–5, Figure 5 · p. 5
  • The framework supports decoder-only and encoder-only Transformers as well as dynamic and static MoE routing.For static routing, experts can be preloaded directly from the input-token instructions without prediction.Source: A. Overview · p. 5
  • EdgeMoE is designed for sparse decoding of a single request and is not applicable when one inference contains more than one token, including prefilling and multiple input requests.The stated optimization scope is the decoding stage rather than these multi-token scenarios.Source: A. Overview · p. 5

B. Expert-wise Quantization

EdgeMoE uses quantization to fit expert weights within a memory budget while preserving accuracy, assigning bitwidths individually according to expert sensitivity. Its lightweight, training-free procedure balances storage against tolerable accuracy loss.

B. Expert-wise Quantization

  • Expert-wise quantization applies dictionary-based quantization to individual experts without modifying pretrained models or retraining them.The approach avoids quantization-aware training and exploits accuracy–bitwidth tradeoffs at expert granularity.Source: B. Expert-wise Quantization · p. 5
  • Channel-wise linear quantization causes only 1.30%–1.44% accuracy degradation at INT4, while decompression is 1.1×–2.5× faster than Gaussian outlier-aware quantization.On SAMsum, experts can be quantized to INT2 with 5.82% loss; dequantization takes 2.7% of pure I/O loading time on Jetson TX2.Source: B. Expert-wise Quantization · p. 5
  • EdgeMoE profiles each expert’s accuracy sensitivity by quantizing it to INT2, then ranks experts so more important experts retain higher bitwidths.For ST-base-8, quantizing the first expert in the first block causes 0.44% loss, versus 0.59% for the second expert.Source: B. Expert-wise Quantization · p. 6
  • Given a tolerable accuracy loss, EdgeMoE selects global bitwidth bounds and progressively lowers the bitwidth of less important experts until the loss approaches the target.The resulting mixed-precision model balances storage and accuracy, while non-expert weights are selected from INT4, INT8, FP16, and FP32 candidates.Source: B. Expert-wise Quantization · p. 6
  • The quantization design is training-free, because EdgeMoE uses a simple offline search rather than retraining or fine-tuning.Its offline phase iteratively enumerates bitwidth configurations, while online inference only applies the selected configuration.Source: B. Expert-wise Quantization · p. 6

1) Preloading and pipeline:

EdgeMoE predicts upcoming expert activations from prior-layer activation history and overlaps expert loading with computation. This pipeline hides loading latency when predictions succeed, while failed predictions fall back to on-demand loading.

1) Preloading and pipeline:

  • Sequential expert activations are statistically correlated, enabling prediction of the next layer’s expert from earlier activations.For ST-base-8 on SAMsum, observing two prior layers gives an 87.1% probability for one reported next-expert choice.Source: 1) Preloading and pipeline · p. 7
  • EdgeMoE builds an offline activation profile and, online, preloads experts in descending estimated probability before the current router finishes.Depending on the compute–I/O gap, it can preload 1–3 experts per layer.Source: 1) Preloading and pipeline · p. 7
  • The engine executes the current transformer block while preloading the subsequent block’s predicted experts, hiding loading time when forecasts are correct.If predictions fail, inference falls back to loading experts on demand; the reported worst case is no longer than on-demand loading.Source: 1) Preloading and pipeline · p. 7, Figure 9 · p. 7
  • Expert weights are served from an in-memory cache on hits; misses fetch from disk and evict an existing expert when the buffer is full.Evicting an expert that will soon be reused can create substantial additional I/O overhead.Source: 2) Cache eviction policy · p. 7

2) Cache eviction policy:

EdgeMoE combines layer-aware expert caching with end-to-end evaluation across edge platforms. The resulting system reduces memory and latency substantially, while exposing trade-offs from prediction failures, storage speed, and energy use.

2) Cache eviction policy:

  • EdgeMoE’s layer-aware eviction policy prioritizes evicting infrequently used experts from layers farther from the current block.Its score combines activation frequency with layer position; encoder experts are assigned zero frequency because they are loaded only once.Source: 2) Cache eviction policy · p. 7

B. End-to-end Results

  • 2.1×–38.6× memory savings versus IO-FREE are reported across models and platforms, while savings versus IO-QFREE range from 0.8× to 24.3×.EdgeMoE dynamically loads and caches active experts rather than retaining all expert weights in memory.Source: B. End-to-end Results · p. 8, Figure 10 · p. 9
  • 2.64×–3.03× speedup over IO-EXP on Jetson TX2 and 2.55×–3.25× on Raspberry Pi 4B are reported for TPOT.Relative speedups are also 1.19×–2.12× over IO-QEXP and 1.22×–2.77× over STI, attributed to quantization and overlapped preloading.Source: B. End-to-end Results · pp. 9–10, Figure 11 · p. 9
  • EdgeMoE achieves 4.49×–4.76× speedup over IO-EXP on HDDs, compared with 2.63×–3.01× on SSDs.The larger HDD improvement is attributed to longer baseline expert-loading times on slower storage.Source: B. End-to-end Results · pp. 9–10
  • On Raspberry Pi 4B, 50% of ST-base-8 samples have latency below 46 seconds with EdgeMoE, versus below 106 seconds with IO-EXP.The comparison comes from the per-sample latency CDF.Source: Figure 12 · p. 9, B. End-to-end Results · p. 10
  • EdgeMoE’s accuracy loss approximately matches the configured 2% tolerance, whereas STI incurs significant loss by quantizing both expert and non-expert weights.Its offline profiling stage completes within half an hour, and storage improves by 3.03× over IO-FREE and 1.11× over IO-QEXP for ST-base-8.Source: B. End-to-end Results · p. 10, Table II · p. 10
  • Compared with IO-FREE, EdgeMoE incurs an average 17% energy increase, with the smallest reported increase at 1.1%.The paper attributes the overhead to compute-dominated inference, where CPU/GPU energy is the major component.Source: B. End-to-end Results · p. 10, Figure 13 · p. 10
  • On Xiaomi 14, EdgeMoE saves 63% memory versus IO-FREE and increases speed by 25% versus IO-QEXP.The mobile-phone gains are smaller than on Jetson devices because stronger compute makes I/O the larger bottleneck.Source: B. End-to-end Results · p. 11, Figure 15 · p. 11

C. Sensitivity Analysis

EdgeMoE adjusts its expert buffer and bitwidth configuration to available memory, storage, latency, and accuracy budgets. Across tested devices and models, larger buffers reduce latency and relaxed accuracy targets enable further resource adaptation.

C. Sensitivity Analysis

  • Larger expert buffers reduce inference latency by retaining more previously activated experts and increasing cache hit ratios.Experiments span memory budgets from 1.6GB to 3.5GB, reflecting different edge-device memory profiles.Source: C. Sensitivity Analysis · p. 11, Figure 16 · p. 11
  • EdgeMoE has lower inference latency than both INT4 and INT2 IO-QEXP across all tested memory-budget configurations.The comparison is reported for two devices.Source: C. Sensitivity Analysis · p. 11
  • The number of cacheable experts grows linearly with the memory limit for Switch Transformers base-8 on Jetson TX2 and MiniCPM MoE 8x2B on Xiaomi 14.The corresponding cache limits are summarized in Table III.Source: C. Sensitivity Analysis · p. 11, Table III · p. 11
  • Across desired accuracy-loss levels from 2% to 20%, EdgeMoE tunes expert bitwidths to adapt storage and inference latency on Jetson TX2.The evaluation covers ST-base-8 and ST-base-16.Source: C. Sensitivity Analysis · pp. 11–12, Figure 17 · p. 11
  • Using different offline datasets, EdgeMoE maintains approximately 2% average accuracy loss relative to the original model across the reported model and dataset settings.Table IV evaluates Switch Transformers and MiniCPM MoE variants with multiple offline datasets.Source: C. Sensitivity Analysis · p. 12, Table IV · p. 12

D. Ablation Study

EdgeMoE’s techniques progressively reduce inference latency and generally reduce memory footprint in the ablation study.

D. Ablation Study

  • 0.789s to 0.245s: combining expert-wise quantization, preloading and pipeline, and expert buffering progressively reduces ST-base-8 latency on Jetson TX2.The corresponding memory footprint changes from 2.5GB to 0.81GB.Source: D. Ablation Study · p. 12

V. RELATED WORK

Prior work addresses memory-efficient neural inference, MoE systems, resource-efficient LLMs, and on-device optimization. EdgeMoE targets autoregressive MoE LLM inference on mobile devices and is presented as complementary to these approaches.

Ablation context

  • Figure 18 evaluates EdgeMoE’s ablations using TPOT and memory footprint, with Q denoting expert-wise quantization, P preloading and pipeline, and C experts buffer.The supplied caption identifies the three ablation components and the evaluated metrics.Source: Figure 18 · p. 12

DNN memory optimizations

  • EdgeMoE shares quantization and swapping rationales with mobile memory optimizations but exploits MoE-specific expert structure for memory saving.Prior examples include Split-CNN, Melon, and SwapNN.Source: §V · p. 12

Systems for MoE models

  • Prior MoE systems target distributed clouds, training, or non-autoregressive models, whereas EdgeMoE accelerates autoregressive MoE-based LLM inference on mobile devices.The comparison includes Edge-MoE, X-MOE, and TA-MoE.Source: §V · p. 12

Resource-efficient LLMs

On-device ML optimizations

  • On-device optimizations span system-level processor, cache, kernel, and offloading techniques and model-level quantization or sparsification, but EdgeMoE is built for resource-intensive MoE LLMs.The passage contrasts these methods with LLMs whose running memory can greatly exceed edge-device capacity.Source: §V · p. 13

VI. CONCLUSIONS

EdgeMoE is presented as an on-device inference engine for MoE LLMs that combines expert-specific compression and predictive preloading. Experiments report real-time edge inference with tolerable accuracy loss.

VI. CONCLUSIONS

  • EdgeMoE integrates expert-specific bitwidth adaptation and expert preloading with a compute-I/O pipeline for on-device MoE LLM inference.The first reduces expert sizes, while the second anticipates activated experts.Source: §VI · p. 13
  • Extensive experiments demonstrate real-time inference for MoE LLMs on edge CPU and GPU platforms while maintaining tolerable accuracy loss.Source: §VI · p. 13

APPENDIX A ED G EMOE EXAMPLES

The appendix includes model-output examples and author biographies, alongside a description of Wellington Monument studies unrelated to EdgeMoE.

APPENDIX A ED G EMOE EXAMPLES

  • The appendix presents Switch Transformers base-8 outputs before and after EdgeMoE processing.Two examples are referenced, but their outputs are not included in the supplied passage.Source: Appendix A · p. 15
  • The remaining supplied appendix passages contain unrelated examples and author biographies rather than EdgeMoE findings.They include rugby, Wellington Monument, and contributor-information passages.Source: Appendix A · pp. 15–16

Related papers

Loading 2308.14352v2…