PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileMoE: Scaling On-Device Mixture of Experts

Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi

arXiv:2605.27358v1cs.LGcs.AIcs.CL

TL;DR

On-device MoE scaling in the sub-billion active-parameter regime remains largely unexplored. MobileMoE addresses this with a mobile-constrained scaling law, a four-stage training recipe, and smartphone deployment, establishing a Pareto frontier for on-device LLMs while improving runtime against a dense baseline.

  • Problem

    Sub-billion-active MoE scaling for on-device LLMs remains largely unexplored despite the need for low-latency, cost-effective, privacy-preserving edge applications.

  • Method

    MobileMoE combines an on-device MoE scaling law for joint memory-compute architectural design with a four-stage training recipe and a custom fused MoE kernel for smartphone inference.

  • Results

    MobileMoE establishes a new Pareto frontier for on-device LLMs and, at comparable INT4 weight memory, achieves 1.8–3.8× faster prefill and 2.2–3.4× faster decode than MobileLLM-Pro.

  • Takeaways & Limitations

    The results demonstrate MoE as a practical path for edge deployment, supporting local intelligence with lower cloud compute demand and private, low-latency inference.

  • Takeaways & Limitations

    The gap with Qwen3.5 2B on instruction following and knowledge & reasoning motivates future improvements such as distillation and thinking-enabled post-training.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4$\times$ fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers $1.8$-$3.8\times$ faster prefill and $2.2$-$3.4\times$ faster decode than the dense baseline MobileLLM-Pro.

1 Introduction

MobileMoE addresses the limited exploration of sub-billion-active MoE models for on-device deployment by jointly optimizing architecture, training, and runtime under mobile constraints. It establishes strong benchmark and smartphone-runtime results relative to dense and MoE baselines.

  • On-device MoE scaling remains largely unexplored despite strict compute and memory constraints and the potential for low-latency, privacy-preserving edge applications.Existing scaling frameworks primarily target much larger cloud-deployed MoE models.
  • MobileMoE-S/M/L use 0.3B/0.5B/0.9B active and 1.3B/2.8B/5.3B total parameters, with less than 3 GB INT4 weight footprints.The family is derived from a scaling law tailored to joint mobile memory and compute constraints.
  • Across 14 benchmarks, smaller MobileMoE models match or exceed dense baselines with 2-4× fewer inference FLOPs at comparable memory.MobileMoE-M matches OLMoE-1B-7B accuracy with approximately 60% fewer active and total parameters, while MobileMoE-L achieves higher accuracy with smaller active parameters and model memory.
  • At comparable INT4 weight memory, MobileMoE-S achieves 1.8-3.8× faster prefill and 2.2-3.4× faster decode than MobileLLM-Pro on flagship smartphones.Deployment uses a custom fused MoE kernel and reports profiling across CPU and GPU backends.
  • Moderate sparsity, fine-grained experts, and a shared expert define the identified on-device MoE sweet spot.These design choices are selected through the generalized on-device MoE scaling law.
  • A four-stage recipe combines pre-training, mid-training, instruction fine-tuning, and INT4 quantization-aware training with MoE-specific stability and efficiency techniques.MobileMoE uses approximately 6T pre-training tokens, fewer than cited dense baselines trained on 9T and 11T tokens.

2 Related Work

Related work establishes dense on-device LLMs and sparse MoE as complementary approaches, while prior scaling laws mainly address larger models and server-oriented settings. MobileMoE explores a compact MoE design space for edge deployment.

  • Dense on-device LLMs target sub-billion to few-billion scales under stringent latency and memory constraints, using architectures such as deep-and-thin backbones.Examples include MobileLLM, MobileLLM-Pro, SmolLM, and Gemma.
  • MoE increases transformer capacity by routing each token to a sparse subset of specialized expert subnetworks while keeping active parameters compact.Expert specialization supports different domains and tasks.
  • Existing MoE scaling laws study factors such as expert count, expert granularity, and allocation under memory constraints, but primarily target much larger models.The related formulations motivate a scaling treatment tailored to on-device regimes.
  • MobileMoE varies sparsity, expert granularity, and shared experts across Small, Medium, and Large models with 0.3B, 0.5B, and 0.9B active parameters.The base architecture uses expansion ratio dff/dmodel = 4, aspect ratio dmodel/nl ≈40, SwiGLU, and GQA with 4 KV heads.

3 Scaling On-Device MoE

MobileMoE formulates an on-device MoE scaling law that jointly accounts for compute and memory, then uses it to identify moderate sparsity, fine-grained experts, and shared experts as effective design choices.

  • On-Device MoE Scaling Law: The generalized scaling law models loss using active parameters, training data, expert count, expert granularity, and shared experts.Expert count controls total parameters and sparsity, while granularity and shared experts represent architectural choices.
  • On-Device Optimization: The optimization minimizes model loss under training compute, inference compute, and device-memory constraints.The memory proxy includes quantized weights and KV-cache memory, with device DRAM roughly capped at 5 GB for app usage.
  • Scaling the Number of Experts: MoE models achieve lower loss than dense models at fixed memory above 0.25 GB, while increasing expert count reduces loss with diminishing returns beyond E = 8.Moderate sparsity with E ∈{4, 8} is identified as the practical sweet spot in the on-device memory regime.
  • Expert Granularity: Fine-grained experts achieve substantially lower loss at fixed compute, with diminishing returns beyond g = 8.The result is attributed to more diverse top-k routing.
  • Shared Experts: Shared experts achieve lower loss than routed-only experts at fixed compute, while also improving training throughput and final loss at the same active and total parameters.The shared expert complements routed specialists as a generalist component.
  • Derived Architectures: MobileMoE adopts E = 8, g = 8, and a shared expert, producing S/M/L models with 0.3B/0.5B/0.9B active and 1.26B/2.82B/5.33B total parameters.All three models fit within a 3–5 GB on-device memory budget under 4-bit quantization.

4 Experiments

Experiments show that MobileMoE establishes strong parameter and token efficiency across base and instruct evaluations, while delivering gains concentrated in knowledge, reading, code, and math. Its advantages persist through training stages and translate to substantial reasoning and comprehension improvements.

  • Data scaling: MobileMoE-L surpasses Llama 3.2 1B at ∼0.5T tokens, SmolLM2-1.7B at ∼1T, and OLMoE-1B-7B at ∼2T on overall average.
  • Model scaling: MobileMoE-L reaches Avg 60 with 922M active parameters, surpassing OLMoE-1B-7B by +8 with 30% fewer active parameters.
  • Model scaling: MobileMoE models improve monotonically across S/M/L, with Overall Avg increasing from 47 to 55 to 60.
  • Instruction tuning: MobileMoE-SFT matches or exceeds larger dense instruct models with 2–4× fewer active parameters across the evaluated scales.
  • Additional capabilities: MobileMoE-L shows clear strengths on code and math, including +13.2 over Qwen3.5 2B on code and leadership over dense baselines on math.
  • Limitations: MobileMoE’s gap with Qwen3.5 2B on instruction following and knowledge & reasoning motivates future distillation and thinking-enabled post-training.
  • Training stages: Mid-training produces the largest knowledge and reading gains, while instruction SFT produces the largest GSM8K improvements.

5 Conclusion

MobileMoE combines an on-device scaling law, an end-to-end training recipe, and efficient smartphone deployment. The resulting models establish a new benchmark frontier and demonstrate MoE as a practical edge-computing alternative.

  • MobileMoE jointly optimizes MoE architecture under mobile memory and compute constraints.
  • The four-stage recipe scales MobileMoE training to a new Pareto frontier in on-device LLM benchmark performance.
  • MobileMoE provides efficient MoE inference on commodity smartphone CPUs with systematic CPU and GPU profiling.
  • The paper identifies distillation, reasoning-oriented post-training, multimodal extensions, dynamic routing, compression, and NPU deployment as future directions.

A Scaling Law Ablation Details

The scaling-law ablations vary the number of experts, expert granularity, and shared experts to support MobileMoE’s on-device architecture derivation.

  • The ablations study number of experts E, expert granularity g, and shared experts s.
  • All ablations use the Figure 2 base architectures and train on up to ∼500B tokens.
  • Each ablation configuration varies its target architecture factor while holding the remaining settings fixed for the scaling-law analysis.

A.1 Parametric Fitting of On-Device MoE Scaling Laws

The fitting procedure estimates on-device MoE scaling-law coefficients from validation-loss sweeps, while constraining expert-count transformations to mobile memory budgets. The resulting fits support interpolation of optimal expert counts.

  • Parametric fitting: The procedure warm-starts nonlinear least-squares curve fitting, then refines coefficients with bounded L-BFGS-B minimization using MSE.
  • Expert-count sweep: Expert-count fitting uses E ∈ {1, 2, 4, 8, 16, 32}, with E = 1 as the dense baseline and Emax = 32 constrained by mobile memory.
  • Fit validation: The fitted transformation yields the same optimal E under a 5 GB on-device memory budget as simplified and free-parameter alternatives.
  • Ablation-specific fitting: The E-sweep jointly fits all coefficients, whereas g- and s-sweeps fit independently with E fixed and absorb E-dependent exponents into effective constants.

B Training Data

MobileMoE uses publicly available data across pre-training, mid-training, and supervised fine-tuning, with domain distributions tailored to each stage. Pre-training emphasizes broad coverage, while later stages concentrate higher-quality domain-specific and instructional data.

  • Data availability: All training data across pre-training, mid-training, and SFT stages are publicly available under permissive open-source licenses.The listed licenses include CC-BY-4.0, Apache 2.0, ODC-BY, MIT, and NVIDIA License.
  • Pre-training data: Pre-training uses the Dolma3 mix and curated MobileLLM data, with a web-heavy mixture for broad linguistic coverage.The pre-training distribution is 62% web, alongside math, knowledge, code, and science domains.
  • Pre-training data: Domain-diverse pre-training data supports language modeling, reasoning, coding, factual knowledge, and expert specialization across token types.The mixture includes 11.6% math, 10% knowledge, 10% code, and 6.4% science.
  • Mid-training data: Mid-training shifts toward higher-quality, domain-specific data and extends context length to 8,192 tokens.Web data decreases from 62% to 9%, while knowledge rises from 10% to 32%, code from 10% to 22%, and math from 12% to 21%.
  • Mid-training data: Domain-concentrated mid-training further sharpens routed experts’ specialization on domain-specific tokens.The data mixture specifically upweights knowledge, code, and math relative to pre-training.
  • Supervised fine-tuning data: SFT combines more than 80M samples from 28 public collections across math, instruction, code, safety, science, tool use, and reasoning.Sampling is proportional to dataset size while a floor ensures small but important domains receive representation.

C.1 Evaluation setup of base and instruct models on foundational competencies

The foundational evaluation covers both pre-trained and instruction-tuned models across 14 benchmarks using standardized lm-eval configurations. Runs use deterministic greedy decoding and automatic batch sizing.

  • Scope: Both pre-trained and instruction-tuned models are evaluated on 14 foundational benchmarks.The evaluation uses the Language Model Evaluation Harness with the vLLM backend.
  • Protocol: Evaluation uses lm-eval with vLLM, automatic batch sizing up to 16, and bfloat16 at model precision.The configuration sets dtype=auto, which resolves to bfloat16 for these models.
  • Protocol: Base and instruction models use standard few-shot settings, with per-task configurations summarized in Table C.1.Instruction-tuned models are evaluated in non-thinking mode.

C.2 Evaluation setup of instruct models on advanced competencies

Instruction-tuned models are evaluated on eight advanced benchmarks spanning math, code, instruction following, and harder knowledge and reasoning. The setup combines lm-eval with official benchmark packages and uses deterministic generation where applicable.

  • Benchmark scope: Eight advanced benchmarks cover math, code, instruction following, and harder knowledge and reasoning.The evaluated tasks include MATH500, GSM-Plus, HumanEval, MBPP, IFEval, GPQA Diamond, MMLU-Pro, and IFBench.
  • Evaluation tools: lm-eval evaluates MATH500, GSM-Plus, HumanEval, MBPP, IFEval, and GPQA Diamond, while official packages evaluate MMLU-Pro and IFBench.Per-task settings are summarized in Table C.2.
  • Protocol: Generation tasks use chat templates and greedy decoding with T = 0 unless otherwise noted, while loglikelihood tasks do not generate.The advanced-benchmark table distinguishes generation from loglikelihood evaluation.
  • Baselines: Baseline models are publicly available, with HuggingFace identifiers listed for both base and instruction-tuned models.The identifiers used in evaluation are provided in Table C.3.

C.4 MMLU-Pro and GPQA Diamond per-protocol ablation

The appendix compares MMLU-Pro and GPQA Diamond under multiple evaluation protocols across 13 instruction-tuned baselines. It reports that chat-template protocols generally reduce baseline scores, with Qwen3.5 2B as the main exception.

  • Protocols: MMLU-Pro is compared in three 5-shot protocols: the official package, lm-eval loglikelihood, and lm-eval Chat.The protocols differ in prompting, chat-template use, and whether they generate chain-of-thought responses.
  • Protocols: GPQA Diamond is compared in two 0-shot protocols: lm-eval loglikelihood and lm-eval Chat.The loglikelihood variant uses multiple-choice scoring, while Chat uses chain-of-thought generation with a chat template.
  • Comparison set: The ablation covers 13 instruction-tuned baselines under multiple evaluation protocols.Main-text reporting uses the official MMLU-Pro package and lm-eval loglikelihood for GPQA Diamond.
  • Protocol effects: MMLU-Pro Chat regresses on 9 of 13 models relative to the original evaluation package, while Qwen3.5 2B improves from 38.8 to 48.9.Examples include Gemma 3 1B declining from 16.1 to 0.0 and Llama 3.2 1B from 20.8 to 13.0.
  • Protocol effects: GPQA Diamond Chat degrades 10 of 13 models and benefits only Qwen3.5 2B, which gains 9.6 points.The reported pattern motivates avoiding protocol-induced bias in the main comparisons.

D Quantitative Analysis

MobileMoE exhibits task-specific expert specialization, while training progressively broadens expert utilization. These patterns imply opportunities for selective expert loading or task-conditional pruning to reduce on-device memory.

  • Different downstream domains activate distinct subsets of fine-grained experts, indicating cross-task specialization.MobileMoE-S shows domain-specific activation patterns across code, math, and knowledge tasks.
  • Expert utilization broadens progressively from pre-training through mid-training and supervised fine-tuning.More experts become activated during downstream training while cross-task specialization is maintained.
  • Math activates a broader set of experts, whereas code and knowledge concentrate on narrower subsets.The task-dependent utilization distributions are reported across all MoE layers after supervised fine-tuning.
  • Task-dependent sparsity enables selective expert loading or task-conditional pruning to save on-device memory.Because not every expert weight needs to be loaded at inference, utilization patterns expose a deployment pathway for memory reduction.
Loading 2605.27358v1…