PaperlayerView exact revision on arXiv

Source-linked AI summary

Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone

Musa Shams

arXiv:2609.29032v1cs.PFcs.LG

TL;DR

Sparse activation does not remove the need to store all experts, motivating a storage-backed MoE runtime for a phone. Routide characterizes cache policies and separates same-runtime cache transparency from cross-runtime numerical comparison. Its evidence supports bounded feasibility, not universal efficiency or equivalence claims.

  • Problem

    Sparse MoE computation still leaves the full expert set to store, while quantization does not close the memory gap for every model/device combination.

    Source: §1 · p. 1

  • Method

    Routide implements a Swift/MLX storage-to-execution path for a pinned checkpoint, using a byte-budgeted expert cache and evaluating cache behavior and comparison boundaries.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Same-runtime cache controls preserve output sequences across cache eviction and prefetch in 2,560 exact token comparisons, while complete resident-Python sequences do not match the recorded phone outputs.

    Source: §5.3 · p. 8

  • Takeaways & Limitations

    The cache cliff is policy-dependent, and bounded feasibility on a phone supports careful reproduction rather than a universal efficiency claim.

    Source: §7 · pp. 11–12

    Cross-framework and cross-device sequence equivalence remains unestablished, and logical expert reads and sampled process footprints do not measure physical storage traffic or continuous memory peaks.

    Source: §6 · p. 11

Abstract

from arXiv · show

Sparse activation reduces mixture-of-experts computation without eliminating the need to store all experts. We present Routide, a Swift/MLX runtime that executes the text path of a pinned public Qwen3.6-35B-A3B quantized checkpoint while keeping expert weights in iPhone storage and a byte-budgeted subset in memory. We characterize cache-policy sensitivity, numerical comparison boundaries, and measurement limits. Across five recorded 128-token workloads, fixed-route replay gives 0.00% demand hits with a 512 MiB LRU cache, 18.80% with seeded random eviction at the same budget, and 38.58% with 576 MiB LRU. The apparent capacity cliff is therefore a policy/workload interaction, not a universal memory requirement. Same-runtime Mac controls preserve generated sequences across eviction and asynchronous prefetch, including 2,560 exact token comparisons and 10,334 speculative loads. In contrast, complete resident-Python versus recorded-phone sequences disagree on all five tested cases, precluding a general numerical equivalence claim. Two separately scoped iOS 27 memory protocols observe sampled process-footprint peaks of 1.87-2.32 GiB on short prompts and 2.39-2.73 GiB on one longer prompt. We retain a thermal stopping event, negative timing comparisons, and a single qualified whole-device power estimate. These results establish bounded feasibility and identify limitations that a deployment claim must not hide.

1 Introduction

Sparse activation leaves storage of the full expert set as a phone-memory challenge, and quantization does not resolve it for every model/device combination. Routide provides an inspectable storage-backed text runtime and characterizes cache behavior while separating correctness and measurement claims.

Motivation

  • Storage-backed MoE execution remains a systems problem involving layout, cache admission and eviction, I/O scheduling, framework arithmetic, and measurement semantics.A text-emitting demonstration alone establishes neither efficient execution, numerical equivalence, nor improved task quality.Source: §1 · p. 1

Contributions and scope

  • The implementation combines a bounded-memory packer, aligned per-expert layout, Swift positional reader, byte-budgeted cache, quantized MLX executor, and on-device benchmark capture.It forms an inspectable storage-to-execution path for the checkpoint’s text path.Source: §1 · p. 1
  • The paper separates same-runtime cache transparency from cross-runtime sequence equivalence, logical expert traffic from physical storage traffic, and sampled process footprint from MLX allocator counters.It also keeps a stopped phone campaign separate from its declared continuation.Source: §1 · p. 2
  • Routide’s evidence supports a narrower, reproducible characterization—not claims of first-phone-runtime status, state-of-the-art throughput, resident-model quality parity, or prefetch energy savings.The implementation and measurements are scoped to the pinned checkpoint’s text path.Source: §1 · pp. 1–2

2 Background and related work

Routide situates its storage-aware, explicitly routed expert paging among work on flash-backed inference, phone-oriented sparse execution, and CPU-GPU MoE offloading. Its fixed-checkpoint Apple-hardware scope is distinct from those systems and does not establish superiority over them.

Flash-backed inference

  • Routide pages explicitly routed, quantized MoE experts, unlike LLM in a flash’s learned neuron-sparsity predictor and activation-function modifications [2].Routide does not reproduce that paper’s physical-storage bandwidth experiments or compare its reported speed with phone timings.Source: §2 · p. 2

Phone-oriented sparse execution

  • PowerInfer-2 combines neuron-granularity caching, heterogeneous execution, and fine-grained I/O/computation pipelining on smartphones [3].Routide instead characterizes a fixed public checkpoint on Apple hardware and does not implement its planner or establish superiority over it.Source: §2 · p. 2

MoE offloading

  • Fiddler orchestrates expert execution across CPU and GPU to reduce data movement [4], whereas Routide’s setting is iPhone unified memory plus storage.Both settings distinguish total parameter size, active arithmetic, and data movement.Source: §2 · p. 2

Runtime and cache references

  • Routide builds on Apple’s MLX Swift ecosystem and LLMEval, while treating farthest-next-use as an offline paging reference.Neither the feasible pin-constrained schedule nor the relaxed unpinned equal-block oracle predicts online without future routing.Source: §2 · p. 2

3 System design

Routide packs a pinned quantized checkpoint into aligned expert blocks and executes its text path by positional reads and a byte-budgeted cache. Optional prefetch changes cache access policy without changing routing or expert arithmetic.

3.1 Checkpoint and storage inventory

  • The checkpoint’s text path uses 40 layers, 256 routed experts per layer, and top-eight selection; its primarily affine INT4 weights use group size 64, but not every stored scalar is four bits.Ten layers use full attention and 30 use recurrent gated-delta attention; router-related overrides and BF16 quantization metadata are present.Source: §3.1 · p. 2
  • The pack stores each expert’s gate, up, and down projections with scales and biases, aligned to 64 KiB boundaries, while the text runtime loads required non-routed tensors and streams the requested embedding row.The vision tower is not executed; packed-file size, text parameter size, and process footprint are distinct quantities.Source: §3.1 · p. 3

3.2 Demand execution and cache semantics

  • The positional range reader avoids mapping the full model, and its cache budgets expert-payload bytes with entries pinned during each layer’s evaluation.Activations, metadata, framework allocations, staging, attention state, and OS behavior add memory beyond that budget.Source: §3.2 · p. 3
  • A single forward requests 320 distinct routed experts and would read 540 MiB of expert payload if all requests miss; this is an over-forward working set, not a simultaneously pinned set.The replay grid has no oversized-expert or pinned-working-set bypasses.Source: §3.2 · p. 3
  • Routide uses serial single-token prefill and greedy generation, performing P + G - 1 forwards for P prepared prompt tokens and G output tokens.Batched prefill is not implemented, so TTFT characterizes this implementation rather than an optimized batched-prefill baseline.Source: §3.2 · p. 3

3.3 Prefetch as a separate, optional policy

  • The confidence-gated prefetcher uses the preceding step’s highest-weight same-layer expert at a normalized weight threshold of 0.20, during prefill only.Actual routing and arithmetic remain unchanged; resident-hit checks can affect recency, while the preserve-resident policy records probes without changing cache state.Source: §3.3 · pp. 3–4

4 Experimental methodology

The evaluation keeps evidence cohorts and measurement scopes separate, using fixed workloads and versioned protocols rather than treating every run as interchangeable. It reports scoped counters and sampled memory observations, retains environmental events, and limits claims about generality.

4.1 Evidence cohorts and scope

  • The evidence unit is a recorded protocol/build, with operating systems, runtime versions, instrumentation, prompts, and generation caps kept distinct rather than pooled.The designed prompts are small workloads, not representative production samples or scored quality tests.Source: §4.1 · p. 4
  • The 300 pinned replays and 30 relaxed bounds are correlated counterfactuals over five traces, not 330 independent model runs.The replay grid includes six budgets, six pin-constrained policies, and a relaxed oracle; random results use five declared seeds.Source: §4.1 · pp. 4–5

4.2 Counters, timestamps, and statistics

  • Logical expert-byte counts exclude embedding-row reads and other runtime I/O, and do not measure physical NAND traffic or filesystem-cache state.With no speculation, expertBytesRead equals demand misses times expert payload size; speculative loads are counted separately.Source: §4.2 · p. 5
  • Elapsed time, TTFT, and decode throughput retain their saved scopes, while paired timing results remain descriptive rather than population-level claims.The study keeps every pair, including an opposite-direction pair and one containing a thermal violation.Source: §4.2 · p. 5

4.3 Memory and environmental observation

  • Phone memory protocols sample resident size and physical footprint every 250 ms, retaining failed readings and the maximum successful-sample gap.These maxima are sampled observations, not continuous high-water marks; footprint, RSS, and MLX allocation counters are distinct metrics.Source: §4.3 · p. 5
  • The final memory protocol stops on any observed non-nominal state, warning, interruption, or validation failure, without automatically retrying or replacing rows.Thermal labels do not prove identical frequencies or temperatures, and manual locking, suspension, autobrightness, or thermal dimming remain possible.Source: §4.3 · p. 5

4.4 AI-assisted review and editing

  • AI tools assisted selected design reviews and manuscript cleanup, while humans retained research and experimental decisions and manually reviewed suggestions.The tools did not execute experiments, select runs, generate measured data, or determine conclusions.Source: §4.4 · p. 5

5 Results

Replay results show that cache outcomes depend on policy and budget, while same-runtime output checks do not establish cross-runtime sequence equivalence. Phone measurements retain a thermal stopping event and qualify the scope of memory and power observations.

5.1 The LRU capacity cliff is not a universal RAM threshold

  • At 512 MiB, seeded random eviction reaches 18.80% mean hits; LRU's 303 fitting payloads fall below the 313-entry reuse threshold.The threshold is an access-pattern condition, not a whole-process memory requirement.Source: §5.1 · p. 6

5.2 Prediction accuracy does not imply useful prefetch

  • Although the frozen prefill rule reaches 61.31% held-out selected-set predictions, the 576 MiB resident-refresh diagnostic loses 74 demand hits on resident checks alone.At 512 MiB, prefetch produced 10,334 loads—6,223 useful and 4,111 wasted—with 1.307% more logical reads than demand-only execution.Source: §5.2 · pp. 6–7

5.3 Cache transparency is not cross-runtime equivalence

  • Same-runtime controls preserve outputs across eviction and prefetch, including 2,560 exact token comparisons, but resident-Python and phone sequences differ on all five cases.Recorded-history inputs still yield 624/640 matching next-token predictions (97.5%), which is not free-generation accuracy.Source: §5.3 · p. 8

5.4 Sustained execution and timing variation

  • Identically configured candidate runs differ by 10.58% elapsed time, and the second nominal code pair is slower at 576 MiB despite lower logical reads.These records show sustained-generation feasibility on one workload; older MLX peaks are not process-footprint measurements.Source: §5.4 · pp. 8–9

5.5 Sampled process memory and a retained thermal stop

  • Sampled process-footprint peaks span 1.87–2.32 GiB on short prompts and reach 2.39 and 2.73 GiB on the two longer-prompt requests.The parent protocol stopped after request 12 reached fair final and peak thermal state; the two longer-context requests were a separately declared follow-up, not completion of the parent study.Source: §5.5 · pp. 9–10

5.6 A qualified whole-device power case study

  • One 512-token trace yielded a duration-weighted 26.505 battery-percent/hour rate and a 1.939-percentage-point source-window rate-time estimate, not an energy-saving result.The estimator was not independently calibrated, and there was no matched no-prefetch power control.Source: §5.6 · p. 10

6 Discussion and threats to validity

The discussion limits the evidence to bounded feasibility and identifies comparison, instrumentation, workload, and implementation boundaries. The results do not establish general efficiency, numerical equivalence, or superiority over smaller resident models.

6 Discussion and threats to validity

  • No completed matched study establishes superiority over smaller resident 4B/8B models, and feasibility alone does not show the best use of device resources.The implementation executes a large quantized checkpoint’s text path on a physical phone, but not a matched comparison of memory, storage, latency, or battery use.Source: §6 · p. 11
  • Serial prefill processes prompt tokens one at a time with full output-head evaluation, shaping TTFT, reuse, and expert traffic; batched prefill or another prompt mix could change cache ranking and latency.The 340-token prompt is longer than the short protocol prompts but is not “long context” of thousands of tokens.Source: §6 · p. 11
  • Process footprint is sampled, expert budgets exclude other allocations, and logical reads omit other runtime reads and physical flash behavior, so the study cannot infer NAND bandwidth or endurance.The samples are not a complete record of instantaneous allocations or system filesystem caches.Source: §6 · p. 11
  • Same-runtime cache transparency does not establish cross-framework or cross-device sequence equivalence, and prediction precision or conditional token agreement is not a task-quality metric.The study evaluates neither lossy routing modifications nor quality benefits; failures and scheduling races are also not comprehensively covered.Source: §6 · p. 11
  • Five replay traces, five additional Mac prompts, and one phone provide mechanisms and counterexamples, not population estimates or coverage of sampling, concurrent users, batching, all context lengths, or background conditions.Greedy generation improves repeatability, while multiple seeds and budgets reuse the same traces.Source: §6 · p. 11
  • Timings are not pooled across changing operating systems, compilers, dependencies, instrumentation, and export schemas; older exports cannot retrospectively supply missing provenance.The paper avoids using legacy output objects as independent token-equivalence evidence and relies on later raw same-runtime matrices and schema-12 captures for stronger comparisons.Source: §6 · p. 11
  • Full system reproduction requires the runtime, packing workflow, licensed model weights, and suitable Apple hardware, although the public runtime, analysis code, and paper-facing artifact are available.The profiling archives and licensed model weights remain outside the paper package.Source: §6 · p. 11

7 Conclusion

Routide demonstrates storage-backed execution of a pinned large MoE checkpoint on an iPhone, while its evidence supports a bounded characterization rather than a universal efficiency claim. The findings motivate careful comparison boundaries and independent reproduction.

7 Conclusion

  • Routide demonstrates storage-backed execution of a pinned large MoE checkpoint on an iPhone, but this does not establish a universal efficiency claim.The cache cliff is policy-dependent, and resident-prefetch checks can damage reuse without useful I/O.Source: §7 · pp. 11–12
  • Exact same-runtime cache transparency does not imply cross-runtime sequence equivalence, and process-memory measurements require their own scopes rather than relabeled allocator counters.The conclusion also preserves negative timing pairs, a thermal stop, and a separately declared successful continuation.Source: §7 · p. 12
  • The authors frame the findings as grounds for independent reproduction and careful comparisons, not repeated benchmarks selected for favorable timing.These distinctions are presented as testable through retained evidence and qualified claims.Source: §7 · p. 12

A Claim boundaries

The claim boundaries support text generation from one pinned flash-paged model on one iPhone, along with scoped measurements and specific cache and output comparisons. They do not establish universal feasibility, general numerical equivalence, energy savings, or task-quality parity.

A Claim boundaries

  • The demonstrated outcome is text generation from the pinned flash-paged model on one iPhone, not universal device/model feasibility or state-of-the-art speed.The boundary table distinguishes supported statements from stronger claims not established.Source: A Claim boundaries · p. 12
  • Cache hits are policy-dependent on five recorded traces; the evidence does not establish an unavoidable RAM threshold or globally optimal 576 MiB budget.The supported output claim is exact tested outputs under same-Swift-implementation cache changes.Source: A Claim boundaries · p. 12
  • The study supports a fixed predictor’s selected-set overlap and read accounting, not general cross-framework or cross-device numerical equivalence.The claim boundary separates these scoped measurements from broader equivalence claims.Source: A Claim boundaries · p. 12
  • The evidence does not establish general latency hiding or energy savings.The supported power result is one whole-device power-estimator integral, not calibrated joules per token or app-only energy benefit.Source: A Claim boundaries · p. 12
  • Memory evidence is sampled process footprint on separately identified protocols, not continuous device-wide maxima or context-length scaling.A separate successful longer-prompt pair follows a retained stop.Source: A Claim boundaries · p. 12
  • The completed observations are not a nominal 14-request parent campaign, and structured prompts with captured text do not establish scored task-quality parity or superiority to smaller resident models.The parent study stopped after its twelfth request; two follow-up rows belong to a separately declared protocol and different executable.Source: A Claim boundaries · p. 12
  • These boundaries define the claims supported by the evidence; stronger claims require additional experiments or independent reproduction.The claim-boundary table identifies supported statements alongside stronger statements not established.Source: A Claim boundaries · p. 12

B Per-request process-memory observations

The appendix retains every completed per-request observation and distinguishes the stopped parent study from its follow-up rows. Its footprint and RSS values are separate sampled maxima, not additive memory components.

B Per-request process-memory observations

  • Every completed observation is retained, while the parent study stopped after its twelfth request and two follow-up rows belong to a separately declared protocol and different executable.The appendix labels footprint and RSS columns as separate sampled maxima.Source: B Per-request process-memory… · p. 13
  • Footprint and RSS columns are separate sampled maxima, not additive memory components.They should not be summed as components of total memory.Source: B Per-request process-memory… · p. 13
  • The table identifies Peak RSS in GiB and Peak thermal as distinct reported fields.Source: B Per-request process-memory… · p. 13
Loading 2609.29032v1…