COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

Motivation

• Model sparsity, offloading mechanisms, and heterogeneous hardware make policy performance configuration-dependent, while implementing every candidate incurs substantial engineering cost.CPU and GPU throughput, interconnect bandwidth, synchronization overhead, and model sparsity can change which policy performs best.Source: §1 · p. 1

Approach

• The Policy Matrix separates expert residency from MoE execution strategy, allowing mechanisms to be reused and exposing four candidate policies absent from existing systems.Residency controls where experts are stored; execution controls where and how routed experts are computed.Source: §1 · pp. 1–2
• VHM profiles target-hardware operators and combined schedules, then replays offline routed and predicted expert IDs to estimate candidate TPOT.The routing traces are timing-free, and the profile captures interactions among compute, data transfer, synchronization, and asynchronous dispatch.Source: §1 · p. 2

Evaluation

• COPE evaluates nine candidates on three CPU–GPU platforms and 11 additional controlled A100 resource configurations, comparing VHM predictions with end-to-end measurements.The measured implementations provide ground truth for policy selection and prediction error.Source: §1 · p. 2

Results

• COPE selects the measured minimum-TPOT candidate in 134/150 A100, 20/20 RTX 4090, and 42/45 RTX 5090 cases.For Qwen-3 on RTX 4090, implementing only the selected Fixed Parallel policy reduces policy-specific code by 74% versus implementing all candidates.Source: §1 · p. 2

2.1 MoE Inference with Offloading

• In partial-fit deployments, resident experts run on the GPU while cache misses can execute on the CPU or move weights to the GPU, with overlap and hardware costs shaping TPOT.The study focuses on batch-1 decode where weights fit in host memory but not entirely in VRAM.Source: §2.1 · p. 2

2.2 Existing Offloading Policies

• Existing offloading approaches vary in execution, placement, and prefetching, while exact-routing methods preserve the router’s choices and stored precision.Popularity-based placement and future-expert prediction can change cached weights without changing which experts the router eventually selects.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 A Growing Space of Models and Configurations

• Models and platforms vary in expert counts, routing, expert size, compute, memory, and bandwidth, so rankings cannot be extrapolated between configurations.The practical question is which feasible policy template to integrate for a particular model–platform without first implementing every candidate.Source: §2.3 · p. 3

2.4 LLM simulators

• Existing simulators address broader serving or training questions, but their evaluations do not compare multiple MoE offloading policies for memory-constrained edge devices within a common model.COPE targets this narrower policy-level prediction problem.Source: §2.4 · p. 3

3.1 Observation I: Model–Platform Dependent Policy Performance

• Resource changes can reorder candidate policies even for one model and physical GPU, and simple Fixed Parallel can beat more adaptive candidates.On the controlled A100 study, layer partition was only 12.2–25.0% slower than Fixed Parallel in the two 16-worker configurations.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Observation II: Composable and Executable Policy Space Modeling

• Surveyed offloading designs factor into expert residency and MoE execution strategy, whose combinations define candidates within the stated policy space.The matrix preserves router decisions, executes every selected expert, and retains original precision; pruning, substitution, and lower-precision miss execution fall outside its scope.Source: §3.2 · p. 4

4 Design

• COPE represents a policy as a combination of expert residency and MoE execution choices, then estimates candidate TPOT without running each candidate end to end.VHM uses target-platform operator and short-schedule profiles together with offline routing traces to replay cache states.Source: §4 · p. 4, Figure 3 · p. 5

4.1 Modularizing Offloading Policies with the Policy Matrix

• The Policy Matrix crosses residency families with execution strategies, yielding eight instantiated cache policies plus a separately evaluated whole-layer partition candidate.Fixed residency retains its initial experts; Mutable per-layer and Mutable cross-layer permit replacement at different scopes, while execution choices coordinate CPU work and GPU staging.Source: §4.1 · p. 5, Figure 4 · p. 6, Figure 5 · p. 6

4.2 Predicting Offloading Performance with the Virtual Heterogeneous Machine

• VHM profiles reusable atomic operators and combined schedules because isolated costs alone miss overlap, serialization, contention, and dispatch effects.Measured operator costs can differ from analytic HRM estimates; for six staged experts, measured cost is 2.4× the analytic estimate.Source: §4.2.1 · pp. 6–7, §4.2 · p. 6, Figure 6 · p. 7

4.2.1 Abstracting Hardware with Atomic-Operator

• VHM replays routed cache hits and misses, uses predicted next-layer IDs for mutable-cache prefetching, and sums execution, prefetch, and shared decode costs into TPOT.The replay assumes selected prefetches complete before the layer is used, and its profile is reusable only while the target operating point remains valid.Source: §4.2.2 · p. 7

5 Implementation

• Nine production paths in llama.cpp validate VHM’s candidate TPOT estimates against end-to-end measurements on RTX 4090, RTX 5090, and A100.The production paths serve only as validation and do not contribute to VHM construction.Source: §5 · pp. 7–8
• Fixed and mutable residency start with the same per-layer popularity order and cache capacity, but only mutable residency updates the resident set through prediction and prefetch.Execution strategies differ in how they handle cache misses; the eight cache candidates share a bounded pinned buffer.Source: §5 · p. 8

5.1 Expert Routing Trace Collection

• A routing-trace hook records each layer’s routed experts and next-layer predictions, which respectively determine replay cache hits and possible updates to the next layer’s cache.The next-layer predictor reads the current layer’s input hidden state.Source: §5.1 · p. 8

6.1 Methodology

• The evaluation compares end-to-end measurements with atomic-operator and HRM estimators, using shared routing traces and replay states; COPE profiles combined-operator schedules.A100 testing covers three models across ten resource configurations; physical RTX testing covers RTX 4090 and RTX 5090.Source: §6.1 · p. 8, §6 · p. 8, §6.3 · p. 9

6.2 Optimal Policy Diversity on A100

• Four candidates win across 15 A100 model-resource cells, and DeepSeek’s winner changes from Mutable ready-first at one CPU thread to Fixed Parallel at 16 threads.Qwen-3.5’s 16-thread winner improves on layer partition by only 1.02×.Source: §6.2 · p. 9, Table 2 · p. 9
• For DeepSeek-Coder-V2-Lite, changing A100 resources alone reorders candidates: mutable policies win at low worker counts, while Fixed Parallel wins at 16 workers.In the latter configurations, layer partition is 12.2–25.0% slower; wait-then-GPU and ready-first are 163–364% slower.Source: §6.2 · p. 9

6.3 Optimal Policy Prediction on A100

• On A100 sweeps, COPE selects 45/50 DS-Coder and 41/50 Qwen-3 measured winners; for Qwen-3.5 it selects 48/50 versus 36/50 for the atomic baseline.For Qwen-3.5, COPE reduces MAPE from 15.12% to 8.71% and raises mean Spearman correlation from 0.884 to 0.944.Source: Table 3 · p. 10, §6.3 · p. 10, Figure 8 · p. 10

6.4 Optimal Policy Prediction on RTX 4090 and RTX

• On RTX 4090, COPE selects all 20 measured winners and recovers 59/60 top-three slots, while the atomic baseline selects none and recovers 24/60.COPE’s MAPE is 9.39–17.43%, versus 31.73–40.62% for the baseline.Source: §6.4 · p. 10, Figure 9 · p. 10, Figure 10 · p. 10
• On RTX 5090, COPE selects all measured winners for Coder and Qwen3, and 12/15 for Chat; for Qwen3, the baseline ranks candidates better while COPE has lower absolute TPOT error.The study covers three VRAM reservations per model; Table 5 reports selection, ranking, and error.Source: Table 5 · p. 11, §6.4 · p. 11, Figure 11 · p. 11

6.5 Engineering Effort and Calibration Cost

• For Qwen3 on RTX 4090, implementing only the selected Fixed Parallel policy requires 3,107 policy-specific lines, versus 8,276 for all options, a 74% reduction.The comparison excludes shared plumbing and prediction/diagnostic code; Figure 12 contrasts the policy-specific implementations.Source: Figure 12 · p. 11, §6.5 · p. 12
• On RTX 4090, atomic-operator profiling takes 2.8–3.5 minutes across four models; exhaustive combined-operator schedule profiling takes 25.9–43.5 minutes, while combined-operator time is projected at 25% of exhaustive time when profiling only replay-reachable states.Source: Table 6 · p. 12, §6.5 · p. 12

Expert caches

• COPE organizes existing demand-loading, prediction, and overlap mechanisms as residency/execution candidates and predicts which complete template to integrate without production-candidate latency.The cited systems include MoE-Infinity, Mixtral-Offloading, ProMoE, DuoServe-MoE, and FreeToken [13] [47] [54] [57].Source: §7 · p. 12

Adjacent mechanisms

• MoE-Lightning’s batch-paged weights, vLLM UVA remote reads, and execution-order module staging do not share the Policy Matrix’s candidate semantics.The passage identifies vLLM Prefetch and SGLang OffloaderV2 as staging modules in execution order.Source: §7 · p. 12

Other representation and routing choices

• COPE fixes whole-expert representation and exact routing, excluding finer-grained or compressed weights and routing changes from its policy comparisons.FineMoE and ZipMoE depart from whole-expert representation; approximate routing changes selected computation.Source: §7 · p. 12

TPOT rather than TTFT

• COPE optimizes decode TPOT rather than TTFT, and its batch-1 results do not predict prefill performance.Longer prefills can shift the CPU/GPU crossover toward staging weights for GPU execution.Source: §8 · p. 12

Batch n versus batch 1

• Batch-n serving has a different CPU/GPU crossover and may rank policies differently from single-user batch-1 decode, requiring batch-size-specific probes.Multiple tokens can share an expert.Source: §8 · p. 12

Unmodified computation versus accuracy-for-speed offloading

• COPE compares unchanged-checkpoint computation: it preserves exact routing, executes every selected expert, and retains stored precision.Routing changes, quantization, and skipping or modifying experts have accuracy-performance trade-offs not directly comparable with TPOT for an unchanged checkpoint.Source: §8 · p. 12

Disk versus host-memory offloading

• COPE assumes the model fits in host RAM; disk offloading requires additional templates and probes.On a shared-memory AI SoC, disk becomes a capacity tier analogous to host RAM below discrete-GPU HBM.Source: §8 · p. 12

• MoE offloading performance depends jointly on expert residency and execution strategy, so policy selection is specific to each model–platform configuration.Source: §9 · p. 12
• COPE uses a Policy Matrix and a Virtual Heterogeneous Machine to evaluate candidate policies before integration.Source: §9 · p. 12
• COPE replaces implementation-intensive trial and error with systematic pre-integration analysis, lowering the engineering barrier to deployment on heterogeneous, resource-constrained hardware.Source: §9 · p. 12