PaperlayerView exact revision on arXiv

Source-linked AI summary

DPA-MoE: A Runtime Precision Control Framework for MoE Inference

Jaeyun Kim, Hoonki Lee, Susie Shin, Hyunmin Kim, Dahyun Lee, Sangyeob Kim

arXiv:2610.21378v1cs.CL

TL;DR

MoE routes each token to a small set of experts, but selected experts are used at fixed precision. DPA-MoE encodes expert weights as prefix-decodable chunks and allocates chunk counts using offline residual profiles and live router scores. Across four MoE models, it improves reconstruction fidelity and the weight-size–quality frontier over uniform chunk allocation, with modest prototype throughput overhead.

  • Problem

    MoE routing selects experts but leaves each selected expert at fixed precision, limiting the weight-size–quality operating points available from a checkpoint.

    Source: §1 · pp. 1–2

  • Method

    DPA-MoE uses prefix-decodable expert chunks and combines offline reconstruction residuals with live router gate scores to allocate precision at inference time.

    Source: §1 · p. 2

  • Results

    Across four MoE models, DPA-MoE improves reconstruction fidelity and the overall weight-size–quality frontier over uniform chunk allocation, with modest steady-state throughput overhead in the prototype.

    Source: §8 · p. 9

  • Takeaways & Limitations

    A single encoded MoE checkpoint can expose a fine-grained weight-size–quality frontier at runtime, making expert precision an inference-time allocation variable.

    Source: §8 · p. 9

    The allocation objective is a surrogate for downstream quality, using imperfect gate scores and residual estimates while ignoring cross-expert residual interactions.

    Source: §9 · p. 10

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models reduce computation by routing each token to a small set of experts, but each selected expert is still used at a fixed precision. We present DPA-MoE, a runtime precision control framework that makes routed experts any-precision. DPA-MoE encodes expert feed-forward weights as prefix-decodable chunks using SharedScale AnyBCQ, profiles per-expert reconstruction residuals offline, and allocates chunks at inference time using live router gate scores. This post-routing allocation lets a single encoded MoE checkpoint operate at multiple effective weight budgets without retraining or re-encoding. Experiments on Qwen3-30B-A3B and DeepSeekMoE-16B show that DPA-MoE improves KL divergence and downstream accuracy over uniform chunk allocation at matched budgets, with modest steady-state throughput overhead in our prototype.

1 Introduction

Diverse workloads and shifting resource constraints create a need for multiple operating points, but fixed-precision checkpoints expose only one. DPA-MoE makes selected experts any-precision and allocates their precision at runtime, improving the weight-size–quality frontier over uniform chunk allocation with modest prototype throughput overhead.

1 Introduction

  • An any-precision checkpoint can provide multiple operating points for workloads with differing latency, throughput, memory, and quality requirements; separate quantized checkpoints increase storage and deployment complexity.For local inference, memory constraints and users’ speed-versus-accuracy preferences can shift while the model remains resident.Source: §1 · p. 1
  • MoE routing selects experts but evaluates each selected expert at fixed precision, leaving the number of weight bits read inside those experts uncontrolled.Expert FFN weights account for about 95% of Qwen3-30B-A3B parameters.Source: §1 · pp. 1–2
  • DPA-MoE assigns variable numbers of precision chunks to selected experts, tracing a fine-grained weight-size–quality frontier without retraining or maintaining multiple quantized variants.Precision k is the count of prefix chunks used; larger counts give higher-fidelity reconstructions.Source: §1 · p. 2
  • DPA-MoE combines offline expert residual profiles with live router gate scores to allocate precision across routed expert activations.The pretrained router and expert identities remain unchanged; the framework controls precision for each selected expert.Source: §1 · p. 2
  • DPA-MoE consistently reduces KL divergence versus uniform chunk allocation and improves the overall weight-size–quality frontier, with modest steady-state throughput overhead in the prototype.The reported model set includes Qwen3-30B-A3B, DeepSeekMoE-16B-Chat, OLMoE-1B-7B-0125, and Qwen1.5-MoE-A2.7B.Source: §1 · p. 2
  • SharedScale AnyBCQ provides a prefix-decodable expert encoding that exposes multiple effective weight-memory budgets from one checkpoint without retraining or re-encoding.Source: §1 · p. 2

2 Related Work

Prior approaches address MoE memory and quality through static quantization, input-adaptive precision, or progressive encodings. DPA-MoE is positioned as training-free, token-aware, and fine-grained in chunk count while leaving the router unchanged.

Static Quantization

  • Static MoE mixed-precision methods assign expert precision before deployment, so their allocations remain fixed regardless of the current input.The section contrasts this with DPA-MoE’s runtime allocation.Source: §2 · p. 2

Input-Adaptive Precision Methods

  • Cache-driven and long-horizon methods adapt precision using cache state or aggregate router traces, while Wang et al. (2025a) use a trained per-expert bit-width gate.Wang et al. (2025a) require an additional adapter and bind precision to a learned policy.Source: §2 · pp. 2–3

Progressive and Multi-Precision Encodings

  • Progressive encodings make multiple precisions accessible without re-encoding, but do not specify how precision should be allocated across experts.The section cites Yu et al. (2021), Park et al. (2024), Wang et al. (2025b), and Park et al. (2026) as examples.Source: §2 · p. 3

Positioning

  • D2MoE is the closest comparison, but exposes discrete tiers and uses a trained per-expert network to select bit-widths; DPA-MoE is presented as training-free, token-aware, fine-grained in k, and router-preserving.The authors qualify the uniqueness claim with “to our knowledge.”Source: §2 · p. 3

3 Background and Problem Formulation

MoE routing combines outputs from selected experts, while this formulation allows each selected expert to use a different prefix length. The problem statement targets uniform precision inside routed experts, not changes to the router, model training, or serving scheduler.

3.1 Precision Chunks

  • A chunk is one binary bitplane with its per-group scale, while a per-group offset is shared across chunks.Source: §3.1 · p. 3
  • Using the first k chunks forms a valid prefix, so increasing k preserves the already-read portion of the encoded weight.Source: §3.1 · p. 3
  • The supported chunk counts range from kmin to kmax, and chunk count is the granularity for spending precision on an expert.Source: §3.1 · p. 3

3.2 MoE Routing

  • For a token state x, the MoE layer selects experts E(x) and combines their outputs using router gate scores.Source: §3.2 · p. 3
  • Larger chunk counts use more encoded expert weight and generally produce a more faithful reconstruction.Source: §3.2 · p. 3

3.3 Problem Statement

  • Uniform precision gives every selected expert the same chunk count despite differences in gate scores and reconstruction error across experts and layers.Source: §3.3 · p. 3
  • Uniform allocation can spend memory and bandwidth on low-contribution or easy-to-reconstruct experts while underserving activations where added precision matters more.Source: §3.3 · p. 3
  • The stated objective is to remove uniform precision within experts selected by the existing router, exposing multiple operating points from one encoded checkpoint.Source: §3.3 · p. 3

4 Method

DPA-MoE encodes expert weights as prefix-decodable chunks, then allocates chunk counts across active experts using offline residual profiles and live router gates. A dynamic-programming allocator chooses assignments under a fixed chunk budget.

4.1 Progressive Expert Encoding

  • SharedScale AnyBCQ jointly optimizes active scales during progressive encoding, so inference can load a longer chunk prefix without refetching previously loaded scales.The default supported range is k=2 to 8 chunks.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2 Two-Stage Framework

  • Offline profiling measures each expert’s reconstruction error at each precision and stores it in a residual lookup table; it is gate-free and computed once.At inference, the allocator combines that table with live gate scores for the active experts.Source: §4.2 · pp. 4–5
  • The approximation changes the objective’s absolute magnitude but preserves allocation rankings with Spearman correlation of at least 0.999; residuals are monotone-projected across k to remove calibration noise..Source: §4.2 · p. 5

4.3 Dynamic Precision Allocator (DPA)

  • DPA formulates chunk assignment as a multi-choice knapsack, minimizing gate-weighted residual costs across active experts under a total chunk budget.The budget is B = round(|A|·¯k), and each expert’s chunk count is constrained to the supported range.Source: §4.3 · p. 5
  • Dynamic programming and backtracking recover the global optimum for the projected residual table under the budget.The method runs in O(n · B · krange) time; for Qwen3-30B-A3B decoding, it evaluates at most 3,584 candidate transitions per dispatch.Source: §4.3 · p. 6

5 Experimental Setup

The evaluation covers four open-weight MoE models, compares DPA with static quantization and uniform chunk allocation, and measures reconstruction quality, downstream accuracy, and inference performance. Speed tests use a controlled full-memory setting and matched average chunk budgets for DPA and Uniform-k.

5.1 Models

  • The evaluation uses four MoE models spanning different scales, routing widths, and shared-expert configurations; the first two receive the full suite including inference-speed tests.The other two are evaluated on KL divergence and downstream accuracy.Source: §5.1 · p. 6

5.2 Baselines

  • Baselines include Q2_K, Q5_K_M, and Q8_0 static quantizations, plus Uniform-k, which uses the same encoding as DPA but assigns one fixed chunk count to every activated expert.Uniform-k is evaluated for k values from 2 to 8.Source: §5.2 · p. 6

5.3 Evaluation

  • Reconstruction quality is measured by mean KL divergence against full-precision output distributions, and downstream accuracy is reported on ARC-Challenge, ARC-Easy, WinoGrande, and PIQA.The paper also reports macro-average accuracy across the four tasks.Source: §5.3 · p. 6

5.4 Inference Evaluation Setup

  • Throughput and latency are measured in a controlled full-memory setting that isolates precision allocation and variable-chunk execution from storage and offloading effects.Generation throughput and time-to-first-token are averaged over 100 sequences from each of WikiText-2, C4-en, and Korean mC4.Source: §5.4 · p. 6
  • DPA and Uniform-k are compared at matched average chunk budgets, isolating runtime and quality effects of non-uniform allocation at the same effective weight-memory budget.Figures 4 and 5 plot mean KL divergence and macro-average accuracy, respectively, against effective model weight for two models.Source: §5.4 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

6 Results

DPA improves reconstruction quality and downstream accuracy over uniform chunk allocation at matched budgets across evaluated MoE models. Its runtime allocation has a throughput and TTFT cost at matched chunk counts, while signal ablations show that gate scores provide most of the gain and residual profiles add further improvement.

6.1 Reconstruction Quality

  • 4.5× lower KLD than Uniform k at matched size is the maximum reported gain on DeepSeekMoE-16B-Chat at k̄=4; DPA’s curve stays below Uniform k across both models.DPA also recovers equivalent fidelity with 2–4 GB less memory, and non-integer operating points interpolate between integer anchors.Source: §6.1 · p. 7

6.2 Downstream Task Performance

  • DPA consistently outperforms Uniform k in macro-average accuracy at matched memory budgets on both evaluated models, with the largest gains at aggressive compression.The accuracy improvements mirror the KLD reductions.Source: §6.2 · p. 7

6.3 Generality across MoE Architectures

  • At k̄=3 on OLMoE, macro-average accuracy rises from 60.60% to 62.40%; at k̄=4 on Qwen1.5-MoE, it rises from 62.04% to 63.45%.Across OLMoE and Qwen1.5-MoE, DPA reduces KLD by 26–69% wherever allocation is not forced; methods coincide at k̄=2 and k̄=8.Source: §6.3 · pp. 7–8

6.4 Inference Speed

  • At matched k̄, DPA incurs 4–6% generation-throughput overhead on Qwen3-30B-A3B and 6–8% on DeepSeekMoE-16B-Chat, with TTFT increases of approximately 30–50%.At comparable quality, DPA at k̄=4 reaches 67.90% accuracy at 48.72 TPS on Qwen3, nearly matching Uniform k=5 at 67.86% and 46.29 TPS.Source: §6.4 · p. 8, Table 1 · p. 8

6.5 Which Signal Drives the Gain?

  • DPA attains the lowest KLD at every budget; Gate-only accounts for most of the gain over Uniform-k, and combining gate scores with residuals further improves on Gate-only by 4.6–5.4%.The residual-only variant contributes little on its own; the ablation stands in for comparison with published methods because D2MoE has no public implementation.Source: §6.5 · p. 8, Table 2 · p. 8

7 Discussion

DPA addresses a mismatch between uniform chunk allocation and the differing importance and quantization sensitivity of routed experts. Its practical fit is memory-constrained, small-to-moderate-batch inference, with allocation overhead and system integration still relevant boundaries.

7 Discussion

  • Routed experts differ in current-token importance and reconstruction sensitivity, yet Uniform-k assigns each the same chunk count.DPA combines gate scores and residual profiles to spend a fixed chunk budget more effectively without modifying the pretrained router or training an additional policy.Source: §7 · pp. 8–9
  • At matched k̄, DPA is not a free speedup: per-dispatch allocation and non-uniform materialization lower steady-state TPS and raise TTFT in the prototype.At comparable quality, DPA can use fewer chunks than Uniform-k and recover part of that overhead.Source: §7 · p. 9
  • DPA suits memory-constrained, small-to-moderate batches; as batch size grows, allocator cost grows while its benefit shrinks.The deployment should fall back to Uniform-k beyond the device-dependent batch-size threshold where allocation stops paying for itself.Source: §7 · p. 9
  • SharedScale AnyBCQ is one possible prefix-stable encoding substrate, and allocation can extend to other progressive encodings with allocatable chunks and measurable residual profiles.Optimized runtime integration and downstream system composition remain future work.Source: §7 · p. 9

8 Conclusion

DPA-MoE adds post-routing precision allocation to MoE inference, using a single encoded checkpoint to expose multiple effective weight budgets. Across four MoE models, it improves reconstruction fidelity and the weight-size–quality frontier over uniform allocation, with modest steady-state throughput overhead in the current prototype.

8 Conclusion

  • DPA-MoE adds a post-routing step that assigns precision chunks to experts selected by the pretrained router under a fixed chunk budget.The router and expert identities remain unchanged.Source: §8 · p. 9
  • SharedScale AnyBCQ exposes multiple effective weight-memory budgets from one MoE checkpoint without retraining or re-encoding, while offline residual profiles and live router gate scores guide chunk allocation.The allocation follows gate-weighted reconstruction costs.Source: §8 · p. 9
  • Across four MoE models, DPA consistently improves reconstruction fidelity over uniform chunk allocation and yields a better overall weight-size–quality frontier.Runtime measurements show modest steady-state throughput overhead in the current prototype.Source: §8 · p. 9
  • DPA-MoE treats expert precision as a runtime allocation variable rather than a fixed property of a deployed checkpoint.A single encoded checkpoint can trace a fine-grained weight-size–quality frontier at runtime.Source: §8 · p. 9

9 Limitations

The stated limitations concern the gap between weight-size and serving latency, prototype-specific runtime costs, surrogate-based allocation, and the scope of the empirical evaluation.

9 Limitations

  • Effective weight size is only a proxy for latency; kernel fusion, batching, dispatch overhead, and hardware-specific scheduling are not modeled.The prototype does not fully fuse allocation and variable chunk materialization with expert dispatch kernels and uses no custom scheduler, so its observed overheads are prototype costs.Source: §9 · pp. 9–10
  • The allocation objective uses gate scores and offline residuals as proxies, omits cross-expert residual interactions, and is optimal for a projected surrogate rather than necessarily true downstream loss.Residual estimates come from finite calibration data and may contain small precision inversions, addressed by monotone projection.Source: §9 · p. 10
  • Evaluation covers four open-weight MoE models and focused benchmarks, not all architectures, routing designs, hardware backends, or task domains.DPA-MoE also requires router gate scores, which may be unavailable in closed-weight deployments.Source: §9 · p. 10

Ethical considerations

The paper reports using public artifacts without collecting human-subject data, and compares storage overhead for two encodings supporting the full 2–8-bit range.

Ethical considerations

  • The ethical considerations report public model and dataset artifacts, no newly collected human-subject data or redistributed weights, and aggregate-only reporting for web-derived text.Web-derived calibration and speed-prompt text may contain offensive content or personally identifying strings inherited from source corpora.Source: Ethical considerations · p. 10
  • AI assistants supported coding, documentation, and editorial checks; the authors reviewed the resulting work and accept responsibility for its correctness and originality.Source: Ethical considerations · p. 10

A Storage Overhead: AnyBCQ vs. SharedScale AnyBCQ

  • For full 2–8-bit support, AnyBCQ stores 35 scale sets versus 8 for SharedScale AnyBCQ, whose scale set is shared across tiers.For one Qwen3-30B-A3B expert, the reported scale overhead is 4.4× larger with AnyBCQ, increasing total per-expert storage by 37.5%; SharedScale requires only one new bitplane and scale set per upward transition.Source: A Storage Overhead: AnyBCQ vs.… · p. 12

B Cross-Term Analysis

The section reports that diagonal costs closely preserve allocation rankings above the minimum precision, while DPA improves per-task accuracy over Uniform k and differs from other dynamic precision methods.

B Cross-Term Analysis

  • For k≥3, cross terms contribute at most 27% of true error, while exact and diagonal objectives have Spearman correlation above 0.999 and Pearson above 0.95.At k=2 cross terms contribute 59.8%, but every expert defaults to this minimum chunk count, so no allocation decision is made there; rank preservation supports using the separable diagonal objective for knapsack allocation.Source: B Cross-Term Analysis · pp. 12–13, Table 4 · p. 12

C Per-Task Benchmark Results

  • DPA consistently outperforms Uniform k at matched memory budgets across four tasks on Qwen3-30B-A3B and DeepSeekMoE-16B-Chat, with largest gains under aggressive compression.Figures 6 and 7 show the per-task results for the respective models.Source: C Per-Task Benchmark Results · p. 13

D Comparison with Dynamic Expert-Precision Methods

  • DPA-MoE is described as training-free, token-aware, fine-grained in k, and router-preserving; D2MoE shares prefix-reuse but offers discrete tiers and requires trained per-expert plug-ins.MxMoE is fixed before deployment, HOBBIT is gated by cache misses, and DynaExq uses long-horizon expert hotness rather than current-token importance.Source: D Comparison with Dynamic… · p. 13
  • The paper reports no head-to-head empirical comparison against the other dynamic or mixed expert-precision methods.Its main comparison is against Uniform-k, with a signal ablation used instead of a cross-method baseline.Source: D Comparison with Dynamic… · p. 13

E Additional MoE Architectures

Additional experiments extend the budget sweep to two MoE architectures, test DPA with dynamic routing, and document the experiment settings and artifacts.

E Additional MoE Architectures

  • Across the interior budget sweep on OLMoE and Qwen1.5-MoE, DPA lowers mean KLD and improves accuracy at most matched budgets, with one accuracy exception at k̄=6 on OLMoE.The largest reported accuracy gains are +1.80 points at k̄=3 on OLMoE and +1.41 points at k̄=4 on Qwen1.5-MoE; at k̄=6 on OLMoE, Uniform-k is ahead by 0.31 points despite DPA's lower KLD.Source: E Additional MoE Architectures · p. 13, Table 6 · p. 15

F Composing with Dynamic Top-k Routing

  • With dynamic top-p routing at p=0.25 and 4.0 active experts per token on average, DPA improves PIQA accuracy by +0.37 percentage points on average across the budget sweep.The gain is not uniform: it is largest at k̄=6 and k̄=7 and slightly negative at k̄=5.Source: F Composing with Dynamic Top-k Routing · p. 15, Table 7 · p. 15
  • At the closest reported memory-read points, DPA at k̄=4 scores 3.22 points higher than top-p routing while reading 8.8% more bytes per token.The comparison is 465.6 MB/token for DPA against 427.8 MB/token for top-p activating 4.0 experts; the reported ordering holds across the swept compression range.Source: Table 8 · p. 15, F Composing with Dynamic Top-k Routing · p. 16
  • The paper describes precision and active expert count as two composable adjustment axes, yielding a two-dimensional operating range from one artifact.Under fixed top-8 routing, DPA varies k from kmin=2 to K=8; dynamic routing adds adjustment of |A|.Source: F Composing with Dynamic Top-k Routing · p. 16

G Detailed Experiment Configuration

H Artifact, License, and Reproducibility Notes

  • The supplementary package contains source code, scripts, and configuration files, but not model weights, datasets, or generated outputs; reproduction requires obtaining external artifacts from their original sources.The appendix identifies public source artifacts and their intended uses.Source: H Artifact, License, and… · p. 16, Table 13 · p. 18, Table 14 · p. 18

Related papers

Loading 2610.21378v1…