PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

Congwei Zhang, Youming Tao, Yifei Zou, Yu Yang, Ruirui Zhang, Xiaoyu Zhang, Falko Dressler, Xiuzhen Cheng, Dongxiao Yu

arXiv:2610.37442v1cs.LG

TL;DR

Edge MoE inference faces tight resource budgets and expert-weight movement costs. CRISP co-designs combination-aware pruning with runtime scheduling, improving throughput while limiting accuracy degradation at the same memory budget.

  • Problem

    Edge MoE inference must operate within limited memory, computation, and communication budgets, while designing combination-aware pruning requires balancing accuracy, storage cost, and scheduling complexity.

    Source: §1 · pp. 1, 3

  • Method

    CRISP co-designs expert representation, combination-aware pruning, and runtime scheduling, using differentiated widths to create heterogeneity the scheduler exploits to overlap loading with computation.

    Source: §1 · p. 3

  • Results

    Up to 2.1× throughput improvement at the same memory budget, with accuracy degradation limited to 3.6% relative to full-width execution across Mixtral-8x7B and DeepSeek-MoE-16B.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Efficient edge deployment of sparse models requires jointly reasoning about model structure, access patterns, and execution behavior.

    Source: §7 · p. 13

    CRISP’s calibration-derived rankings, combination frequencies, and expert roles depend on model parameters and deployment workload, so substantial changes require reassessing offline artifacts.

    Source: §3.1 · pp. 5–6

Abstract

from arXiv · show

Mixture-of-Experts (MoE) inference on memory-constrained edge platforms is often dominated by expert-weight movement rather than computation. Although each token activates only a small subset of experts, routing is input-dependent, so all experts must remain accessible and memory movement dominates latency. Existing approaches such as uniform pruning and expert dropping overlook two properties of MoE execution: expert combinations are highly skewed in frequency, and experts contribute asymmetrically within each combination. We present CRISP, a combination-aware framework that co-designs pruning and runtime scheduling for efficient MoE serving on memory-constrained edge platforms. CRISP reorders neurons by importance so one stored expert can support multiple execution widths, assigns differentiated execution to hot, warm, and cold combinations based on frequency and expert role, and exploits heterogeneous widths to overlap expert loading with computation. Across Mixtral-8x7B and DeepSeek-MoE-16B, CRISP improves throughput by up to $2.1\times$ at the same memory budget while limiting accuracy loss to 3.6\% relative to full-width inference. Our results highlight that efficient edge MoE serving requires jointly optimizing which expert capacity is preserved and how expert execution is scheduled.

1 Introduction

CRISP addresses edge MoE inference costs by co-designing combination-aware pruning and runtime scheduling. Its differentiated expert widths enable scheduling that overlaps loading with computation.

1 Introduction

  • Under 50% uniform pruning, frequently activated combinations suffer 2.3× greater accuracy degradation than rare ones, while expert dropping misses combination-dependent expert roles.Figure 1 contrasts frequently and rarely activated pairs; Figure 2 illustrates role variation across combinations.Source: §1 · p. 2, Figure 1 · p. 2, Figure 2 · p. 2
  • Importance-aware neuron reordering lets one stored expert support multiple execution widths through prefix truncation, avoiding duplicated weight copies.CRISP ranks neurons on a calibration set and places more important neurons first.Source: §1 · p. 3
  • CRISP assigns full width to both experts in hot combinations, reduces auxiliary width in warm combinations, and skips auxiliary experts in cold combinations.The tiers are based on combination frequency and expert role.Source: §1 · p. 3
  • The pruning-aware scheduler prioritizes wider experts so their longer computation windows can overlap loading narrower experts and reduce exposed I/O latency.Differentiated widths create the heterogeneous execution times that the scheduler exploits.Source: §1 · p. 3
  • MoE combinations have skewed activation frequencies and co-activated experts contribute asymmetrically, motivating pruning by both combination frequency and expert role.Source: §1 · p. 3
  • On Mixtral-8x7B and DeepSeek-MoE-16B, CRISP improves throughput by up to 2.1× at the same memory budget while limiting accuracy degradation to 3.6% relative to full-width execution.Source: §1 · p. 3

2 Empirical Observations

On Mixtral-8x7B, the empirical studies identify skewed combination usage, unequal expert contributions within combinations, and width-dependent I/O and computation times as opportunities for differentiated pruning and scheduling. The section motivates these observations with the dominance of expert loading in edge inference.

2 Empirical Observations

  • Loading two experts dominates single-layer latency on an NVIDIA RTX 4090 with PCIe 4.0, motivating reductions in expert size.The observations use Mixtral-8x7B and a calibration set of 2,048 sequences sampled from C4 validation.Source: §2 · p. 4

2.1 Power-law Distribution of Expert Combinations

  • The top 25% of expert combinations account for over 60% of token activations, while the bottom 29% handle less than 10%.The skew holds across model layers, although the hot combinations vary by layer.Source: §2.1 · p. 4, Figure 3 · p. 4

2.2 Asymmetric Expert Roles in Combinations

  • Masking dominant experts raises perplexity by an average of 2.64, versus 0.71 for auxiliary experts—a 3.7× difference.The distinction is based on which expert’s removal causes greater accuracy degradation; Figure 4 examines the top 20 most frequent combinations.Source: §2.2 · p. 4, Figure 4 · p. 5

2.3 Opportunity for I/O-Computation Overlap

  • A full-width expert requires 7.5 ms for I/O and 0.8 ms for computation, while a 50%-width expert requires 3.8 ms and 0.4 ms, respectively.The simulated I/O-computation bubble is 90% for sequential execution versus 38% with ideal overlap under tiered pruning; wider-expert computation can potentially mask subsequent narrower-expert I/O.Source: §2.3 · p. 5, Figure 5 · p. 5

3 System Design

CRISP profiles combinations and expert roles offline, then uses tier assignments to choose execution widths during inference. Its three components are coupled: reordering enables width flexibility, tiering selects widths, and scheduling exploits the resulting heterogeneity.

3.1 System Overview

  • CRISP profiles combinations, neuron importance, and expert roles offline, then selects tier-specific widths online; its three components are designed to enable one another.Profiling artifacts should be reassessed when the model or deployment workload changes substantially.Source: §3.1 · pp. 5–6

3.2.1 Problem Formulation.

  • A width-k expert requires retaining k intermediate neurons, but storing separate weight matrices for every supported width would add storage overhead.The gated expert uses gate, up, and down projection matrices.Source: §3.2.1 · p. 6

3.2 Importance-Aware Neuron Reordering

  • Neuron importance is measured by average activation magnitude after gating, and descending-importance reordering makes the first k neurons the width-k sub-expert.The reordered matrices preserve the original expert’s outputs; runtime truncation accesses the corresponding leading columns and rows.Source: §3.2.2 · p. 6, §3.2.3 · p. 7

3.3.1 Combination Tier Classification.

  • Combination frequencies determine Hot, Warm, and Cold tiers, with thresholds α=0.6 and β=0.9 yielding approximately 25%, 46%, and 29% of combinations, respectively.The thresholds are based on calibration-set activation frequencies and remain fixed across experiments.Source: §3.3.1 · p. 7

3.3.2 Dominant Expert Identification.

  • Offline ablation identifies the dominant expert by which expert’s individual masking causes the larger perplexity increase; Hot executes both experts fully, Warm narrows the auxiliary to 50%, and Cold skips it.The Warm and Cold policies preserve full width for the dominant expert.Source: §3.3.2 · p. 7, §3.3.3 · p. 7

3.3.4 Memory Footprint Analysis.

  • With Hot, Warm, and Cold token fractions of 0.60, 0.30, and 0.10, and a 50%-width Warm auxiliary, average per-token memory access is reduced by 13% versus baseline.The analysis expresses average access as a tier-weighted sum of full-width and narrow-width expert footprints.Source: §3.3.4 · p. 7

3.4.1 Dependency-Preserving Scheduling Architecture.

  • After routing, CRISP schedules expert loading on an I/O stream alongside ready-expert computation, while preserving layer dependencies and prioritizing wider experts.Wider experts begin computing earlier, creating longer windows to overlap subsequent loads.Source: §3.4 · p. 7, §3.4.1 · p. 8, §3.4.2 · p. 8

3.4.3 Overlap Efficiency Analysis.

  • A full-width expert computes in 0.8 ms while loading a 50%-width expert takes 3.8 ms, so one computation cannot fully hide that transfer, but the scheduler overlaps part of it.Batched execution can supply additional routed tasks; the scheduler reports hiding a significant portion of narrow-auxiliary I/O behind full-width dominant-expert computation.Source: §3.4.3 · p. 8

4 Implementation

CRISP prepares experts offline for variable-width execution, then uses width-aware loading, computation, and scheduling online. Its implementation is modular and designed to integrate with existing inference frameworks.

4 Implementation

  • The prototype combines an offline preparation pipeline with an online inference engine built on PyTorch and custom CUDA kernels.The implementation is approximately 5,000 lines of Python and CUDA.Source: §4 · p. 8, §4.1 · p. 8, §4.2 · p. 9

4.1 Offline Preparation Pipeline

  • Offline profiling assigns combination tiers, estimates neuron importance and dominant experts, then reorders each expert’s weights by importance.Profiling uses calibration data; dominant experts are identified by pairwise ablations in Warm and Cold combinations.Source: §4.1 · pp. 8–9

4.2.1 Expert Loading.

  • Width-aware loading transfers only the required expert-weight portions, using strided copies to skip pruned neurons and asynchronous CUDA streams to overlap transfers with computation.For width k, the engine loads the first k columns of Wgate and Wup and the first k rows of Wdown.Source: §4.2.1 · p. 9

4.2.2 Truncated Expert Computation.

  • Custom CUDA kernels compute gated FFN outputs using the first k neurons, with optimized full-width and 50%-width configurations.For Cold combinations, the second expert is skipped.Source: §4.2.2 · p. 9

4.2.3 Dependency-Preserving Scheduling.

  • Separate I/O and computation streams, plus priority queues for pending and ready experts, let the scheduler load selected experts while executing ready ones.Execution advances after the current layer’s expert outputs are aggregated.Source: §4.2.3 · p. 9

4.2.4 Tier Lookup.

  • A GPU-resident lookup table maps each activated combination to its tier and, for Warm or Cold tiers, its dominant expert.The runtime configuration requires one lookup.Source: §4.2.4 · p. 9

4.3 Integration and Portability

  • CRISP’s components are drop-in replacements with adapter interfaces for Hugging Face Transformers and vLLM.The kernels target Ampere and Ada Lovelace GPUs, while the algorithmic approach is described as architecture-agnostic.Source: §4.3 · p. 9

5 Evaluation

CRISP is evaluated on Mixtral-8x7B and DeepSeek-MoE-16B across edge platforms and memory budgets. It improves throughput while preserving accuracy, and ablations report benefits from importance-aware reordering, tiered execution, and width-aware scheduling.

5.2.1 Model Accuracy Preservation.

  • On Mixtral-8x7B, CRISP’s WikiText-103 perplexity is 3.98 versus 3.84 for full-precision execution, while Uniform Pruning reaches 4.67.CRISP preserves 97.4% of original average downstream accuracy versus 90.6% for Uniform Pruning; DeepSeek-MoE perplexity is 6.89 versus 8.21 for Uniform Pruning.Source: §5.2.1 · p. 10, Figure 8 · p. 10

5.2.2 Iso-Memory Comparison.

  • At 40% expert-memory budget, CRISP achieves 73.4% average accuracy versus 70.5% for Uniform Pruning and 69.6% for Expert Dropping, with 47 ms/token latency.The reported latency is close to Expert Dropping at 46 ms/token and below Uniform Pruning at 95 ms/token.Source: §5.2.2 · pp. 10–11, Figure 9 · p. 11

5.2.3 Tier-conditioned accuracy.

  • Hot, Warm, and Cold routing events retain 99.9%, 99.1%, and 94.7% of full-width accuracy, respectively.Cold combinations account for 10% of routing events; skipping their auxiliary expert reduces I/O and latency at the cost of a larger conditional accuracy drop.Source: §5.2.3 · p. 11

5.2.4 Robustness to workload shift.

  • Transferred profiling metadata decreases target-workload accuracy by at most 2.7 percentage points, while refreshing it recovers accuracy to within 0.5 points of target-matched profiling.The refreshed setting has similar throughput.Source: §5.2.4 · p. 11

5.3 Throughput Evaluation

  • 8.3 tokens/second at 10% expert-memory budget is 3.9× Full Offloading and 2.1× MoE-Offload; CRISP reaches 14.7 tokens/second versus 8.6 for MoE-Offload at 30%.Figure 10 presents throughput across memory budgets.Source: §5.3 · p. 11, Figure 10 · p. 11

5.4 Energy Efficiency Across Edge Platforms

  • CRISP reaches 8.2 and 5.2 tokens/s on the RTX 4090 and Jetson Orin, consuming 28.9 and 7.5 J/token, respectively.Relative to full offloading, energy per token falls by 76.7% on PCIe and 73.9% on UMA; Figure 11 reports throughput and energy with 95% confidence intervals.Source: §5.4 · p. 11, Figure 11 · p. 11

5.5 Ablation Study

  • Importance-aware reordering achieves 4.21 perplexity versus 6.83 for random and 5.92 for original neuron ordering.The tiered policy reduces I/O volume by 58% versus all full-width execution with a 0.14 perplexity increase, and improves perplexity by 0.69 over uniform 50%-width pruning at similar I/O volume.Source: §5.5.1 · p. 12, Figure 12 · p. 12, §5.5.2 · p. 12

5.5.3 Effect of Pruning-Aware Scheduling.

  • Width-aware scheduling achieves 78% I/O-computation overlap versus 52% for FIFO and improves throughput by 1.44× over FIFO.Figure 14 compares scheduling strategies at a 20% memory budget.Source: §5.5.3 · p. 12, Figure 14 · p. 12

6 Related Work

Related systems improve expert management, pruning, quantization, or scheduling, but CRISP focuses on combination-specific expert capacity and the execution heterogeneity it creates.

Efficient MoE Inference Systems

  • Existing inference systems optimize expert management and runtime efficiency but do not reduce the amount of expert-weight data that must be loaded.CRISP instead targets expert-weight access through combination-aware pruning.Source: §6 · p. 12

Neural Network Pruning

  • MoE pruning must account for skewed expert usage and the fact that an expert’s contribution can vary across combinations.CRISP selects execution widths according to combination frequency and expert role.Source: §6 · p. 13

LLM Quantization

  • Quantization reduces weight representation cost, whereas CRISP reduces combination-specific expert capacity, so the approaches can be combined.The passage describes CRISP as orthogonal to quantization.Source: §6 · p. 13

Scheduling and I/O-Computation Overlap

  • Unlike approaches relying on pipeline structure or request-level batching, CRISP creates heterogeneous expert execution times through differentiated pruning and schedules around them.The scheduler uses this heterogeneity to overlap expert loading with computation.Source: §6 · p. 13

7 Conclusion

The conclusion frames edge MoE serving as a problem of deciding where expert capacity can be reduced according to runtime use. CRISP jointly designs capacity preservation and execution scheduling, improving throughput under tight memory budgets while maintaining a favorable quality-latency trade-off.

7 Conclusion

  • The main challenge is deciding where expert capacity can be reduced to match how experts are used at runtime.The conclusion argues for pruning by expert combinations rather than experts in isolation.Source: §7 · p. 13
  • Importance-aware reordering supports multiple widths, combination-aware pruning selects widths by frequency and role, and scheduling exploits the resulting heterogeneity to reduce exposed I/O latency.The conclusion presents these elements as a co-design.Source: §7 · p. 13
  • The conclusion identifies jointly reasoning about model structure, access patterns, and execution behavior as important for efficient sparse-model deployment on edge platforms.The stated scope is edge deployment.Source: §7 · p. 13

A.1 Tier-Conditioned Accuracy

Accuracy retention differs by routing tier, and the Cold-tier auxiliary-expert analysis exposes a quality–latency trade-off.

A.1 Tier-Conditioned Accuracy

  • The analysis conditions accuracy retention on Hot, Warm, and Cold routing events and varies retained auxiliary-expert capacity for Cold combinations.Figure 15 presents tier accuracy and routing-event share alongside Cold-tier accuracy, I/O volume, and latency across auxiliary-expert settings.Source: §A.1 · p. 16, Figure 15 · p. 16
  • Hot and Warm combinations retain 99.9% and 99.1% of full-width accuracy, respectively, while Cold combinations retain 94.7%.Cold combinations account for 10% of routing events, limiting their larger conditional degradation's effect on dataset-level accuracy.Source: §A.1 · p. 16
  • Executing only the Cold-tier dominant expert cuts I/O volume by 50% and latency from 86 ms to 45 ms, while accuracy retention falls from 100% to 94.7%.Retaining 50% of the auxiliary expert provides an intermediate operating point.Source: §A.1 · p. 16, Figure 15 · p. 16

A.2 Robustness to Workload Shift

CRISP's profiling metadata is tested across transfers among MMLU, HellaSwag, and WinoGrande. Direct transfer limits accuracy loss, while refreshing metadata recovers most of it with similar throughput.

A.2 Robustness to Workload Shift

  • The evaluation compares target-matched metadata, direct source-to-target transfer, and metadata refreshed using the target workload.It covers all cross-domain transfers among MMLU, HellaSwag, and WinoGrande.Source: §A.2 · p. 16, Figure 16 · p. 16
  • Directly transferred metadata limits accuracy decrease to at most 2.7 percentage points across evaluated workload pairs while maintaining comparable or slightly higher throughput.Figure 16 compares target-matched, directly transferred, and refreshed metadata for cross-domain transfers among MMLU, HellaSwag, and WinoGrande.Source: §A.2 · p. 16, Figure 16 · p. 16
  • Refreshing profiling metadata on the target workload recovers accuracy to within 0.5 percentage points of target-matched performance and retains similar throughput.The refresh updates source-workload metadata using the target workload.Source: §A.2 · p. 16

A.3 Throughput Across Sequence Lengths

On Mixtral-8x7B with a 20% memory budget, CRISP maintains a throughput advantage across the evaluated sequence lengths, with larger gains at longer lengths.

A.3 Throughput Across Sequence Lengths

  • CRISP maintains its throughput advantage across input sequence lengths from 128 to 2,048 tokens on Mixtral-8x7B under a 20% memory budget.Figure 17 reports throughput in tokens per second across these lengths.Source: §A.3 · p. 16, Figure 17 · p. 16
  • Throughput improvements are greater for longer sequences as the cumulative benefit of reduced I/O becomes more pronounced.The comparison uses Mixtral-8x7B under a 20% memory budget.Source: §A.3 · p. 16
  • The sequence-length evaluation measures throughput in tokens per second on Mixtral-8x7B with a 20% memory budget.Figure 17 spans the evaluated input lengths from 128 to 2,048 tokens.Source: Figure 17 · p. 16, §A.3 · p. 16

A.4 System Overhead Analysis

CRISP's preparation is offline, its runtime scheduling overhead is small relative to token-generation latency, and asymmetric Warm-tier execution outperforms a symmetric alternative at the same I/O budget.

A.4 System Overhead Analysis

  • Importance-aware neuron reordering and combination classification are performed offline and incur no runtime cost.The overhead analysis covers both offline preparation and runtime overhead.Source: §A.4 · p. 16

A.4.1 Offline Preparation Overhead.

  • Offline preparation takes 48 minutes for Mixtral-8x7B, a one-time cost amortized across subsequent inferences.Importance Computation takes 18 minutes, Weight Reordering 7 minutes, Combination Profiling 15 minutes, and Dominant Expert Identification 8 minutes; additional storage is less than 2 MB.Source: §A.4.1 · p. 17, Table 2 · p. 17

A.4.2 Runtime Scheduling Overhead.

  • Runtime scheduling overhead is 48.1 μs per token for Mixtral-8x7B, or 0.05% of total token-generation latency.Tier Lookup takes 0.8 μs; Stream Coordination adds 45 μs and enables I/O prefetching that saves significantly more time.Source: §A.4.2 · p. 17, Table 3 · p. 17

A.5 Asymmetric Execution

  • At the same 75% I/O budget, asymmetric Warm-tier execution achieves 4.02 perplexity versus 4.18 for symmetric 75%-width execution.The comparison supports preserving full capacity for the dominant expert rather than balanced pruning.Source: §A.5 · p. 17, Table 4 · p. 17
Loading 2610.37442v1…