Source-linked AI summary
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
Lei Xin, Bin Gu, Peize Li, Zitong Wang, Jianbo Zhao, Changjiang Jiang, Yanyue Xie, Chao Huang, Xuyang Zhao, Zunhai Su, Fanhu Zeng, Zhenglun Kong
TL;DR
Trained recommendation MoEs retain a full expert bank even when serving requires a smaller budget. UniMoMo converts the checkpoint through calibration-conditioned graph coarsening that uses functional similarity and routing-aware protection, then adapts the compressed standard MoE. Across three datasets and depths, four-expert checkpoints retain 99.92%–102.30% of source mean NDCG@10 with 1.28×–1.63× speedup, while two-expert endpoints provide a more aggressive trade-off.
Problem
A trained recommendation MoE still stores and routes over its full expert bank, while existing methods do not convert such checkpoints to a chosen deployment budget.
Method
UniMoMo partitions experts through calibration-conditioned constrained coarsening, using functional affinities, routing-exposure protection, traffic-weighted reconstruction, and brief supervised adaptation.
Results
99.92%–102.30% of Origin MoE five-run mean NDCG@10 and 1.28×–1.63× speedup are achieved by four-expert checkpoints across the evaluated datasets and depths.
Takeaways & Limitations
The results establish checkpoint conversion as a practical way to export trained recommendation MoEs at multiple serving budgets without a compression-specific online module.
Takeaways & Limitations
The evidence targets RankMixer checkpoints with eight source experts, 2–6 MoE blocks, sampled ranking, and single-A100 inference; calibration behavior depends on representative traffic.
Abstract
from arXiv · showhide
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28$\times$--1.63$\times$. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47$\times$--2.21$\times$. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.
1 Introduction
The paper frames expert-bank reduction as converting a trained recommendation MoE to a chosen serving budget while preserving a standard sparse-MoE interface. UniMoMo uses calibration-conditioned expert behavior and routing exposure to guide the conversion, achieving strong quality retention with substantial speedups.
- Deployment problem: Post-training conversion asks how much of a trained recommendation MoE’s expert bank is necessary for serving.Existing sparse architectures expand capacity or optimize execution but do not answer this checkpoint-level deployment question.
- UniMoMo: UniMoMo formulates expert reduction as constrained graph coarsening under an explicit expert budget.The output remains a smaller standard MoE rather than requiring a compression-specific online module.
- UniMoMo: Functional similarity is measured from shared unlabeled calibration interactions instead of parameter distance.The calibration stream forms expert affinities from how experts respond to recommendation states.
- UniMoMo: Routing exposure protects high-traffic experts from repeated merging during the partitioning process.This combines behavioral compatibility with traffic-aware risk control under one expert-count constraint.
- Results: 99.92%–102.30% of Origin MoE five-run mean NDCG@10 and 1.28×–1.63× speedup are obtained by four-expert checkpoints.The experiments cover Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks.
- Results: 98.36%–104.24% and 1.47×–2.21× are obtained at the aggressive two-expert operating point.These results evaluate the converted models after the stated adaptation workflow.
2 Related Work
Prior recommendation MoEs and sparse-execution systems establish conditional capacity and efficiency techniques but generally determine expert organization during training or retain the full bank. UniMoMo instead defines a traffic-conditioned, recommendation-native partition for resizing an existing checkpoint while preserving standard routing.
- Recommendation model scaling: Large recommendation architectures increase interaction depth, token mixing, or sparse per-token capacity, making serving efficiency a first-order constraint.The cited models establish high-capacity ranking workloads in which deployment cost matters.
- Recommendation MoEs: Recommendation-specific MoEs specialize capacity for heterogeneous behavior and multitask objectives, but their expert organization is fixed during architecture design or training.This leaves open how to resize an existing recommendation-MoE checkpoint to a requested serving budget.
- Acceleration: Existing acceleration mechanisms primarily alter routing, expert execution, or capacity utilization while retaining the trained expert bank.Examples include routing objectives, MegaBlocks, ScatterMoE, shared experts, and conditional-computation granularities.
- Post-training compression: Pruning and quantization compress weights within a fixed network, while model fusion combines independently trained checkpoints or updates.Neither objective determines how jointly trained experts under one recommendation router should share a smaller bank.
- UniMoMo’s distinction: Expert-bank reducers provide grouping and reconstruction operations, but UniMoMo defines the deployment partition directly on ranking states and serving traffic.Its unified partition uses shared unlabeled interactions for functional affinities, exposure protection, and reconstruction weighting.
- UniMoMo’s distinction: The resulting checkpoint preserves standard top-k routing and adds no compression-specific online module.This positions the method as checkpoint conversion rather than a new online serving mechanism.
3 Method
UniMoMo converts a trained recommendation MoE into a smaller standard MoE under a serving budget by combining calibration-based behavioral similarity with routing-exposure constraints. It reconstructs compressed experts and routers, then adapts the complete compressed model without retaining a method-specific online module.
- Checkpoint conversion: Post-training compression partitions each layer’s original experts into M_l clusters and reduces the router from E_l rows to M_l rows.Compression begins from a completed backbone checkpoint; calibration initializes the compressed weights, followed by matched supervised fine-tuning.
- Dual-view profiling: UniMoMo evaluates all experts on shared calibration states to derive response summaries and functional affinities, while routing mass identifies high-exposure experts for protection.Shared inputs make pairwise differences attributable to expert behavior, and normalized routing concentration guides admissibility during coarsening.
- Constrained coarsening: Greedy graph coarsening repeatedly combines the most similar admissible pair until the target expert count is reached.Protected vertices cannot collapse together or be modified repeatedly; a deterministic fallback handles cases where protection conflicts with the target count.
- Expert reconstruction: Each cluster becomes one expert through routing-mass-weighted reconstruction, least-squares correction, and arithmetic router-row aggregation.The corrections are folded into stored parameters rather than retained as an inference-time operator.
- Compressed deployment: Compressed inference uses M_l experts with standard top-k routing and existing expert kernels, without retaining the affinity graph, profiling summaries, or a UniMoMo-specific module.The complete compressed recommender is fine-tuned after all layers are replaced and selected by validation NDCG@10.
- Practical boundary: The two-expert operating point is aggressive: when protected experts exceed the target count, UniMoMo uses a stepwise fallback rather than claiming hard-protection feasibility.The standard four-expert target remains feasible without protection relaxation for layers with eight experts.
4 Experiments
Experiments evaluate UniMoMo across datasets, depths, expert budgets, and matched baselines, measuring ranking quality, latency, and the roles of its two partition signals. The results show strong source-relative quality retention, faster serving, and a balanced four-expert operating point.
- Setup: Experiments span Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, using five-run means and matched conversion settings.The evaluation includes an eight-expert Origin MoE, a matched MergeMoE baseline, and UniMoMo deployments.
- Matched comparison: 0.0004, 0.0028, and 0.0051 NDCG@10 gains over MergeMoE are reported on Amazon Beauty, KuaiRec, and TenRec, respectively.The comparison uses the same four-expert, top-2 serving shape and adaptation budget, and is interpreted as matched five-run gains rather than statistical superiority.
- Main results: 99.92%–102.30% of Origin MoE mean NDCG@10 is retained by four-expert checkpoints across nine dataset-depth pairs.These source-relative endpoints include the fixed supervised adaptation stage.
- Ablation: 0.0018 and 0.0034 are the complete pipeline’s NDCG@10 and HR@10 changes relative to the signal-free ablation row.The sweep holds 8→4 compression and top-2 routing fixed while restoring calibrated behavior and exposure signals.
- Sensitivity: 40.36% lower latency and 99.37% retained NDCG@10 make four experts the balanced default, whereas two experts reduce latency further but lower NDCG@10 by 1.68%.The four-expert setting improves NDCG@10 per unit latency by 66.61% versus eight experts.
- Sensitivity: 6.04% higher NDCG@10 and 6.29% higher HR@10 result from increasing depth from two to six blocks, while eight blocks add only 0.63% NDCG@10 and 145.70% latency.These results support six blocks as the main quality–efficiency operating point.
- Diagnostics: Figure 3 separates routing exposure from functional affinity, showing graded routing relationships within and across displayed groups.Darker cells indicate larger routing-vector similarity, and dashed lines show the displayed routing groups.
5 Limitations and ethical considerations
The evaluation covers RankMixer checkpoints with eight source experts, 2–6 MoE blocks, sampled ranking, and single-A100 inference. UniMoMo depends on representative calibration traffic, which also remains subject to privacy and fairness controls.
- Evaluation is limited to RankMixer checkpoints with eight source experts, 2–6 MoE blocks, sampled ranking, and single-A100 inference.
- UniMoMo’s partition depends on representative calibration traffic, so behavior absent from that stream cannot influence affinities or exposure estimates.
- Calibration logs remain subject to the underlying recommender’s privacy and fairness controls, although UniMoMo collects no new data.
6 Conclusion
UniMoMo treats expert-bank reduction as checkpoint conversion under a deployment budget, reconstructing a smaller ordinary MoE without a compression-specific online module. Across nine dataset-depth pairs, it preserves near-source NDCG@10 while improving measured speed, with portability also supported on four additional backbones.
- UniMoMo converts an expert bank under a deployment budget into an ordinary MoE without a compression-specific online module.
- 99.92%–102.30% of Origin MoE mean NDCG@10 accompanies 1.28×–1.63× measured speedup for final four-expert checkpoints across nine dataset-depth pairs.
- The two-expert endpoints obtain 98.36%–104.24% of Origin MoE mean NDCG@10 and reach 2.21× measured speedup.
- Results on four additional backbones further support UniMoMo’s portability.
A Additional Recommendation Backbones and MoE Baselines
An additional comparison evaluates UniMoMo across four recommendation backbone types against reproduced Vanilla and MoE baselines. UniMoMo ranks first across all 24 metrics, supporting portability while matched compression and serving-cost evidence remains in Table 1.
- Table 5 compares UniMoMo with reproduced Vanilla and MoE baselines across four recommendation backbone types.
- UniMoMo ranks first in all 24 metrics, gaining 0.51–1.37 percentage points over the strongest alternative.
- The consistency across four backbone types supports portability, while Table 1 provides the matched compression and serving-cost comparison.