PaperlayerView exact revision on arXiv

Source-linked AI summary

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu

arXiv:2605.10933v3cs.LGcs.CL

TL;DR

Large MoE parameter footprints create storage and memory-access bottlenecks for end-side deployment, despite sparse activation reducing computation. DECO revises routing and expert design with ReLU routing, learnable expert-wise scaling, NormSiLU, and non-gated MLPs. At 20% routed-expert activation, it matches dense performance under matched parameter and token budgets and surpasses established MoE baselines.

  • Problem

    End-side MoE must achieve high performance and low computation while avoiding the storage and memory-access costs of massive total parameter footprints.

  • Method

    DECO uses differentiable ReLU-based routing with learnable expert-wise scaling, NormSiLU activation, non-gated MLP experts, and adaptive sparsity regularization.

  • Results

    At 20% routed-expert activation, DECO matches dense performance under matched total parameters and training tokens and outperforms established MoE baselines.

  • Takeaways & Limitations

    DECO’s sparsity translates into practical deployment efficiency, including a 2.93× speedup over vanilla dense inference on Jetson AGX Orin.

  • Takeaways & Limitations

    The study does not evaluate supervised fine-tuning or reinforcement learning, where MoE post-training challenges may arise.

Abstract

from arXiv · show

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93$\times$ speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO.

1 Introduction

End-side MoE deployment must balance performance, computation, and storage, motivating DECO’s test of dense-comparable performance under matched parameter and token budgets. DECO combines flexible routing, revised experts, and practical acceleration to pursue this balance.

  • Motivation: End-side MoE must jointly target high performance, low computation, and small storage overhead because large expert pools create storage and memory-access costs.These costs arise when experts or offloaded parameters move through memory hierarchies.
  • Research question: DECO asks whether sparse MoE can match dense performance with the same total parameter budget and training-token count.This is a stricter comparison than settings that use more tokens under an equal computation budget.
  • Architecture: DECO uses differentiable ReLU-based routing with learnable expert-wise scaling to support token-dependent activation and balance routed-expert contributions.The scaling factors calibrate individual routed experts against shared experts.
  • Architecture: NormSiLU applies dual-stage normalization before SiLU, stabilizing activation trends and reducing routed-expert activation ratios.The design addresses rising activation ratios and vanishing SiLU output magnitudes observed with vanilla SiLU.
  • Results: 20% routed-expert activation yields performance comparable to dense models under matched total parameters and training tokens, while surpassing same-scale MoE baselines.The comparison is reported at the same activation ratio for the MoE baselines.
  • Deployment: 2.93× speedup over vanilla dense inference is achieved on Jetson AGX Orin with a tailored kernel exploiting sparse activation.The kernel uses tensor cores and reduces memory-access overhead.

2 Related Works

Prior MoE work primarily emphasizes performance and computation, while DECO’s related-work framing adds routing flexibility, expert design, and sparsity control as relevant design dimensions. DECO adopts ReLU routing, learnable expert-wise scaling, NormSiLU, non-gated experts, and adaptive regularization.

  • MoE components: MoE combines a router, experts, and an auxiliary training objective to increase performance while controlling computational growth.The auxiliary objective is distinct from the language-modeling loss.
  • Router design: TopK routing fixes the number of active experts per token and is nondifferentiable, whereas TopP permits token-dependent activation through score thresholds.Other approaches introduce zero-computation experts or alternative sparsification objectives.
  • Router design: DeepSeek-V3 uses Sigmoid followed by unitsum normalization and a scalar score scale, while DECO replaces the fixed scale with learnable expert-wise factors.DECO’s factors accommodate heterogeneous routed-expert output scales.
  • Expert design: Mainstream MoE experts generally use gated SiLU MLPs, but DECO investigates NormSiLU and non-gated MLP experts with ReLU-based routing.The proposed expert choices target activation stability and efficiency.
  • Training objective: DECO uses adaptive sparsity regularization inspired by ReMoE and replaces L1 regularization with router entropy for numerical stability.Its coefficient auto-scales to control the sparsity level.

3 Methodology of DECO

DECO’s methodology combines ReLU routing, learnable expert-wise scaling, NormSiLU-based experts, non-gated MLPs, and adaptive entropy regularization. These components are designed to provide flexible routing, stable activations, and precise sparsity control.

  • Router design: ReLU-based routing is differentiable, inherently sparse, and supports token-dependent activation ratios.It replaces conventional TopK routing’s fixed activation pattern.
  • Router design: Learnable expert-wise router scaling assigns distinct coefficients to routed experts to balance routed and shared output scales.The vectorized scaling extends a fixed scalar factor and accommodates expert heterogeneity.
  • Expert design: Non-gated experts produce a more stable activation-ratio trend than gated variants under ReLU-based routing.This stability requires a lower regularization penalty to reach a target sparsity threshold.
  • Expert design: NormSiLU prepends inter-expert mean normalization and intra-expert RMS normalization to the SiLU nonlinearity.The two stages center pre-activation inputs and maintain consistent activation magnitudes.
  • Expert design: SparseLinear operations involve only active experts during inference, reducing the computation associated with inactive expert modules.The expert weights are organized across expert and intermediate dimensions for sparse operations.
  • Adaptive sparsity regularization: Router entropy regularization is multiplied by λ and adaptively rescales λ according to the current sparsity relative to a target.Increasing or decreasing λ steers activation toward the desired ratio.

4 Experiments

Experiments compare DECO with dense and established MoE baselines under matched parameter, token, and activation settings. Ablations show that expert-wise scaling, NormSiLU, and non-gated experts support stable sparse routing and dense-comparable performance.

  • 4 Experiments: Experiments evaluate Dense, TopP, DeepSeek-V3, ReMoE, and BlockFFN across four parameter scales using C4 perplexity and downstream average accuracy.The scales are Small (0.11B), Medium (0.24B), Large (0.53B), and XLarge (1.18B).
  • 4 Experiments: All settings hold total parameters, training tokens, non-FFN components, routed-expert activation ratio, and shared-expert dimensions consistent within each comparison group.Training uses around 40 times the parameter count in tokens, with routed-expert activation near 20% on training data.
  • 4.1 Main Results: 20% average routed-expert activation lets DECO match Dense performance under the same total parameter budget and training-token volume.This result is reported as dense comparability with reduced active computation.
  • 4.1 Main Results: Under the same routed-expert activation ratio, shared-expert dimensions, and expert granularity, DECO surpasses existing MoE baselines in perplexity and downstream task performance.The comparison includes the established MoE settings evaluated in the main experiment.
  • 4.2 Effect of Expert-Wise Router Scaling: Learnable expert-wise router scaling outperforms fixed and shared-scalar scaling, consistent with routed experts having heterogeneous output norms.In DECO (Medium)'s first MoE layer, average expert output norm is 0.23 while the maximum exceeds 1.14.
  • 4.3 Effect of NormSiLU: NormSiLU stabilizes routing and activation behavior: RMS normalization limits activation-ratio growth, mean normalization promotes sparsity, and both normalization steps improve results.Without normalization or without RMS, activation ratios surge and regularization coefficients become significantly higher; SiLU and w/o Mean also show lower output magnitudes.
  • 4.4 Effect of Expert Gating: For ReLU-based routing, non-gated MLP experts generally outperform gated counterparts, while gated experts provide only a marginal gain with standard TopK routing.The gated DECO variant shows unstable activation growth requiring aggressive sparsity regularization, whereas the non-gated variant remains more stable.
  • 4.5 Effect of Activation Ratio: Dense-comparable activation thresholds vary by scale: Small reaches parity at 15%, whereas Medium requires 10%.The paper notes that larger-scale models may attain dense-comparable parameter efficiency at lower activation ratios, but further validation is required.

5 Practical Inference Acceleration

DECO’s tailored sparse-inference kernel reduces computational and memory-access overhead from inactive routed experts, producing substantial speedups on GPUs and edge hardware.

  • The kernel exploits sparse activation to reduce computation and memory-access latency from inactive routed experts.
  • 2.56× average speedup over Dense on RTX 4090 and 2.93× on Jetson AGX Orin demonstrates practical inference gains.The evaluation uses Spec-Bench and compares single-GPU decoding speeds.
  • DECO also outperforms standard TopK at identical 20% average activation ratios.The TopK baseline and DECO kernel share the same average activation ratio.
  • Element-wise in-place ReLU avoids the vector-wise sorting required by TopK, yielding lower computational complexity.

6 Discussion

DECO’s dense-comparable performance is motivated by activation sparsity in dense models but appears sensitive to training-data heterogeneity.

  • 30%∼40% of neurons in a standard SwiGLU FFN provide non-negligible per-token contributions, leaving 60%∼70% with apparently limited contribution.
  • DECO matches or exceeds dense baselines across parameter scales when trained on a diverse mixture of web, code, math, and other data.
  • On FineWeb, DECO (Small) has slightly higher PPL than Dense (Small), 30.14 vs. 29.76, while exceeding task accuracy, 38.65 vs. 38.03.
  • The authors state that more studies are needed to verify why heterogeneous datasets may better suit sparse MoE.

7 Conclusion

DECO is presented as a sparse MoE architecture that maintains dense-comparable performance with low activation while improving deployment efficiency through architectural refinements and hardware acceleration.

  • At a 20% activation ratio, DECO matches dense models with equivalent parameter counts and training-token budgets.
  • DECO consistently outperforms existing MoE baselines under the reported evaluation setting.
  • Its design combines ReLU-based routing with learnable expert-wise scaling, non-gated MLP experts, and NormSiLU activation.
  • Real-hardware acceleration confirms that DECO’s sparsity can translate into deployment efficiency gains.

Limitations

The paper identifies unresolved scope boundaries involving post-training behavior, model-size scaling, data distributions, and inference-task variation.

  • The study does not evaluate supervised fine-tuning or reinforcement-learning stages, where DECO may encounter issues requiring mitigation.
  • The scaling of the activation-ratio threshold with model size remains undetermined.
  • DECO’s intrinsic sparsity and performance across diverse data distributions or inference tasks require further investigation.

Ethics Statement

DECO targets efficient end-side deployment, but its dense-comparable performance depends on selected architectural settings. The paper also acknowledges inherited risks from large-scale web pretraining data.

  • Ethics Statement: DECO reduces active parameter scale and memory-access overhead while maintaining dense-comparable performance for resource-constrained edge hardware.The paper links this efficiency to more energy-efficient inference, reduced carbon footprint, and broader access to advanced AI capabilities.
  • Ethics Statement: Large-scale web corpora expose DECO’s downstream models to inherited historical biases, toxicity, and representational harms.The authors report cleaning and detoxification, removal of personal information and offensive content, and recommend safety alignment and bias mitigation.
  • Ethics Statement: DECO’s dense-comparability holds only under certain conditions because performance is sensitive to activation ratio, expert granularity, and shared expert size.These hyperparameters are identified as important determinants of MoE performance.
  • Ethics Statement: When the shared-expert dimension is 1∼2 times the routed-expert dimension, DECO remains comparable to the dense baseline and relatively insensitive to shared-expert size.The routed-expert dimension is fixed at 64 in this comparison.
  • Ethics Statement: For g > 60, DECO consistently achieves dense parity, while finer granularity improves stability and competitiveness; below 60, performance fluctuates near the dense baseline.In the Medium setting, perplexity improves monotonically when g > 120, whereas the Small setting is less sensitive to granularity.

B Detailed Experimental Settings

The experiments use shared pretraining and evaluation procedures across models, adjust BlockFFN regularization for fair comparison, and tune hyperparameters through small-scale searches and scaling assumptions. Additional studies examine repeated-run reliability and expert-scaling initialization.

  • Training and evaluation: All settings use a mixture of FineWeb, Nemotron-CC, the Pile, Wikipedia, and other corpora spanning web text, mathematics, and code.The data mixing ratio is tuned through small-scale experiments.
  • Training and evaluation: Accuracy is evaluated with LMEval on PIQA, SIQA, HellaSwag, ARC-C, ARC-E, WinoGrande, and LAMBADA.
  • Baseline adjustments: For fair comparison, BlockFFN replaces its original chunk-sparsification and activation-locality losses with adaptive sparsity regularization.The original losses target consecutive-token union sparsity and neighboring-token activation similarity, which are outside this work’s objectives.
  • Baseline adjustments: At the same target activation ratio of 20%, the one-loss BlockFFN setting consistently outperforms the original two-loss regularization.The authors use this result to support their adjusted experimental setting.
  • Hyperparameters: Hyperparameters are selected by small-scale grid search and extrapolated to larger models using assumed power-law relationships with computation budget.The extrapolated quantities are the optimal learning rate and batch size.
  • Hyperparameters: All models use a dense first-layer FFN, sparse MoE in the remaining Nlayer −1 layers, and a WSD learning-rate schedule.The schedule includes 100 warmup steps and 1,000 decay steps.
  • Scaling initialization: DECO performs best when learnable expert-wise scaling factors are initialized around 0.1∼0.25; experiments use 0.1.Performance does not vary monotonically with initialization.

D Repeated Experiments

Repeated runs show stable DECO performance across Small and Medium scales, including slight Medium-scale gains over Dense. The NormSiLU analysis provides a theoretical gradient-bound rationale while exposing and addressing a sparsity-related implementation constraint.

  • Repeated experiments: Five independent runs at Small and Medium scales show narrow standard deviations and no high variance for DECO.These repetitions use different random seeds.
  • Repeated experiments: DECO tightly matches Dense at Small scale and slightly outperforms it at Medium scale.
  • Repeated experiments: 27.53 ± 0.276 perplexity and 39.13 ± 0.217 task score are the reported Medium-scale DECO averages.The paper presents these results as evidence of comparable or occasionally superior performance to Dense.
  • NormSiLU justification: The theoretical analysis considers concatenated up-projection weights, an input hidden state, and the post-activation intermediate state across all experts.The derivation begins by assuming Norm is vanilla layer normalization across the concatenated expert dimension.
  • NormSiLU justification: The gradient bound is ||∇WupL||F ≤ ||g||2 · O(||Wup||−1), under the stated norm assumptions.The result follows from bounding the normalization Jacobian and applying sub-multiplicativity of matrix norms.
  • NormSiLU justification: With properly initialized Wup, normalization keeps gradient scale bounded and invariant to input magnitude, preventing gradient explosion.
  • NormSiLU implementation: Global layer normalization would materialize z across all experts and force computation of inactive experts, violating MoE sparsity.
  • NormSiLU implementation: DECO instead uses inter-expert mean normalization plus intra-expert RMS normalization applied only to activated experts.This preserves the theoretical gradient-bounding stability while respecting sparse-inference constraints.
Loading 2605.10933v3…