Source-linked AI summary
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Ning Liao, Xiaoxing Wang, Xiaohan Qin, Junchi Yan
TL;DR
Fine-grained MoE designs face a performance ceiling when intermediate-dimension granularity exceeds its optimal threshold. FineRMoE extends fine-grained experts to the intermediate and output dimensions with bi-level sparse computation, single-router routing, and generalized upcycling. Across ten benchmarks, it outperforms baselines while delivering substantially higher parameter efficiency and improved inference latency and throughput.
Problem
Fine-grained MoE performance stops improving beyond an optimal intermediate-dimension granularity, limiting further gains from single-dimension design.
Method
FineRMoE extends fine-grained expert design to intermediate and output dimensions using bi-level sparse computation, a single specialized router, and generalized upcycling.
Results
Across ten benchmarks, FineRMoE outperforms carefully curated baselines and achieves 6× higher parameter efficiency, 281× lower prefill latency, and 136× higher decoding throughput than the strongest baseline.
Takeaways & Limitations
FineRMoE demonstrates that extending fine-grained design to the output dimension can deliver additional expert-specialization gains beyond intermediate-dimension design.
Abstract
from arXiv · showhide
As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
1 INTRODUCTION
FineRMoE extends fine-grained expert design from the intermediate dimension to the output dimension to address the performance ceiling of single-dimension designs. It combines bi-level sparse computation, single-router activation, and generalized upcycling, achieving strong benchmark and inference-efficiency results.
- Fine-grained MoE performance improves with intermediate-dimension granularity only within a valid parameter regime, then declines beyond an optimal threshold.
- FineRMoE applies fine-grained design to both intermediate and output dimensions through four hyperparameters controlling granularity and expansion rates.
- Output-dimension fine-graining creates dimensional-consistency challenges because conventional weighted-sum fusion cannot directly combine dimension-reduced expert outputs.
- The bi-level sparse paradigm restores output dimensions by concatenating selected reduced vectors, each formed from weighted sums of sparsely activated experts.
- 6× higher parameter efficiency, 281× lower prefill latency, and 136× higher decoding throughput were reported against the strongest baseline across ten benchmarks.The models used 128 total experts with 2 activated experts and were built from Qwen2.5 models using the upcycling method.
- A single specialized router governs activation in both sparse layers, promoting consistent activations while reducing routing parameter cost.
- The generalized upcycling method flexibly partitions and expands pretrained FFNs along intermediate and output dimensions and also applies to conventional MoE architectures.
2 RELATED WORK
Related work situates FineRMoE within the development of Mixture-of-Experts and upcycling approaches for contemporary language models. Prior work moved from larger-capacity experts toward fine-grained intermediate dimensions, while upcycling commonly reuses pretrained FFNs.
- Mixture-of-Experts: Mixture-of-Experts was introduced to scale model capacity while curbing computational overhead and is now prevalent in contemporary LLMs.
- FineRMoE: FineRMoE combines a shared expert with multiple sparse experts and applies fine-grained design to both intermediate and output dimensions.
- Mixture-of-Experts: Earlier MoE models favored larger intermediate dimensions, whereas recent LLMs have adopted fine-grained experts with lower intermediate dimensions.
- Upcycling: Upcycling addresses the computational and data demands of training MoE models from scratch by reusing pretrained dense-model components.
3 METHOD
FineRMoE expands fine-grained expert design across output and intermediate dimensions through a configurable sparse architecture, unified routing, and generalized upcycling. Its method reduces dimensions before restoring them and constructs experts from pretrained dense FFNs rather than training from scratch.
- FineRMoE architecture: FineRMoE combines a shared expert with sparse finer-grained experts whose outputs are processed through sparse concatenation and sparse summation.Each expert uses up, gate, and down projection matrices.
- FineRMoE architecture: Output granularity GO measures the ratio between the LLM hidden dimension h and each sparse expert’s reduced output dimension he.The output is formed by concatenating GO dimension-reduced components.
- FineRMoE architecture: Intermediate granularity GI and expansion rate RI control sparse experts’ intermediate dimensions and their total intermediate width within each group.Each group contains Ng finer-grained experts whose weighted sum produces a candidate vector.
- Router mechanism: A single router jointly selects experts within groups and dimension-reduced vectors for concatenation, avoiding conflicts from two separate routers.The final mask combines expert-activation and vector-selection masks before activating GOTI experts.
- Router mechanism: Top1Select chooses the highest-scoring candidate vector for each concatenation component, while WeightedSum aggregates activated experts within each group.The candidate score is the sum of scores from experts in its corresponding group.
- Upcycling for FineRMoE: The generalized upcycling method constructs FineRMoE from pretrained dense FFNs by partitioning and expanding weights using GI, RI, GO, and RO.The same configurable protocol can also implement replication-based and intermediate-dimension partitioning upcycling methods.
4 EXPERIMENTS
Experiments evaluate FineRMoE against baselines and through ablations of its dimensions, router, architecture, and configuration. FineRMoE achieves the strongest benchmark and inference-efficiency results, while ablations support its design choices.
- Baseline comparison: FineRMoE delivers 281 times faster TTFT and 136 times higher decoding throughput than C32A2.TTFT is 178.3 ms versus 50245.9 ms, while throughput is 27.3 tokens/s versus 0.2 tokens/s.
- Baseline comparison: FineRMoE achieves superior parameter efficiency and inference efficiency while outperforming all baseline methods.The comparison evaluates both parameter-performance trade-offs and inference efficiency.
- Finer-grained design: Fine-grained design across intermediate and output dimensions achieves the best average performance and the lowest expert similarity.Output-only design improves average performance by 0.73 over the pre-trained model, while the combined design further improves performance with fewer activated parameters.
- Router design: A unified router outperforms using two separate routers because it keeps selected concatenation vectors aligned with highly scored experts.Separate routers can select inconsistent experts across the sparse sum and concatenation layers, degrading performance.
5 CONCLUSIONS
FineRMoE extends fine-grained expert design to both intermediate and output dimensions and combines this architecture with bi-level sparse computation, specialized routing, and generalized upcycling. Experiments show strong performance and efficiency across the evaluated model sizes and benchmarks.
- Conclusions: FineRMoE extends fine-grained expert design from the intermediate dimension to the output dimension.The architecture uses a bi-level sparse forward computation paradigm with sparse concatenation and sparse sum layers.
- Conclusions: The proposed generalized upcycling method builds FineRMoE without training the architecture from scratch.The method is also described as compatible with existing MoE architectures.
- Conclusions: FineRMoE uses 128 total experts and activates 2 experts in models based on Qwen2.5 at 0.5B, 1.5B, and 7B sizes.The models are constructed through the proposed upcycling method.
B EXPERIMENTAL SETUP
FineRMoE training combines language-modeling loss with a weighted load-balancing loss, whose formulation follows DeepSeek-V2 and uses expert-selection scores and indicators.
- Training optimizes the sum of language-modeling loss and weighted load-balancing loss.The load-balancing design follows DeepSeek-V2.
- The load-balancing loss uses the number of experts, activated experts per token, sequence length, assignment scores, and selection indicators.
- The load-balancing-loss weight α is set to 0.001 throughout all experiments.
B.2 IMPLEMENTATION DETAILS
Experiments use Megatron-LM with Qwen2.5, consistently prepared mixed pre-training data and settings, and evaluation across knowledge, reasoning, code, and math benchmarks.
- FineRMoE is implemented and trained with Megatron-LM based on Qwen2.5 for parallelization flexibility.
- Training data mixes refined public corpora spanning webpages, knowledge, code, math, The Pile, Wikipedia, and books, with settings kept consistent across experiments.Warmup steps equal 1% of total training steps.
- Evaluation uses Language Model Evaluation Harness across benchmarks covering knowledge, reasoning, code, and math.The benchmark suite includes MMLU, BBH, HellaSwag, WinoGrande, ARC, AGIEval, MBPP, and GSM8K.
C INFERENCE EFFICIENCY ANALYSIS
Inference analysis compares FineRMoE with several continued-training baselines across prefill and decoding efficiency. FineRMoE combines the strongest reported benchmark performance with substantially better inference efficiency than the parameter-heavy C32A2 baseline.
- Inference efficiency is evaluated for prefill latency and decoding throughput using models continued-trained on 50B tokens from Qwen2.5-7B.The comparison includes C32A2, S16A4, Drop-Upcycling, NVShard, and FineRMoE.
- FineRMoE achieves the highest throughput among the compared models.
- 281 times faster TTFT gives FineRMoE 178.3 ms versus C32A2’s 50245.9 ms, while FineRMoE also outperforms C32A2 in benchmark performance.
- FineRMoE achieves optimal performance across ten benchmarks while maintaining parameter and inference efficiency.
D EXPERT SIMILARITY ANALYSIS
Expert-similarity analysis compares fine-grained designs applied to intermediate dimensions, output dimensions, both dimensions, or neither. FineRMoE’s two-dimensional design produces the lowest expert similarity, indicating reduced redundancy relative to the alternatives.
- The analysis computes average pairwise cosine similarity among sparse experts in each layer across four fine-grained design settings.The corresponding models are trained on Qwen2.5-1.5B with 10B tokens for efficiency.
- Removing fine-grained design yields the highest expert similarity, indicating severe redundancy and the poorest Table 2 performance.
- Output-only fine-grained design produces slightly higher expert similarity than intermediate-only design because GI = 32 exceeds GO = 2.
- FineRMoE’s fine-grained design on both intermediate and output dimensions yields the lowest expert similarity across all four settings.This signifies alleviated expert redundancy and greater specialization.
E DETAILED EVALUATION ON FINE-GRAINED CONFIGURATIONS
The ablation study reports detailed evaluation results for fine-grained configurations across ten benchmarks in Table 7.
- Table 7 presents the detailed evaluation results for the ablation study of fine-grained configurations.
- The evaluation covers ten benchmarks.
- The study examines how fine-grained configurations affect model performance.
F ABLATION STUDY ON TI, RI, RO
The ablation study evaluates TI, RI, and RO against a base configuration, finding stable improvements over the pretrained model and minor variation across settings. The base configuration is preferred because it balances performance gains and parameter efficiency.
- Main settings: The main settings use TI = 1, RI = 1, and RO = 2 to minimize total and activated parameters while preserving sparsity.
- Base configuration: The base configuration uses GI = 32, RI = 1, GO = 2, and RO = 2.
- Ablation setup: TI, RI, and RO are varied in quick ablations on Qwen2.5-1.5B trained for 10B tokens.
- Results: All ablation settings improve performance over the pretrained model.
- Results: Changing TI, RI, or RO causes only minor performance fluctuations relative to the base configuration.
- Conclusion: The base configuration is selected as optimal because it trades off performance gains against parameter efficiency.