Source-linked AI summary
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng, Ming Zhang, Ning Ding, Qipeng Guo, Youbang Sun, Biqing Qi, Bowen Zhou
TL;DR
MoE models improve parameter efficiency along width, while similarity among layer-wise experts motivates examining efficiency along depth. This paper studies cross-layer expert sharing and reports lower redundancy and improved performance, with more flexible residual pathways for information retrieval.
Problem
Although sparse MoE improves parameter efficiency along width, the introduction identifies depth-wise efficiency as an open question amid evidence of similar knowledge across layers.
Source: §1 · p. 2Method
The paper studies Cross-Layer Expert Sharing, using Meta Experts shared across layers instead of conventional layer-private Local Experts while keeping recursion count and per-layer activated computation unchanged.
Source: §1 · p. 2Results
At fixed activated experts and expert granularity, Meta Experts match Vanilla MoE performance with only 70% as many experts, while reducing redundancy and improving cross-entropy loss and downstream accuracy.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that inter-layer redundancy can be converted into effective capacity, while shared experts provide hidden states more flexible residual pathways for retrieval.
Source: §1 · p. 2, §5 · p. 10
Abstract
from arXiv · showhide
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged \textbf{Local Expert} in existing mainstream models, we hypothesize that inter-layer shared \textbf{Meta Expert} hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, \textbf{our model matches the performance of Vanilla MoE using only 70--80\% of the experts}, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
1 INTRODUCTION
The introduction motivates improving parameter efficiency across MoE depth and proposes sharing experts across layers. It reports reduced redundancy and better performance, alongside a residual-pathway explanation for more flexible information retrieval.
1 INTRODUCTION
- Across varying model depths and scales, Meta-Expert models achieve comparable loss with 70–80% as many experts or 75–90% as many parameters as Vanilla baselines.The reported expert and parameter savings grow as models become deeper and larger.Source: Figure 1 · p. 1
- Sparse MoE activation addresses dense feed-forward computation costs, but parameter efficiency along model depth remains an open question.The introduction notes that inactive parameters still participate in dense feed-forward computation.Source: §1 · p. 2
- Similar knowledge across layers suggests depth-wise redundancy, while parameter-looped Transformers can add recursions and serial depth, increasing computation and latency.The introduction presents cross-layer sharing as another way to reuse parameters.Source: §1 · p. 2
- Cross-Layer Expert Sharing keeps the recursion count unchanged while increasing each layer’s candidate experts and preserving activated parameters and computation per layer through sparse activation.The shared experts are Meta Experts; conventional layer-private experts are Local Experts.Source: §1 · p. 2
- At fixed activated experts and expert granularity, Meta Experts match Vanilla MoE performance with only 70% as many experts while reducing redundancy and improving cross-entropy loss and downstream accuracy.The analysis also attributes more flexible storage and retrieval to additional residual pathways.Source: §1 · p. 2
- The paper analyzes Meta-Expert activation and total-number ratios, layer presence, and partial hierarchy, and links the design to bidirectional access to representations across depth.The proposed backward residual connection operates at the activation level within one forward pass.Source: §1 · p. 2
2 META EXPERT
The section contrasts layer-private Local Experts with a Meta-Expert design shared across layers. In the latter, each layer can use a common router and expert pool.
2 META EXPERT
- A Local-Expert MoE layer has its own router and expert set, initialized and accessed independently across layers.For an incoming hidden state, the router selects the top-k experts belonging to that layer.Source: §2 · p. 3
- For Local Experts, the layer-specific router selects top-k experts from the current layer to process the token’s hidden state.Source: §2 · p. 3
- The Meta-Expert design replaces layer-private MoE blocks with one shared router and a pool of experts available to every layer.The shared pool can be activated by each layer.Source: §2 · p. 3
3 EMPIRICAL COMPARISON BETWEEN META AND LOCAL EXPERTS
The experiments compare Meta and Local Expert ratios and topologies under controlled expert and activation budgets, then evaluate selected designs across compute scales and downstream benchmarks. Meta-based configurations generally improve loss and redundancy, and outperform Vanilla MoE on downstream tasks, though outcomes vary by topology and dataset.
3.1 SETUPS
- The study varies total and per-layer activation ratios, layer placement, and Meta-pool hierarchy while keeping expert and activation totals fixed to support fairer FLOPs comparisons.It evaluates cross-entropy loss, downstream accuracy, and expert redundancy.Source: §3.1 · pp. 3–4
3.2 META/LOCAL EXPERTS’ RATIO
- Under fixed total expert and activation counts, higher Meta Expert proportions reduce loss and redundancy; higher Local Expert activation reduces redundancy but worsens loss.The authors suggest Local Experts may be harder to train sufficiently because each is activated in only one layer, potentially contributing to redundancy.Source: Figure 4 · p. 4, §3.2 · p. 5
- With 4 Meta and 4 Local Experts activated per layer, the 75% Meta ratio is the best partial-Meta setting for loss and redundancy at every compute budget.It remains slightly worse than the Pure-Meta setting; the authors select 75% for subsequent experiments because an all-Meta model may be unstable to train.Source: Figure 5 · p. 5, §3.2 · p. 5
3.3 META/LOCAL EXPERTS’ TOPOLOGY
- Local-only edge layers and hierarchical Meta pools generally degrade loss, but restricting the first and last two layers or using two loops slightly improves it.The authors relate the edge-layer result to differentiated shallow and deep functions, and leave unexplained why hierarchy-related redundancy does not raise loss.Source: §3.3 · p. 5, Figure 6 · p. 6
- 20–30% fewer experts (15–25% fewer parameters) than Vanilla MoE are needed by Meta Expert models, with the advantage growing as model size increases.The combined configuration consistently outperforms individual configurations in loss.Source: §3.3 · p. 6
- Across compute budgets, Local-only first and last two layers plus two Meta sub-pools perform best; deeper models also benefit from more such layers and larger hierarchies.The authors suggest the relative share of specialized edge layers may stay roughly constant with depth, while the hierarchy should grow.Source: §3.3 · pp. 5–6, Figure 7 · p. 7
- Every refinement improves on Vanilla MoE across compute budgets, with the combined F2L2 + Loop2 design yielding the largest gains in loss and redundancy.The advantage in both measures becomes more pronounced as training compute grows.Source: §3.3 · p. 6, Figure 8 · p. 7
3.4 DOWNSTREAM PERFORMANCE OF META EXPERTS-BASED MODEL
- Meta Expert variants consistently outperform Vanilla MoE on downstream benchmarks, although different variants lead on different datasets under the limited training budget.The comparison uses 6B-A1.6B models trained on 200B tokens; Table 1 reports 5-shot accuracy, normalized accuracy, or generation exact-match depending on the benchmark.Source: §3.4 · p. 7, Table 1 · p. 8
4 META EXPERTS AS BIDIRECTIONAL RESIDUAL PATHWAYS
Meta Experts share one expert block and router across layers, creating shared storage that layers can read through more flexible residual pathways. Unlike layer-wise MoE, this provides hidden states bidirectional access to expert content across depth.
4 META EXPERTS AS BIDIRECTIONAL RESIDUAL PATHWAYS
- Meta Experts replace layer-private MoE blocks with one expert block and router learned once and shared across all layers.Each layer still has its own attention block; the shared expert block is inserted into the residual stream at each layer.Source: §4 · p. 8
4.1 LAYER-WISE MOE.
- In layer-wise MoE, hidden states include only earlier layers’ expert content, while deeper-layer signals reach shallow layers only as parameter updates during training.The gradient channel reaches layer-private parameters rather than providing activation-level content in the shallow layer’s readout.Source: §4.1 · p. 8
4.2 MOE WITH META EXPERTS.
- A layer’s shared-pool gradient contribution contains contributions from every layer, unlike layer-wise MoE.The shared-parameter influence is analyzed through local first-order changes to a layer’s readout accumulated over training; an analogous router-update term is omitted because top-k selection is piecewise constant in R.Source: §4.2 · p. 9
- A single forward pass with Meta Experts gives hidden states bidirectional access to expert content across depth through one shared storage.Deep hidden states can reach shallow expert content, and shallow hidden states can reach deep expert content.Source: §4.2 · p. 9
5 CONCLUSION
The paper compares inter-layer shared Meta Experts with hierarchical Local Experts empirically and theoretically. It reports reduced redundancy and improved loss and accuracy, alongside more flexible residual pathways; it identifies further scaling and architecture questions for future work.
5 CONCLUSION
- At fixed activated-expert count and expert granularity, Meta Expert models match Vanilla MoE performance with only 70–80% of its expert budget.The comparisons also report lower expert redundancy and cross-entropy loss, with higher downstream task accuracy.Source: §5 · p. 9
- A shared expert block widens forward residual connections and gives hidden states bidirectional access to expert content across depth.The paper presents this as improving both storage and retrieval of information.Source: §5 · p. 9
- The authors identify scaling to larger models and broader architectures, and explaining redundancy in mildly hierarchical shared experts, as future directions.They note that mildly hierarchical shared experts introduce redundancy without degrading performance.Source: §5 · p. 10
- The paper reports that its scaling-ladder configurations and compute-versus-loss evaluation protocol are documented, and that its models use Megatron-LM.The protocol averages language-modeling loss over the final ten training steps.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 RELATED WORK
Prior work shares parameters across depth in recurrent blocks, attention, and feed-forward networks, or shares activations through KV caches. Meta Experts instead share MoE experts during training, with flexible access topologies.
A.1 RELATED WORK
- Looped Transformers reuse the same block across multiple recursions, whereas Meta Experts reuse only the expert block across layers within one feed-forward pass.The paper contrasts this with additional recursions and greater serialization in looped models; Meta Experts retain standard Transformer parallelism and may keep attention layer-specific.Source: §A.1 · pp. 15–16
- Attention weight sharing reuses projections or blocks, while KV-cache sharing reuses activations at deployment rather than learned weights.The paper says KV-cache sharing trades representational independence for inference memory and throughput; it does not establish that layers should store the same content.Source: §A.1 · p. 16
- Existing FFN-sharing studies focus on dense FFNs, report mixed results, and typically share uniformly across layers [Lan et al., 2019; Thawakar et al., 2024].The paper identifies FFN Weight Sharing as the closest line of work.Source: §A.1 · p. 16
- Meta Experts generalize MoE weight sharing with flexible topologies, including local boundary layers or chunked and interleaved access to shared sub-pools.The paper says these arrangements are more friendly to communication and memory than the every-layer sharing topology common in recent MoE weight-sharing work.Source: §A.1 · p. 16
A.2 REDUNDANCY METRIC
The redundancy metric builds pairwise expert similarity from linear CKA on centered response covariances, then measures concentration in the similarity matrix’s spectrum with a Gini coefficient. Its ratio form is intended to compare redundancy across models of different sizes and depths.
A.2 REDUNDANCY METRIC
- The metric uses linear CKA on centered expert-response covariances to construct a pairwise similarity Gram matrix.Expert responses come from a shared probe set, and the matrix covers the experts under analysis.Source: §A.2 · p. 16
- The Gini coefficient of the similarity matrix’s eigenspectrum measures how concentrated similarity energy is across principal directions.The paper describes the matrix as positive semi-definite, with eigenvalues distributing total similarity energy across directions.Source: §A.2 · p. 17
- The resulting Expert Redundancy metric ρ ranges from zero for uniform energy and complementary experts toward one for concentrated energy and strong redundancy.For a uniform rank-k spectrum, ρ equals 1 − k/E, the fraction of expert directions collapsed into redundant ones; the ratio is comparable across models with different expert counts, depths, and training budgets.Source: §A.2 · p. 17
A.3 SCALING-LADDER EXPERIMENTAL SETTINGS
The scaling ladder compares Meta Expert variants across four depths while holding each rung’s backbone and compute configuration fixed. It varies expert allocation and sharing topology, then tests how far shared Meta capacity can be reduced without changing activated compute.
A.3 SCALING-LADDER EXPERIMENTAL SETTINGS
- Across four depths, each variant shares its rung’s backbone and training configuration with the vanilla baseline, with top-8 routing and a 128 × L routed-expert budget.The ladder uses Qwen3.5-style text-only MoE models; width is scaled jointly with depth, and each variant is trained independently from scratch.Source: §A.3 · pp. 17–18, Table 2 · p. 17
- The architecture uses a period-4 pattern of three GDN layers followed by one full-attention layer, with 128 routed experts and one gated shared expert per block.The full-attention layers use grouped-query attention with head dimension 256; GDN layers use key/value head dimension 128 and kernel width 4.Source: §A.3 · p. 17, Table 2 · p. 17
- The variants vary Meta share from 25% to 75%, boundary-layer Local-only regions, and the number of interleaved Meta sub-pools; a pure-Meta variant is also included.Every variant retains the 128 × L routed-expert budget and top-8 activation; the combined F2L2+Loop2 configuration applies both topology changes.Source: §A.3 · pp. 18–19, Table 4 · p. 19
- The param-reduction ablation shrinks Meta width from the F2L2+Loop2 r75 setting without refilling Local experts, reducing stored parameters while keeping top-8 activation and training FLOPs fixed.The sweep runs at all four depths, with Local width held at 32 and four Meta plus four Local experts routed per token.Source: §A.3 · p. 19, Table 5 · p. 20
A.4 DISCUSSION FOR THE RESIDUAL ANALYSIS.
The shared-pool pathway has constant hop length across layer distance, unlike the layer-wise routes through successive Jacobians. This provides access to shared content, but does not guarantee that signals avoid vanishing, and it does not change within-pass causal direction.
A.4 DISCUSSION FOR THE RESIDUAL ANALYSIS.
- The shared-pool route has O(1) length regardless of layer distance, while layer-wise forward carrying and the gradient highway pass through |m − l| Jacobians.The effective local gain still depends on the spectrum of the cross-layer tangent factor, so the analysis characterizes access and path length rather than guaranteeing against vanishing signals.Source: §A.4 · p. 20
- A shallow layer cannot read deeper-layer activations in the same forward pass; it can access deeper content written into the shared pool during training.The shared pool acts as a carrier of information across depth in a common code space.Source: §A.4 · p. 20
- Layer-wise gradient reweighting gates the backward pathway by controlling how much each layer’s write influences other layers’ reads.The write is ∆θ(t), and its effect on other layers’ reads passes through ∆y(t).Source: §A.4 · p. 20