PaperlayerView exact revision on arXiv

Source-linked AI summary

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu

arXiv:2610.07809v1cs.LG

TL;DR

Dense-to-MoE upcycling commonly builds routed experts by adapting copies of pretrained FFNs. MASKerade instead jointly learns token-level routing and binary masks over frozen FFN weights; it achieves the highest reported performance among compared methods across five benchmarks, supporting mask learning as an alternative to independent expert weights.

  • Problem

    Dense-to-MoE upcycling often constructs experts by copying pretrained FFNs, while whether experts can instead be learned through connectivity selection remains an open question.

    Source: §1 · p. 1

  • Method

    MASKerade jointly learns binary expert masks and a token-level router over frozen pretrained FFN weights, with prescribed sparsity constraints.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    MASKerade achieves the highest reported performance among external methods across five benchmarks.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The experiments and controlled ablations support learning expert connectivity over frozen weights as an effective alternative to updating independent expert weights.

    Source: §5 · p. 9

    The pretrained parameters remain frozen throughout upcycling.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.

1 INTRODUCTION

Dense-to-MoE upcycling reuses pretrained FFNs, but existing approaches typically adapt expert weights or construct experts by partitioning or pruning. MASKerade instead learns routed binary masks over frozen FFN weights, and the paper reports strong benchmark performance without updating those weights.

1 INTRODUCTION

  • MoE layers select a small subset of experts per input, while dense-to-MoE upcycling converts pretrained FFNs into routed experts.This reuses investment in dense pretraining.Source: §1 · p. 1
  • Prior upcycling approaches copy FFNs and learn separate weights, partition or prune FFNs, or add expert-specific adapters and low-rank updates.Some copied experts are partially reinitialized to encourage differentiation.Source: §1 · p. 1
  • MASKerade treats expert identity as learned connectivity over shared weights, allowing several masks to define alternative sparse computations.This replaces expert-specific weight adaptation with expert-specific connectivity learning.Source: §1 · p. 1
  • MASKerade achieves the highest reported performance among external methods across five benchmarks.The experiments evaluate multimodal instruction tuning with Qwen and Gemma backbones.Source: §1 · p. 2
  • MASKerade learns token-routed binary masks over frozen pretrained FFNs instead of updating independent expert weights.The method jointly optimizes expert masks and token routing under prescribed sparsity constraints.Source: §1 · p. 2
  • The method accommodates neuron-structured, semi-structured, and unstructured masks.The experiments and analysis support dense-to-MoE adaptation without updating pretrained FFN weights.Source: §1 · p. 2

2 RELATED WORK

Prior work spans routed MoE conversion, learned sparse subnetworks, and mask-based or interpolated experts. MASKerade distinguishes itself by token-routing hard-masked versions of the frozen pretrained FFN itself.

2 RELATED WORK

  • Dense-to-MoE methods initialize experts from dense checkpoints, while MASKerade learns masks directly on frozen FFN tensors rather than fine-tuning separate copies.Related approaches include Sparse Upcycling, Drop-Upcycling, MoEfication, LLaMA-MoE, DIVE, and ToMoE.Source: §2 · p. 2
  • Sparse-subnetwork methods learn or select connectivity, but MASKerade maintains multiple masks and selects among them conditionally for each token.The cited approaches include supermasks, Wanda, SparseLLM, MaskLLM, MFT, and learned masks for pretrained vision-language models.Source: §2 · p. 2
  • Mask-based and interpolated expert methods vary in training and routing, whereas MASKerade routes tokens through hard-masked pretrained FFNs without a trainable shared delta.SIMoE applies structured masks to a shared trainable parameter delta and merges experts at the instance level.Source: §2 · p. 2

3 METHOD

MASKerade builds routed experts from learned masks over a frozen pretrained FFN. The method combines token-level top-k routing, several mask granularities, and joint mask-and-router training.

3.1 EXPERTS AS FROZEN-WEIGHT SUBNETWORKS

  • Each expert applies a binary mask to a frozen pretrained FFN, changing its connectivity without learning new retained weight values.The method covers the gated FFNs’ gate, up, and down projections.Source: §3.1 · p. 3

3.2 TOKEN-LEVEL ROUTING

  • A learned linear router maps each token’s hidden state to probabilities, selects experts by deterministic top-k, and combines their outputs.For k ≥2, the selected experts’ coefficients sum to one.Source: §3.2 · p. 3

3.3 MASK PARAMETERIZATION AND GRANULARITY

  • Masks are binary selections from trainable scores and support neuron-structured, semi-structured N:M, and unstructured projection-level sparsity.N:M masks retain the N largest score magnitudes in each contiguous M-entry group; unstructured masks retain q = ⌈ρP⌉ scores across a projection tensor.Source: §3.3 · pp. 3–4

3.4 JOINT MASK AND ROUTER TRAINING

  • Importance initialization produces substantially lower initial loss and lower training loss near the end of the run for 2:4 masks on Qwen3-1.7B.The authors adopt importance initialization as the default for subsequent experiments.Source: Figure 2 · p. 4, §3.4 · p. 4
  • Mask scores and routers are learned jointly using supervised autoregressive loss and a load-balancing term, with hard masks in the forward pass and straight-through gradients.For neuron-structured masks, gradients from incident connections accumulate into shared neuron scores.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate MASKerade for multimodal instruction tuning and report the highest performance among compared methods on five benchmarks.

4.1 EXPERIMENTAL SETUP

  • Experiments use Qwen3 and Gemma vision-language backbones, a 665K instruction mixture, and five benchmarks; alternating FFN layers receive four mask experts with top-2 routing.The converted layers are even-indexed; most experiments use three random seeds and report means and sample standard deviations.Source: §4 · p. 5, §4.1 · p. 6

4.2 MAIN RESULTS

  • MASKerade achieves the highest reported performance among compared methods on all five benchmarks, with nominal activated-parameter counts matching Dense MLP and below most baselines.Activated parameters count overlapping masked weights separately, rather than unique weights, exact FLOPs, or measured latency.Source: §4.2 · p. 6
  • Learned 2:4 masks achieve higher reported means than random, magnitude-based, and Wanda-based fixed masks on all five benchmarks; unstructured and semi-structured masks outperform neuron-structured masks.Because NVIDIA GPUs support accelerated 2:4 sparsity, the paper uses 2:4 masks as its primary configuration.Source: Table 3 · p. 6, §4.2 · p. 7

4.3 ABLATIONS AND ANALYSIS

  • Jointly learning masks and routers improves reported means over frozen-random routers, while random-router replacement and label-permuted mask selection reduce performance.Routing frequencies differ by training source in individual layers, while the all-layer mean is closer to balanced usage and obscures larger layer-local contrasts.Source: §4.3 · pp. 7, 9, Table 4 · p. 7, Figure 4 · p. 9
  • Importance initialization scores highest across all three mask families, while retention changes which weight-level granularity performs best on Qwen3-1.7B.Importance initialization is particularly advantageous over random initialization for 2:4 masks; at 25% retention unstructured masks lead on most benchmarks, whereas at 50% and 75% results are split between unstructured and semi-structured masks.Source: §4.3 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 7
  • At the same nominal budget, four-expert top-1 with 2:4 masks improves all five benchmarks over a single learned 2:4 mask; 2:4/top-1 also outperforms 1:4/top-2.Both low-budget variants use half the nominal arithmetic of one dense pass; default 2:4/top-2 improves further while increasing nominal activated parameters from 1.46B to 1.72B.Source: §4.3 · p. 8, Table 7 · p. 8
  • MASKerade latency relative to Vanilla is 0.93× and 0.99× at Qwen3-1.7B and Gemma3-1B shapes, and 0.79× and 0.75× at Qwen3.6-27B and Gemma4-31B shapes.The standalone SwiGLU microbenchmark used bfloat16, H100, CUDA-graph replay, balanced synthetic routing, and included router, dispatch, expert, and combination operations.Source: §4.3 · p. 8, Table 8 · p. 8
  • Learned masks retain overlapping but nonidentical connectivity, and semi-structured and unstructured masks have an all-four-expert shared core exceeding half their retained entries.Neuron-structured masks have a smaller all-four intersection and larger partially shared component; all mask families contain both shared and expert-specific connections.Source: §4.3 · pp. 8–9, Figure 3 · p. 9

5 CONCLUSION

The conclusion presents MASKerade as token-routed mask experts over frozen pretrained FFNs, with experiments and ablations supporting connectivity learning as an alternative to updating independent expert weights.

5 CONCLUSION

  • MASKerade learns token-routed mask experts over frozen pretrained FFN weights.Source: §5 · p. 9
  • The learned masks contain a shared connectivity core alongside expert-specific connections, and routing adapts their use to individual tokens.Source: §5 · p. 9
  • Future work will investigate kernels that reuse overlapping computations while preserving expert-specific nonlinear transformations, and assess end-to-end latency and memory trade-offs.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The default setup converts alternating FFNs into four 2:4 mask experts with top-2 routing while keeping pretrained language-model weights frozen.

A IMPLEMENTATION DETAILS

B.1 SCORE INITIALIZATION

Initialization effects vary by mask granularity and backbone, while importance initialization leads across Gemma3-1B mask families. Other analyses compare activation budgets, mask structure, routing choices, weight updates, and layer placement.

Initialization dynamics

  • Importance initialization lowers initial loss for unstructured and 2:4 masks on both backbones, but Gemma3-1B random initialization remains lower near the end of training.The unstructured gap narrows, 2:4 importance masks retain lower loss with fewer large fluctuations, and neuron-structured curves are closer on Qwen3-1.7B.Source: §B.1 · p. 14, Figure 5 · p. 15, Figure 6 · p. 15

B.1 SCORE INITIALIZATION

  • Importance initialization has the highest reported score on all five Gemma3-1B benchmarks within each mask family, while permuting importance scores reduces performance in every family.Its advantage over random initialization is particularly pronounced for 2:4 masks on GQA and TextVQA.Source: §B.1 · pp. 14–15, Table 10 · p. 16

Matched activation budgets

  • At 0.84B nominal activated parameters on Gemma3-1B, 1:4/top-2 exceeds the single 2:4 mask and all four external baselines on all five benchmarks.The 2:4/top-1 variant exceeds the single mask on four benchmarks, with a slightly lower GQA mean (58.24 versus 58.30).Source: §B.1 · p. 15, Table 12 · p. 16

Structure and functional diversity

  • Gemma3-1B learned 2:4 masks have pairwise Jaccard overlap of 0.489–0.492, and the shared all-four core is smaller for neuron-structured masks than for weight-level families.Shared, partially shared, and expert-specific connections coexist; distinct outputs and replacement interventions separately assess functional differences and assignment usefulness.Source: §B.1 · pp. 15–18, Figure 7 · p. 17

Updating shared FFN weights

  • Updating shared FFN weights improves all five Gemma3-1B benchmarks in both configurations and four Qwen3-1.7B benchmarks for 4e/top-2; frozen multi-expert MASKerade exceeds Wanda prune-then-tune on both backbones.For Qwen3-1.7B single-mask configuration, weight updates improve all five benchmarks.Source: §B.1 · p. 16, Table 13 · p. 18

Number of activated experts

  • On Gemma3-1B, top-2 leads on four benchmarks and top-3 gives a slightly higher MME score; activating all four experts lowers every score relative to top-2.Across backbones, no top-k setting leads uniformly; for Qwen3-1.7B, all four experts lead on GQA, SQA, and TextVQA, while top-2 leads on MME and POPE.Source: §B.1 · p. 16, Table 14 · p. 18, Table 16 · p. 18

Expert bank and layer placement

  • Increasing the 2:4 expert bank does not yield uniform gains across backbones or tasks, and converting every FFN raises GQA but lowers the other four scores relative to even-layer conversion.Qwen3-1.7B improves on all five benchmarks from one to four experts; eight experts further improve MME and POPE but reduce the other three scores.Source: §B.1 · pp. 16–17, Table 15 · p. 18, Table 17 · p. 18

Training policy and sensitivity

  • Deterministic top-2 training has higher reported means than Gumbel-sampled second-expert selection on all five benchmarks for both backbones; no tested hyperparameter alternative improves every benchmark over the default.Both routing controls use deterministic evaluation; the higher 1 × 10^-3 learning rate lowers all five Gemma3-1B scores.Source: §B.1 · p. 18, Table 22 · p. 20, Table 21 · p. 20
Loading 2610.07809v1…