PaperlayerView exact revision on arXiv

Source-linked AI summary

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Tianxiang Pan, Jiale Yan, Mo Guang, Kaiwen Long

arXiv:2604.23996v2cs.CV

TL;DR

MoE-VLM routing still lacks guidance for how evolving modality signals should shape expert specialization. SMOES uses layer-dependent soft modality scores, expert bins, and inter-bin mutual-information regularization; across four backbones, it improves multimodal and language-only results over soft routing.

  • Problem

    How modality-specific signals should guide expert routing in MoE-VLMs remains under-explored, while existing strategies rely on rigid or uncontrolled routing assumptions.

    Source: §1 · p. 1

  • Method

    SMOES combines dynamic soft modality scores, expert binning, and inter-bin mutual-information regularization to encourage coherent modality specialization.

    Source: §1 · p. 2

  • Results

    SMOES improves over the soft-routing baseline by 2.2% average, including 0.9% on multimodal tasks and 4.2% on language-only tasks.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    Experiments across four MoE backbones and 16 benchmarks validate consistent task-accuracy improvements alongside reduced communication overhead and increased throughput in expert-parallel deployments.

    Source: §1 · p. 2, §5 · p. 8

    The simplified Gaussian distribution may miss nonlinear dependencies and multimodal structures in fused representations.

    Source: §2.2 · p. 14

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency.

1. Introduction

SMoE-VLM routing must account for modality signals that evolve across layers, rather than relying on fixed separation or uncontrolled mixing. SMOES combines layer-wise soft modality scores, expert binning, and mutual-information regularization to guide specialization and deployment.

1. Introduction

  • MoE-VLM routing remains under-explored in how modality-specific signals should guide experts, while hard and soft routing impose rigid separation or uncontrolled mixing.Existing strategies offer little guidance for how specialization should evolve with network depth.Source: §1 · p. 1, Figure 1 · p. 1
  • Fusion varies across models, layers, and tokens, with no clear boundary between modality-specific and cross-modal states.Text forms several semantic clusters, while vision tokens roughly separate into content and padding clusters; JS-divergence trajectories also vary.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • Modality-agnostic routing can scatter tokens across expert-parallel devices, inflating communication overhead amid sharp differences in token quantity and information density.The passage identifies vision tokens as numerous but lower-density, while text tokens are fewer and more semantically concentrated.Source: §1 · p. 2
  • SMOES estimates dynamic soft modality scores from attention or Gaussian statistics, bins experts into coherent groups, and applies mutual information regularization to encourage distinct modality preferences.The bins also provide natural device-placement units for modality-aware expert-parallel deployment.Source: §1 · p. 2
  • Experiments across four MoE-VLM backbones and 16 benchmarks demonstrate accuracy and efficiency improvements, including reduced expert-parallel communication overhead.The contribution summary reports state-of-the-art performance in accuracy and efficiency.Source: §1 · p. 2

2. Related Works

Prior work studies sparse expert routing, modality specialization, and distributed efficiency, but leaves modality-aware expert partitioning and communication under MoE-VLM fusion dynamics open.

2.1. Mixture-of-Experts

  • MoE research addresses sparse conditional computation, routing efficiency, load balancing, communication, and mutual-information constraints for expert-task or expert-token alignment.The cited MI approaches do not address modality differentiation in multimodal models.Source: §2.1 · p. 2

2.2. Modality Specialization in MoE-based VLMs

  • Modality routing ranges from hard assignment and manually partitioned hybrid routing to unguided soft mixing and auxiliary-objective approaches.SMOES instead combines mutual-information constraints, layer-adaptive soft modality scores, and expert binning for distributed deployment.Source: §2.2 · pp. 2–3, Figure 3 · p. 3

2.3. Efficient Deployment of MoE Models

  • Distributed MoE work targets load imbalance and All-to-All overhead, but modality-fusion characteristics remain underexplored for guiding expert partitioning and communication in MoE-VLM deployment.Prior strategies include predefined balanced groups, heterogeneous expert sizes, token dropping, and asynchronous queuing with re-batching.Source: §2.3 · p. 3

3. Method

SMoES combines soft token-level modality scores, modality-aligned expert bins, and inter-bin mutual information regularization to guide specialization and support expert-parallel deployment.

3.1. Preliminaries: MoE in VLM

  • Each MoE layer routes token features to experts through top-k gating, with an auxiliary loss used to prevent routing collapse.Source: §3.1 · p. 3

3.2. Soft Modality Scores

  • SMoES uses soft token scores to track layer-dependent modality fusion, with attention-accumulated and Gaussian-statistics estimators capturing local interactions and global regularities.Source: §3.2 · p. 3

3.2.1. Attention-Accumulated Score

  • Attention-accumulated scores propagate modality evidence through attended tokens and combine it with the input score using a residual-weighted update.The attention matrix is averaged across heads, and feature norms weight the attention and residual contributions.Source: §3.2.1 · p. 4

3.2.2. Gaussian-Statistics Score

  • Gaussian-statistics scores infer token modality from log-likelihoods under per-layer modality distributions, then convert them to soft scores with a temperature-scaled softmax.The modality distributions are updated from token batches using an EMA variant of Welford’s algorithm.Source: §3.2.2 · p. 4

3.3. Expert Binning

  • Experts are sorted by EMA-derived modality preferences into bins that can match the device count, enabling modality-aligned placement and reduced communication.The text-bias score is based on each expert’s routed text and vision token loads.Source: §3.3 · p. 4

3.4. Inter-Bin Mutual Information Regularization

  • Maximizing mutual information between modality and bin encourages distinct modality preferences, while bin-level placement aligns specialization with devices and reduces cross-device communication.Source: §3.4 · p. 5

3.5. Training Objective and Implementation Details

  • The training objective combines language-modeling loss, per-bin load balancing, and inter-bin mutual-information regularization.The reported setup uses eight A800 GPUs and eight bins, with EMA decay β = 0.99 for Gaussian updates and momentum-adaptive binning.Source: §3.5 · p. 5

4. Results

SMOES improves results across multimodal and language-only tasks, while its ablations and visualizations examine how soft modality scores, expert binning, and inter-bin specialization shape routing. Its expert-parallel evaluation also assesses communication and speed.

4.1. Experimental Setup

  • Evaluation uses four MoE backbones, four routing baselines, and 10 multimodal plus 6 language benchmarks under a shared finetuning protocol.The models use a CLIP ViT-L/14 vision encoder and a 2-layer MLP projector.Source: §4.1 · p. 5

4.2. Main Results

  • Across four backbones, SMOES improves over the soft routing baseline by 2.2% on average, with gains of 0.9% on multimodal tasks and 4.2% on language-only tasks.Source: §4.2 · p. 5
  • Hard routing loses 4.4% on multimodal and 22.2% on language-only tasks; hybrid MoIIE loses 3.1% and 17.4%, while SMAR loses 1.0% and 15.1%, respectively.SMAR disables load balancing in its final model, which the authors identify as an incompatibility with its KL regularization.Source: §4.2 · p. 6

4.3.1. Modality Score and Specialization Objective

  • Attention-soft and Gaussian-soft scores outperform hard scores, while MI-based specialization with expert binning adds +3.5% and +2.4% for the respective soft variants.Hard scores with MI-based specialization also exceed the best SMAR configuration.Source: §4.3.1 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.3.2. Expert Binning Strategy and Granularity

  • Adaptive modality-aware binning consistently outperforms fixed expert-order binning; too many bins risk deployment imbalance, while too few weaken specialization.Figure 5 presents the ablation on the number of expert bins.Source: §4.3.2 · p. 7, Table 4 · p. 7, Figure 5 · p. 7

4.4. Visualization and Analysis

  • SMOES shows sharper modality separation in early layers and more balanced, fusion-aware distributions in deeper layers.Figure 6 depicts token routing distributions for the soft baseline and SMOES.Source: §4.4 · p. 7, Figure 6 · p. 7
  • Both soft-score estimators show increasing fusion toward deeper layers; attention-soft is text-biased, while Gaussian-soft is vision-biased in high-fusion regimes.Figure 7 shows specialization evolving during training, and Figure 8 plots soft modality scores across layers.Source: §4.4 · pp. 7–8, Figure 7 · p. 8, Figure 8 · p. 8

4.5. Efficiency Analysis

  • In expert-parallel deployment, specialization reduces cross-GPU token transfers and improves TTFT and TPOT through less communication and overlapping computation with communication.Table 5 reports transfer ratios at prefill and decode; Table 6 reports TTFT and TPOT speed improvements, with larger prefill gains at larger batch sizes and stable decode improvement ratios.Source: §4.5 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion

SMOES combines dynamic soft modality scores, expert binning, and inter-bin mutual-information regularization to specialize experts in MoE-VLMs. Experiments across four backbones and 16 benchmarks show accuracy and deployment-efficiency improvements, while the Gaussian estimator's simple density model remains a limitation.

5. Conclusion

  • SMOES combines layer-dependent soft modality scores, deployment-aligned expert bins, and inter-bin mutual-information regularization to encourage coherent modality specialization.The modality scores use attention-based or Gaussian-statistics estimators.Source: §5 · p. 8
  • Experiments across four MoE backbones and 16 benchmarks show consistent task-accuracy improvements, reduced communication overhead, and increased throughput in expert-parallel deployment.Source: §5 · p. 8
  • The Gaussian-soft estimator uses a unimodal Gaussian with diagonal covariance for efficiency; richer density models remain an open problem.The authors report preliminary exploration of GMM in supplementary material.Source: §5 · p. 8

1. Detailed Experimental Setup

The experiments use a shared vision front end with four varied MoE backbones, a two-stage LLaVA training protocol, and an implementation of attention-score aggregation compatible with efficient attention kernels.

1.1. Vision Encoder and Projector

  • All configurations use CLIP ViT-L/14 at 336×336, yielding 576 visual tokens, and a two-layer GELU MLP projector to map 1024-dimensional features to each backbone’s hidden size.The shared encoder and projector keep backbone and routing changes as the experimental variables.Source: §1.1 · p. 12

1.2. MoE Backbone Architectures

  • The four backbones span DeepSeekMoE, OLMoE, Moonlight-MoE, and Qwen3-MoE, varying in router, expert, attention designs, and scale from 7B to 30B parameters.DeepSeekMoE has shared experts and top-6 routing; OLMoE has 64 experts with top-8 routing; Moonlight-MoE uses sigmoid routing with score bias; Qwen3-MoE has 128 experts and top-8 routing.Source: §1.2 · p. 12

1.3. Training Configuration

  • Training follows LLaVA’s two stages: projector-only feature alignment on 558K image-caption pairs, then instruction tuning on LLaVA-Instruct-665K with the vision encoder frozen.Experiments use AdamW; default method-specific settings include 8 bins, αbal = 0.001, αMI = 0.0001, EMA decay β = 0.99, and Gaussian temperature τ = 0.5 · D.Source: §1.3 · p. 12, Table S1 · p. 12

1.4. Efficient Implementation of SMoES

  • Attention-accumulated modality scores are concatenated to value vectors so efficient attention kernels can aggregate them implicitly without materializing the full attention matrix.The implementation preserves aggregation semantics without additional memory or computational overhead.Source: §1.4 · pp. 12–13

2. Supplementary Results

Supplementary experiments report consistent routing gains across additional backbones, probe estimator and specialization choices, and examine deployment efficiency and learned routing patterns.

2.1. Main results on Moonlight-MoE and Qwen3-MoE

  • SMoES consistently outperforms routing baselines on Moonlight-MoE and Qwen3-MoE; on Moonlight-MoE, MSI rises to 0.442 from 0.380 with vanilla soft routing.The authors attribute Moonlight-MoE’s moderate MSI improvement partly to architectural and pretraining characteristics that already foster modality differentiation.Source: §2.1 · p. 13, Table S4 · p. 14

2.2. Gaussian Estimator

  • A two-component Gaussian mixture yields measurable gains on OLMoE, while the authors suggest simple Gaussian estimation can miss complex or heterogeneous token distributions.Full covariance could capture cross-dimension correlations but would introduce O(D^2) parameters and online update and likelihood-computation overhead.Source: §2.2 · pp. 14–15, Table S5 · p. 14

2.3. Ablation study

  • Attention-soft and gaussian-soft scores outperform hardcoded 0/1 modality scores on all four backbones, and MI-based differentiation is more effective than KL-based approaches in this ablation.The inter-bin MI objective improves specialization and model performance; KL-based differentiation can raise specialization yet degrade performance.Source: Table S3 · p. 13, §2.3 · pp. 15–16, Table S6 · p. 15
  • Adaptive binning outperforms fixed binning in model performance, while 0.5D Gaussian temperature balances score confidence and smoothness; larger αMI may improve performance but risks instability.The authors choose αMI = 1e−4 for stable gains and αbal = 1e−3 for stable performance across models, although preferred balance-loss strength varies by backbone.Source: Table S7 · p. 15, §2.3 · p. 16, Table S8 · p. 16

2.4. Efficiency Analysis

  • SMoES reduces cross-GPU EP communication overhead in most cases, especially for vision tokens during prefill, and its latency gains mainly come from lower communication time and overlapping communication with expert computation.The prefill improvement grows with batch size as communication takes a larger share of time; decode improvement remains constant at small batch sizes.Source: Table S10 · p. 17, §2.4 · pp. 18–19, Table S11 · p. 18

2.4.1. Larger-Scale EP

  • In a 16-GPU, two-node Qwen3-MoE setup, SMoES reduces inference time by 5.6% versus the baseline, even without deep deployment optimization.The authors frame this modality-based arrangement as demonstrating potential rather than solving cloud-scale scheduling.Source: Table S12 · p. 19, §2.4.1 · p. 19

2.4.2. Training Overhead

  • SMoES adds no inference overhead and leaves TFLOPS and peak GPU memory almost unchanged, though it slightly increases training time.The Gaussian Estimator adds an approximately 60M buffer and contributes more training overhead through global feature synchronization; this becomes less noticeable as models scale.Source: §2.4.2 · p. 19, Table S13 · p. 19

2.5. Visualizations

  • SMoES preserves distinct vision- and text-specialized experts alongside mixed-modal experts, while specialization generally decreases with depth as modality representations fuse.Shallow-layer scores tend toward pure-modality extremes, but fusion trajectories vary by backbone; DeepSeekMoE and Qwen3-MoE often show multiple vision-token peaks, while Moonlight-MoE shows one.Source: §2.5 · pp. 19–20, Figure S5 · p. 23
Loading 2604.23996v2…