Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

1 Introduction

• MoE sparsity was introduced for computational efficiency, while consistent routing may also induce implicit modularity among jointly learned experts.The introduction asks whether this organization spans modality, domain, and visual concept in multimodal models.Source: §1 · p. 1
• Multimodal MoEs show specialization across modalities, semantic domains, and fine-grained visual concepts without explicit modularity training.Medical, math, and remote sensing imagery form distinct structures not explained by low-level properties such as color or spatial position.Source: §1 · p. 1
• ExpertLens identifies domain-specialized experts from pretrained router weights by decoding them into meaningful vocabulary tokens, without domain-specific examples.The method addresses how to identify experts specializing in a given domain.Source: §1 · p. 1
• Across math, medical, and remote sensing tasks, selective adaptation matches or surpasses full fine-tuning while updating 21.7–47.0% of parameters and averaging 4.0× training speedup.ExpertLens also outperforms LoRA in adaptation performance and training speed (2.9× faster).Source: §1 · p. 2

2 Related Work

• MoEs scale capacity by routing inputs through sparse subsets of experts, from early conditional computation to large-scale Transformers and multimodal foundation models.The passage traces MoE use across language and multimodal models.Source: §2 · p. 2
• Representation-interpretability methods include Logit Lens, which applies the output head to intermediate residual-stream representations to expose how predictions evolve across layers.The passage relates this approach to the residual stream’s approximately linear, additive structure and vocabulary-aligned MLP updates.Source: §2 · p. 2
• Prior MoE interpretability work studies expert importance, routing interventions, and output-space decoding; ExpertLens extends this with data-free semantic decoding for domain-specific adaptation.The related work describes MoE interpretability as comparatively underexplored.Source: §2 · p. 2
• Text-only MoEs have shown low-level expert structure, while multimodal organization remains less explored; this work finds structure spanning modalities, domains, and visual concepts.Prior work includes specialization by token type and part of speech, while EMO modifies pretraining to encourage domain-level modularity.Source: §2 · p. 2
• LoRA adds trainable low-rank parameters, whereas ExpertLens selectively updates experts already relevant to the target domain and outperforms LoRA in performance and training time.The comparison reports higher adaptation performance and more substantially reduced training time for ExpertLens.Source: §2 · p. 2
• Figure 1 shows experts rarely balance image and text activations; QWEN3-VL-30B-A3B’s text-share distribution is bimodal (dip = 0.0136, p < 0.001).The figure reports each expert’s image-versus-text share among top-K activations and the distribution of text-token activation share.Source: Figure 1 · p. 3

3.1 Expert Specialization by Modality

• Across layers, experts rarely balance image and text activations; QWEN3-VL-30B-A3B shows bimodal specialization, while KIMI-VL-A3B is more nearly unimodal.In QWEN3-VL-30B-A3B, the two modes contain 462 image-leaning experts and 433 text-leaning experts; GEMMA-4 26B-A4B is similarly bimodal.Source: §3.1 · pp. 3–4

3.2 Domain Specificity in Vision and Text Experts

• Image activations cluster more tightly by domain than text, and image patches align best with semantic domain while text tokens align best with part of speech.The pattern holds across layers and both QWEN3-VL-30B-A3B and KIMI-VL-A3B; image organization is not explained by color or spatial position.Source: Figure 2 · p. 4, §3.2 · p. 4, Figure 3 · p. 5

3.3 Visual Expert Specialization by Object Semantics

• Semantically related object classes recruit more similar expert sets, with overlap decreasing as semantic distance grows and related pairs remaining above unrelated-class baselines despite visual variation.Several pairs exceed Z(a, b) > 3 in intermediate layers.Source: Figure 4 · p. 5, §3.3 · pp. 5–6

3.4 ExpertLens: Identifying Domain-Specialized Experts

• ExpertLens decodes router weights into vocabulary-token semantics and identifies domain cohorts that preferentially activate on matching datasets, with weaker math alignment on text tokens.The domain correspondence also holds across layers, and representative experts’ decoded tokens align with their most activating image domains.Source: §3.4 · pp. 6–7
• ExpertLens matches or surpasses FFT while outperforming LoRA, with a 4.0× average speedup over FFT and 2.9× over LoRA.Figure 6 compares performance and training time during adaptation.Source: Figure 6 · p. 7

4 Domain-Specialized Experts for Efficient Adaptation

• The sequential protocol adapts one model across math, medical, and remote sensing, training selected experts and shared decoder parameters while freezing other expert MLPs.Training uses 4,000 examples per domain, 200 steps per domain, and three seeds.Source: §4 · p. 7

4.2 The Importance of Expert Selection

• Against same-budget Random (full), ExpertLens scores 64.6% vs 60.8% on math, 53.7% vs 52.0% on medical, and 51.9% vs 51.3% on remote sensing; Random (disjoint) matches math but trails on the other domains.Source: Figure 7 · p. 8, §4.2 · p. 8

4.1 ExpertLens for Efficient Multimodal Adaptation

• 64.6% vs 63.8% on math, 53.7% vs 53.6% on medical, and 51.9% vs 51.6% on remote sensing; ExpertLens matches or surpasses FFT while updating 21.7–47.0% of model parameters.Source: §4.1 · p. 8
• 3.2× on math, 4.0× on medical, and 4.8× on remote sensing speedups over FFT yield a 4.0× average; ExpertLens is 2.9× faster than LoRA.Source: §4.1 · p. 8

4.2 The Importance of Expert Selection

• ExpertLens reaches comparable final accuracy to Frequency Selection and the privileged oracle without requiring target-domain examples or future-domain data.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Ablations and Analyses

• Adding the router improves math by 11.2 points; QKV projections and the language-model head add gains across domains, and the full configuration performs best.Source: Table 2 · p. 9, §4.3 · p. 9
• ExpertLens exceeds FFT by 1.7, 2.0, and 0.2 percentage points across three domain orders, while reducing across-order standard deviation from 1.37 to 0.78 points (43%).Source: Table 3 · p. 9, §4.3 · p. 9

4.4 Experiments on Kimi-VL

• On KIMI-VL-A3B, ExpertLens achieves 48.0% mean final accuracy versus 47.4% for FFT and 45.0% for random selection, with a 1.53× speedup over FFT.Source: Table 4 · p. 9, §4.4 · p. 9

• Multimodal MoEs develop semantic specialization across modalities, domains, and visual concepts, which can be exploited for efficient and effective adaptation.Source: §5 · p. 10
• ExpertLens uses the model’s existing organization by identifying and updating experts already specialized for the target domain.Source: §5 · p. 10
• The authors release the data, code, and trained models to encourage further progress and model development.Source: §5 · p. 10

• MoE models replace a transformer block’s dense feedforward network with a set of expert MLPs.Source: §A.1 · p. 14
• For each token, the router computes expert probabilities and selects the K experts with the largest probabilities.Source: §A.1 · p. 14
• The layer output is a weighted combination of the selected experts; some architectures also evaluate shared experts for every token.Source: §A.1 · p. 14
• Sparse routing evaluates only a small subset of expert parameters per token, increasing model capacity while improving throughput.Source: §A.1 · p. 14
• In multimodal MoEs, visual and textual tokens pass through MoE layers with routing performed independently for each token.Source: §A.1 · p. 14
• The paper studies whether multimodal MoEs develop semantic specialization and whether that structure can support efficient, targeted adaptation.Source: §A.1 · p. 14

• Section A.2.1 discusses Optimal Leaf Ordering and the datasets used for Figure 1.Source: §A.2 · p. 15
• Section A.2.2 provides additional details on the KNN purity analysis in Figure 3.Source: §A.2 · p. 15
• Section A.2.3 provides details on the semantic specialization analysis in Figure 4.Source: §A.2 · p. 15

A.2.1 Modality Specialization Addendum

• The modality-affinity analysis samples image patches and text tokens equally from datasets spanning medical, math, and other domains.A score near 0.5 indicates no strong preference; values approaching 1 or 0 indicate increasingly image- or text-specific activation, respectively.Source: §A.2.1 · p. 15
• Optimal leaf ordering arranges experts within each layer by router-weight similarity, without using modality-specific activation statistics.Pairwise cosine distances form a hierarchical clustering; OLO chooses a leaf ordering that minimizes adjacent-expert distances while preserving that clustering.Source: §A.2.1 · p. 15
• Experts with similar router weights tend to appear near one another along the x-axis, but their modality preferences are computed independently.OLO affects only the horizontal ordering of experts in Figure 1A.Source: §A.2.1 · p. 15

A.2.2 kNN Purity

• Higher kNN purity means nearby activation vectors share a label more often, indicating that the corresponding factor better explains local representation structure.The analysis compares dataset or part-of-speech labels for text, and dataset, color, or spatial-position labels for image patches.Source: §A.2.2 · p. 15
• The comparisons use image and text samples from 12 datasets each, with text part-of-speech labels drawn from 12 categories; sample counts are balanced across categories.Balancing prevents purity from being driven by differences in category frequency.Source: §A.2.2 · pp. 15–16
• For each activation, the method finds its k nearest neighbors in the original activation space using Euclidean distance and measures the fraction sharing its categorical label.The sample itself is excluded from its neighbor set.Source: §A.2.2 · p. 16

A.2.3 Semantic Specialization Analysis

• An expert is associated with a class if it activates for at least 60% of that class’s images, and the analysis repeats this test across five patch-count thresholds.For an image, an expert is active when routed at least m of its patch tokens; thresholds are m ∈ {3, 5, 8, 12, 16}.Source: §A.2.3 · p. 16
• The Jaccard similarity measures overlap between two classes’ associated expert sets; larger values mean they consistently recruit more of the same experts.Raw overlap is standardized because some classes may overlap broadly with many others.Source: §A.2.3 · p. 16
• A positive Z(a, b) means two classes share more experts with each other than they typically do with other classes.The standardization uses the typical pairwise overlap and the standard deviation of off-diagonal entries in the pairwise Jaccard matrix.Source: §A.2.3 · p. 16

A.3 ExpertLens Classification

• At SEL ≥0.9, ExpertLens selects 2,768 math, 1,103 medical, and 1,290 remote-sensing experts; token-level misclassifications can add noise to inferred labels.The threshold was used across domains rather than tuned separately; ambiguous tokens, fragments, punctuation, multilingual fragments, and control tokens can be mislabeled.Source: §A.3 · p. 17, Table 5 · p. 17
• ExpertLens-identified medical, math, and remote-sensing cohorts preferentially activate on matching-domain inputs across decoder layers, despite variation in preference strength.The analysis extends the layer-18 result across layers and both text and image inputs.Source: §A.3 · p. 17, Figure 9 · p. 18

A.4.1 Routing Sharpness Analysis

• Routing weights decline across selected experts rather than collapsing onto one; text routing is sharper than image routing, while profiles are similar across domains within each modality.No model has mean top-1 routing weight above 0.5, and even the lowest-ranked selected experts retain non-negligible routing mass.Source: §A.4.1 · p. 19, Figure 10 · p. 19

A.4.2 Additional Modality Specificity Analysis

• GEMMA-4 26B-A4B shows bimodal modality specialization, while KIMI-VL-A3B appears mostly neutral; modality preferences remain stable across image-scale settings.The reported Hartigan dip value for GEMMA-4 26B-A4B is 0.0214 (p < 0.001); the authors note KIMI-VL-A3B has 64 experts per layer versus 128 in the other models, possibly explaining its pattern.Source: §A.4.2 · pp. 19–20, Figure 11 · p. 20, Figure 12 · p. 20

A.4.3 Additional kNN Purity Analysis

• With three domains and two datasets per domain, domain grouping is most kNN-pure for image activations, while POS narrowly exceeds domain grouping for text.Both groupings exceed chance; purity values are higher than in the prior analysis because there are fewer classes.Source: §A.4.3 · p. 20, Figure 13 · p. 21

A.4.4 Additional Semantic Specialization Visualizations.

• For GEMMA-4 26B-A4B and KIMI-VL-A3B, semantic distance correlates negatively with expert overlap (−0.343 and −0.349); related class pairs share more experts than unrelated pairs.The reported QWEN3-VL-30B-A3B coefficient is −0.380.Source: §A.4.4 · p. 21, Figure 14 · p. 22

A.4.5 Routing Stability Analysis

• After three training stages, the final router retains 90.1% top-8 expert-set overlap with the pretrained router, with similarly high retention across the three domains.Consecutive checkpoints preserve 92.0–94.0% overlap; the final overlap is 90.2% for Math, 90.1% for Medical, and 89.9% for Remote Sensing.Source: §A.4.5 · p. 21

A.4.6 Additional t-SNE Projections

• Across additional t-SNE projections, image-patch activations cluster by domain while text activations are more diffuse; ExpertLens generally shows lower intra-seed variation in domain-ordering results.The adaptation experiments use QWEN3-VL-30B-A3B with GSPO, 4,000 training examples per domain, and three seeds.Source: §A.4.6 · p. 21, Figure 15 · p. 23, Figure 16 · p. 24