Source-linked AI summary
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Johannes Kaiser, Florian Braunmiller, Daniel Rückert, Georgios Kaissis
TL;DR
Multimodal imaging models must balance modality-specific learning with cross-modal sharing, a tension that matters for low-prevalence pathologies. GS-MoE pairs a cross-modal generalist with modality-specific specialists. It recovers detection of six pathologies missed by every baseline, with gains up to +0.60 F1.
Problem
Multimodal imaging must balance specialization, which limits negative transfer, with cross-modal sharing, which may provide useful signals to low-prevalence classes.
Source: §1 · pp. 1–2Method
GS-MoE combines an always-active cross-modal generalist with modality-routed specialists and domain-specific output heads.
Source: §2.2 · p. 3, §2.3 · p. 4Results
GS-MoE recovers six low-prevalence pathologies on which every baseline scores F1 = 0, with gains up to +0.60 F1; it achieves overall MCC 0.770.
Source: §1 · p. 2Takeaways & Limitations
Metadata-supervised specialization and learned generalist features contribute complementary gains toward rare-pathology-aware multimodal imaging.
Source: §5 · p. 9The work leaves extensions to volumetric segmentation, report-grounded supervision, hierarchical experts, and model-size ablations for future study.
Source: §5 · p. 9
Abstract
from arXiv · showhide
AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 $=$ 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ${\sim}53\%$ fewer active parameters at inference than the strongest investigated dense model.
1 Introduction
Multimodal imaging creates competing transfer effects: modality differences can hinder learning, while shared anatomical structures may help low-prevalence classes with limited data. GS-MoE combines modality-specific experts with a cross-modal generalist, and the paper reports improved detection of six pathologies that all baselines miss.
1 Introduction
- Large multimodal datasets enable generalized representations, but monolithic architectures struggle with heterogeneity across demographics, anatomy, pathology, and modalities.The introduction frames RadImageNet as 1.35 million instances across 165 pathologies.Source: §1 · p. 1
- Joint training across modalities can create conflicting gradients that hinder learning, while shared anatomical structures can provide a complementary signal to low-prevalence classes.Here, low prevalence means few training samples and is used as a data-driven proxy for rarity, not as a claim of clinical rarity.Source: §1 · pp. 1–2
- GS-MoE couples modality-specific experts trained on related data with a global generalist trained on all available multimodal data.The paper presents this dual-stream design as a way to limit negative transfer while retaining cross-modal positive transfer.Source: §1 · p. 2
- Six low-prevalence pathologies missed by every dense and specialist-only MoE baseline become detectable with GS-MoE, with gains up to +0.60 F1.GS-MoE also leads all baselines in overall MCC (0.770) while using ∼53% fewer active inference parameters than the strongest dense baseline at matched total parameters.Source: §1 · p. 2
2 Methods
Explicit routing limits negative transfer but also limits positive cross-modal transfer, while implicit routing can collapse to a single expert. GS-MoE combines a shared generalist with modality-routed specialists and domain-specific output heads.
2.1 Routing Strategies: Implicit vs. Explicit
- Explicit routing uses metadata to assign modality-specific experts, whereas implicit routing learns assignments but can collapse to one expert without careful initialization.Explicit routing limits negative transfer but positive transfer; neither approach alone ensures interpretable routing, cross-modal sharing, and reduced negative transfer.Source: §2.1 · p. 3
2.2 Generalist-Specialist-MoE (GS-MoE)
- GS-MoE concatenates an always-active, all-modality generalist representation with features from the metadata-routed modality expert, then jointly fine-tunes with the router frozen.A cosine orthogonality penalty encourages complementary branch representations; the generalist is pretrained across modalities and also seeds the specialists.Source: §2.2 · p. 3
2.3 Domain-Constrained Optimization: Why Shared Heads Fail
- A shared output head computes loss over labels from every domain, so GS-MoE instead uses modality-specific heads restricted to each modality’s valid labels.The passage reports that this change raises low-prevalence MR-class macro-F1 from 0.4092 to 0.4537 (+0.045).Source: §2.3 · p. 4
3 Experimental Setup
The evaluation uses RadImageNet’s official splits to compare GS-MoE with dense and MoE baselines. MCC is the primary metric, with macro-F1 also reported.
3 Experimental Setup
- RadImageNet contains 1.35M 2D images across CT, MR, and ultrasound, covering 165 classes; models use the official train, validation, and test splits.GS-MoE uses DenseNet121 branches, a lightweight modality router, and a three-stage training curriculum; baselines include dense models, ExplicitMoE, and ImplicitMoE.Source: §3 · p. 4
- MCC is the primary metric for its robustness to class imbalance, and macro-F1 is reported alongside it.Models are trained until convergence with AdamW and cosine annealing on 224×224 inputs with random affine transforms and horizontal flips.Source: §3 · pp. 4–5
4 Results
Across RadImageNet results, GS-MoE combines strong aggregate performance and parameter efficiency with recovery of low-prevalence pathologies that baselines miss. The results also show distinct failure patterns for explicit and implicit routing, and benefits from domain-constrained heads.
4.1 Aggregate Results
- GS-MoE achieves the top MCC/F1 and uses ∼53% fewer active parameters than DenseNet161Ext; Fig. 2 places it on the parameter-efficiency Pareto frontier.At inference, GS-MoE activates ∼16.4M parameters versus 35.0M for DenseNet161Ext.Source: Table 1 · p. 5, §4.1 · p. 5, Figure 2 · p. 5
4.2 Routing Strategies Face Mutually Exclusive Failure Modes
- Scaling DenseNet161Ext over DenseNet121 adds only +1.2 pp MCC at 4.9× more parameters, whereas GS-MoE exceeds DenseNet121 by +2.6 pp with matched total and ∼53% fewer active parameters.The comparison describes diminishing returns from increasing monolithic capacity and an aggregate advantage for GS-MoE.Source: §4.2 · p. 6
- ExplicitMoE improves CT-Lung and US group F1 but degrades all eight MR groups; implicit routing distributes inputs after sparse upcycling yet both routing strategies fail on low-prevalence classes.ImplicitMoE outperforms DenseNet161 across all 11 anatomical groups, while ExplicitMoE improves on none of the 20% lowest-performing classes.Source: §4.2 · p. 6, Figure 3 · p. 7
4.3 Hard Class Recovery: A Qualitative Capability Gap
- GS-MoE recovers six low-prevalence pathologies where every baseline scores F1 = 0, with gains up to +0.60 F1; it significantly outperforms baselines on low-prevalence classes (p < 0.001).The strongest group recovery is MR-Abdomen (+0.057), and a parameter-matched DenseNet161Ext does not recover the tail.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4 Ablation
- Generalist, Specialist, Domain Head, and Lortho each improve MCC; domain-constrained heads raise MR-class-averaged F1 from 0.4092 to 0.4537 versus a shared head.The shared head is identical to the expert-head variant except for the output head, isolating that design choice.Source: §4.4 · p. 7, Table 3 · p. 8
5 Discussion
GS-MoE outperforms the investigated baselines, with its largest margin in low-prevalence classes, and the reported ordering and recovery persist with transformer backbones. The authors identify further extensions for future work, including volumetric segmentation and report-grounded supervision.
5 Discussion
- GS-MoE outperforms all investigated baselines, with the margin most pronounced in low-prevalence classes.Source: §5 · p. 8
- MCC was 0.745 for GS-MoE-ViT-T, 0.741 for ExplicitMoE-ViT-T, and 0.726/0.737 for ViT-T/S baselines.The ordering and low-prevalence recovery were preserved with transformer backbones despite lower absolute MCC than DenseNet.Source: §5 · pp. 8–9
- The authors describe metadata-supervised specialization and learned generalist features as contributing complementary gains toward rare-pathology-aware multimodal imaging.Source: §5 · p. 9
- Future work includes volumetric segmentation, report-grounded supervision, hierarchical experts conditioned on anatomy and demographics, and ablations across model sizes.Source: §5 · p. 9