PaperlayerView exact revision on arXiv

Source-linked AI summary

Expandable Modality Experts for Medical Vision-Language Learning

DM BAPPY, Eunsu Kim, Hanjoo Lee, Sung Hak Lee, Sangjeong Ahn

arXiv:2610.29881v1cs.CV

TL;DR

Medical vision–language learning must use images without paired text while accommodating new modalities without changing existing representations. The paper combines unpaired visual pretraining, contrastive alignment, and expandable modality experts, achieving best or tied-best results in 24 of 25 benchmark settings. Its preservation argument depends on unchanged computation paths and inference conditions, with retention checked on evaluated inputs.

  • Problem

    Medical vision–language learning must use images without paired text and accommodate new modalities while retaining existing representations.

    Source: §1 · p. 1

  • Method

    The technique combines masked pretraining on unpaired medical images, contrastive image–text alignment, and expandable modality experts.

    Source: §1 · p. 2, §9 · p. 11

  • Results

    The model supports nine imaging modalities and is best or tied-best in 24 of 25 classification, linear-probe, and retrieval settings.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    The results support modular expert expansion as a way to extend medical vision–language representations while retaining existing embeddings.

    Source: §9 · p. 11

    Preservation depends on unchanged existing computation paths and inference conditions; bitwise equality is verified only on evaluated inputs.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

Learning across medical imaging modalities requires making use of images without paired text and accommodating new domains while retaining existing representations. We propose a medical vision-language technique built around an expandable bank of modality experts. Its Multiway Transformer combines shared visual computation with deterministically routed feed-forward experts and a separate text pathway. A modality-aware visual tokenizer supplies targets for masked pretraining on 19.4 million unpaired image rows across seven modalities; contrastive alignment then uses 3.23 million image-text pairs. Additional fundus and optical coherence tomography experts learn from paired data while the original image and text embedding paths remain frozen. With a ViT-Large backbone, the proposed technique achieves the best or tied-best result in 24 of 25 benchmark settings across nine modalities: 10 of 11 zero-shot classification benchmarks, all 10 linear probes, and all four retrieval benchmarks. Mean balanced accuracy is 0.674 for zero-shot classification and 0.856 for linear probing, and mean bidirectional retrieval recall is 20.70%. A separate three-seed sequential experiment with matched adaptation data yields fundus and OCT retrieval gains of 14.95 and 9.31 percentage points, with no measured loss on preceding fundus or pathology tasks. The preservation argument requires unchanged existing computation paths and inference conditions, and feature-identity checks verify retention on evaluated inputs. The proposed technique provides a modular approach to learning and expanding medical vision-language representations.

1 Introduction

The introduction frames medical vision–language learning around two constraints: many images lack paired text, and adding modalities can alter existing representations. It presents an expandable expert architecture and reports broad benchmark performance alongside retention evaluation.

1 Introduction

  • Medical imaging spans visually different domains, while many available images lack paired text and paired-data collection may not reflect clinical-image distributions.The introduction motivates separating visual learning from language alignment across medical imaging domains.Source: §1 · p. 1
  • A Multiway Transformer combines shared visual computation, modality-routed feed-forward experts, and a separate text pathway, with masked pretraining and contrastive alignment.The core uses 19.4M unpaired image rows and 3.23M image–text pairs; fundus and OCT experts are then added using paired data while existing embedding paths remain frozen.Source: §1 · p. 2
  • The setting does not require patient-matched images across modalities and evaluates text-prompted classification, linear probing, cross-modal retrieval, and retention of earlier pathways.At inference, each image is encoded through its modality route.Source: §1 · p. 2
  • The expansion contribution formalizes conditions for preserving existing image and text embedding functions while distinguishing matched-data sequential adaptation from the final full-data assembly.The introduction identifies changed representations as a risk when shared parameters are updated for a new modality.Source: §1 · pp. 1–2
  • The proposed technique is best or tied-best in 24 of 25 benchmark settings, with feature-identity checks and retrieval tests assessing retention on evaluated inputs.The evaluation spans nine modalities and includes controlled ViT-B comparisons and three-seed adaptation results.Source: §1 · p. 2

2 Modality experts for medical vision–language learning

The proposed technique combines unpaired visual pretraining, image–text alignment, and expandable modality experts. Its expansion rule preserves existing embedding functions only under specified conditions.

2 Modality experts for medical vision–language learning

  • The technique combines a Multiway backbone with domain-aware visual experts, a two-stage training pipeline, and conditional preservation of existing embedding functions.These are its three stated ingredients.Source: §2 · p. 2

2.1 Backbone and Domain-Aware Mixture-of-Experts

  • A modality ID deterministically routes each image to one modality-specific expert alongside a shared visual pathway.The seven-modality core uses routed experts at six layers; fundus and OCT extensions use experts spanning all 24 layers.Source: §2.1 · pp. 3–4

2.2 Two-stage training pipeline

  • Stage 1 uses frozen tokenizer codes for masked image modeling and unpaired text for masked language modeling; Stage 2 aligns pooled image and text embeddings over 3.23M pairs.The image and text streams are unpaired during Stage 1.Source: §2.2 · p. 4

2.3 Expansion with preserved embedding functions

  • Expansion appends an independent expert and trains only it on paired image–text data while freezing parameters on existing image and text embedding paths.Fundus and OCT extensions use no additional Stage-1 masked pretraining.Source: §2.3 · p. 5
  • Existing embedding functions are preserved only if original parameters, persistent state, routing, and inference operations remain unchanged.Unchanged predictions and metrics additionally require unchanged preprocessing, scoring rules, candidate sets, and downstream classifiers.Source: §2.3 · p. 5

3 Data and evaluation protocols

The study combines large unpaired-image and paired image–text corpora in a nine-modality ViT-Large assembly. Evaluation uses task-specific benchmarks, while expansion is assessed through separate matched-data sequential and full-data assembly designs.

3.1 Training data and model configurations

  • 19,393,075 unpaired image rows support masked pretraining across seven modalities, followed by alignment on 3,229,073 image–text pairs.Fundus and OCT experts use 88,143 and 19,396 additional image–text pairs, respectively, without further masked pretraining.Source: §3.1 · p. 5
  • The main model uses ViT-Large/16; its nine-modality assembly stores 1.280B parameters, while one selected image path contains 355.40M.The selected-path count excludes the text encoder and unselected experts.Source: §3.1 · p. 5

3.2 Transfer benchmarks and metrics

  • Evaluation covers 11 zero-shot classification benchmarks, 10 linear probes, and four retrieval benchmarks, using balanced accuracy for classification and cosine similarity for retrieval.Retrieval averages R@1, R@10, and R@50 across image-to-text and text-to-image; means are task-specific, not directly comparable across task families.Source: §3.2 · p. 6

3.3 Expansion experiments

  • The matched-data sequential experiment uses 20,000 fundus pairs followed by 16,000 OCT pairs, whereas the expanded full-data model assembles independently trained experts.The full-data assembly is not a matched-training comparison with sequential LoRA.Source: §3.3 · p. 6

4 Transfer across medical imaging modalities

The proposed technique performs strongly across zero-shot classification, linear probing, and cross-modal retrieval. It leads or ties on nearly all reported benchmarks, with task-specific means reported for each evaluation family.

4.1 Zero-shot classification

  • 0.674 mean balanced accuracy and 10 of 11 outright wins characterize zero-shot classification; BMCA leads on Pneumonia.The mean is 0.080 above ConceptCLIP, the strongest baseline at 0.594.Source: §4.1 · p. 6

4.2 Linear probe

  • 0.856 mean balanced accuracy accompanies best-or-tied-best performance on all ten linear-probe benchmarks.The ties are CT-Kidney at 0.993 with MerMED-FM and ColorFundus10 at 0.712 with ConceptCLIP.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Cross-modal retrieval

  • 20.70 mean bidirectional retrieval recall leads all four benchmarks, exceeding the 12.09 baseline mean by 8.61.The largest margins are on OpenI and MR-RATE; margins are smaller on derm7pt and PathGen-600.Source: Table 3 · p. 7, §4.3 · p. 7

5 Modality expansion and representation retention

Sequential adaptation with matched data tests learning and retention, while a separate full-data assembly evaluates independently trained experts. The results show retention on evaluated prior tasks, with the experiments differing in training design and scope.

5.1 Sequential adaptation with matched data

  • 14.95 percentage points on fundus retrieval and 9.31 on OCT are gained under matched sequential adaptation, with no measured loss on preceding fundus or PathGen tasks.All methods use 20,000 fundus pairs followed by 16,000 OCT pairs, three seeds, and common validation-based checkpoint selection; baselines use LoRA.Source: §5.1 · p. 8
  • Fundus retrieval rises from 5.22% to 20.17% and stays fixed after OCT training, while OCT reaches 13.59%, below adapted baselines’ 15.46%–15.88%.Derm7pt retrieval remains 9.63%, with byte-identical saved features within each seed.Source: §5.1 · p. 8

5.2 Full-data expert assembly

  • The full-data assembly is best or tied-best on all five fundus/OCT rows and retains PathGen-600 at 25.75% mean bidirectional recall with byte-identical saved image and text features.This is descriptive rather than a matched-training comparison: the experts are independently trained, while baselines use sequential LoRA on matched-data subsets.Source: §5.2 · p. 9

5.3 Checks on existing representations

  • On a balanced 4096-image PCam probe, core, +fundus, and +fundus+OCT checkpoints have identical AUROC (0.8691), balanced accuracy (64.26%), features, cosine scores, and predictions.The full-data assembly also retains evaluated retrieval results across all seven base modalities.Source: §5.3 · p. 9

6 Ablations and supporting analyses

Supporting analyses compare modality-specific capacity and visual tokenization under controlled but limited training settings, alongside selected qualitative examples. The analyses report gains while stating limits on what the comparisons establish.

6.1 Modality experts under a matched training budget

  • 15 of 18 benchmarks improve with modality experts over the variant without them, with mean gains of 0.0433 zero-shot balanced accuracy, 0.0201 linear-probe balanced accuracy, and 0.1330 percentage points retrieval.The comparison uses a matched ViT-B setup, but a five-epoch-per-stage final-checkpoint schedule does not establish effects at convergence or isolate routing from added expert parameters.Source: §6.1 · p. 9

6.2 Multi-teacher visual tokenization

  • Multi-teacher supervision raises mean zero-shot balanced accuracy from 32.74% to 35.60%, linear-probe balanced accuracy from 74.56% to 76.34%, and retrieval from 2.17% to 2.35%.The one-seed, short-schedule experiment does not establish statistical significance or the effect under full-data training.Source: §6.2 · pp. 9–10

6.3 Qualitative evidence

  • Appendix H shows individual ranking changes from fundus and OCT integration, while Appendices I and J visualize spatial responses to finding descriptions and paired reports; these complement quantitative evaluations, not localization measures.For linear-probe examples, the saliency target is image–prompt similarity, not the image-only classifier’s class score.Source: §6.3 · p. 10

7 Discussion and limitations

In the matched-data sequential experiment, new experts learn fundus and OCT while earlier tasks are retained, although OCT retrieval remains below adapted baselines. The discussion distinguishes this retention result from broader performance comparisons, whose causes are not isolated, and states conditions and evaluation limits for the preservation claim.

7 Discussion and limitations

  • The matched-data experiment learns fundus and OCT while retaining earlier tasks, but its OCT retrieval score remains below those of adapted baselines.The discussion says retention and new-task performance need to be assessed together.Source: §7 · p. 10
  • Broader baseline comparisons do not isolate performance differences, and the matched-data experiment does not match trainable parameter counts or computational cost.The ViT-B routing and tokenizer studies use limited subsets and short schedules, so they do not establish effect sizes after converged ViT-L training.Source: §7 · p. 10
  • Preservation assumes supplied modality labels and fixed preprocessing, persistent state, routing, inference operations, scoring rules, candidate sets, and downstream classifiers.Feature-identity checks cover evaluated inputs; unchanged features do not establish identity of every scaled logit when the learned logit-scale parameter changes.Source: §7 · p. 10
  • Some linear-probe splits are at image or data-row level rather than patient or slide level, retention benchmarks sample base-modality capabilities, and qualitative examples lack quantitative localization evaluation.The paper says clinical use would require prospective validation in the intended setting.Source: §7 · p. 10

8 Related work

Prior biomedical vision-language models provide text interfaces but do not explicitly separate modality computation paths, while vision-only models lack a text interface. The proposed technique combines modality-specific routing with a language interface in one encoder, extending multiway pretraining and expert expansion to medical imaging.

Biomedical vision–language models

  • Biomedical vision-language models support text-prompted classification and image-text retrieval, but do not explicitly separate computation paths for different imaging modalities.Vision-only models cover several modalities but lack a text interface; CTF connects domain-specific radiology and pathology encoders through clinical concepts and cross-domain prompting.Source: §8 · p. 10

Multiway pretraining and visual tokenization

  • The proposed technique extends Multiway Transformer and VQ-KD designs to multiple modalities with one tokenizer supervised by six visual teachers and modality-specific visual FFNs.Teachers supervise tokenizer training; after it is frozen, the VLM receives no further teacher-feature or teacher-logit loss.Source: §8 · p. 10

Mixture-of-Experts and continual expansion

  • Deterministic modality routing uses known modality labels and needs no load-balancing loss, while preservation follows from keeping existing computation paths fixed.Unlike learned token routing, each image uses its known modality label; the technique uses full-rank modality experts in a contrastive encoder rather than low-rank experts in a generative medical VLM.Source: §8 · p. 11

9 Conclusion

The proposed technique combines masked pretraining on unpaired medical images, contrastive image-text alignment, and expandable modality experts. It performs best or tied-best across most evaluated settings, and matched-data expansion improves new-modality retrieval while retaining evaluated earlier tasks under explicit inference conditions.

9 Conclusion

  • The model achieves the best or tied-best result in 24 of 25 classification, linear-probe, and retrieval settings across nine imaging modalities.In the matched-data sequential experiment, new experts improve fundus and OCT retrieval while evaluated preceding fundus and pathology tasks remain unchanged.Source: §9 · p. 11
  • Masked pretraining on unpaired medical images, contrastive image-text alignment, and expandable modality experts form the proposed approach.Source: §9 · p. 11
  • The preservation argument depends on retaining original computation paths under explicit inference conditions, and the reported results support modular expert expansion for medical vision-language representations.Source: §9 · p. 11

Appendices

The appendices provide data and implementation details, controlled comparisons, adaptation and retention protocols, and qualitative examples. They describe the evaluation tasks and training data, including an image-level HEST split rather than a slide-level split.

Appendices

  • The appendices provide data and implementation details, controlled comparisons, adaptation and retention protocols, and qualitative image examples.Source: Appendices · p. 17

A Datasets and evaluation tasks

  • Evaluation covers zero-shot classification, linear probing, and image-text retrieval across nine imaging modalities, with models compared on the same evaluation set and scoring procedure within each experiment.Source: A Datasets and evaluation tasks · p. 17
  • The ViT-L core uses 19,393,075 image rows for masked pretraining and 3,229,073 image-text pairs for alignment, followed by fundus and OCT experts trained on additional paired data.The extension stages use 88,143 fundus pairs and 19,396 OCT pairs without further masked pretraining.Source: A Datasets and evaluation tasks · p. 17
  • The classification and probe benchmarks include tissue, MRI, and pediatric chest-radiograph tasks, while four main retrieval benchmarks cover X-ray, MRI, dermatology, and pathology.The broader retention experiment also includes ultrasound and endoscopy retrieval.Source: A Datasets and evaluation tasks · p. 17
  • HEST uses an image-level rather than slide-level split in the ViT-B linear-probe evaluation.The ViT-B evaluation uses stratified cross-validation.Source: A Datasets and evaluation tasks · p. 17

B Method and implementation details

The ViT-B comparison tests DA-MoE under a limited, matched training budget, while the tokenizer experiment compares multi-teacher and single-teacher supervision. The implementation uses modality routing and frozen existing pathways for expert expansion.

B Method and implementation details

  • Masked-image and masked-language prediction precede symmetric contrastive alignment, using a frozen tokenizer and modality routing in both stages.The tokenizer uses six modality-specific visual teachers; the core schedule is 130,000 masked-pretraining steps followed by 15 alignment epochs.Source: B Method and implementation details · p. 17
  • A shared image FFN output is added to the FFN output of the expert selected by the image modality.Only the selected expert contributes to the layer output, and the formulation does not claim that all computation associated with unselected experts is avoided.Source: B Method and implementation details · p. 18
  • During new-expert training, existing visual experts, the shared visual pathway, text encoder, and projection heads are frozen.Preservation requires unchanged routes, preprocessing, inference operations, scoring rules, candidate sets, and downstream classifiers; feature and prediction identity was verified on evaluated inputs.Source: B Method and implementation details · p. 18
  • New fundus and OCT experts are independently trained from the seven-modality base using 88,143 and 19,396 image-text pairs, respectively.These full-data runs are separate from the matched-data sequential experiment.Source: B Method and implementation details · p. 18

C ViT-B comparison with and without DA-MoE

  • DA-MoE performs better on 15 of 18 benchmarks, ties on BreaKHis zero-shot classification, and performs worse on Pneumonia zero-shot classification and HAM10000 linear probing.The matched ViT-B variants use five epochs per training stage, and the paper notes that this budget does not necessarily establish convergence.Source: C ViT-B comparison with and without… · p. 19

D Multi-teacher visual tokenization

  • The ViT-B tokenizer comparison holds architecture, data subsets, initialization, and training schedule fixed while varying multi-teacher versus single-teacher supervision.Both tokenizers use 8,192 visual codes and are evaluated on zero-shot classification, linear probing, and image-text retrieval.Source: D Multi-teacher visual tokenization · p. 19
  • Multi-teacher supervision raises mean zero-shot balanced accuracy from 32.74% to 35.60%, while also improving mean linear-probe and retrieval scores.The experiment uses one seed, a limited training subset, and a short schedule, so it does not establish statistical significance or performance under converged full-data training.Source: D Multi-teacher visual tokenization · p. 19

E Matched-data adaptation: setup and results

The matched-data experiment sequentially adapts fundus and OCT experts while holding the original pathways fixed. All methods improve on the new tasks, while only the proposed technique shows no measured loss on preceding fundus or PathGen tasks.

E Matched-data adaptation: setup and results

  • The experiment sequentially trains fundus and then OCT experts from the seven-modality ViT-L checkpoint using 20,000 and 16,000 pairs, respectively.The proposed technique trains new modality experts while the shared backbone, original experts, text encoder, projection heads, and contrastive temperature remain frozen.Source: E Matched-data adaptation: setup and… · p. 20
  • Baseline results are means across three training seeds, with LoRA settings and learning rates selected using validation performance.Evaluation uses fixed sets of 588 fundus images, 576 OCT images, and 600 PathGen report rows.Source: E Matched-data adaptation: setup and… · p. 20
  • All methods improve on new fundus and OCT tasks, while the proposed technique shows no forgetting on fundus or PathGen after OCT adaptation.The baselines lose some fundus performance and perform worse on PathGen.Source: E Matched-data adaptation: setup and… · p. 20

F Matched-data learning and retention

This section evaluates matched-data expansion and retention across existing modalities, alongside qualitative retrieval and saliency examples for classification and retrieval tasks. It clarifies that feature checks establish preservation only on evaluated inputs and that heatmaps are descriptive image–text responses, not quantitative localization evidence or validated findings.

F Matched-data learning and retention

  • Fundus retrieval rises from 5.22% to 20.17% and remains unchanged after OCT integration; OCT rises from 4.28% to 13.59%, below baselines’ 15.46%–15.88%.All baselines improve after fundus training but decline after OCT training; the newly initialized experts learn OCT without the highest OCT score in this subset experiment. Adaptation data are matched, but model size, pretraining history, trainable parameters, and computational cost are not.Source: F Matched-data learning and retention · p. 20

G Retention across seven base modalities

  • The proposed technique is unchanged on all seven base-modality retrieval benchmarks, while 27 of 28 baseline model–benchmark results decrease and UniMed-CLIP gains 0.57 points on MR-RATE.Saved image and text features are byte-identical across evaluated checkpoints, verifying preservation on tested inputs; this does not establish unchanged parameters, statistical significance of every baseline decrease, or retention of every capability.Source: G Retention across seven base modalities · p. 21, Table 13 · p. 35

H Qualitative examples

  • One retrieval example per original modality compares before and after expansion using the same query, gallery, preprocessing, and cosine scoring; top-three examples illustrate individual ranking changes, not aggregate performance.Green marks a report correct under the benchmark matching rule; an unmarked report may still be clinically relevant. The separate CT-Kidney classification example is not part of aggregate retrieval results.Source: H Qualitative examples · pp. 21–22

I Saliency overview across nine modalities

  • Similarity-gradient Grad-CAM maps show prompt-conditioned spatial responses across nine modalities, including fundus and OCT, by backpropagating image–text cosine similarity through the routed expert to final-block patch tokens.These maps are qualitative examples, not a quantitative localization evaluation.Source: I Saliency overview across nine… · p. 22, Figure 11 · p. 30

J Comparative image–text similarity-gradient patch saliency

  • The comparative saliency set contains 66 selected examples from 22 datasets, spanning 33 zero-shot, 21 linear-probe, and 12 retrieval cases, and illustrates spatial-response differences rather than dataset-level performance.Each dataset contributes three examples; all five models use the same image and text prompt within an example, with each model’s own preprocessing.Source: J Comparative image–text… · p. 36

J.1 Saliency construction and task-specific prompts

  • Patch saliency backpropagates image–text cosine similarity to patch features at the fourth transformer block from the end, using the image’s modality route without updating model weights.Overlays combine image and heatmap equally; colors are not a common numerical scale across models, and these maps differ from Figure 11’s final-block maps.Source: §J.1 · p. 36
  • For zero-shot, linear-probe, and retrieval examples, saliency targets the specified ground-truth-class prompt, a class-related text prompt, or the paired caption/report, respectively—not necessarily the model’s prediction or retrieval alternative.Linear probes use image features without text; their heatmaps show image–prompt similarity, not gradients of the linear-probe score. Retrieval maps explain response to the reference paired text, not an alternative report.Source: §J.1 · p. 36

J.2 Reading the comparative examples

  • Selected saliency examples are descriptive responses to text and do not quantify localization, establish causal necessity, or show that one model localizes disease better across a dataset.Responses within an apparent lesion do not by themselves establish correct classification or successful retrieval, and the examples do not isolate routing from other model differences.Source: §J.2 · p. 36
  • For TN5000, the proposed technique responds within and around a thyroid nodule, while its heatmap reflects image–prompt similarity, not the image-only linear-probe class score.The compared methods highlight different central or peripheral locations.Source: Figure 29 · p. 54
  • Across Derm7pt, MR-RATE, and OpenI retrieval examples, methods show different spatial responses to paired text, but heatmaps do not establish correct gallery ranking or validate report findings.The maps respond to the paired caption or full report, rather than necessarily to one diagnostic term.Source: Figure 30 · p. 55, Figure 31 · p. 56, Figure 32 · p. 57
  • In PathGen-600 examples, the proposed technique distributes responses across cellular regions, but these maps are not instance-level annotations of reported changes.BMC and ConceptCLIP emphasize different location groups, while BMCA and UniMed-CLIP include prominent peripheral foci.Source: Figure 33 · p. 58

Related papers

Loading 2610.29881v1…