PaperlayerView exact revision on arXiv

Source-linked AI summary

Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillation

Chu Zhang, Haoyu Jiang, Hongyuan Zhang, Hongbin Liu, Dong Yi

arXiv:2609.37682v1cs.CV

TL;DR

Medical generalists have less training data and less domain-specific depth than specialists, leaving a gap between cross-modality baselines and domain ceilings. Med-RADIO distills a generalist and modality-specific teachers into one encoder using modality-aligned streams and balanced loss. It improves on strong generalists and matches or surpasses specialists on four of five modalities under linear probing.

  • Problem

    Medical generalists have less training data than natural-image generalists and less domain-specific depth than medical specialists, whose expertise is modality-specific.

    Source: §1 · p. 1

  • Method

    Med-RADIO distills a generalist and modality-specific specialists into one encoder using modality-aligned data streams, translator projectors, and contribution-aware balanced loss.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    85.28% mean linear-probing AUC/ACC, +3.38 pp over UniMed-CLIP large, with Med-RADIO matching or surpassing specialists on four of five modalities.

    Source: §5 · p. 9, §6 · p. 9

  • Takeaways & Limitations

    Heterogeneous distillation offers a practical alternative to ensembles and weight merging, with one forward pass for the unified encoder.

    Source: §6 · p. 9

    Evaluation covers 2D classification across five modalities, leaving chest X-ray, fundus, volumetric 3D understanding, dense prediction, and VQA unassessed systematically.

    Source: §5 · p. 9

Abstract

from arXiv · show

The rapid expansion of large-scale medical datasets and computational resources has driven significant progress in medical foundation models. Given the inherent heterogeneity of medical imaging modalities, current research mainly follows two paths: specialized models optimized for specific modalities, and generalist models designed to handle multiple modalities. However, medical generalist models suffer from both insufficient training data scale relative to natural image generalists and inadequate domain-specific depth relative to medical specialists. Empirically, generalist models establish a cross-modality performance baseline, while specialists define the performance ceiling within their respective domains. To elevate this baseline toward these ceilings, we propose Med-RADIO, a medical multi-teacher distillation framework that Reduces All Domains Into One by compressing complementary expertise from multiple domain-specific teachers into a unified medical vision foundation model. Our method curates both generalist and specialist teachers, allocates modality-aligned distillation streams to reorganize generalist pretraining data so it matches specialist domains, and uses a balanced loss to prevent any single teacher from dominating the distillation process. On internal and external classification benchmarks spanning five modalities, Med-RADIO improves over strong medical generalists under linear probing and remains competitive with representative specialists on most evaluated modalities. Code is available at https://github.com/CAIR-HKISI/Med-RADIO.

1 Introduction

Medical generalists provide cross-modality baselines, while specialists set domain-specific ceilings; the paper asks whether one unified model can narrow that gap. Med-RADIO distills complementary teacher expertise and improves over strong generalists while remaining competitive with specialists on most evaluated modalities.

1 Introduction

  • Medical generalists trail specialists partly because they use millions rather than billions of images, limiting their domain-specific depth.The paper characterizes generalists as cross-modality baselines and specialists as performance ceilings within their domains.Source: §1 · p. 1
  • Multi-teacher distillation offers a unified alternative to costly specialist ensembles and infeasible weight merging, but medical settings bring architectural, modality, and optimization barriers.Specialists need modality-specific inputs, teacher output spaces may be misaligned, and stronger teachers can suppress weaker complementary signals.Source: §1 · pp. 1–2
  • Med-RADIO repartitions generalist pretraining data into modality-specific streams to distill specialist expertise without retrieving the original specialist corpora.It freezes teachers and trains the student encoder with lightweight projectors.Source: §1 · p. 2
  • Its three innovations are complementary teacher curation with domain-specific projectors, modality-aligned data streams, and contribution-aware balanced loss with a mandatory generalist anchor.The streams reorganize UniMed-CLIP [19] pretraining data into single-modality subsets for specialist-native distributions.Source: §1 · p. 2
  • Across internal and external classification benchmarks spanning five modalities, Med-RADIO improves over strong generalists and remains competitive with dedicated specialists on most modalities.The authors frame it as a practical route to single-model deployment across heterogeneous medical imaging workflows.Source: §1 · p. 2

2 Related Work

Related work describes a gap between medical generalists and modality specialists, then motivates distillation as an alternative to ensemble or weight-merging approaches. Med-RADIO extends foundation-model distillation to heterogeneous medical teachers and addresses alignment, optimization, and modality-distribution challenges.

2.1 Medical Vision Foundation Models

  • Medical specialists establish strong domain-specific ceilings, while generalists trained on millions rather than billions of images tend to underperform specialists on single-modality benchmarks.Ensembling brings multiplicative inference costs, and architectural heterogeneity prevents direct parameter fusion.Source: §2.1 · pp. 2–3

2.2 Distilling Foundation Models

  • Prior natural-image multi-teacher distillation unified heterogeneous encoders, but used homogeneous photographic data rather than medical modality-divergent inputs.Medical distillation efforts were limited to single teachers, identical architectures, or one modality; Med-RADIO targets heterogeneous multi-teacher transfer.Source: §2.2 · p. 3

2.3 Multi-Teacher Knowledge Distillation

  • Representation alignment matters because directly enforcing feature consistency between structurally dissimilar teachers and students can impose overly rigid constraints.Uniform weighting risks collapse or dominance, motivating adaptive weighting approaches.Source: §2.3 · p. 3
  • Medical multi-teacher distillation must address misaligned backbone outputs, dominance from heterogeneous pretraining scales, and specialists’ need for native modality exposure.Med-RADIO uses modality-specific projection heads, contribution-aware balanced loss, and modality-aligned data streams.Source: §2.3 · p. 3

3 Methodology

Med-RADIO trains one shared medical encoder to combine cross-modality representations from a generalist teacher with modality-specific expertise from specialist teachers.

3.1 Framework Overview

  • A shared student encoder aggregates specialist supervision on dedicated modality streams and generalist supervision on the full heterogeneous corpus.The pipeline combines the teachers through a balanced distillation objective.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Pretraining Dataset Construction

  • The method repartitions UniMed-CLIP into single-modality subsets for specialists while retaining a heterogeneous mixture for the generalist teacher.Five specialist streams cover US, MRI, CT, histopathology, and dermatology; the original corpus spans seven modalities.Source: §3.2 · p. 4

3.3 Teacher Model Composition

  • The teacher ensemble pairs one cross-modality medical model with specialists for US, CT, MRI, pathology, and dermatology.The student encoder is trained solely via multi-teacher distillation.Source: §3.3 · p. 4

3.4 Modality-Specific Translator Heads

  • Translator heads project student features into each teacher’s output space, aligning heterogeneous representations without modifying frozen teachers.Separate generalist- and specialist-aligned projectors target their respective output dimensions, and all heads are discarded after pretraining, leaving downstream inference to the encoder alone.Source: §3.4 · p. 5

3.5 Balanced Distillation Loss

  • Static normalization by each teacher’s angular dispersion balances per-teacher losses, amplifying low-dispersion signals and attenuating high-dispersion ones.Dispersion is estimated offline from 10,000 samples per teacher’s modality stream; the balancing mechanism aims for equitable gradient magnitudes without manual weight tuning or online adaptation.Source: §3.5 · p. 5

4 Experiments

Med-RADIO improves on strong medical generalists and matches or exceeds the strongest specialist on most evaluated modalities under linear probing. Its ablations identify dispersion balancing as the most critical ingredient.

4.1 Implementation Details

  • The experiments use a ViT-Large/16 student with modality-specific MLP translator heads, trained for 10 epochs; downstream evaluation uses the bare student backbone.Evaluation covers nine internal classification benchmarks across five modalities and an external five-modality protocol, using frozen-encoder linear probing and full fine-tuning.Source: §4.1 · p. 6

4.2 Main Results

  • 85.28% mean AUC/ACC under linear probing, +3.38 pp over UniMed-CLIP (large) and over 7 pp above all other baselines.The largest reported margin is +11.60 pp on Breast-US AUC.Source: §4.2 · p. 6, Table 1 · p. 7
  • Med-RADIO matches or exceeds the best specialist on four of five modalities, with its largest gains on Thyroid ultrasound (+10.8 pp) and CT Sagittal/Axial (+4.1/+3.7 pp).Its largest deficit is histopathology (−5.8 pp vs. GPFM).Source: §4.2 · p. 7, Figure 3 · p. 7
  • Under fine-tuning, Med-RADIO leads on five of nine tasks, especially Thyroid and Breast ultrasound (+7.0/+5.3 pp), while PCam and Meniscus remain about 2–3 pp behind.Figure 4 reports the gaps against the best specialist per modality in percentage points.Source: §4.2 · p. 7, Figure 4 · p. 8
  • 84.66% mean on the internal/external linear-probing protocol, +3.71 pp over MMKD-CLIP and +0.52 pp over Uncertainty, the strongest balancing baseline.External tasks are held out from teacher-pool construction.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Ablation Studies and Analysis

  • Removing the generalist teacher lowers the mean by −1.74 pp (85.28%→83.55%), while scaling from ViT-Base to ViT-Large raises it by +1.55 pp on average.The generalist’s contribution is concentrated on ultrasound Thyroid and CT Axial; scaling gives the largest lifts on CT Sagittal and MRI Meniscus.Source: §4.3 · pp. 8–9, Figure 5 · p. 8
  • Removing dispersion balancing drops the mean by −8.94 pp (85.28%→76.34%), with severe losses on CT and ultrasound Thyroid.The analysis attributes this to high-dispersion teachers dominating and suppressing complementary streams.Source: Figure 5 · p. 8, §4.3 · p. 9

5 Discussion and Limitation

Under linear probing, Med-RADIO improves on the generalist baseline and remains competitive with specialists on most modalities. Ablations highlight the roles of a generalist teacher, balanced weighting, and modality-native teacher assignment, while the evaluation and training setup impose important scope limits.

Discussion

  • 85.28% mean AUC/ACC (+3.38 pp over UniMed-CLIP large); Med-RADIO remains competitive with specialists on four of five modalities under linear probing.It also outperforms a medical MTKD baseline and matched balancing alternatives on the internal/external protocol.Source: §5 · p. 9
  • Removing the generalist teacher lowers average performance by −1.74 pp, with ultrasound and CT hit hardest, consistent with cross-modal regularization.The generalist teacher helps stabilize domains where specialist coverage alone is insufficient.Source: §5 · p. 9
  • Angular-dispersion weighting outperforms unweighted and generic adaptive losses, while modality-native teacher assignment improves most clearly on ultrasound and CT.Unweighted optimization can suppress low-dispersion teachers or let high-dispersion teachers dominate; standard medical MTKD without modality-native assignment also lags on ultrasound and CT.Source: §5 · p. 9

Limitations

  • The largest remaining specialist gap is in histopathology, attributed primarily to noisier Quilt-1M data than the curated pathology corpora used by GPFM.The authors suggest higher-quality, distribution-matched pathology data and optionally multi-scale supervision as possible remedies.Source: §5 · p. 9
  • Evaluation covers 2D classification across five modalities, while chest X-ray, fundus, volumetric 3D understanding, dense prediction, and VQA lack systematic assessment.Student quality also remains bounded by the frozen teacher pool and public stream corpora; frozen linear probing with CLS angular matching under-emphasizes fine-grained texture cues.Source: §5 · p. 9

6 Conclusion

Med-RADIO distills a generalist and modality-specific specialists into one encoder using translator heads, modality-aligned streams, and balanced loss. On a restructured 4.5M-image corpus, it improves linear-probing performance and matches or surpasses specialists on four of five modalities.

Conclusion

  • 85.28% mean linear-probing AUC/ACC (+3.38 pp over the strongest generalist); Med-RADIO matches or surpasses dedicated specialists on four of five modalities in a single forward pass.The 4.5M-image corpus is restructured without new specialist corpora.Source: §6 · p. 9
  • The authors identify extending Med-RADIO to additional modalities, volumetric inputs, and multi-scale pathology supervision as future work.Source: §6 · p. 9

Supplementary Material

The supplementary material documents corpus reconstruction, benchmark protocols, teacher selection, feature-dispersion balancing, and implementation details for reproducing the distillation pipeline.

Pretraining Corpus Reconstruction

  • UniMed-CLIP’s 5.3M multimodal mixture is reorganized into six streams, with modality-specific streams assigned to specialists and the mixture supervised by the generalist.The streams comprise US, MRI, CT, histopathology, dermatology, and a multimodal mixture; Table S1 reports 4.54M pairs.Source: Pretraining Corpus Reconstruction · p. 14, Table S1 · p. 14

Downstream Benchmarks

  • Nine internal classification tasks span five modalities, using official splits where available or stratified 75:10:15 train/validation/test partitions otherwise.Metrics follow domain conventions: AUC for ultrasound/MRI and accuracy for CT, histopathology, and dermatology.Source: Downstream Benchmarks · p. 14, Table S2 · p. 14

Teacher Model Selection Details

  • The final teacher ensemble combines UniMed-CLIP large with URFM for ultrasound, Curia for MRI and CT, GPFM for histopathology, and PanDerm for dermatology.Specialist selection uses the modality-specific candidates under linear probing, excluding UniMed-CLIP from specialist selection.Source: Teacher Model Selection Details · p. 15, Table S3 · p. 15

Teacher Feature Statistics Computation

Hyperparameters and Training Pseudocode

  • The implementation specifies hyperparameters across distillation, linear probing, and fine-tuning, and Algorithm 1 details proportional batch assembly, balanced angular loss, and optimization.The algorithm initializes the student from ImageNet-21k ViT-L/16, trains for 10 epochs, and outputs the student encoder while discarding translator heads.Source: Hyperparameters and Training Pseudocode · pp. 16–17, Table S5 · p. 16
Loading 2609.37682v1…