PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniRad: A Radiological Foundation Model for Multi-Task Medical Image Analysis

Luca Zedda, Andrea Loddo, Cecilia Di Ruberto

arXiv:2602.04547v1cs.CVcs.AI

TL;DR

Radiological analysis needs transferable representations because current approaches often use task-specific strategies and underuse shared structure across heterogeneous imaging tasks. OmniRad learns a shared self-supervised encoder from diverse radiological data and systematically improves classification, segmentation, and report-generation benchmarks with frozen downstream adaptation.

  • Problem

    Current radiological models often lack modality and anatomical diversity, use task-specific adaptation, and underuse shared structure, limiting representation reuse across tasks.

  • Method

    OmniRad uses self-supervised pretraining on heterogeneous multi-modal radiological data, then adapts one shared encoder to classification and segmentation tasks.

  • Results

    Across diverse benchmarks, OmniRad showed systematic improvements in classification, segmentation, and report generation while retaining a frozen encoder during downstream adaptation.

  • Takeaways & Limitations

    A single task-agnostic encoder can support stable, reusable representations across diverse radiological objectives without joint multi-task optimization.

  • Takeaways & Limitations

    Evaluation is limited to retrospective public benchmarks without prospective clinical validation and omits multi-view, longitudinal, and clinical-metadata integration.

Abstract

from arXiv · show

Radiological analysis increasingly benefits from pretrained visual representations that can support heterogeneous downstream tasks across imaging modalities. In this work, we introduce OmniRad, a self-supervised radiological foundation model pretrained on 1.2 million medical images, designed with radiology-inspired principles emphasizing representation reuse and cross-task transferability. We evaluate the pretrained encoder under multiple downstream adaptation regimes, including lightweight task-specific adapters with a frozen backbone as well as full end-to-end fine-tuning for classification, allowing us to assess both representation quality and task-specific performance. OmniRad is evaluated on a broad suite of public benchmarks spanning classification and segmentation across multiple modalities. On the MedMNISTv2 collection, OmniRad improves classification F1 by up to 2.05% over competing foundation models. For dense prediction, OmniRad attains mean Dice score improvements across six MedSegBench datasets when using frozen representations. Qualitative analyses and latent-space visualizations suggest improved feature clustering and modality-related separation.

1 Introduction

Radiological image interpretation is challenging because of observer variability, subtle pathology, and heterogeneous data, while supervised task-specific models require extensive annotations and transfer poorly. OmniRad addresses these limitations with a self-supervised, heterogeneous-data-pretrained shared encoder designed for stable representation reuse across radiological tasks.

  • Motivation: Radiological interpretation is challenged by inter- and intra-observer variability, subtle pathological patterns, and the scale and heterogeneity of contemporary imaging data.These challenges motivate computational methods that extract quantitative and reproducible information from radiological images.
  • Motivation: Supervised deep radiomics models require large annotated datasets and show limited robustness when transferred across modalities, organs, or institutions.They are typically trained in a fully supervised and task-specific manner.
  • Motivation: Radiological workflows involve related classification, delineation, and exploratory report-generation tasks that rely on shared visual evidence and complementary information.Developing these tasks independently with inconsistent representations undermines feature reuse.
  • OmniRad: OmniRad learns stable and transferable representations through self-supervised pretraining on large-scale, heterogeneous radiological data spanning multiple modalities and anatomical regions.A single shared encoder is pretrained once and subsequently adapted separately to classification and segmentation, with exploratory vision–language evaluation.
  • OmniRad: OmniRad uses a task-agnostic representation paradigm in which one pretrained encoder is consistently adapted across multiple radiological analysis tasks, promoting feature stability and reuse.The work reports improvements over state-of-the-art medical foundation models across heterogeneous radiological benchmarks.

2 Related Works

Radiological analysis has progressed from interpretable handcrafted radiomics to deep learning and self-supervised foundation models, but representation stability, transferability, and reuse across modalities and tasks remain unresolved. These limitations motivate unified, task-agnostic encoders that support coherent multi-task adaptation.

  • Classical radiomics: Classical radiomics uses interpretable, reproducible descriptors of intensity, morphology, and texture for prognosis, treatment response, and disease characterization.Its features are extracted from segmented regions of interest.
  • Classical radiomics: Classical radiomics is sensitive to acquisition, reconstruction, preprocessing, and segmentation choices, limiting reproducibility and scalability across centers, modalities, organs, and tasks.Adapting handcrafted pipelines requires substantial manual engineering.
  • Deep radiomics: Deep learning learns complex anatomical and pathological patterns directly from images and performs strongly on supervised classification and segmentation tasks.Convolutional and transformer-based architectures extend beyond handcrafted descriptors.
  • Deep radiomics: Task-specific supervised training on limited annotations can couple deep radiomics representations to datasets or protocols, reducing stability, transferability, and cross-task generalization.Features learned for one task or modality may not reliably generalize to other clinical scenarios.
  • Foundation models: Self-supervised medical foundation models reduce annotation reliance and provide transferable representations, but modality or anatomy-restricted pretraining and independent downstream adaptation can fragment feature spaces.This creates inconsistent representations across classification and segmentation.
  • Motivation for OmniRad: Unified radiological foundation models are needed to integrate radiomics principles with self-supervised learning and coherent multi-task adaptation for robust, transferable, clinically reliable representations.A shared encoder pretrained once and reused across heterogeneous visual tasks prioritizes representation stability and transferability while avoiding joint-training optimization conflicts.

3 Method

OmniRad combines modified self-supervised pretraining with task-specific adaptation for classification, dense segmentation, and exploratory image captioning. Its dense pathway uses lightweight hierarchical convolutional features and a frozen encoder.

  • Radiological pretraining: OmniRad modifies DINOv2 pretraining by using only global crops on diverse RadImageNet radiological imagery, improving training stability without rollbacks or feature-collapse interventions.Both small and base variants use the shared configuration described for ViT-S and ViT-B backbones.
  • Global classification: Classification experiments evaluate OmniRad small and base models with standard supervised cross-entropy training for consistent baseline comparisons.Inputs are resized to 224 × 224 pixels and trained for 40 epochs with AdamW, learning rate 1 × 10−5, and augmentations.
  • Dense segmentation: Dense adaptation adds a parallel convolutional branch that produces lightweight hierarchical representations from 1/8 to 1/32 resolution while keeping the OmniRad encoder frozen.Intermediate transformer features from layers L = {3, 7, 11} are reshaped into spatial maps and aligned with convolutional priors.
  • Dense segmentation: The segmentation decoder uses two lightweight upsampling blocks with 2× bilinear interpolation, convolutions, normalization, ReLU, and finer-scale feature concatenation.A final 1 × 1 convolution produces segmentation logits at H/8 × W/8 resolution.
  • Exploratory captioning: In exploratory captioning, the frozen OmniRad base encoder feeds BART through projected and pooled visual tokens, with captions generated autoregressively using beam search.The Patch Merger produces K = 64 latent visual tokens; inference uses 5 beams and a maximum length of 64 tokens, targeting semantic alignment rather than clinical validation.

4 Experimental Evaluation · 4.1 Datasets

The experimental evaluation assesses OmniRad across classification, segmentation, and exploratory multimodal tasks using complementary datasets. The datasets use official train, validation, and test splits to support fair comparisons and representation-transfer assessment.

  • 4 Experimental Evaluation: The evaluation covers classification, segmentation, and exploratory multimodal experiments across the paper’s considered tasks.Classification results are reported in section 4.2, segmentation results in section 4.4, and exploratory BART-based experiments in section 4.5.
  • 4.1 Datasets: Official train, validation, and test splits are used for all datasets to encourage fair comparisons and evaluation.The dataset list and brief descriptions are provided in table 2.
  • 4.1 Datasets: RadImageNet provides over one million radiological images across 165 annotated classes spanning CT, MR, and ultrasound.Its scale and heterogeneity support visual pretraining across multiple modalities and anatomical regions.
  • 4.1 Datasets: Classification evaluation uses PneumoniaMNIST, BreastMNIST, OrganAMNIST, OrganCMNIST, and OrganSMNIST for binary and multi-class radiological image classification.The benchmarks cover chest X-ray, breast ultrasound, and abdominal CT images.
  • 4.1 Datasets: MedSegBench aggregates 35 datasets comprising more than 60,000 images for binary and multi-class segmentation across diverse medical imaging modalities.The benchmark includes tasks with up to 19 classes and standardized train, validation, and test splits.
  • 4.1 Datasets: ROCOv2 contains nearly 80,000 radiological image–text pairs for evaluating visual–semantic alignment across multiple radiological modalities.The dataset includes textual captions and medical concepts extracted from the PMC Open Access Subset.

4.2 Classification Results · Multi–organ CT benchmarks · Model capacity effects

OmniRad delivers strongest or near-strongest classification performance across heterogeneous benchmarks, with dataset-dependent capacity effects: small excels on localized structures, while base benefits heterogeneous organ recognition.

  • 4.2 Classification Results: OmniRad is evaluated using Accuracy, F1 score, and AUC across classification benchmarks.
  • 4.2 Classification Results: 89.42% F1 is achieved by OmniRad small on BreastMNIST, exceeding convolutional, transformer, and foundation-model baselines, including Radio DINO base.
  • 4.2 Classification Results: 94.85% F1 is achieved by OmniRad small on PneumoniaMNIST, establishing the best reported performance and demonstrating strong generalization on chest radiographs.
  • Multi–organ CT benchmarks: OmniRad records the highest F1 scores across OrganAMNIST, OrganCMNIST, and OrganSMNIST, including 97.30%, 95.45%, and 80.97%, respectively.On OrganCMNIST, OmniRad base improves over DINOv3 small by +0.71%; on OrganSMNIST, it improves over DINOv3 small by +2.05%.
  • Model capacity effects: The small variant dominates BreastMNIST, PneumoniaMNIST, and OrganAMNIST, whose limited anatomical diversity and localized structures favor its capacity.
  • Model capacity effects: The base variant outperforms on OrganCMNIST and OrganSMNIST, where greater anatomical heterogeneity and richer class distributions benefit increased representational capacity.
  • Model capacity effects: OmniRad establishes strongest classification performance across heterogeneous MedMNIST benchmarks, improving over existing foundation models and prior domain-specific architectures.The passage attributes this overall result to OmniRad’s radiological pretraining strategy for general-purpose radiological recognition.

4.3 Classification Ablation Study: Head-Only and LoRA Fine-Tuning

The ablation study compares full fine-tuning, head-only fine-tuning with a frozen backbone, and LoRA adaptation across MedMNIST classification benchmarks. Results are reported as F1 scores with standard deviations, with head-only fine-tuning consistently achieving strong performance.

  • Fine-Tuning Configurations: The study evaluates three strategies: full fine-tuning, head-only fine-tuning with a frozen backbone, and LoRA adaptation with two parameter settings.Full fine-tuning results are reported in Tables 3 to 7, while the ablation results appear in Table 8.
  • Ablation Findings: Head-only fine-tuning consistently achieves strong performance across the evaluated MedMNIST classification benchmarks.This configuration keeps the backbone frozen while adapting the classification head.
  • Evaluation Protocol: Table 8 reports F1 scores with standard deviations for the three fine-tuning configurations across the MedMNIST classification benchmarks.The best score in each row is highlighted in bold.

4.4 Segmentation Results

OmniRad demonstrates strong dense-prediction performance across segmentation benchmarks and evaluation metrics. Its base variant achieves the highest aggregated mIoU, Dice, and F1 scores, while leading on several modality-diverse datasets.

  • Evaluation metrics: Segmentation performance is evaluated using mIoU, Dice coefficient, and F1 across multiple dense-prediction benchmarks.Results are summarized in Tables 9–11.
  • Aggregated results: 87.93% average mIoU, 92.95% Dice, and 93.03% F1 score are achieved by OmniRad base, the strongest aggregated results among evaluated foundation models.The comparison is conducted within a unified experimental setting.
  • Benchmark results: 86.19% peak mIoU is reached on MosMedPlusMSBench, while OmniRad base leads Promise12MSBench with 93.25% and USforKidneyMSBench with 97.25%.The leading results span CT, MRI, and ultrasound benchmarks.

Model capacity effects and generalization

OmniRad’s base variant generally benefits dense prediction on large-scale and anatomically complex datasets, while the small variant remains competitive and transferable. Across heterogeneous clinical imaging domains, OmniRad’s radiomics-aware representations support segmentation, with tighter predictions but occasional failures.

  • Model capacity effects: The base variant generally outperforms the small model on large-scale and anatomically complex datasets, where greater representational capacity benefits dense anatomical delineation.The small variant remains highly competitive and often surpasses existing foundation models.
  • Generalization: OmniRad shows systematic improvements across all segmentation benchmarks, indicating radiomics-aware representations that generalize to dense prediction across heterogeneous clinical imaging domains.These findings support OmniRad’s role as a universal radiological foundation model for segmentation-driven clinical pipelines.
  • Qualitative segmentation results: Qualitatively, OmniRad often produces tighter predictions with minimal over- or under-segmentation, although occasional failures occur on datasets such as ParadentalMSBench.Compared with general radiological foundation models, DINOv3 exhibits a higher rate of missed positives.

4.5 Exploratory Captioning Results

OmniRad was explored as a frozen visual encoder for ROCOv2 captioning using a unified BART-based decoder and fixed protocol. It achieved the strongest reported captioning results and remained robust across decoding configurations, supporting visual-semantic alignment in radiological imagery.

  • Experimental setup: OmniRad was evaluated as a frozen visual encoder on ROCOv2 using a unified BART-based decoding framework and fixed experimental protocol.Performance was reported with METEOR, BLEU, BLEU–1, BLEU–4, and ROUGE–L.
  • Captioning performance: 2.97 BLEU, 19.39 BLEU–1, 0.54 BLEU–4, and 17.48 ROUGE–L were achieved by OmniRad base with beam size 5 and maximum generation length 64 tokens.OmniRad base also reached 22.45 METEOR when maximum generation length increased to 128 tokens.
  • Comparison with foundation models: OmniRad consistently produced higher automatic metric scores than CLIP, DINOv2, DINOv3, and Radio DINO under the same decoding configuration.The improvements were systematic across all major metrics.
  • Comparison with foundation models: OmniRad maintained superior BLEU, ROUGE–L, and METEOR scores across all beam sizes, including when DINOv2 and DINOv3 performed competitively under optimized decoding.This indicates greater stability across decoding settings for language grounding in radiological imagery.
  • Effect of decoding configuration: Increasing beam size from 1 to 5 consistently improved performance across encoders, while OmniRad remained robust as maximum generation length increased.The results support effective visual-semantic alignment and encoding of long-range anatomical and pathological semantics.

5 Discussion

OmniRad is presented as a stable, broadly reusable radiological backbone whose architectural adaptations and learned representations support competitive performance across classification and segmentation. The discussion also identifies limitations involving domain coverage, clinical validation, frozen-encoder evaluation, scaling, and exploratory captioning assessment.

  • Limitations and future work: Captioning remains exploratory and uses a controlled shared-decoder setting with automatic metrics, excluding task-specific fine-tuning and human assessment of clinical relevance, semantic accuracy, and hallucinations.The controlled design isolates visual representations, while prospective and expert validation are identified as necessary future directions.
  • Overall implications: The segmentation branch enables multi-scale feature extraction, allowing the transformer backbone to remain competitive with or outperform established U-Net–based architectures.This demonstrates how architectural adaptations can address limitations of plain transformer backbones in dense prediction.
  • Overall implications: OmniRad supports multiple radiological tasks with minimal architecture-specific adaptation, reducing training complexity and computational cost.The authors suggest this stability could accelerate research workflows and potential clinical deployment.
  • Statistical robustness: OmniRad consistently matches or exceeds prior models, with statistically significant gains on OrganSMNIST, MosMedPlusMSBench, and USforKidneyMSBench.The statistical comparisons used t-tests for repeated runs and one-sample t-tests for single-value baselines.
  • Representation analysis: UMAP projections show semantically coherent clusters, including adjacent left and right lung classes and clear separation of the heart.These qualitative structures are consistent with the reported robustness, generalization, low variance, and consistent downstream performance, although UMAP remains approximate.
  • Limitations and future work: OmniRad’s limitations include RadImageNet initialization, retrospective public-benchmark evaluation, predominantly frozen-encoder experiments, and computationally demanding larger-scale training.These constraints may leave domain biases, omit real-world workflow factors, limit fine-grained performance, and leave scaling saturation unexamined.

6 Conclusions

OmniRad is a radiology-driven foundation model using one self-supervised encoder across classification and segmentation, with an exploratory extension to image captioning. Across diverse benchmarks, it showed systematic improvements with a frozen encoder, supporting reusable representations for integrated clinical applications.

  • Contributions: OmniRad uses a single self-supervised encoder across classification and segmentation, with an exploratory extension to image captioning.Its task-agnostic design emphasizes stable and transferable visual representations.
  • Results: OmniRad demonstrated systematic improvements across benchmarks covering multiple modalities, anatomical regions, and task formulations.Reported gains included image-level classification, dense anatomical segmentation, and report generation.
  • Implications: Improvements were achieved with a frozen encoder during downstream adaptation, underscoring representation robustness and generalizability.The approach supports coherent feature reuse across tasks in longitudinal studies, multi-center deployments, and integrated clinical pipelines.
Loading 2602.04547v1…