Source-linked AI summary

nnFoundation: 3D Foundation Models for Radiology

Constantin Ulrich Harsy, Tassilo Wald, Karol Gotkowski, Yannick Kirchhoff, Marcel Knopp, Maximilian Rokuss, Elisa Stegmeier, Philipp Schader, Dasha Trofimova, Raphael Stock, Kim-Celine Kahl, Stephen Schaumann, Selen Erkan, David Zimmerer, Stefan Denner, Moritz Langenberg, Sebastian Ziegler, Katharina Eckstein, Maximilian Fischer, Jonathan Suprijadi, Bálint Kovács, Benjamin Hamm, Anand Deshpande, Dimitrios Bounias, Nico Disch, Shuhan Xiao, Jessica Kächele, Jan Sellner, Rajesh Baidya, Jeremias Traub, Lars Krämer, Maximilian Zenk, Tim Rädsch, Stefan Dvoretskii, Robin Peretzke, Jonathan Deissler, Alexandra Ertl, Partha Ghosh, Kris Dreher, Stefan Dinkelacker, Annika Reinke, Evangelia Christodoulou, Numan Saeed, Yoland Savriama, Santiago Estrada, David Kügler, Laura Alexandra Daza Barragan, Cristina Isabel Gonzalez Osorio, Jan Peeken, Michael Baumgartner, Marvin Teichmann, Guillaume Chabin, Matthias Kirchler, Valentin Koch, for the ALFA study, Markus Hohenhaus, Dimitri Koslov, Nina Decker, Mohammad Yaqub, Arnd Heuser, Martin Reuter, Julia A. Schnabel, Tobias Heimann, Florin Ghesu, Paul Brachmann, Claus P. Heußel, Alexander Radbruch, Gianluca Brugnara, Aditya Rastogi, Martha Foltyn-Dumitru, Heinz-Peter Schlemmer, Ignaz Reicht, Julius C. Holzschuh, Michael Bach, Bram Stieltjes, Kai Schlamp, Lena Maier-Hein, Marco Nolden, Ralf Floca, Paul F. Jäger, Philipp Vollmuth, Fabian Isensee, Klaus H. Maier-Hein

arXiv:2609.26924v1cs.CVeess.IV

TL;DR

Existing radiological foundation models are limited in scale and narrowly evaluated, while diverse downstream tasks motivate broader evidence. nnFoundation pairs complementary 3D CNN and transformer models and consistently outperforms prior 3D foundation models and training from scratch across task types.

  • Problem

    Existing radiological foundation-model approaches are limited in scale and narrowly evaluated, leaving their support across the diagnostic spectrum insufficiently assessed.

    Source: §1 · p. 3
  • Method

    nnFoundation comprises complementary convolutional and transformer-based 3D models trained on 2.1 million CT, MRI, and PET volumes from 125 institutional and public datasets.

    Source: §1 · p. 4
  • Results

    Across task types, nnFoundation models outperform prior 3D foundation models and training from scratch; CNNs favor localized tasks, while transformers favor global reasoning and frozen-feature settings.

    Source: §1 · p. 5
  • Takeaways & Limitations

    Transferable 3D radiological performance is not served by one universal model, but depends on scalable pretraining, complementary architectures, and dataset-aware adaptation.

    Source: §1 · p. 5

    Corpus heterogeneity and variable metadata bound the study's scope, and the authors cannot determine whether observed gains arise primarily from scale or diversity.

    Source: §3 · p. 23

Abstract

from arXiv · show

Radiological artificial intelligence has advanced rapidly, yet most systems remain narrowly task-specific, data-intensive, and fragile under domain shift. Foundation models promise more transferable and data-efficient solutions, but existing approaches are limited in scale, evaluated narrowly, and often assume that a single pretrained model can support diverse downstream tasks. Here we present nnFoundation, complementary convolutional and transformer-based 3D radiological foundation models. Developed within the Human Radiome Project (THRP), nnFoundation is trained on 2.1 million CT, MRI, and PET image volumes from 125 institutional and public datasets. We evaluate them across 108 tasks spanning segmentation, detection, classification, report generation, and image retrieval, including evaluations under domain shift, by external partners and in low-data and low-compute regimes. Across all task types, our convolution- and transformer-based nnFoundation models consistently outperform both prior 3D foundation models and training from scratch, establishing state-of-the-art performance for radiological imaging. However, performance follows a consistent task-dependent structure: the convolutional nnFoundation model dominates spatially localized tasks, whereas the transformer-based nnFoundation model excels in tasks requiring global semantic reasoning and in frozen-feature settings. Dynamically aligning the foundation model topology with the dataset characteristics post-hoc further improves transfer across heterogeneous 3D settings. These results show that transferable 3D radiological performance is governed not by a single universal model, but by the interplay of scalable pretraining, complementary architectures, and dataset-aware adaptation. We release nnFoundation models integrated into nnU-Net and nnDetection, enabling immediate application across established radiology workflows.

1 Introduction

nnFoundation introduces complementary CNN- and ViT-based 3D radiology models at large pretraining scale, with results showing that transfer depends on task and dataset alignment rather than one universal model.

  • nnFoundation comprises complementary ViT- and CNN-based 3D models, pretrained on 2.1 million CT, MRI, and PET volumes from 125 institutional and public sources.The models were developed within the Human Radiome Project.Source: §1 · p. 4
  • Evaluation spans 108 downstream tasks and 160,000 volumes across body regions, CT, MRI, and PET, five task types, domain shift, external validation, and low-data and low-compute settings.Source: §1 · p. 5
  • nnFoundation models consistently outperform prior 3D foundation models and training from scratch, with masked autoencoding the most effective objective and peak performance varying by task and architecture.Source: §1 · p. 5
  • Aligning model topology with dataset characteristics improves transfer across heterogeneous 3D data through dynamic dataset-specific adaptation without additional training.Source: §1 · p. 5
  • Pretrained models are integrated into nnU-Net and nnDetection for application to downstream tasks.Source: §1 · p. 5

2 Results

The Results section examines how nnFoundation’s pretrained CNN and ViT representations transfer across radiological tasks and settings, including domain shift, external validation, limited data and compute, and downstream workflows. It also covers task-dependent architectural strengths and dataset-aware topology adaptation.

general-purpose pretraining

  • The 2.1-million-volume pretraining corpus spans CT, MRI, and PET from 125 institutional and public sources and was described as the largest 3D radiological collection reported to date.Its broad scanner, protocol, anatomy, and population coverage addressed the comparatively narrow datasets used by prior radiological foundation models [33] [34] [35] [36] [38] [39].Source: general-purpose pretraining · p. 7

A complementary pair of pretrained 3D radiological foundation models

  • Masked autoencoding was the most performant pretraining strategy in internal benchmarking, motivating complementary CNN and ViT models trained on the full 2.1M-volume corpus.The architectures showed complementary strengths across task types.Source: A complementary pair of pretrained 3D… · p. 9

representations

  • The downstream evaluation covered 108 datasets across five task types, including domain-shift and biological imaging evaluations and external validation.The authors characterize this as the most comprehensive combination of 3D radiological pretraining scale and downstream evaluation breadth reported to date.Source: representations · p. 9
  • Across task types, pretrained nnFoundation models consistently outperformed scratch-trained and existing foundation models; for segmentation, nnFoundationCNN achieved the best average across datasets, beating next-best CTFM by 3.0 Dice points.Source: representations · p. 10
  • 8.3 AUROC points over scratch training: nnFoundationViT achieved the strongest classification mean rank (2.63), ahead of CTFM (3.63), CURIA (3.81), and VISTA3D (4.06).The pairwise rank differences among the top-performing models were within confidence intervals.Source: representations · p. 10
  • nnFoundationViT achieved the best mean ranking in image-to-image retrieval, placing first or second on every dataset, with Merlin showing comparable performance.The ViT also led tasks requiring global semantic understanding, while the CNN excelled on spatially localized tasks.Source: Figure 2 · p. 8, representations · p. 11
  • 43.7 F1 on CT-RATE: nnFoundationViT exceeded CURIA’s 42.8 and achieved a MERLIN-dataset BLEU score of 0.172 versus 0.170 for Merlin.nnFoundationViT had the best overall report-generation performance across both datasets despite Merlin having trained on MERLIN.Source: representations · p. 11
  • Results across healthy-anatomy and pathology-focused targets support nnFoundation’s large-scale pretraining as a general strategy for transferable 3D representations.The paper presents this as the first of three factors governing performance.Source: representations · p. 11

strengths

  • CNNs excel at spatially precise segmentation and detection, whereas ViT leads global classification and frozen-feature tasks, showing that architecture strengths remain task-dependent after pretraining.The results attribute this pattern to complementary spatial modeling and global semantic representations.Source: strengths · p. 12

heterogeneous 3D imaging settings

  • 1.2 Dice points across 34 segmentation datasets and 1.2 mAP points across 9 detection datasets are gained through dynamic weight adaptation with dataset-specific topology planning.The method maps pretrained encoders to dataset-specific target topologies during weight transfer, enabling reuse across heterogeneous datasets without additional training.Source: Figure 3 · p. 13, heterogeneous 3D imaging settings · pp. 14–15

robustness

  • 3 Dice points of average improvement over training from scratch are achieved by pretrained nnFoundationCNN with only 10 training cases.Across six segmentation datasets, the gains are largest at very small training-set sizes.Source: robustness · p. 15, Figure 4 · p. 16
  • 99.16% of full fine-tuning performance is retained by nnFoundationCNN using only 15% of training compute for segmentation.It still outperforms all baselines, including models trained from scratch with the full schedule.Source: Figure 4 · p. 16, robustness · p. 17
  • nnFoundationViT outperforms its from-scratch counterpart across short and extended classification schedules, although convergence is slower than for some baselines.The paper attributes slower adaptation to higher model capacity and reconstruction-based pretraining.Source: Figure 4 · p. 16, robustness · p. 17
  • Across seven transfer settings, nnFoundationCNN achieved the strongest overall robustness: 74.9 OOD Dice points, 88.8% ID retention, and a 9.5-Dice-point drop from 84.4 ID Dice points.The evaluations introduced shifts in site, scanner, acquisition protocol, and patient population.Source: robustness · p. 17, Figure 5 · p. 18
  • Independent partner evaluations found consistent pretraining benefits across local datasets, tasks, and pipelines, with the largest gains in low-data regimes despite variation between sites.Partners conducted training and evaluation locally without the core authors accessing downstream data; effect sizes varied with fine-tuning protocols, task optimization, and pipeline integration.Source: Figure 5 · p. 18, robustness · p. 19
  • Against training from scratch, nnFoundation pretraining improved robustness by +1.4% and absolute accuracy by +4.5 OOD Dice, yielding the strongest evaluated domain-shift performance.Source: robustness · p. 18
  • Exploratory 3D microscopy experiments suggested that nnFoundation representations transfer beyond radiology, with fine-tuning improving performance despite shifts in modality, acquisition scale, and biological targets.Source: robustness · p. 18

Framework integration and usability

  • Pretrained models are integrated into nnU-Net and nnDetection for segmentation and detection, with dedicated fine-tuning pipelines for classification, retrieval, and vision-language tasks.Source: Framework integration and usability · p. 19

3 Discussion

The discussion frames transferable 3D radiological performance as arising from scalable pretraining, complementary task-aligned architectures, and dataset-aware adaptation rather than a single universally optimal model. It also notes evaluation gaps and unresolved questions about transfer, representativeness, and robustness.

3 Discussion

  • Across 108 downstream tasks, nnFoundation achieved state-of-the-art performance across major radiological task types and improved transfer in low-data, low-compute, and domain-shift settings.The authors attribute transferable performance to the interplay of scalable pretraining, task-aligned architectures, and dataset-aware adaptation.Source: §3 · p. 20
  • Under the benchmark protocol, no competing foundation model surpassed nnU-Net for segmentation; only nnFoundationCNN exceeded this dataset-adaptive baseline.The discussion contrasts this broad evaluation with prior reports of competitive performance on narrower benchmarks.Source: §3 · p. 20
  • Convolutional models performed best on localized tasks such as segmentation and detection, whereas transformers excelled at global-semantic tasks, including classification, retrieval, and frozen-feature report generation.The authors conclude that no single foundation-model architecture is uniformly optimal across radiological tasks.Source: §3 · p. 20
  • Masked autoencoding was the most effective pretraining strategy overall, but it improved nnFoundationViT classification while reducing nnFoundationCNN classification performance versus training from scratch.The authors attribute this negative transfer to a mismatch between reconstruction-based pretraining and the global semantic representations required for classification in convolutional models.Source: §3 · p. 21
  • The authors frame foundation models as starting points for specialized solutions, not replacements for task-specific modeling, and release nnFoundation integrated into nnU-Net and nnDetection.They argue domain-specific optimization remains valuable and that integration lowers the barrier to practical deployment and evaluation.Source: §3 · pp. 21–22
  • Dynamic dataset-specific adaptation improved or matched fixed-topology transfer on 27 of 34 segmentation datasets, particularly under strong anisotropy or extreme aspect ratios.The method specializes a pretrained generalist to downstream datasets without additional training.Source: §3 · p. 22
  • Evaluation evidence remains uneven across task families, detection is excluded from the main cross-model rankings, and prospective validation in clinical workflows remains necessary.High-quality detection, retrieval, and report-generation datasets are scarce; baseline models also lack dedicated detection frameworks and protocols, while current external validation is retrospective.Source: §3 · p. 22
  • The study cannot determine which corpus factors drive the gains; the corpus is imbalanced across modalities and sources, and formal subgroup or fairness audits and systematic tests of artifact and rare-pathology robustness remain outstanding.The authors also note that representativeness across populations, modalities, and acquisition protocols is important before clinical translation.Source: §3 · pp. 23–24
  • Expanded and standardized benchmarks are needed to establish the remaining finding.Source: §3 · p. 24
  • Transferable 3D radiological performance depends on the combined effect of scalable pretraining, complementary task-aligned architectures, and dataset-aware adaptation.The conclusion contrasts this combination with relying on a single universal model.Source: §3 · p. 24
  • The findings may extend to volumetric biomedical imaging, where heterogeneous data challenge fixed architectures and universal transfer strategies.Source: §3 · p. 24
  • Foundation models for volumetric imaging should be viewed as adaptable representation systems whose success depends on scale, architecture, and downstream-data alignment.Source: §3 · p. 24

4 Methods

The Methods section describes the curation and training of complementary CNN and transformer models and the controlled benchmarking used to evaluate them. It details task-specific protocols across segmentation, classification, detection, report generation, and retrieval, along with efficiency and domain-shift experiments, model adaptation, and evaluation procedures.

4.1 Curating real-world clinical and internet-scale data

  • The pretraining dataset aggregated 3,271,454 volumetric imaging series from publicly available small datasets, large public cohorts, and clinical partner institutions; systematic filtering and quality control yielded 2,158,570 3D imaging volumes.Source: §4.1 · p. 24

4.2 Controlled benchmarking motivates a complementary model set

  • CNN MAE performed best on spatially localized tasks, while transformer MAE performed best on classification, motivating complementary architectures for large-scale training.The controlled benchmark covered nine validation tasks across segmentation, detection, and classification; Fig. 6a–c presents the comparison.Source: §4.2 · p. 26, Figure 6 · p. 27

4.3 Model training and scaling

  • Both architecture families used masked autoencoding, with complementary CNN and transformer backbones scaled to the full dataset; all models trained at global batch size 96 on 192 × 192 × 192 patches for 625,000 steps.Model-size scaling substantially improved transformer performance, whereas no consistent benefit was observed for convolutional models.Source: §4.3 · pp. 26–27

4.4 Baselines

  • Baselines spanned convolutional, Swin-transformer, ViT, and hybrid architectures and included contrastive, self-distillation, reconstruction, volume-fusion, vision–language, and supervised-segmentation pretraining.All baselines used the same downstream evaluation pipelines, with task-specific learning rates selected through validation sweeps.Source: §4.4 · pp. 28–29

4.5.1 Segmentation

  • Segmentation evaluation in nnU-Net covered 34 large-scale downstream datasets, excluding the five initial development-stage datasets, and used a 50/50 train-test split.Unless noted otherwise, baselines and fixed-topology nnFoundationCNN models were fine-tuned for 1,000 epochs using 250 steps per epoch and a batch size of 2.Source: §4.5.1 · p. 30
  • Segmentation transferred pretrained encoder weights while randomly initializing decoders, except for SwinUNETR and VOCO; reduced-compute runs used 150 epochs.nnFoundationCNN used SGD for 1,000-epoch fine-tuning and AdamW for the 150-epoch reduced-compute experiments.Source: §4.5.1 · p. 30

4.5.2 Classification

  • Classification was evaluated with full fine-tuning on eight datasets using 40/10/50 train/validation/test splits, except CT-RATE used its official test split.An adaptive-average-pooled encoder output fed an MLP classification head.Source: §4.5.2 · p. 31

4.5.3 Detection

  • Detection covered nine datasets using nnDetection’s default DETR detector, with a shared setup to isolate encoder pretraining and only pretrained encoder weights transferred.The remaining detector components were randomly initialized.Source: §4.5.3 · pp. 31–32

4.6 Field of View for report generation and image-to-image retrieval

  • Volumes were resampled to each encoder’s native target spacing (or 1 mm), cropped to an anatomical ROI, and evaluated across three field-of-view modes; resize mode was evaluated on MERLIN.Source: §4.6 · pp. 32–33

4.6.1 Report Generation

  • Report generation replaced Qwen2.5-VL-3B’s vision encoder with a frozen image encoder and used a 3D adapter that pooled 23 neighboring representations into the language-model space.Learning rate and field of view were swept on internal validation splits, and the first supervised next-token training stage aligned the adapter for 1.2 dataset epochs.Source: §4.6.1 · pp. 33–34
  • Report generation used a two-stage procedure, with language-model alignment followed by LoRA fine-tuning while the image encoder remained frozen.The supplied passage fragments state that alignment was intended to avoid language-model feature collapse; LoRA fine-tuning ran for 10.8 additional epochs, and the adapter remained trainable in both stages.Source: §4.6.1 · p. 34

4.6.2 Image-to-image retrieval

  • Retrieval followed the MedImageInsight benchmark, extended with MAMA-MIA tumor-stage targets, and evaluated frozen encoders without task-specific training.MAMA-MIA cases were grouped into three tumor-volume-based stages; retrieval used pooled encoder embeddings with alternative pooling strategies.Source: §4.6.2 · pp. 34–35

4.7 Label and compute efficiency experiments

  • Label efficiency used six segmentation datasets with 10, 20, 40, or 100 training cases, while compute efficiency reduced segmentation and classification training budgets.Low-data experiments used a fixed 50/50 train/test split, three seeds, shared sampled subsets across methods, and 300 nnU-Net epochs; reduced-compute budgets were 150 segmentation and 20 classification epochs.Source: §4.7 · p. 35

4.8 Robustness under domain shift

  • Domain-shift robustness was tested in seven segmentation transfer settings using reduced-compute fine-tuning and aligned source- and target-label definitions.Settings spanned held-out-center, cross-dataset, and cross-geographic, population, country, and site shifts.Source: §4.8 · pp. 35–36

4.9 Dynamic adaptation for nnFoundationCNN

  • CNN weights were adapted at initialization to nnU-Net’s dataset-planned target topology, including kernel, stage, input-channel, and positional-embedding adjustments.When target kernels were smaller, weights were averaged along reduced spatial axes; channel mismatches were handled by repeating and rescaling input projection weights, and changed patch sizes triggered trilinear positional-embedding resizing.Source: §4.9 · p. 36

4.10 Evaluation

  • Evaluation covered 58 tasks across five task families using complementary task-native metrics.Metrics included segmentation DSC and NSD, classification AUROC and AUPRC, detection mean average precision and FROC, report-generation BLEU, and retrieval average precision; dataset-level ranks were equally weighted in task-family rankings.Source: §4.10 · pp. 36–37

Affiliations

The paper lists affiliations across academic, clinical, and industry institutions and organizes credit attribution by work package and category. It distinguishes sustained central contributions from meaningful support, with core contributors and contributors sorted alphabetically.

Affiliations

  • Affiliations include German Cancer Research Center (DKFZ), Heidelberg University, and institutions in Germany, the United Kingdom, the United States, Switzerland, and the United Arab Emirates.The listed organizations include academic and clinical institutions, research centers, and industry affiliations.Source: Affiliations · pp. 38–39

Authorship and Credit Attribution

  • Credit attribution is organized by work package and category, distinguishing sustained central contributions from meaningful support and alphabetizing contributors within those groups.Source: Affiliations · p. 41

Data Acquisition & Governance

  • Data acquisition and governance credits cover clinical, open-source, and big-cohort sourcing, dataset preparation, compliance, conversion, database and query systems, storage, and IT coordination.Source: Affiliations · p. 41

Data Cleaning & Quality Control

  • Data cleaning and quality-control credits cover out-of-distribution detection, visual inspection, LLM-assisted metadata curation, deduplication, and other curation steps.Source: Affiliations · pp. 41–42

Self-supervised learning methods

  • Pretraining and downstream work packages credit contributors for self-supervised methods, compute infrastructure, data preparation, segmentation and classification fine-tuning, detection, partner coordination, and baseline implementation.Source: Affiliations · p. 42

Vision-Language Models & Retrieval

  • Separate credits recognize vision-language model development and retrieval, alongside dataset selection, validation strategy, results reporting, external validation, and data provision.Source: Affiliations · p. 42

Supplementary information … Appendix A Initial development stage

Appendix A describes exploratory experiments used to shape final pretraining, including the decision to omit fixed-spacing resampling. It reports no individual benchmark results because experimental conditions changed during iteration.

Supplementary information

  • The supplementary contents list optimization sections on MAE, contrastive learning, and DINOv2.The listed sections begin on S2, S5, and S6, respectively.Source: A Initial development stage · p. 49

B Scale comparison with prior foundation models in radiology

  • The contents list sections on generalization, exploratory out-of-domain biology segmentation, and external validation.They are listed across pages S10–S13.Source: C Additional Experiment Details · p. 49

E Downstream Datasets

  • The contents list rank summaries by task type and modality, followed by extended results for Figures 2–6.The listed material spans pages S22–S100.Source: F Detailed Results · p. 49

Appendix A Initial development stage

  • Exploratory optimization used an internal proxy benchmark with five segmentation, two detection, and two classification datasets to guide design before full-scale training.The experiments summarized tested configurations, empirical findings, and observations informing the large-scale model design.Source: Appendix A · p. 50
  • Because downstream settings, preprocessing, and optimization details changed during iterative development, the appendix does not report individual benchmark results.The proxy benchmark was intended for controlled method development, but not all experimental conditions remained fixed.Source: p. 1, Appendix A · p. 50
  • Without fixed target spacing, downstream performance did not substantially decline, so final large-scale training omitted fixed-spacing resampling.The authors expected this setting to yield more robust representations across heterogeneous acquisition spacings.Source: Appendix A · p. 50

A.1 MAE Optimization · A.2 Contrastive learning Optimization

MAE optimization favored 0.8 masking and deeper encoders, with decoder effects depending on task; CNN ablations yielded no consistent gains. Contrastive learning benefited from adjusted attention-head and learning-rate settings, larger batches with smaller crops, and model scaling.

A.1 MAE Optimization

  • A 0.8 masking ratio yielded the strongest overall downstream behavior among those tested and was selected for subsequent experiments.Source: §A.1 · pp. 50–51
  • Increasing ViT encoder depth improved performance through depth 28, whereas 12–16 attention heads and moderate embedding-dimension changes made little measurable difference.Source: §A.1 · p. 51
  • At encoder depth 16, decoder depth 2 favored segmentation while classification favored deeper decoders; at depth 28, decoder depth 4 best balanced both tasks.Source: §A.1 · p. 51
  • Both MAE architectures recovered anatomy across imaging modalities, with CNN reconstructions smoother and ViT reconstructions preserving finer structural detail.Source: Figure A1 · p. 52, §A.1 · p. 53
  • Despite extensive CNN MAE ablations, no modification consistently improved downstream performance, so the original configuration was retained with 192 × 192 × 192 input patches.Source: §A.1 · p. 53

A.2 Contrastive learning Optimization

  • Increasing contrastive-learning attention heads to 24 while lowering the learning rate to 3 × 10−5 stabilized training and improved performance; reducing projection-head dimensions slightly improved performance while reducing memory use and epoch times.Changing minimum crop overlap from 0.25 to 0.75 had no measurable influence.Source: §A.2 · pp. 53–54
  • Within a fixed compute budget, batch size 256 with crop size 32 cubic gave the best contrastive performance, particularly for segmentation and detection.Source: §A.2 · p. 54
  • Contrastive pretraining-time extensions yielded minimal gains within the compute budget, while scaling model depth to 28 had the strongest impact.Source: §A.2 · p. 54

A.3 DINOv2 optimization … C.1 Generalization

DINOv2’s downstream performance improved chiefly through longer, more stable optimization and modest model scaling, while its default crop design remained unchanged; the other sections compare model scale and describe the ID/OOD evaluation setup.

A.3 DINOv2 optimization

  • DINOv2 pretraining already provided a clear benefit over training from scratch, motivating its use in subsequent experiments.Source: §A.3 · p. 54
  • A 50-epoch warm-up, weight decay of 0.005, and per-GPU batch size of 6 each improved downstream behavior; extending pretraining to 400 epochs added a strong improvement.These changes were combined in the final selected configuration.Source: §A.3 · pp. 54–55
  • Increasing attention heads from 12 to 16 while keeping depth at 24 was part of the best-performing configuration and was adopted in the final model.The default Primus-M-style configuration started with 24 transformer blocks and 12 attention heads.Source: §A.3 · pp. 54–55
  • Alternative crop settings did not provide sufficiently robust evidence to replace the default global and local crops of 96 and 48 voxels, respectively.The default crop scale range was [0.7, 1.3].Source: §A.3 · p. 55
  • DINOv2 performance improved mainly through a longer, more stable optimization schedule and a moderate increase in model capacity, while the final selected configuration kept the default crop and augmentation design unchanged.Increasing pretraining from 100 to 400 epochs yielded an additional strong improvement in this setting. Depth fixed at 24 was part of the best-performing configuration.Source: §A.3 · p. 55

Appendix B Scale comparison with prior foundation

  • nnFoundation is reported as the largest radiology foundation model by 3D pretraining scale and as evaluated on the largest number of downstream datasets.The comparison expresses 2D slices as 3D-volume equivalents using a divisor of 200.Source: Figure B2 · p. 56, Table B1 · p. 57

C.1 Generalization

  • Generalization evaluation covered six ID/OOD axes from five datasets, with models fine-tuned on ID training splits and tested on held-out ID and OOD partitions.Table C2 reports cases as OOD/ID/finetuning counts and classes as multiclass or binary.Source: §C.1 · p. 58, Table C2 · p. 58

C.2 Exploratory Out-of-Domain Biology Segmentation

The section tests nnFoundation beyond radiology on 3D biology segmentation tasks with acquisition characteristics, spatial scales, contrast mechanisms, and target structures that differ from CT, MRI, and PET. Fine-tuning still yields improvements, suggesting radiology-scale pretraining can transfer beyond clinical imaging.

C.2 Exploratory Out-of-Domain Biology Segmentation

  • The evaluation covered 3D bioimaging with acquisition characteristics, spatial scales, contrast mechanisms, and biological targets distinct from CT, MRI, and PET.Source: §C.2 · p. 58
  • Fine-tuning nnFoundation improved performance on exploratory out-of-domain biology segmentation tasks, suggesting radiology-scale pretraining benefits transfer beyond clinical imaging.The CREMI nnFoundationViT run used 30 fine-tuning epochs; its scratch comparator and all other downstream runs used 150 epochs.Source: §C.2 · p. 58, Figure C3 · p. 59

C.3 External validation

Independent partner evaluations tested nnFoundation in fine-tuning and existing custom pipelines, finding benefits in low-data settings while revealing greater variation across datasets and institutions. External lung-nodule detection extended validation beyond segmentation to an object-detection workflow.

C.3 External validation

  • Partners independently conducted preprocessing, integration, training, and evaluation, comparing pretrained models with the same architectures trained from scratch.Partners used nnU-Net fine-tuning in most cases, while Siemens integrated the pretrained model into its existing downstream pipeline.Source: §C.3 · p. 62
  • External results fluctuated more across datasets and institutions than the internal benchmark, reflecting diverse data and pipelines, saturation, and no centralized troubleshooting.These conditions provide a realistic assessment of transferability that complements the controlled internal benchmark.Source: §C.3 · p. 62
  • Pretrained encoders showed benefits in low-data settings in an external validation beyond dense segmentation, with chest CT detection using an existing RetinaNet-based pipeline and comparison against the same architectures trained from scratch.Source: §C.3 · pp. 62–63
  • External validation covered partner datasets spanning anatomical structures and pathologies across CT and MRI, using the small nnFoundationViT because the larger model exceeded available compute budgets.The dataset overview reports modalities, segmentation labels, and train/test sizes.Source: §C.3 · pp. 61–62, Table C6 · p. 64

Appendix D Pretraining Datasets

Appendix D documents the datasets used for nnFoundation pretraining, reporting curated image counts and available modalities.

Appendix D Pretraining Datasets

  • Table D7 inventories pretraining datasets, the number of images used after curation, and available modalities.The table continues across three passages.Source: Table D7 · pp. 65–67

Appendix E Downstream Datasets · Appendix F Detailed Results

The supplied passages describe Table E8 as an overview of downstream datasets, listing their names, image counts, modalities, targets, and task types.

Appendix E Downstream Datasets

  • Table E8 covers downstream dataset names, image counts, modalities, targets, and task types.The second passage identifies the table as continued.Source: Table E8 · pp. 68–69

F.1 Rank Summary by Task Type and Modality · Figure 2 Extended Results · F.2.1 Summary Tables

The supplementary tables specify how nnFoundation ranks and Figure 2 metric scores are summarized across tasks, modalities, and datasets. The supplied passages describe table scope and comparison methods but provide no cell values to establish numerical results or winners.

F.1 Rank Summary by Task Type and Modality

  • Global ranks use equal task-type weighting, while task-specific ranks cover segmentation, classification, retrieval, and report generation, split by overall, CT, and MRI where applicable.The reported nnFoundation model is CNN for segmentation and ViT for classification, retrieval, and report generation.Source: Table F1 · p. 70
  • Pairwise overall-rank differences are comparator-minus-nnFoundation, with positive values favoring nnFoundation and daggers marking comparisons whose paired 95% CI does not lie entirely above zero.Global paired differences average task-specific differences with equal task-type weighting within each bootstrap replicate.Source: Table F2 · p. 71

Figure 2 Extended Results

  • Figure 2 ranking tables report lower-is-better mean panel ranks with 95% confidence intervals, and their model pools differ by task.Pairwise comparator-minus-nnFoundation differences use paired 95% CIs; positive values favor nnFoundation.Source: Table F3 · p. 72, Table F4 · p. 73

F.2.1 Summary Tables

  • Semantic-segmentation metric summaries cover 34 datasets, and paired comparisons report nnFoundationCNN-minus-comparator differences for DSC and NSD.Scores are dataset-equal means with fixed-dataset within-dataset bootstrap 95% CIs; positive paired differences favor nnFoundationCNN.Source: Table F5 · p. 74, Table F6 · p. 75
  • Classification metric summaries cover 8 datasets, with paired comparisons reporting nnFoundationViT-minus-comparator differences for AUROC and AUPRC.Scores are dataset-equal means with fixed-dataset within-dataset bootstrap 95% CIs; positive paired differences favor nnFoundationViT.Source: Table F7 · p. 76, Table F8 · p. 77
  • Report-generation metric summaries cover 2 datasets, with paired comparisons reporting nnFoundationViT-minus-comparator differences for BLEU, F1, and CRG.Scores are dataset-equal means with fixed-dataset within-dataset bootstrap 95% CIs; SD is reported where at least two datasets contribute.Source: Table F9 · p. 78, Table F10 · p. 79
  • Image-retrieval metric summaries cover 5 datasets, with paired comparisons reporting nnFoundationViT-minus-comparator differences in AP.Scores are dataset-equal means with fixed-dataset within-query bootstrap 95% CIs; positive paired differences favor nnFoundationViT.Source: Table F11 · p. 80, Table F12 · p. 81

F.2.2 Dataset-Level Results … F.6.2 Dataset-Level Results

This section presents dataset-level and summary results across benchmark tasks and analyses, including Figure 6 scores for segmentation, classification, and detection reported as means with 95% confidence intervals, as well as results on dynamic adaptation, efficiency, robustness, external validation, and pretraining development.

F.2.2 Dataset-Level Results

F.3.1 Summary Tables

F.3.2 Dataset-Level Results

F.4.1 Low-Data Summary

F.4.3 Low-Compute Summary

F.5.1 OOD Robustness Summary

F.5.3 External Partner Validation Summary

F.6.1 Summary Tables

F.6.2 Dataset-Level Results

  • Table F44 presents dataset-level Figure 6 segmentation scores as means and 95% confidence intervals.Source: Table F44 · p. 154
  • Table F45 presents dataset-level Figure 6 classification scores as means and 95% confidence intervals.Source: Table F45 · p. 155
  • Table F46 presents dataset-level Figure 6 detection scores as means and 95% confidence intervals.Source: Table F46 · p. 155
Loading 2609.26924v1…