PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Design of Deep Learning Methods for Biomedical Image Segmentation

Fabian Isensee, Paul F. Jäger, Simon A. A. Kohl, Jens Petersen, Klaus H. Maier-Hein

arXiv:1904.08128v2cs.CV

TL;DR

Biomedical segmentation pipelines are difficult to design because dataset properties, hardware constraints, and task requirements vary widely. The paper introduces nnU-Net, which uses systematic heuristic rules to automate pipeline design for new datasets. Across 19 competitions and 49 tasks, it surpasses most specialized pipelines and reaches state-of-the-art performance in the majority of tasks.

  • Problem

    Designing effective biomedical segmentation methods requires difficult, task-specific manual adaptation across datasets with varying properties and hardware constraints.

  • Method

    nnU-Net represents datasets and pipeline choices as fingerprints, then uses explicit heuristic rules to generate a high-quality pipeline for each dataset without further optimization.

  • Results

    Across 19 datasets and 49 segmentation tasks, nnU-Net outperforms most specialized pipelines and achieves state-of-the-art performance in the majority of evaluated tasks.

  • Takeaways & Limitations

    nnU-Net provides an out-of-the-box, standardized baseline and framework for evaluating segmentation methods across new biomedical datasets without manual task-specific adaptation.

  • Takeaways & Limitations

    nnU-Net may be suboptimal for tasks requiring domain-specific metrics or dataset properties not represented in its automatic adaptation.

Abstract

from arXiv · show

Biomedical imaging is a driver of scientific discovery and core component of medical care, currently stimulated by the field of deep learning. While semantic segmentation algorithms enable 3D image analysis and quantification in many applications, the design of respective specialised solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We propose nnU-Net, a deep learning framework that condenses the current domain knowledge and autonomously takes the key decisions required to transfer a basic architecture to different datasets and segmentation tasks. Without manual tuning, nnU-Net surpasses most specialised deep learning pipelines in 19 public international competitions and sets a new state of the art in the majority of the 49 tasks. The results demonstrate a vast hidden potential in the systematic adaptation of deep learning methods to different datasets. We make nnU-Net publicly available as an open-source tool that can effectively be used out-of-the-box, rendering state of the art segmentation accessible to non-experts and catalyzing scientific progress as a framework for automated method design.

1 Introduction

Biomedical semantic segmentation is important for scientific discovery and clinical applications, but designing effective 3D pipelines is difficult because datasets and hardware vary substantially. nnU-Net addresses this challenge by systematically adapting a basic segmentation architecture to new datasets without manual intervention.

  • Semantic segmentation converts raw biomedical images into spatially structured information for scientific discovery and clinical applications.
  • Task-specific segmentation design requires substantial expertise because small configuration errors can cause major performance drops across varying 3D datasets.Relevant variations include imaging modality, image size, voxel spacing, and class ratio.
  • Approximately 12000 studies cite U-Net, making the literature on architectural extensions difficult for non-experts and challenging even for experts to evaluate.
  • nnU-Net automatically adapts to arbitrary datasets by using dataset and pipeline fingerprints to represent data properties and algorithmic design choices.
  • Heuristic rules map dataset fingerprints to high-quality pipeline fingerprints while accounting for associated hardware constraints.
  • Without manual intervention, nnU-Net achieves state-of-the-art results across numerous international challenges and supports out-of-the-box use on custom datasets.

2 Results

nnU-Net systematically adapts a deep-learning segmentation pipeline to diverse biomedical datasets through dataset- and pipeline-fingerprint-based configuration. Across varied tasks, it provides competitive or state-of-the-art performance while showing that dataset-specific configuration and broad evaluation matter more than architectural variation alone.

  • Automated adaptation: nnU-Net defines dataset and pipeline fingerprints, then uses heuristic rules to generate data-dependent configurations for new datasets.The framework combines inferred parameters with blueprint parameters and hardware constraints.
  • Broad evaluation: 19 biomedical datasets and 49 segmentation tasks span diverse organs, tumors, lesions, cellular structures, MRI, CT, and electron microscopy.The experiments cover 10 international biomedical image-segmentation challenges.
  • Broad evaluation: nnU-Net sets a new state of the art in 29 of 49 target structures and otherwise performs on par with or close to top leaderboard entries.This result is reported across the quantitative results from all 49 segmentation tasks.
  • Configuration versus architecture: On KiTS, contributions using the same residual 3D U-Net architecture span the full range of scores and rankings, indicating strong effects from configuration choices.The analyzed methods used co-dependent pipeline parameters and showed drastically varying configurations.
  • Configuration versus architecture: A plain 3D U-Net with well-chosen hyperparameters achieves a new state of the art on the KiTS open leaderboard.The result supports the reported importance of hyperparameters over architectural variations on KiTS.
  • Cross-dataset generalization: Across ten datasets, no tested variant improves consistently, while the original nnU-Net configuration ranks first after aggregating results across datasets.The findings show why methodological conclusions based on too few datasets can be misleading.

3 Discussion

nnU-Net replaces manual, task-specific pipeline optimization with interpretable heuristic rules that adapt a basic U-Net to diverse biomedical segmentation datasets. It achieves strong cross-task performance without user intervention, while remaining suboptimal for some domain-specific tasks.

  • Automated design: nnU-Net automates 3D segmentation model design with explicit heuristic rules rather than a new architecture, loss function, or training scheme.Its design choices are inferred from dataset properties and hardware constraints, replacing complex manual optimization.
  • Performance: Across 49 diverse tasks, nnU-Net sets a new state of the art in the majority while outperforming the respective specialized pipelines.The framework was evaluated across a broad range of biomedical imaging tasks without manual intervention.
  • Interpretation: Common architectural modifications may not be necessary for state-of-the-art performance when the segmentation pipeline is configured appropriately.The authors report that performance varies widely among methods using the same network type, and hyperparameter configuration can overshadow architectural changes on unseen datasets.
  • Practical role: nnU-Net provides a standardized framework for evaluating methodological modifications across arbitrary datasets and a strong baseline requiring no task-specific adaptation.This supports large-scale comparisons without repeatedly optimizing each pipeline for each task.
  • Limitations: Automatic adaptation can be suboptimal for tasks requiring domain-specific metrics or unconsidered dataset properties.For such cases, the authors position nnU-Net as a starting point for manual modifications, including loss-function changes or specialized preprocessing.

Methods

The methods configure a plain U-Net pipeline from dataset fingerprints and hardware constraints using heuristic rules. Adaptation covers preprocessing, spacing, architecture, memory, training, inference, and model selection.

  • Preprocessing: Images are cropped to their nonzero regions and normalized using modality-dependent schemes before model training and inference.Cropping can substantially reduce brain-image sizes, while z-scoring is the default for non-CT modalities.
  • Pipeline configuration: nnU-Net generates a pipeline fingerprint from dataset properties and hardware constraints, reducing design choices to essential decisions inferred by heuristic rules.The data fingerprint captures relevant dataset parameters, while the pipeline fingerprint records the resulting method configuration.
  • Training: Training uses a plain U-Net template with 1000 epochs, Nesterov momentum, a poly learning-rate schedule, combined cross-entropy and Dice losses, and foreground oversampling.The reported optimizer uses µ = 0.99 momentum and an initial learning rate of 0.01; 33.3% of patches are guaranteed to contain foreground.
  • Inference and selection: Inference uses overlapping sliding-window predictions with Gaussian center weighting, while model selection chooses a single configuration or ensemble using cross-validated foreground Dice.The selected configurations may include 2D, 3D full-resolution, 3D low-resolution, or cascade models.
  • Spacing: Target spacing balances detail preservation against contextual coverage, with full-resolution spacing defaulting to the per-axis median training-case spacing.Larger spacing reduces image detail, whereas smaller spacing increases image size and can limit contextual information under GPU-memory constraints.
  • Architecture and memory: Patch size, batch size, and U-Net topology are adapted to voxel spacing and GPU memory while maintaining a minimum batch size of 2.The configuration prioritizes large patches for contextual information, then reduces them iteratively when memory limits are exceeded.
  • Cascade: A cascade is activated when the full-resolution patch covers less than 12.5% of the median image shape, combining global low-resolution context with local full-resolution refinement.The first network predicts on downsampled images, and the second uses those predictions as an additional input channel.

Reporting summary

Additional research-design information is available in the linked Nature Research Reporting Summary.

  • Research design: The Nature Research Reporting Summary provides further information on the research design.
  • Research design: The reporting summary is linked to the article as an additional source of methodological information.
  • Research design: Readers seeking further research-design details are directed to the linked reporting summary.

A Dataset details

The manuscript’s dataset overview identifies the challenge datasets and provides the basis for dataset fingerprints computed from training cases.

  • The dataset values are computed from training cases and form the basis of the dataset fingerprints in Figure 5.
  • The dataset overview includes references for accessing the data.
  • Table A.1 overviews the challenge datasets used in the manuscript.

B.1 Blueprint Parameters

nnU-Net uses a small family of U-Net configurations and systematic rules to adapt architecture, preprocessing, training, and inference to dataset properties and hardware constraints. It selects among configurations using validation performance while retaining a largely standardized pipeline.

  • Selecting the best U-Net configuration: nnU-Net generates three U-Net configurations and automatically selects the best one or ensemble using cross-validation.The best configuration cannot be predicted perfectly for every task from general experience alone.
  • Architecture Design decisions: The 3D full resolution U-Net is overall the best-performing configuration, but limited GPU memory can make its patches too small for large data.The 3D U-Net cascade addresses large-data cases by learning coarse low-resolution maps before full-resolution refinement.
  • Training and optimization: The pipeline combines fixed-length training, stochastic gradient descent, extensive augmentation, foreground oversampling, and averaged Dice-plus-cross-entropy loss.Training uses 1000 epochs of 250 iterations, with learning-rate decay through the polyLR schedule.
  • Inference and ensembling: Inference uses patch-based predictions with overlap and Gaussian importance weighting, while five cross-validation models are ensembled for test prediction.Fully convolutional inference is not recommended because of zero padding and instance-normalization issues.
  • Dynamic Network adaptation: Dataset fingerprints drive on-the-fly choices including target spacing, anisotropic pooling, patch shape, and network depth.The receptive field is adapted so it covers the entire training input patch.
  • Configuration of the input patch size: Patch size is maximized subject to GPU memory while retaining a minimum batch size of 2 to increase available context and support robust optimization.If memory remains available, nnU-Net increases batch size after configuring the patch size.

C Analysis of exemplary nnU-Net-generated pipelines

The exemplary pipelines show how nnU-Net adapts preprocessing and architecture to contrasting cardiac MRI and liver CT datasets. Cross-validation then compares configurations to select an effective pipeline.

  • ACDC: ACDC segments cardiac structures from cine MRI, with a typical median image shape of 9 × 237 × 256 voxels and spacing of 10 × 1.56 × 1.56 mm.
  • ACDC: For ACDC, nnU-Net applies per-image z-score normalization and configures a 2D U-Net whose patch covers the typical in-plane image shape.The 3D U-Net cascade is omitted because the 3D U-Net already covers the median image shape.
  • LiTS: LiTS segments liver and tumors in abdominal CT scans, using a global intensity normalization scheme based on foreground intensities across training cases.This differs from ACDC, where images are normalized individually.
  • LiTS: For LiTS, nnU-Net configures the 3D full-resolution U-Net with 128 × 128 × 128 voxel patches and batch size 2 after resampling.The patch size is prioritized to capture contextual information under the GPU memory constraint.
  • LiTS: LiTS cross-validation scores were 0.7625 for 2D, 0.8044 for 3D full resolution, 0.7796 for 3D low resolution, and 0.8017 for the cascade’s full-resolution model.The configurations are evaluated using average Dice across all foreground classes.

D Details on nnU-Net’s Data Augmentation

nnU-Net uses a standardized, stochastic augmentation pipeline whose spatial operations match the dimensionality and anisotropy of the input patches. Augmentations are sampled on the fly from predefined ranges and probabilities.

  • Pipeline design: All augmentations run on the fly on the CPU, and their parameters are not varied between datasets.Background workers support the augmentation pipeline.
  • Patch handling: Patches are initially sampled larger than the training size and center-cropped after rotation and scaling to reduce out-of-boundary values.
  • Spatial augmentation: Spatial augmentations are applied in 3D for 3D U-Nets but in 2D for 2D or anisotropic 3D patches.A patch is anisotropic when its largest edge is at least three times the smallest.
  • Spatial augmentation: Rotation and scaling are stochastic, with each applied at probability 0.2 and isotropic 3D rotation angles sampled from U(−30, 30) degrees.Applying both together yields probabilities of 0.16 for only scaling, 0.16 for only rotation, and 0.08 for both.
  • Cascade mask augmentation: Cascade training additionally augments low-resolution predicted masks using binary morphology and connected-component removal.Binary operators are applied with probability 0.4, while small-component removal uses probability 0.2.

E Network Architecture Configuration

nnU-Net configures segmentation architectures from dataset properties through a systematic workflow rather than manual trial and error. The evaluation spans diverse biomedical competitions selected under explicit semantic-segmentation and comparability criteria.

  • Workflow: nnU-Net runs applicable 2D, 3D full-resolution, 3D low-resolution, and 3D cascade configurations in 5-fold cross-validation.Models are trained from scratch using only challenge training data, after which cross-validation selects configurations for ensembling.
  • Workflow: Architecture configuration is iterative because GPU memory changes discontinuously with pooling operations and network depth.The workflow therefore cannot be solved analytically.
  • Evaluation design: Evaluation challenges required 3D semantic segmentation, provided training cases, separate test cases, and participant comparisons through standardized evaluation or public leaderboards.These criteria were used to select competitions for evaluation.
  • Architecture representation: Generated architectures are stored in compact kernel-size and stride lists that can be fully reconstructed when desired.Figure F.1 illustrates this representation with six kernel-size entries and five stride entries.
  • Evaluation design: The selected competitions included 19 datasets spanning diverse biomedical imaging applications, with CREMI identified as the only qualifying biological-domain competition known to the authors.The challenge-selection process aimed to demonstrate robustness and flexibility across datasets.

F.2 Compact Architecture Representation

The compact representation encodes nnU-Net architectures through kernel sizes and downsampling strides, while the surrounding evaluation reports Decathlon configurations and results. These configurations cover multiple datasets and include cross-validation-based model selection details.

  • Compact representation: nnU-Net represents each architecture with lists of convolutional kernel sizes and downsampling strides that fully specify the network.The condensed form reduces the need for many architecture figures.
  • Compact representation: Each kernel-size list entry corresponds to one encoder resolution, while each stride entry describes an encoder downsampling step.A stride of 2 halves the corresponding axis, whereas a stride of 1 leaves that axis unchanged.
  • Compact representation: The initial stride changes spacing from 5 × 1.56 × 1.56 mm to 5 × 3.12 × 3.12 mm in the second resolution step.This anisotropic stride reflects the spacing discrepancy in the example architecture.
  • Decathlon results: The Medical Segmentation Decathlon covers 10 segmentation tasks, and the preliminary nnU-Net achieved first rank in both competition phases.The current version was refined using all ten tasks and later achieved first rank on the open leaderboard.
  • Decathlon results: The current nnU-Net was applied to all ten Decathlon tasks without manual intervention and used dataset-specific normalization procedures.Reported result tables include five-fold cross-validation scores and online-platform test scores for individual datasets.

F.10 Kidney and Kidney Tumor Segmentation (KiTS) (D17)

The KiTS challenge evaluates nnU-Net on kidney and kidney-tumor segmentation using original variable-spacing data. The submission set a new state of the art on the open leaderboard and exceeded the authors’ earlier manually optimized solution.

  • Challenge setup: KiTS provided 210 training and 90 test cases in both original variable-spacing geometry and interpolated common spacing.The challenge targets kidneys and kidney tumors, with online test-set evaluation.
  • Challenge setup: The authors’ original KiTS submission used a manually designed residual 3D U-Net and achieved first rank in the challenge.The training data was modified by replacing masks for two organizer-confirmed faulty cases.
  • Results: nnU-Net set a new state of the art on the open KiTS leaderboard, outperforming the authors’ initial manually optimized solution.The open leaderboard included many competitors who entered after the original challenge.
  • Method: nnU-Net used the original non-interpolated images and handled voxel-spacing homogenization automatically without manual intervention.This application relied on nnU-Net’s ability to deal with varying voxel spacings within the dataset.
  • Reporting: The KiTS configuration and results were documented as compact architecture and result tables, including cross-validation and test-set reporting conventions.The test-set kidney Dice calculation used the union of kidney and tumor labels on the challenge platform.

F.12 Challenge on Circuit Reconstruction from Electron Microscopy Images (CREMI) (D19)

CREMI contributes a semantic segmentation task on synaptic cell membranes in serial-section electron microscopy. Limited training data prevented standard cross-validation and configuration selection, so only the 3D full-resolution configuration was used.

  • Challenge and task: The CREMI synaptic-cleft task is semantic segmentation of cell membranes at locations where cells form synapses.The data consists of serial-section transmission electron microscopy scans of Drosophila melanogaster.
  • Method: Because three training volumes were fewer than the number of splits, the authors trained five models on all volumes and ensembled their predictions.This training scheme left no validation data for selecting among nnU-Net configurations.
  • Method: The authors intervened by configuring and training only the 3D full-resolution configuration.Standard post-cross-validation selection of the best among three model configurations was therefore unavailable.
  • Architecture representation: The CREMI architecture was recorded in the compact nnU-Net configuration format used for generated network descriptions.The corresponding table points readers to the compact representation’s decoding procedure.
  • Results: The CREMI test-set submission achieved a CREMI score of 74.96, where lower is better.No validation performance estimate or test-set Dice score was available under this training scheme.

G Using nnU-Net with limited compute resources

nnU-Net was designed to reduce the computational complexity of automatically generating and evaluating multiple segmentation configurations, while keeping the effort manageable for most users.

  • Reducing computational complexity was a key motivation for nnU-Net's design.The framework aims to keep the effort of running its generated configurations manageable for most users and researchers.

G.1 Reducing the number of network trainings

nnU-Net can require substantial computation because it trains multiple configurations with five-fold cross-validation, but users can reduce this burden through manual selection or fewer training runs.

  • 10 or 20 U-Net trainings may be required, depending on whether the 3D cascade is configured, with each training taking a couple of days on one GPU.The 10-training setup uses 2D and 3D U-Nets; the 20-training setup additionally includes low- and high-resolution cascade models.
  • The 3D full-resolution U-Net is the best default starting point overall, and users may train one model on all cases or ensemble five cross-validation models.This manual choice avoids automatically training every available configuration.
  • Other configurations can perform best in some datasets, but choosing them requires dataset-specific domain knowledge.Highly anisotropic images may favor 2D U-Net, while very large images may favor the cascade, although these relationships are not guaranteed.
  • A computational shortcut is to run only one split per configuration instead of five-fold cross-validation, except where cascade processing requires five folds.The low-resolution cascade U-Net must use five-fold cross-validation to generate training-case segmentation maps for the second full-resolution cascade model.
  • nnU-Net is configured for 11GB of GPU memory and is recommended with this default setting because it was tested extensively.The requirement is described as realistic for several modern deep-learning GPUs.
Loading 1904.08128v2…