Source-linked AI summary
nnInteractive: Redefining 3D Promptable Segmentation
Fabian Isensee, Maximilian Rokuss, Lars Krämer, Stefan Dinkelacker, Ashis Ravindran, Florian Stritzke, Benjamin Hamm, Tassilo Wald, Moritz Langenberg, Constantin Ulrich, Jonathan Deissler, Ralf Floca, Klaus Maier-Hein
TL;DR
Existing 3D interactive segmentation methods are limited in volumetric awareness, prompting diversity, generalization, and integration into established imaging platforms. nnInteractive addresses these gaps with diverse 2D prompts for full 3D masks, training on over 120 multimodal datasets, and integration into Napari and MITK; extensive benchmarking reports superior performance and usability across imaging tasks.
Problem
Existing interactive segmentation approaches remain limited in volumetric awareness, interaction types, supported structures and modalities, generalization, and integration into established imaging platforms.
Method
nnInteractive is a 3D interactive open-set framework that converts points and structured 2D annotations, including scribbles, bounding boxes, and lassos, into complete 3D segmentation masks.
Results
Extensive benchmarking reports that nnInteractive delivers superior segmentation performance and usability across diverse imaging tasks while far surpassing existing methods.
Takeaways & Limitations
Training on 120+ multimodal 3D datasets and integration into Napari and MITK support practical use across clinical and research workflows.
Takeaways & Limitations
3D bounding boxes were excluded because they were cumbersome, offered no measurable advantage over 2D boxes, and often produced false positives from excess empty space.
Abstract
from arXiv · showhide
Accurate and efficient 3D segmentation is essential for both clinical and research applications. While foundation models like SAM have revolutionized interactive segmentation, their 2D design and domain shift limitations make them ill-suited for 3D medical images. Current adaptations address some of these challenges but remain limited, either lacking volumetric awareness, offering restricted interactivity, or supporting only a small set of structures and modalities. Usability also remains a challenge, as current tools are rarely integrated into established imaging platforms and often rely on cumbersome web-based interfaces with restricted functionality. We introduce nnInteractive, the first comprehensive 3D interactive open-set segmentation method. It supports diverse prompts-including points, scribbles, boxes, and a novel lasso prompt-while leveraging intuitive 2D interactions to generate full 3D segmentations. Trained on 120+ diverse volumetric 3D datasets (CT, MRI, PET, 3D Microscopy, etc.), nnInteractive sets a new state-of-the-art in accuracy, adaptability, and usability. Crucially, it is the first method integrated into widely used image viewers (e.g., Napari, MITK), ensuring broad accessibility for real-world clinical and research applications. Extensive benchmarking demonstrates that nnInteractive far surpasses existing methods, setting a new standard for AI-driven interactive 3D segmentation. nnInteractive is publicly available: https://github.com/MIC-DKFZ/napari-nninteractive (Napari plugin), https://www.mitk.org/MITK-nnInteractive (MITK integration), https://github.com/MIC-DKFZ/nnInteractive (Python backend).
1. Introduction
nnInteractive addresses the limitations of 2D and narrowly specialized interactive segmentation through a 3D open-set framework built for diverse prompts, volumetric context, and practical use.
- Existing interactive medical segmentation models primarily operate on 2D slices and fail to account for volumetric structure.
- nnInteractive supports points, scribbles, bounding boxes, and lasso prompts, including positive and negative guidance for full 3D segmentation.
- The model uses an nnU-Net-based UNet design with early prompt incorporation rather than introducing a new architecture.
- The framework converts intuitive 2D annotations placed on any plane into complete 3D masks, reducing annotation effort.
- Its training process simulates iterative user interactions by identifying false-positive and false-negative regions and sampling corrective prompts.
- Auto Zoom adaptively expands the region of interest to prevent patch-wise truncation of structures larger than the model patch size.
3. Training Data
nnInteractive is trained on a large, heterogeneous collection of volumetric segmentation data and supplements real labels with foundation-model-derived SuperVoxels to improve robustness to unseen structures.
- The training collection contains over 120 publicly available 3D datasets, 64,518 volumes, and 717,148 objects across multiple imaging modalities and structures.
- Pseudo-labels are sampled with probability 0.2 to increase robustness to unseen structures.
- SAM’s automatic segmentation generates high-confidence SuperVoxels with confidence of at least 92%.
4. Experiments
The experiments compare nnInteractive with established 2D and 3D methods across prompting styles, expert scribbles, out-of-distribution data, and clinical usability scenarios.
- The evaluation includes comparisons with established 2D and 3D models, expert scribbles, supported prompting styles, and a medical-doctor user study.
- The primary comparison uses the RadioActive Benchmark to provide a comprehensive and reproducible evaluation against existing methods.
- Point and 3D box evaluations account for different prompt burdens by giving 2D models slice-wise prompts and 3D models volumetric prompts.
- Expert-scribble evaluation uses unseen MS-CMRSeg data with annotations for the left ventricle, right ventricle, and myocardium.
- The study also evaluates advanced interactions that lack direct 3D analogues in competing methods, including lasso and 2D boxes and scribbles.
- The user study measures segmentation of 12 tumor lesions and compares prompt-guided and manual annotation against two expert raters.
- Single-prompt testing reports mean Dice scores for points, 3D boxes, and out-of-plane scribbles on unseen test data.
5. Results & Discussion
Across static and interactive evaluations, nnInteractive consistently outperforms competing methods, supports flexible prompting, and maintains practical efficiency and clinical-level performance.
- Comparison with SOTA: 10.1 Dice points separate nnInteractive from ScribblePrompt for point prompts, while its advantages reach 14.9 points for boxes and 23.8 points for scribbles.For interactive refinement, nnInteractive also exceeds ScribblePrompt by 11.2 Dice points and SegVol by 14.9 Dice points.
- Comparison with SOTA: Nearly 7 Dice points higher on average, nnInteractive outperforms ScribblePrompt on expert scribbles across three target structures.With scribbles on only three annotated slices, it reaches a Dice score of 84.3, exceeding ScribblePrompt by 3 points.
- Prompting Styles: An AUC of 83.42 makes lasso the strongest prompting style, while points are weakest at 71.76 and mixed interactions remain supported.Scribbles surpass boxes after five iterations, whereas boxes provide stronger initial guidance but weaker later refinement.
- AutoZoom: AutoZoom improves large-object performance, raising AUC from 91.92 to 95.40 for HCC Tace liver and from 92.54 to 94.81 for InsectAnatomy.Overall Test and OOD improvement is minimal because those datasets contain mostly small objects.
- Inference Time: Below 10 GB of VRAM and 120–200 ms inference for small structures support efficient deployment, while large or high-resolution cases take longer.Reported times reach 1160 ms for a liver CT and 3700 ms in rare high-resolution cases.
- Real-World Impact: In clinical tumor annotation, nnInteractive matches specialist-consistency levels while reducing per-case segmentation time by 72%.Experts required 179±114 seconds with nnInteractive versus 635±343 seconds for manual annotation.
6. Conclusion
The paper presents nnInteractive as a universal 3D promptable segmentation framework that combines diverse interactions, multimodal training, strong performance, and integration into established imaging platforms.
- 6. Conclusion: nnInteractive supports points, scribbles, bounding boxes, and lasso to convert intuitive 2D interactions into full 3D volumetric segmentations.The framework is trained on more than 120 multimodal 3D datasets and is designed for broad imaging-task coverage.
- 6. Conclusion: Extensive benchmarking reports state-of-the-art accuracy and reduced annotation effort across a wide range of imaging tasks.The conclusion describes performance as surpassing existing methods while improving usability.
- 6. Conclusion: Integration with Napari and MITK Workbench supports adoption within clinical and research workflows.The paper frames these integrations as enabling seamless real-world use of interactive 3D segmentation.
Overview
The appendix provides supporting implementation details, training configurations, dataset descriptions, and additional experimental results for nnInteractive.
- Overview: The appendix documents implementation specifics, training configurations, dataset descriptions, and supplementary experimental results.It also includes sections on preprocessing, training, augmentation, label conversion, patch sampling, and user simulation.
A1. Implementation Details
nnInteractive uses image-level normalization, native-resolution processing, extensive patch-based training, instance labels, object-focused sampling, and simulated follow-up interactions.
- Intensity Normalization: Image-level z-score normalization subtracts each image’s mean intensity and divides by its standard deviation.This normalization is applied independently for each image.
- Resampling: Native-resolution processing avoids resampling voxel spacing to preserve compatibility with out-of-distribution datasets.This accommodates datasets whose spacing lies outside common 3D medical-image ranges.
- Model Training: Training runs for 5000 epochs, with 250 iterations per epoch, batch size 24, and 192x192x192-voxel patches used during training and inference.Augmentation extends the default nnU-Net scheme with broader and partly axis-independent scaling.
- Training Labels: The model performs instance segmentation, converting semantic masks through connected-component analysis with optional morphological operations.Training labels encode object instances rather than semantic class identities.
- Patch Sampling: Patch sampling selects a case, an object, and then a pixel within that object, using dataset- and case-level sampling heuristics.This procedure directs training toward objects rather than uniformly sampled image locations.
- User Simulation and Follow-up Interactions: A separate process pool simulates follow-up interactions by refining predictions using the current patch, prior interactions, and a selected user agent.The follow-up probability increases linearly from 0.3 at epoch 0 to 0.75 during training.
A2. 3D Bounding Box Variant of nnInteractive
The 3D bounding-box variant extends nnInteractive’s design principles to volumetric boxes but was excluded from the final system because the interaction is cumbersome and introduced practical drawbacks.
- The 3D bounding-box variant replaced 2D boxes and lasso interactions and was trained for 2,000 rather than 5,000 epochs due to computational constraints.Despite the shorter training schedule, it was used to test whether nnInteractive’s design principles extend to 3D bounding boxes.
- 3D bounding boxes were considered impractical because users must create them volumetrically, and they offered no measurable advantage over 2D boxes.
- The final nnInteractive version excluded 3D boxes because they added complexity and negatively affected 2D-box and lasso performance when sharing input channels.Supporting them would have increased the input channels from 7 to 9.
A3. Dataset
The dataset section describes the training and evaluation resources and highlights nnInteractive’s ability to resolve ambiguities across structures and datasets.
- Table A1 summarizes the datasets used for training, while Table A2 presents the test and out-of-distribution datasets.
- nnInteractive dynamically adapts to user input and resolves ambiguities with minimal interaction.
- Vista3D produces plausible kidney or kidney-tumor segmentations but cannot segment the kidney including the tumor when that combination conflicts with its training labels.
- nnInteractive successfully resolves the liver ambiguity shown in the MSD Task 3 test example.
A5. Run Time
Inference is rapid for objects within the model patch size, while larger objects require AutoZoom and refinement steps that increase runtime; later refinements are typically faster.
- Objects fitting within the 192x192x192 patch are predicted in 120–200 ms.
- AutoZoom increases inference time for larger objects because it requires zoom-out and refinement steps.For liver 201, the reported zoom-out factor is 1.77–2.25 with 4–5 refinement boxes.
- Inference time can exceed 3500 ms for large organs such as the liver.
- The initial interaction typically takes longest, while subsequent refinements are quicker as smaller regions are edited.
- No measurable inference-speed difference was observed among the supported interaction types.
A6. Additional results
Additional results show strong performance on large objects, diverse benchmarks, and far out-of-distribution data, with supporting dataset, runtime, and qualitative-result resources.
- A6.1. AutoZoom: AutoZoom provides substantial benefits for large objects, achieving high Dice scores with fewer iterations.It is described as essential for large objects such as livers in CT scans.
- A6.2. Qualitative Results: Across tested objects and prompting styles, nnInteractive consistently delivers maximum segmentation accuracy while competing methods often produce severe artifacts.
- A6.2. Qualitative Results: 2D methods show inconsistent results between slices, causing major drops in measured performance, while SegVol achieves the second-best results after nnInteractive.
- A6.2. Qualitative Results: On far out-of-distribution examples, nnInteractive segments dinosaur jawbones and individual sandstone grains.
- A6.1. AutoZoom: Figure A3 evaluates AutoZoom’s effect across test and OOD datasets and highlights larger gains for large-object datasets.
- Tables A1–A3 document the training datasets, held-out evaluation datasets, and inference runtimes measured with the Napari plugin.
- A6.2. Qualitative Results: Figure A4 presents 2D cross-sections above 3D renderings, with crossed fields marking unsupported prompting styles.