PaperlayerView exact revision on arXiv

Source-linked AI summary

MONAI: An open-source framework for deep learning in healthcare

M. Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, Vishwesh Nath, Yufan He, Ziyue Xu, Ali Hatamizadeh, Andriy Myronenko, Wentao Zhu, Yun Liu, Mingxin Zheng, Yucheng Tang, Isaac Yang, Michael Zephyr, Behrooz Hashemian, Sachidanand Alle, Mohammad Zalbagi Darestani, Charlie Budd, Marc Modat, Tom Vercauteren, Guotai Wang, Yiwen Li, Yipeng Hu, Yunguan Fu, Benjamin Gorman, Hans Johnson, Brad Genereaux, Barbaros S. Erdal, Vikash Gupta, Andres Diaz-Pinto, Andre Dourson, Lena Maier-Hein, Paul F. Jaeger, Michael Baumgartner, Jayashree Kalpathy-Cramer, Mona Flores, Justin Kirby, Lee A. D. Cooper, Holger R. Roth, Daguang Xu, David Bericat, Ralf Floca, S. Kevin Zhou, Haris Shuaib, Keyvan Farahani, Klaus H. Maier-Hein, Stephen Aylward, Prerna Dogra, Sebastien Ourselin, Andrew Feng

arXiv:2211.02701v1cs.LGcs.AIcs.CV

TL;DR

Healthcare AI needs software that accounts for medical-data particularities while supporting safe, reproducible, and robust model development. MONAI addresses this need by extending PyTorch with healthcare-focused architectures, transformations, utilities, and workflows. The framework is presented as an open-source, community-supported platform intended to simplify and accelerate healthcare AI research, development, and clinical deployment.

  • Problem

    Healthcare AI requires software that handles medical-data characteristics while meeting safety, reproducibility, and robustness requirements.

  • Method

    MONAI extends PyTorch with medical-data-aware architectures, transformations, utilities, and workflow components for healthcare AI.

  • Results

    MONAI provides a standardized, flexible, open-source framework used across healthcare AI applications and supported by research, clinical, and industrial communities.

  • Takeaways & Limitations

    MONAI is intended to accelerate and simplify AI model development and contribute to research, development, and clinical deployment.

Abstract

from arXiv · show

Artificial Intelligence (AI) is having a tremendous impact across most areas of science. Applications of AI in healthcare have the potential to improve our ability to detect, diagnose, prognose, and intervene on human disease. For AI models to be used clinically, they need to be made safe, reproducible and robust, and the underlying software framework must be aware of the particularities (e.g. geometry, physiology, physics) of medical data being processed. This work introduces MONAI, a freely available, community-supported, and consortium-led PyTorch-based framework for deep learning in healthcare. MONAI extends PyTorch to support medical data, with a particular focus on imaging, and provide purpose-specific AI model architectures, transformations and utilities that streamline the development and deployment of medical AI models. MONAI follows best practices for software-development, providing an easy-to-use, robust, well-documented, and well-tested software framework. MONAI preserves the simple, additive, and compositional approach of its underlying PyTorch libraries. MONAI is being used by and receiving contributions from research, clinical and industrial teams from around the world, who are pursuing applications spanning nearly every aspect of healthcare.

1 Introduction

Healthcare AI requires medical-data-aware, safe, and reproducible software, while existing general-purpose and specialized frameworks leave technical and coordination gaps. MONAI addresses these gaps as a consortium-led framework intended to standardize and unify healthcare AI development.

  • Motivation: Healthcare AI must demonstrate safety and efficacy that outweigh potential risks while supporting detection, diagnosis, prognosis, intervention, and operational tasks.Applications also include image reconstruction, data curation, clinical safety auditing, and patient scheduling.
  • Software gap: Existing general-purpose frameworks provide mature capabilities, but they do not directly supply the medical-data functionality needed for healthcare AI.The paper identifies TensorFlow, Keras, PyTorch, JAX, and Apache MXNet as examples of general-purpose frameworks.
  • Motivation: Medical data require purpose-specific architectures, augmentation, and training mechanisms because their geometry, metadata, and acquisition physics affect processing.Building this functionality on a general-purpose framework increases development and testing demands across the R&D lifecycle.
  • Software gap: Multiple healthcare-specific platforms fragmented development, diluted effort, reduced code quality, and slowed research.The paper cites academic platforms including NiftyNet, DLTK, and DeepNeuro, alongside industry initiatives.
  • MONAI: Project MONAI unifies this field through shared best practices and components including MONAI Core, MONAI Label, and MONAI Deploy.The paper focuses on MONAI Core while noting that its design and community philosophies extend across Project MONAI components.

2 Methods

MONAI is designed as an incremental, PyTorch-native extension that preserves familiar workflows while adding healthcare-specific capabilities. Its ecosystem strategy emphasizes interoperability, low coupling, and compatibility with existing libraries and tools.

  • Design principles: MONAI follows three principles: PyTorch-like design, opt-in incremental adoption, and full integration with the PyTorch ecosystem.These principles guide the framework’s design and addition of functionality.
  • MONAI looks and feels like PyTorch: PyTorch conventions reduce the learning curve and cognitive burden for developers already familiar with PyTorch.MONAI implements features according to PyTorch and Ignite guidelines where applicable.
  • Opt-in and incremental: MONAI’s additive extensions preserve familiar transform behavior while exposing additional options for deterministic workflows.The paper gives random-number-generating transforms as an example, with behavior typical of Torchvision and extra determinism controls.
  • Opt-in and incremental: Users can incrementally add MONAI transforms, layers, loss functions, and other components to existing PyTorch or Ignite-based pipelines.The framework emphasizes low coupling rather than rebuilding complete network architectures.
  • PyTorch ecosystem collaboration: MONAI combines medically focused PyTorch libraries through a shared compose command and extends Ignite using an Ignite-native, two-way-compatible approach.This approach integrates otherwise incompatible tools and their user communities.

2.2 Open-Source Strategy

MONAI uses a permissive open-source strategy to support research and commercial development while preserving broad access to healthcare AI tools and community contributions.

  • Licensing: MONAI is licensed under Apache-2.0, allowing use, distribution, and modification for any purpose without royalties or licensing fees.The license is intended to support both research and commercial product development.
  • Licensing: Copyleft licenses were avoided because redistribution requirements were expected to reduce commercial adoption and contributions.The paper links contributions to improvements in MONAI’s quality, capabilities, and impact.
  • Integration: MONAI uses wrappers and adaptors to expose popular healthcare AI tools while requiring only PyTorch and NumPy as core dependencies.This preserves software integration and quality while improving access to external tools.
  • Consortium ecosystem: The Project MONAI consortium also develops tools for annotation, clinical deployment, federated learning, and education.Examples include MONAI Label, MONAI Deploy, MONAI FL, and MONAI Education.

2.3 System Overview and components

MONAI Core organizes healthcare AI development around specialized modules for data, losses, networks, transforms, extensions, and visualization, with workflow components for training and inference.

  • MONAI Core modules: MONAI Core includes modules for datasets and I/O, loss functions, networks, preprocessing and postprocessing transforms, C++/CUDA extensions, and visualization.These modules provide the foundational capabilities used to construct healthcare AI workflows.
  • System overview: The system overview separates MONAI Core modules at the top from workflow components at the bottom.Figure 1 presents this organization pictorially.
  • Workflow components: MONAI engines define training and evaluation workflows, while handlers implement functionality at stages of training and inference.These components assemble and integrate the core modules into workflows.

2.4 Transforms

MONAI provides composable, medical-image-specific transforms for loading, spatial and intensity processing, augmentation, and geometry-aware workflows. These transforms support reproducible pipelines across data formats and computing devices.

  • Medical-image transforms account for high-dimensional data, complex formats, metadata, and acquisition-specific geometry.Physics-informed operations include localized radiofrequency spike noise applied in MR k-space.
  • MONAI supplies transforms for medical-image I/O, spatial operations, intensity changes, cropping, padding, and augmentation.Examples include LoadImage, Spacing, Orientation, and RandGaussianNoise.
  • Transforms can be inverted to restore previous geometry for test-time augmentation, domain adaptation, or preserving original segmentation geometry.
  • Array and dictionary transforms support simple inputs or paired data with shared random operations and tracked metadata.Dictionary storage helps preserve physical image attributes after resizing.
  • Compose and OneOf enable chained pipelines and probabilistic selection among alternative transforms.
  • Most transforms support PyTorch and NumPy inputs, CPU and GPU execution, device switching, and adapters for third-party libraries.ToDevice can place computationally heavy transforms on the GPU while preserving GPU memory for training.

2.5 Engines, Loss Functions, and Metrics

MONAI extends PyTorch and Ignite workflows with medical-imaging training utilities, specialized loss functions, and problem-aware evaluation metrics. These components remain usable within existing PyTorch training loops.

  • MONAI components integrate into explicit PyTorch training loops and add functionality on top of Ignite.The framework provides specialized transform handling, default training functions, and extra event handling.
  • MONAI loss functions follow PyTorch API conventions while adding Dice variants, focal, Tversky, contrastive, and image-registration losses.
  • MONAI metrics assess model performance during training or validation and support problem-aware metric selection through Metrics Reloaded.

2.6 Network Architectures

MONAI offers published reference architectures and configurable general-purpose networks for one-, two-, and three-dimensional medical data. Its network implementations emphasize reuse, portability, and compatibility with PyTorch workflows.

  • MONAI provides reference implementations such as ResNet, BasicUNet, EfficientNet, and transformer-based networks that follow published definitions.Many implementations support configurable spatial dimensions and channel counts.
  • General-purpose networks use reusable, configurable structures for spatial dimensions, depth, layers, and network blocks.
  • UNet can operate on one-, two-, or three-dimensional inputs and outputs, with adjustable levels, channels, strides, blocks, and activations.Its default blocks combine convolution, normalization, dropout, and activation layers, which can be replaced with residual units.
  • UNet downsampling uses strided convolutions, and matching input-output dimensions require spatial dimensions that satisfy skip-connection constraints.With four stride values of (2, 2, 2, 2), activation maps are halved at each of the first four layers; input dimensions must be multiples of 24.
  • General-purpose networks are TorchScript-compatible, enabling training or inference without MONAI installed and reducing host-environment dependencies.
  • Network factories select built-in or custom layer types and appropriate spatial dimensionality, while passing constructor arguments for configuration.

2.7 Datasets and IO

MONAI represents medical images with PyTorch-compatible metadata-aware tensors and provides dataset extensions that reduce preprocessing costs and simplify access to reference datasets. These features preserve imaging information while supporting scalable training workflows.

  • MetaTensor inherits from torch.Tensor while storing and updating medical-image metadata such as DICOM or NIfTI orientation.
  • MetaTensor records applied transformations and their parameters, enabling inspection of random augmentations and inverse transforms.
  • Medical imaging datasets are memory-intensive, often have limited high-quality labels, and can impose substantial preprocessing overhead.
  • MONAI dataset extensions provide integrated caching and persistence to reduce preprocessing expense without adding user complexity.
  • CacheDataset stores deterministic preprocessing results in memory, while PersistentDataset stores them on an intermediary filesystem for large or 3D datasets.Only preprocessing steps before stochastic operations can be cached.
  • Dataset extensions simplify downloading, storage, and partitioning for MedNIST and Medical Segmentation Decathlon, while TciaDataset retrieves public Cancer Imaging Archive datasets with DICOM segmentations.

2.8 Training, Inference Engines and Event Handlers

MONAI provides workflows and engines that simplify training and evaluation setup while supporting extensible event-driven execution, distributed processing, and memory-flexible inference for large volumes.

  • MONAI workflows significantly simplify training and evaluation setup for rapid prototyping.
  • Unified workflow APIs separate domain-specific components from generic machine-learning processes and support higher-level applications such as AutoML and Federated Learning.
  • MONAI event handlers can attach independently to trainers or evaluators and trigger custom operations at defined execution events.
  • MONAI distributed data-parallel APIs work with PyTorch, Ignite, Horovod, XLA, and SLURM across multiple GPU devices and nodes.
  • Sliding-window inference analyzes large volumes in batched windows, aggregates outputs into a segmentation map, and supports overlap and blending configurations.

2.9 Visualisations and utilities

MONAI includes visualization and interpretability utilities for tracking medical-imaging workflows, recording transformation history, and examining which image regions influence classification decisions.

  • MONAI extends TensorBoard visualization for 3D medical images through GIFs, 2D slices, and interactive 3D renderings.
  • Dictionary-based transforms can retain image metadata and the stack of transformations applied to each image.
  • MONAI supports occlusion sensitivity, GradCAM, and Smoothgrad to visualize image regions important for classification decisions.
  • Test-time augmentation repeatedly applies augmentations and averages inferred outputs, using inverse spatial transformations when required.
  • Interpretability: Figure 3 shows correctly classified MedNIST head CT, chest x-ray, and hand x-ray examples alongside occlusion-sensitivity and GradCAM++ outputs.Blue marks regions on which the network relied more heavily during decision-making.
  • The blend image and matshow3d utilities support superimposed image-label visualizations and grid-based 3D volume figures.

3 Experiments and Applications

The applications demonstrate MONAI’s use across supervised segmentation, interpretability, registration, and performance-oriented training workflows, with abstractions intended to improve reproducibility and computational efficiency.

  • MONAI applications are presented to demonstrate the breadth and benefits of the framework.
  • 3.1 Segmentation: A typical supervised segmentation workflow feeds data into a network, optimizes parameters, and evaluates predictions against annotated data using a metric such as Dice.
  • 3.1 Segmentation: MONAI’s SupervisedTrainer and SupervisedEvaluator encapsulate training and validation, reducing workflow creation to object instantiation followed by execution.
  • 3.1 Segmentation: Encapsulated workflows support reproducibility and correctness by limiting manually implemented loops, method calls, and structural definitions.
  • 3.2 Determinism: MONAI routines manage random seeds and backend flags so stochastic transforms and network processes such as dropout can produce identical results between runs.
  • 3.2 Determinism: MONAI combines classification networks, losses, metrics, visualization routines, and deterministic training to support interpretation of network behavior.
  • 3.3 Registration: In registration, MONAI estimates deformation fields and combines image similarity, multiscale Dice, and bending-energy regularization losses.
  • 3.3 Registration: MONAI registration tools provide resampling options, including compiled code and interpolation models such as tricubic that are not present in PyTorch.

4 Conclusions

MONAI is presented as an open-source, community-supported, consortium-led PyTorch framework that standardizes healthcare AI development. The authors argue that its platform can simplify and accelerate biomedical and clinical AI development, research, and deployment.

  • MONAI is an open-source, community-supported, consortium-led PyTorch framework providing a standardized and flexible platform for biomedical and clinical applications.
  • The paper describes MONAI’s design considerations, implementation choices, open-source strategy, major components, and example use cases.
  • The authors believe MONAI can help accelerate and simplify AI model development and contribute to research, development, and clinical deployment.
Loading 2211.02701v1…