PaperlayerView exact revision on arXiv

Source-linked AI summary

MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images

Andres Diaz-Pinto, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Ihsani, Muhammad Asad, Fernando Pérez-García, Pritesh Mehta, Wenqi Li, Mona Flores, Holger R. Roth, Tom Vercauteren, Daguang Xu, Prerna Dogra, Sebastien Ourselin, Andrew Feng, M. Jorge Cardoso

arXiv:2203.12362v2cs.HCcs.CVcs.LGeess.IV

TL;DR

Annotating medical images for AI development takes time, and MONAI Label provides an open-source framework for building interactive and automatic labeling applications. In a reported DeepGrow evaluation, combined 2D and 3D use enabled 3D-volume annotation in approximately 1–2.5 minutes, reported as 10x faster than manual contouring.

  • Problem

    Annotating new datasets takes time and effort, motivating workflows designed to shorten labeling time as AI applications are developed.

    Source: §4 · p. 10, §7 · p. 14

  • Method

    MONAI Label is an open-source framework for building annotation apps with interactive and automatic segmentation that can learn from user interactions.

    Source: §1 · p. 3, §4 · p. 10, §7 · p. 14

  • Results

    At the fourth stage, combined DeepGrow 2D & 3D enabled 3D-volume annotation in approximately 1–2.5 minutes, reported as 10x faster than manual contouring.

    Source: §5.1 · p. 11

  • Takeaways & Limitations

    MONAI Label supports continuous improvement of AI annotation apps and their availability to researchers and clinical staff for evaluation.

    Source: §4 · p. 10, §7 · p. 14

Abstract

from arXiv · show

The lack of annotated datasets is a major bottleneck for training new task-specific supervised machine learning models, considering that manual annotation is extremely expensive and time-consuming. To address this problem, we present MONAI Label, a free and open-source framework that facilitates the development of applications based on artificial intelligence (AI) models that aim at reducing the time required to annotate radiology datasets. Through MONAI Label, researchers can develop AI annotation applications focusing on their domain of expertise. It allows researchers to readily deploy their apps as services, which can be made available to clinicians via their preferred user interface. Currently, MONAI Label readily supports locally installed (3D Slicer) and web-based (OHIF) frontends and offers two active learning strategies to facilitate and speed up the training of segmentation algorithms. MONAI Label allows researchers to make incremental improvements to their AI-based annotation application by making them available to other researchers and clinicians alike. Additionally, MONAI Label provides sample AI-based interactive and non-interactive labeling applications, that can be used directly off the shelf, as plug-and-play to any given dataset. Significant reduced annotation times using the interactive model can be observed on two public datasets.

1. Annotation Approaches

MONAI Label supports interactive click- and scribble-guided annotation as well as non-interactive automated segmentation. Its approaches range from whole-volume DeepGrow interaction to DeepEdit’s automatic inference and optional click-based editing.

1. Annotation Approaches

  • MONAI Label offers interactive DeepGrow, DeepEdit, and scribble-based methods alongside non-interactive automatic segmentation.The interactive methods use clicks or scribbles, while the automatic approach uses a segmentation network.Source: §1 · p. 3

1.1.1. DeepGrow Model

  • DeepGrow uses positive and negative clicks; its 2D variant annotates one slice at a time, while its 3D variant annotates whole volumes.Training simulates click guidance from false negatives and false positives; both variants annotate one label at a time.Source: §1.1.1 · pp. 3, 5

1.1.2. DeepEdit Model

  • DeepEdit combines click-free segmentation inference with optional click-based editing, and trains with both standard and simulated-click iterations.For half of training iterations, foreground and background inputs are zero; clicks are simulated for the other half.Source: §1.1.2 · p. 5

1.1.3. Scribbles-based Segmentation Model

  • Scribble workflows support online likelihood segmentation from scribbles or refinement of a deep-learning segmentation using scribbles.Scribbles can provide minimal interaction for simpler delineations or more detailed input for difficult segmentations; MONAI Label provides sample apps for both modes.Source: §1.1.3 · p. 5

1.2.1. Automated Segmentation Model

  • Automated segmentation uses a non-interactive CNN without foreground or background click channels, and users can edit its output with client-side tools.Researchers can use networks available in MONAI Core.Source: §1.2.1 · p. 5

2. Active Learning

MONAI Label’s active-learning workflow uses model uncertainty and acquisition functions to prioritize unlabeled volumes for expert annotation. DeepEdit can estimate uncertainty on unlabeled images without clicks, unlike DeepGrow.

Uncertainty Estimation

  • MONAI Label supports dropout-based uncertainty estimation because a single model is relatively cheaper to compute than an ensemble.Users can instead build applications using ensemble or Bayesian models.Source: §2 · p. 6

Acquisition Functions

  • MONAI Label provides common acquisition functions, including selecting the most uncertain volumes, while allowing users to define their own.Other acquisition-function examples include maximum cover, knapsack, and combinations of uncertainty and data representativeness via mutual information.Source: §2 · p. 6

DeepEdit and active learning

  • DeepEdit can run inference on unlabeled images for uncertainty-based selection without clicks, whereas DeepGrow needs clicks to produce a predicted mask first.Source: §2 · p. 6

Active-learning cycle

  • The active-learning cycle adds expert-labeled volumes to the training pool, trains a model, ranks unlabeled volumes by uncertainty, and repeats with selected volumes.The cycle can continue until a model of desired performance is achieved.Source: §2 · p. 6, Figure 5 · p. 9

3. Graphical User Interfaces

MONAI Label connects to both locally installed 3D Slicer and web-based OHIF interfaces. Both support user interaction with images and labels, with different integration and customization options.

3.1. Locally-installed: 3D Slicer

  • The 3D Slicer module handles user events and supports click interaction plus image and label upload.Researchers can modify the module’s interaction types and buttons to consume the MONAI Label REST API or customize applications.Source: §3.1 · p. 10

3. Graphical User Interfaces

  • Both interfaces are open-source viewers: 3D Slicer is a multi-platform research package, while OHIF is web-based and works with DICOMWeb archives.Source: §3.1 · p. 10, §3.2 · p. 10

3.2. Web-based: Open Health Imaging Foundation (OHIF)

  • OHIF is embedded as a web-based viewer that works with DICOMWeb servers and lets users create label masks and interact with MONAI Label Server.Its integration with XNAT and other large initiatives or centers can ease integration into complex workflows.Source: §3.2 · p. 10

4. Developing and Deploying MONAI Label Apps

MONAI Label lets researchers build deployable annotation apps around models, user interactions, and active learning. Its interfaces and ready-to-use utilities support both custom app behavior and template applications.

4. Developing and Deploying MONAI Label Apps

  • Researchers can develop novel annotation methods as apps and share them with clinicians and other researchers for ongoing evaluation of user experience and annotation performance.A simple Python API specifies the models, how they learn from user interactions, and how active learning is used as work progresses.Source: §4 · p. 10
  • Apps are exposed as services through a RESTful API, while the server manages metadata and caches images and labels from local datasets or DICOM servers.DICOM-server data are accessible via DICOMWeb.Source: §4 · p. 10

4.1. Application Call-flow

  • App developers can implement inference, training, and active-learning interfaces, choosing interaction behavior, models, preprocessing, data splits, and validation metrics.Apps may also define custom transforms, a main module, and external library requirements.Source: §4.1 · pp. 10–11
  • Ready-to-use template apps cover DeepGrow, DeepEdit, and automated segmentation, with Scribbles-based segmentation support across the templates.Developers can combine custom tasks and selection strategies with utilities such as DeepGrow inference and uncertainty-based image selection.Source: §4.1 · pp. 10–11

5. Experiments and Results

Experiments on spleen and cardiac MRI tasks report faster annotation with interactive MONAI Label approaches. DeepGrow and scribbles-based annotation reduce per-volume time, while DeepEdit supports training as annotation proceeds.

5.1. DeepGrow Performance

  • In the DeepGrow spleen experiment, 41 volumes were split into six validation volumes, five initial training volumes, and remaining unlabeled volumes for a four-stage annotation process.AI-model training time was tracked separately from annotation time in comparisons with traditional approaches.Source: §5.1 · p. 11
  • At stage 4, DeepGrow 2D and 3D annotated 3D volumes in approximately 1–2.5 minutes, 10× faster than advanced manual contouring.The experiment compared annotation time across stages on the Spleen MSD dataset.Source: §5.1 · p. 11, Table 1 · p. 12
  • On average, scribbles-based annotation took 2 minutes per sample, 12.5× faster than paintbrush and 6.25× faster than contour-based annotation.The method also minimized the user interactions required.Source: §5.1 · p. 11, Table 1 · p. 12

5.2. DeepEdit Performance

  • For DeepEdit on 20 cardiac MRI images, clinicians could begin training after labeling one or two images rather than manually labeling all 16 training images first.Manual annotation was measured at 10 minutes per image; the early model may be weak initially, but click-based edits to its labels reduced time spent on later images.Source: §5.2 · p. 13
  • The reported comparison of continuous annotation plus training with manual annotation of all data plus training indicates reduced annotation time and effort with MONAI Label.The comparison is described with the green and red lines in Figure 7 and Table 2.Source: §5.2 · p. 13

6. Latest MONAI Label Use Cases

MONAI Label has been used in telestroke and neurosurgical planning settings. These examples include a developing workflow for stroke lesion assessment and reported annotation-time reductions for diseased-brain segmentation.

6.1. Telestroke Service

  • In the telestroke service, an automated quantitative-labeling model was initially trained with MONAI Label and 3D Slicer for brain hemorrhage CT data.Once fully trained, it is intended to identify and measure lesion volumes during acute-stroke and follow-up imaging.Source: §6.1 · p. 13
  • Telestroke clinicians are to approve or reject predicted labels during consultation, with relabeled images used for continuous model training and cases reviewed weekly.The described workflow places the review in weekly clinical meetings involving consultants and stroke coordinators.Source: §6.1 · pp. 13–14

6.2. Neurosurgical Atlas

  • Neurosurgeons using MONAI Label active learning and its non-interactive model could reduce diseased-brain annotation time by 50% to 80% for surgical planning.The work covered 220 lesions across meningiomas, low- and high-grade gliomas, and brain metastases, using multimodality MR images.Source: §6.2 · p. 14
  • MONAI Label segmentations of diseased brains were registered to The Atlas for surgery planning using technology designed by the Neurosurgical Atlas.The use case involved tumor and surrounding structures deformed by tumors across multimodality MR images.Source: §6.2 · p. 14

7. Conclusion and Future Work

MONAI Label combines interactive and automatic annotation approaches with extensible libraries, pathology examples, and a heuristic planner. The authors identify additional interaction types as future work.

7. Conclusion and Future Work

  • MONAI Label provides a free, open-source labeling platform with interactive and non-interactive annotation approaches.The interactive methods include DeepGrow, DeepEdit, and scribbles-based segmentation; the non-interactive approach is standard deep-learning segmentation.Source: §7 · pp. 14, 16
  • Figures 8 and 9 show predicted MONAI Label masks for brain hemorrhage, brain tumor, and deformed brain regions.Source: Figure 8 · p. 15, Figure 9 · p. 16
  • The platform supports work with different deep-learning frameworks and high-level libraries, including Ignite and PyTorch Lightning.Source: §7 · p. 16
  • The XNAT datastore interface and Pathology App include interactive and automated segmentation examples for whole-slide pathology images.The examples include nuclei multi-label segmentation across several cell categories.Source: §7 · p. 16
  • The heuristic planner uses available GPU and training-set intensity and spatial information to define training and inference transforms and hyperparameters.Source: §7 · p. 16
  • Further work includes adding ROI and/or closed-curve interactions to all interactive models.Source: §7 · p. 16

Code and Data Availability Statements

MONAI Label is freely available as open-source software under the Apache-2.0 license. The paper also identifies supported environments and publicly available demonstration images.

Code and Data Availability Statements

Conflict of Interest

The authors disclose that S.O. and T.V. have company interests, and that T.V. also holds shares in Mauna Kea Technologies.

Conflict of Interest

  • S.O. and T.V. are co-founders and shareholders of Hypervision Surgical; T.V. also holds shares in Mauna Kea Technologies.Source: Conflict of Interest · p. 17
Loading 2203.12362v2…