PaperlayerView exact revision on arXiv

Source-linked AI summary

NiftyNet: a deep-learning platform for medical imaging

Eli Gibson, Wenqi Li, Carole Sudre, Lucas Fidon, Dzhoshkun I. Shakir, Guotai Wang, Zach Eaton-Rosen, Robert Gray, Tom Doel, Yipeng Hu, Tom Whyntie, Parashkev Nachev, Marc Modat, Dean C. Barratt, Sébastien Ourselin, M. Jorge Cardoso, Tom Vercauteren

arXiv:1709.03485v2cs.CVcs.LGcs.NE

TL;DR

Medical-imaging researchers face duplicated, incompatible infrastructure because general deep-learning platforms lack functionality tailored to medical data and applications. NiftyNet provides an open-source modular platform with domain-specific pipeline components and demonstrates it across segmentation, regression, and ultrasound generation, while its small model zoo and a GAN mode-collapse example mark current boundaries.

  • Problem

    General-purpose deep-learning platforms lack medical-imaging-specific functionality, requiring substantial implementation effort and producing duplicated, incompatible infrastructure across research groups.

  • Method

    NiftyNet provides an open-source modular medical-imaging pipeline with tailored data handling, augmentation, architectures, losses, metrics, and reusable application components.

  • Results

    NiftyNet supports and illustrates segmentation, regression, image generation, and representation learning applications, including abdominal-organ segmentation, CT-map regression, and simulated ultrasound generation.

  • Takeaways & Limitations

    The platform lets researchers focus implementation on application-specific innovations while leveraging shared pipeline infrastructure and extending it to new applications.

  • Takeaways & Limitations

    The current model zoo contains only a small number of proof-of-concept models, and ultrasound interpolation includes a sharp transition suggesting GAN mode collapse.

Abstract

from arXiv · show

Medical image analysis and computer-assisted intervention problems are increasingly being addressed with deep-learning-based solutions. Established deep-learning platforms are flexible but do not provide specific functionality for medical image analysis and adapting them for this application requires substantial implementation effort. Thus, there has been substantial duplication of effort and incompatible infrastructure developed across many research groups. This work presents the open-source NiftyNet platform for deep learning in medical imaging. The ambition of NiftyNet is to accelerate and simplify the development of these solutions, and to provide a common mechanism for disseminating research outputs for the community to use, adapt and build upon. NiftyNet provides a modular deep-learning pipeline for a range of medical imaging applications including segmentation, regression, image generation and representation learning applications. Components of the NiftyNet pipeline including data loading, data augmentation, network architectures, loss functions and evaluation metrics are tailored to, and take advantage of, the idiosyncracies of medical image analysis and computer-assisted intervention. NiftyNet is built on TensorFlow and supports TensorBoard visualization of 2D and 3D images and computational graphs by default. We present 3 illustrative medical image analysis applications built using NiftyNet: (1) segmentation of multiple abdominal organs from computed tomography; (2) image regression to predict computed tomography attenuation maps from brain magnetic resonance images; and (3) generation of simulated ultrasound images for specified anatomical poses. NiftyNet enables researchers to rapidly develop and distribute deep learning solutions for segmentation, regression, image generation and representation learning applications, or extend the platform to new applications.

1. Introduction

Medical deep learning spans many tasks but requires overlapping, domain-specific infrastructure that research groups often reimplement. NiftyNet addresses this duplication with an open-source platform tailored to medical imaging and intervention.

  • Deep learning models compose simple functions into parameterized architectures optimized against loss functions, typically using gradient descent.
  • Deep learning has expanded across medical image analysis tasks and anatomical sites, while many applications share substantial software-pipeline overlap.
  • Medical imaging pipelines require specialized data loading, sampling, augmentation, networks, optimization, and evaluation because images are often large, three-dimensional, anisotropic, and metadata-rich.
  • NiftyNet provides open-source common infrastructure, pre-trained networks, and adaptation tools to facilitate efficient research and reduce duplicated implementation effort.

2. Background

General-purpose deep-learning frameworks provide flexible, efficient computational building blocks but limited medical-imaging-specific support. Related medical tools address selected building blocks or distribution needs, leaving room for a broader platform.

  • Medical image analysis and computer-assisted intervention have limited deep-learning support in existing domain-specific software infrastructure.
  • General-purpose frameworks support complex networks, automatic differentiation, and efficient hardware execution, but direct use can be challenging despite their flexibility.
  • The Deep Learning Toolkit targets fast prototyping and reproducibility through medical-imaging modules, but appears focused on building blocks rather than complete analysis pipelines.
  • NifTK and Slicer3D with DeepInfer provide infrastructure for distributing trained deep-learning pipelines.

3. Typical deep learning pipeline

A typical deep-learning project moves from model selection through evaluation to distribution, but implementing and comparing models requires substantial supporting infrastructure around the network itself.

  • The machine-learning pipeline comprises model selection, model evaluation, and model distribution phases.
  • Figure 1 represents software infrastructure as boxes and data flow as arrows in typical deep-learning projects.
  • Network implementation is only one source of complexity because researchers must also implement prior networks for comparison and infrastructure for training, evaluation, and distribution.

4. Design considerations for deep learning in medical imaging

Medical imaging imposes domain-specific design requirements arising from costly, high-dimensional, metadata-rich data and clinically meaningful errors. These constraints shape data handling, computation, evaluation, and latency requirements.

  • Medical image data acquisition, annotation, and sharing are costly because images require expertise, generation is expensive, and privacy complicates cross-institutional distribution.
  • Medical imaging data commonly spans 2D to 5D, including volumetric and longitudinal images used to assess anatomy, organ function, and disease progression.
  • High-resolution multidimensional data create large sample sizes and computational costs, motivating slices, subvolumes, anisotropic convolutions, or combinations of partial images.
  • Specialized formats such as DICOM, NIfTI, and Analyze store spatial, patient, and acquisition metadata critical to image interpretation.
  • Controlled acquisition and calibrated intensities can make medical-image distributions more predictable and spatial or intensity relationships physically meaningful.
  • Different clinical error types motivate specialized losses and evaluation metrics, while real-time intervention applications impose additional latency constraints.

5. NiftyNet: a platform for deep learning in medical imaging

NiftyNet is an open-source, modular platform that augments TensorFlow with medical-imaging-specific infrastructure and application interfaces. Its architecture supports reusable components, multiple application types, and extension to new workflows.

  • Platform scope: NiftyNet provides a high-level medical-imaging pipeline with components for data loading, sampling, augmentation, networks, losses, evaluation, and model reuse.It is built on TensorFlow and targets medical segmentation, classification, regression, image generation, and representation learning.
  • Design goals: The platform’s design goals include broad application support, component-level experimentation, ease of use, TensorFlow integration, best practices, and model distribution.These goals are intended to reduce the need to recreate unrelated parts of a deep-learning pipeline.
  • Pipeline architecture: NiftyNet applications connect Readers, Samplers, Networks, and output handlers within a common ApplicationDriver structure.Training uses losses and optimizers, while inference and evaluation use aggregators for outputs.
  • Pipeline architecture: The ApplicationDriver instantiates applications and data pipelines, distributes computation, manages variables and logging, and supports reproducible configuration.Saved configurations include default parameters so training or inference can be recreated.
  • Supported applications: NiftyNet currently supports segmentation, regression, image representation, and image generation, while its modular Application classes support adding new application types.Application classes encapsulate typical workflows and connect application-specific data, networks, losses, samplers, and training regimens.

5.5. Component details: Networks and Layers

NiftyNet organizes network construction and medical-image data flow through reusable layers, Readers, Samplers, and output handlers. These components support both modular architecture reuse and task-dependent processing during training and inference.

  • Networks and Layers: NiftyNet represents reusable conceptual network blocks as encapsulated Layer classes or TensorFlow-scoped compositions.Composite layers and complete networks can be assembled from NiftyNet layers and TensorFlow operations.
  • Data loading: Readers load corresponding medical image files and apply image-wide preprocessing, using filename matching for simple datasets or tabulated files for complex structures.The latter supports structures such as cross-validation studies.
  • Visualization: TensorBoard visualizes NiftyNet computational graphs as interconnected, color-coded conceptual blocks that can be expanded interactively.The example shows expanded generator, discriminator, and residual-block structures in a generative adversarial network.
  • Sampling and outputs: Samplers map datasets into computation-ready packets, sampling randomly during training and systematically during inference and evaluation.Packets may contain images, labels, classifications, noise samples, or other inputs required by the computation.
  • Sampling and outputs: Output handlers compute training losses and optimization updates, while inference handlers aggregate network outputs and perform postprocessing.Postprocessing can include resizing outputs to the original image size.

5.8. Component details: data normalization and augmentation

NiftyNet addresses limited medical-imaging training data through normalization and augmentation, and supports reuse through standardized model-zoo entries. These facilities target variability, reproducibility, and adaptation across datasets and applications.

  • Normalization and augmentation: Data normalization and augmentation compensate for sparse training datasets that do not represent the variability of medical images.Normalization imposes invariant intensity properties, while augmentation artificially increases training variation.
  • Normalization and augmentation: NiftyNet supports mean, variance, and histogram intensity normalization together with flip, rotation, and scaling spatial augmentation.These operations provide built-in options for modifying image intensity and spatial variation.
  • Evaluation: Reference implementations of descriptive and error metrics reduce implementation burden and inconsistencies when comparing image-analysis pipelines.Multiple metrics are reported because individual metrics capture different aspects of performance.
  • Model reuse: A model zoo enables trained networks to be used directly, fine-tuned for different data distributions, or transferred to other applications.It also hosts untrained networks and conceptual blocks for reuse.
  • Model reuse: NiftyNet can restore its own trained networks and adapt externally trained networks by wrapping them in Network classes and loading external weights.Weights can be restored for complete networks or individual conceptual blocks.
  • Model reuse: The model zoo includes untrained segmentation networks and trained models for multi-organ abdominal CT segmentation, brain-tumor segmentation, and ultrasound generation.Entries use standardized code, configuration, documentation, and, for trained models, checkpoints and training-data information.

6. Results: illustrative applications

NiftyNet was used to build illustrative pipelines for abdominal CT organ segmentation, MRI-to-CT image regression, and conditional ultrasound image generation. These applications demonstrate both practical medical-imaging workflows and diagnostic evaluation capabilities, including evidence of mode collapse in the generative model.

  • Abdominal organ segmentation: A Dense V-Net in NiftyNet segmented eight abdominal organs from 90 CT scans using 9-fold cross-validation.The training data included manual segmentations from two public datasets plus additional center-specific annotations.
  • Abdominal organ segmentation: NiftyNet’s segmentation evaluation action aggregated metrics across all folds, and Figure 4 displays the case with median-nearest Dice scores.The visualization combines posterior surface renderings with organ labels overlaid on a transverse CT slice.
  • Image regression: NiftyNet’s regression application predicted pseudo-CT attenuation maps from brain T1 MRI using a high-resolution residual-dilated-convolution network.Training used RMSE loss, 80×80×80 patches, 15,000 iterations, and 5-fold cross-validation.
  • Ultrasound simulation: A conditional GAN ported into NiftyNet generated ultrasound images for specified fetal-phantom views from conditioning images and 100-dimensional Gaussian model parameters.The original model was trained on 26,000 optically tracked ultrasound frames and added to NiftyNet’s model zoo.
  • Ultrasound simulation: Linear interpolation through the generative model produced smooth changes in ultrasound shadowing in one example but a sharp transition suggesting mode collapse in another.Both behaviors were observed while keeping the conditioning image fixed and interpolating model parameters between random samples.

7. Discussion

NiftyNet’s design emphasizes simplicity, modularity, and separation of common infrastructure from application-specific logic. The platform supports reuse and adaptation, while its roadmap identifies model coverage and experimental-design support as remaining needs.

  • Design principles: NiftyNet’s design targeted novice, intermediate, and advanced users by simplifying common use cases while permitting progressively deeper customization.The implementation choices were guided by simplicity across these three user categories.
  • Design principles: Named conceptual blocks simplify pre-trained weight adaptation and hierarchical TensorBoard visualization by grouping computation subgraphs with their associated variables.A complete conceptual block’s weights can be loaded into a new model through a single scope reference.
  • Modular architecture: Separating common ApplicationDriver logic from application-specific Application logic facilitated rapid development of new application types.Flexible Sampler and Aggregator components support differing inputs and outputs for image mapping and image generation.
  • Distribution: NiftyNet is distributed as open-source software through its website, Git repository, and Python package under an Apache 2.0 license.The consortium also welcomes external contributions and new members.
  • Future direction: The roadmap prioritizes new application types, a larger model zoo, and more advanced experimental design.Current limitations include a small proof-of-concept model zoo and user-managed data partitioning and model selection, including hyper-parameter tuning.

8. Summary of contributions and conclusions

NiftyNet provides a modular, medical-imaging-tailored deep-learning pipeline that supports rapid development and extension across several application types.

  • NiftyNet provides medical-imaging-tailored implementations for data loading, augmentation, network architectures, loss functions, and evaluation metrics.
  • Its modular pipeline lets researchers focus implementation effort on specific innovations while leveraging existing components for the rest of the workflow.
  • The platform supports segmentation, regression, image generation, and representation learning applications, while allowing extensions to new applications.
Loading 1709.03485v2…