PaperlayerView exact revision on arXiv

Source-linked AI summary

An overview of deep learning in medical imaging focusing on MRI

Alexander Selvikvåg Lundervold, Arvid Lundervold

arXiv:1811.10052v2cs.CVcs.LGstat.ML

TL;DR

Medical imaging generates complex, heterogeneous data that challenges traditional analysis, while deep learning’s expanding MRI applications and limitations require synthesis. This paper surveys deep learning across the MRI processing chain and concludes that its valuable results support continued use despite challenges.

  • Problem

    Medical imaging produces large, heterogeneous datasets that traditional analysis cannot fully process, motivating synthesis of rapidly expanding deep-learning applications and challenges in MRI.

  • Method

    The paper provides a focused overview of deep learning in MRI, spanning acquisition, reconstruction, quantitative mapping, restoration, segmentation, prediction, and supporting resources.

  • Results

    Deep learning has been applied across MRI processing, including reconstruction, quantitative susceptibility mapping, and direct estimation of tissue or pharmacokinetic parameters.

  • Takeaways & Limitations

    Deep learning in medical data analysis is positioned as an enduring approach whose valuable results justify continued development for clinical use.

  • Takeaways & Limitations

    Deep neural networks can produce accurate predictions while remaining difficult to interpret, creating challenges for trust in medical applications.

Abstract

from arXiv · show

What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI. Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of machine learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging.

1. Introduction

Recent advances in deep learning, enabled by abundant data, accessible software, and greater computing power, have made neural networks dominant across machine learning and computer vision. These developments create substantial potential for medical imaging and healthcare, where machine learning can analyze large, heterogeneous datasets.

  • Recent advances: Deep learning advances were driven by neural-network breakthroughs alongside increased access to big data, user-friendly software frameworks, and expanded compute power.
  • Computer vision: In 2012, a convolutional neural network halved the second-best ImageNet classification error rate, later surpassing human performance on the benchmark.
  • Broader impact: Deep learning is now the de facto standard for many computer-vision problems and also outperforms other approaches in language, speech, and unstructured tabular-data analysis.
  • Healthcare motivation: Healthcare providers generate enormous amounts of valuable, heterogeneous data that machine learning can integrate, analyze, and use for prediction.

2. Machine learning, artificial neural networks, deep learning

Machine learning develops models that learn from training data to produce useful outputs and generalize to unseen data. This section introduces artificial neural networks and deep learning, emphasizing learned representations and convolutional neural networks in medical imaging.

  • Machine learning models learn from training data through optimization to produce useful outputs and generalize their learned expertise to new data.The models are tuned for accurate training-data predictions, but their main goal is generalization.
  • Artificial neural networks use connected neurons arranged in input, hidden, and output layers to transform data and produce predictions.Training compares output predictions with known labels while the network identifies patterns in labeled data.
  • ANNs are flexible enough to model complicated problems but were difficult and computationally expensive to train; big data and powerful processors helped make them dominant.The passage describes this shift as occurring only recently, after practical limitations had reduced their utility.
  • Deep learning automatically learns useful representations and features directly from raw data, merging feature discovery and task performance into one training problem.This bypasses manually designed feature extraction and allows both components to improve during the same training process.
  • Medical-imaging interest in deep learning is largely driven by convolutional neural networks, which learn image representations directly from structured data instead of relying on handcrafted features.Before CNNs could be used efficiently, image features were typically engineered manually or produced by less powerful models.

3. Deep learning, medical imaging and MRI

Deep learning is increasingly applied across medical imaging, spanning radiology, MRI, radiotherapy, PET-MRI, radiomics, and deformable registration. In MRI, applications cover entire workflows from acquisition and image retrieval through segmentation and disease prediction, including both signal processing and image- and report-based systems.

  • Deep learning is increasingly used in medical imaging, prompting broad application across modalities, organs, and clinical problems rather than a comprehensive survey here.The section presents only a sketch of the landscape before a systematic exposition of deep learning in MRI.
  • CNNs support radiology protocol determination from short-text classification and reduce gadolinium dose in contrast-enhanced brain MRI by an order of magnitude without significant image-quality reduction.Related applications include radiotherapy, PET-MRI attenuation correction, and radiomics.
  • Deep learning enables advanced deformable registration for quantitative analysis across physical imaging modalities and time, including MRI–ultrasound prostate biopsy guidance and brain-MRI displacement estimation.The cited brain-MRI approach uses a cue-aware deep regression network to learn displacement vectors from paired reference-subject patches.
  • MRI applications span entire workflows—from acquisition and image retrieval to segmentation and disease prediction—and include image restoration, multimodal registration, and image-and-report systems.Examples address the brain, kidney, prostate, and spine.

3.1. From image acquisition to image registration

Deep learning is extending MRI analysis beyond reconstructed-image segmentation and classification into acquisition, reconstruction, quantitative parameter estimation, denoising, synthesis, and registration. Applications include learning reconstructions from undersampled complex-valued k-space, estimating tissue properties from MR fingerprints, and processing complex MRI data with specialized networks.

  • From image acquisition to image registration: Deep learning now spans the MRI processing chain from complex-valued k-space acquisition and reconstruction through restoration, synthesis, and multimodal image registration.This represents an expansion beyond the field’s earlier emphasis on segmentation and classification of reconstructed magnitude images.
  • Image reconstruction: CNN- and RNN-based reconstruction methods learn spatio-temporal dependencies from highly undersampled complex-valued k-space data and can outperform 3D CNN and compressed-sensing reconstructions.The research area is rapidly expanding, with applications including dynamic and cardiac MR reconstruction.
  • Quantitative tissue parameter estimation: QSMnet uses a three-dimensional U-Net-based CNN to generate high-quality susceptibility source maps from MRI measurements.QSM estimates biological-tissue magnetic susceptibility by addressing an ill-posed inverse problem relating local magnetic fields to susceptibility.
  • Quantitative tissue parameter estimation: Deep learning reformulates MR fingerprinting as a learned mapping from recorded signal magnitudes to tissue parameters, replacing or complementing dictionary-based pattern matching.MRF uses pseudo-randomized acquisitions whose tissue-dependent signal evolutions are matched to predicted signal dictionaries.
  • Quantitative tissue parameter estimation: Complex-valued neural networks may better uncover information in complex MRI data than analogous networks that separately process real and imaginary components.This conclusion arose in MR fingerprinting experiments comparing complex-valued networks with dictionary matching and real-valued alternatives.
  • Image restoration: Deep learning has also been introduced for MRI denoising and may enable direct estimation of pharmacokinetic parameter maps from highly undersampled dynamic DCE-MRI data.An autoencoder with skip connections was tested for denoising T1-weighted brain MRI, while direct mapping from (k, t)-space data is presented as a prospective approach.

3.2. From image segmentation to diagnosis and prediction

This section surveys higher-level deep learning applications in MRI, including image segmentation, disease prediction, and content-based image retrieval. It briefly focuses on applications in brain, kidney, prostate, and spine imaging.

  • Scope: Deep learning in MRI supports downstream tasks including fast, accurate image segmentation, disease prediction, and content-based image retrieval.These applications are considered after lower-level methods and are typically applied to reconstructed magnitude images.
  • Image segmentation: Image segmentation partitions images into regions with similar attributes, enabling anatomical localization and quantification.Segmentation has become a major target for deep learning in medical imaging and is also crucial for functional imaging.
  • Organ-specific applications: Organ-specific MRI applications discussed here are restricted to the brain, kidney, prostate, and spine.The overview adds more recent work using deep learning for these four organs.
  • Organ-specific applications: CNNs have been used to improve segmentation of the left kidney, right kidney, liver, spleen, and stomach in T2-weighted MR images.The cited abdominal-organ application addresses multiple organs in T2-weighted MRI.
  • Organ-specific applications: An artificial multi-observer deep neural network enabled fully automated segmentation of polycystic kidneys.The application concerns cyst segmentation in polycystic kidneys.
  • Organ-specific applications: A stacked non-negative constrained autoencoder classifier distinguished rejected from non-rejected renal transplants in DWI recordings.This renal-transplant application uses a deep-learning-based classifier.

PROSTATE

Deep learning methods were applied across prostate and spinal MRI analysis, including segmentation, cancer classification, lesion localization, vertebral labeling, stenosis grading, and spondylitis or metastasis detection. The section also notes emerging uses of deep learning for radiology image retrieval and report generation.

  • Prostate: Nested edge-detection and fully convolutional networks enabled end-to-end prostate segmentation, improving boundary detection on MRI scans from 250 patients.The method used holistic image-to-image nested networks and reported very good Dice results with 5-fold cross-validation.
  • Prostate: CNN-based systems classified prostate cancer versus noncancerous tissue using prostate MRI, multiparametric MRI, and clinical or biopsy-linked datasets.Studies used data from 301 patients, the PROSTATEx database, 172 patients with more than 2500 T2-w images, and 160-patient multiparametric MRI data.
  • Prostate: Multimodel CNNs addressed prostate cancer diagnosis using multimodal MRI, lesion and noncancerous patches, and strategies for augmenting limited training data.The study used 364 patients, 463 prostate cancer lesions, and 450 identified noncancerous image patches while investigating augmentation and multimodal MRI design.
  • Spine: CNN architectures supported vertebral detection, labeling, intervertebral disc localization and segmentation, lumbar segmentation, disc-level designation, and spinal stenosis grading from MR images.Approaches included clinical-annotation training, 3D multi-scale fully connected CNNs with random modality voxel dropout, and DeepSPINE’s U-Net with spine-curve fitting.
  • Spine: Deep multiscale multitask learning simultaneously localized and graded neural foramina, vertebrae, and intervertebral discs for pathogenesis-based diagnosis of lumbar neural foraminal stenosis.The proposed DMML-Net combined multiscale multi-output learning with multitask regression in a fully convolutional network.
  • Other MRI and radiology applications: Deep learning was also used to distinguish tuberculous from pyogenic spondylitis, detect spinal metastases, retrieve similar radiology cases, and generate or identify findings in radiology reports.Applications included comparison with three skilled radiologists on 80 patients, metastasis evaluation on 26 cases, CBIR for radiologist decision support, and LSTM-based report generation.

4. Open science and reproducible research in machine learning for medical imaging

Open science helps machine learning keep pace with rapid progress despite limits of conventional peer review, through shared preprints, code, data, and educational resources. Medical imaging increasingly follows this model, with openly available implementations, datasets, and challenges supporting experimentation and reproducibility.

  • Open science and reproducible research: Rapid machine-learning progress has outpaced conventional peer review, while shared code and data partly mitigate the drawbacks of relying on preprints and delayed conference publications.Many influential papers are available only as preprints or appear in proceedings after their results are already widely known and used.
  • Open science and reproducible research: Researchers can start projects by combining an arXiv preprint, GitHub implementation, openly available dataset, and online educational resource.The passage identifies arXiv, GitHub, public repositories, and online courses as accessible entry points to the field.
  • Open science and reproducible research: Competitions attract many participants, stimulate new ideas, and often advance the state of the art, with ILSVRC driving deep-learning progress in computer vision since 2012.Competitions also provide natural entry points for researchers, although the passage truncates the explanation of that role.
  • Open science and reproducible research: Machine learning for medical image analysis increasingly embraces openness through publicly available implementations, datasets, repositories, and challenges.The authors state that they list examples in Tables 3, 4, and 5 and provide an accompanying GitHub repository.
  • Open datasets: OpenNeuro provides public-domain neuroimaging data from 168 studies and 4,718 participants across imaging modalities and acquisition protocols.It is described as an open platform for sharing neuroimaging data.
  • Open datasets: UK Biobank contains health data from half a million participants, including MRI images from 15.000 participants, with a target of 100.000.These figures describe the dataset’s current MRI coverage and stated expansion aim.

5. Challenges, limitations and future perspectives

Deep learning offers valuable capabilities for medical imaging but faces unresolved challenges involving three-dimensional computation, privacy, data requirements, interpretability, and clinical workflow integration. Despite these obstacles, its results are valuable enough that deep learning is expected to remain important in medical data analysis.

  • Technical challenges: Medical imaging remains less settled than standard computer vision because three-dimensional CNNs impose substantial memory and computational demands.Researchers address this by treating 3D data as stacks of 2D images or using patch- or segment-based training.
  • Privacy and data protection: Deep neural networks raise major data-access, privacy, and data-protection challenges, including the possibility of inferring training-set details from the model or its prediction interface.Federated learning and differential privacy are identified as likely areas of future research.
  • Data requirements: Deep neural networks require large, representative labeled datasets, while expert annotations are expensive or unavailable.Transfer learning, data augmentation, and data synthesis can partially alleviate these requirements, although standard high-performing networks remain data-hungry.
  • Interpretability and trust: The black-box nature of hierarchical deep neural networks makes accurate predictions difficult to interpret and trust.Developing explainability methods and robust uncertainty estimates are major efforts aimed at increasing trustworthiness.
  • Clinical workflow integration: Successful clinical adoption requires integrating machine-learning systems into established workflows and involving end users in system creation and evaluation.Otherwise, technically clever systems may be practically useless to clinicians.
  • Future perspectives: Deep learning in medical data analysis is expected to persist because its valuable results are demonstrated by numerous high-impact medical-imaging publications.The authors also view medical imaging’s attention to machine learning as an opportunity to strengthen computational medicine.
Loading 1811.10052v2…