Source-linked AI summary
Med3D: Transfer Learning for 3D Medical Image Analysis
Sihong Chen, Kai Ma, Yefeng Zheng
TL;DR
Large annotated datasets enable effective pretraining, but 3D medical imaging lacks sufficiently large public datasets because acquisition and annotation are difficult. The paper builds the heterogeneous 3DSeg-8 dataset and trains Med3D models for transfer across medical tasks, reporting superior results, including 94.6% Dice on LiTS without post-processing.
Problem
3D medical imaging lacks sufficiently large public datasets for pretraining because data acquisition and detailed 3D annotation are difficult.
Method
The paper aggregates datasets across medical domains into 3DSeg-8 and heterogeneously trains Med3D networks whose features transfer to other 3D medical tasks.
Results
Med3D networks achieve superior results to natural-video-pretrained and scratch-trained networks across transferred medical tasks, including 94.6% Dice on LiTS without post-processing.
Takeaways & Limitations
Med3D pretrained features are effective and generalized for other tasks with small training datasets.
Takeaways & Limitations
The source dataset combines MRI and CT volumes with varied scan regions, target organs, and multicenter characteristics.
Abstract
from arXiv · showhide
The performance on deep learning is significantly affected by volume of training data. Models pre-trained from massive dataset such as ImageNet become a powerful weapon for speeding up training convergence and improving accuracy. Similarly, models based on large dataset are important for the development of deep learning in 3D medical images. However, it is extremely challenging to build a sufficiently large dataset due to difficulty of data acquisition and annotation in 3D medical imaging. We aggregate the dataset from several medical challenges to build 3DSeg-8 dataset with diverse modalities, target organs, and pathologies. To extract general medical three-dimension (3D) features, we design a heterogeneous 3D network called Med3D to co-train multi-domain 3DSeg-8 so as to make a series of pre-trained models. We transfer Med3D pre-trained models to lung segmentation in LIDC dataset, pulmonary nodule classification in LIDC dataset and liver segmentation on LiTS challenge. Experiments show that the Med3D can accelerate the training convergence speed of target 3D medical tasks 2 times compared with model pre-trained on Kinetics dataset, and 10 times compared with training from scratch as well as improve accuracy ranging from 3% to 20%. Transferring our Med3D model on state-the-of-art DenseASPP segmentation network, in case of single model, we achieve 94.6\% Dice coefficient which approaches the result of top-ranged algorithms on the LiTS challenge.
1 Introduction
Deep learning benefits from massive annotated datasets, but constructing sufficiently large 3D medical datasets is difficult. The paper addresses this gap by aggregating diverse data and training transferable Med3D models.
- 1 Introduction: Massive annotated datasets underpin strong deep-learning performance across vision tasks.ImageNet contains more than 14 million images, while MS COCO contains more than a million images with instance-segmentation annotations.
- 1 Introduction: 3D medical imaging lacks comparable public datasets because acquisition, imaging duration, and annotation are difficult.2D transfer from ImageNet improves over training from scratch but discards 3D spatial information.
- 1 Introduction: The paper aggregates small datasets across medical domains, modalities, organs, and pathologies into a larger 3D medical dataset.This dataset-building step is paired with training a baseline network intended for transfer to other medical problems.
- 1 Introduction: Med3D is a heterogeneously trained encode-decoder segmentation network with a multi-branch decoder for incomplete annotations.Its encoder is transferred to lung segmentation, pulmonary nodule classification, and liver segmentation, with experiments reporting improved convergence and accuracy.
- 1 Introduction: The work proposes a heterogeneous network for extracting general 3D features and transfers its backbone to three new medical-image tasks.The authors also report plans to release pretrained models and source code.
2 Related Work
Natural-image pretraining is supported by large datasets, whereas biomedical 3D datasets are usually too small for stable transfer learning. The paper motivates aggregating heterogeneous medical datasets because source-target similarity matters for transfer.
- 2 Related Work: Pretrained models from large natural-image datasets provide general features and accelerate convergence for downstream vision tasks.Examples include ImageNet, PASCAL VOC, and MS COCO.
- 2 Related Work: Biomedical-image challenges commonly provide only tens or hundreds of 3D samples, making individual datasets too small to stably pretrain 3D models.The paper therefore aggregates many small 3D datasets into 3DSeg-8.
- 2 Related Work: Prior approaches converting videos or volumes to 2D do not fully leverage third-dimensional structural information.Many 3D medical-image studies instead train small 3D convolutional networks from scratch.
- 2 Related Work: Transfer learning is generally stronger when source and target data distributions are more similar.The paper argues that 3D medical pretraining should therefore outperform natural-scene video pretraining for 3D medical tasks.
- 2 Related Work: Natural-image cross-domain methods cannot be directly applied to medical imaging because medical domains differ substantially in pixel presentation and value ranges.This motivates a multi-domain 3D medical pretraining approach.
3 Method
Med3D builds a transferable 3D medical-image backbone by co-training heterogeneous segmentation data, using normalization and domain-specific decoders to address variation and incomplete annotations.
- 3.1 Data Selection and Normalization: 3DSeg-8 aggregates eight public datasets spanning CT and MR scans, diverse organs or tissues, and tumor or tissue segmentation targets.The dataset combines multiple medical domains rather than relying on a single classification dataset.
- 3.1 Data Selection and Normalization: Spatial normalization resamples each volume to the median voxel spacing of its domain to reduce scanner- and protocol-related spacing variation.The domain-specific median is intended to avoid excessive interpolation while preserving target spatial characteristics.
- 3.1 Data Selection and Normalization: Intensity normalization truncates voxel values to the 0.5th–99.5th percentiles and then normalizes them using statistics from each volume.This processing reduces the influence of outliers and differences in intensity ranges across modalities and domains.
- 3.2 Med3D Network: Med3D uses a heterogeneous encode-decoder network with eight dataset-specific decoder branches to handle incomplete organ annotations during joint training.Each branch corresponds to one 3DSeg-8 dataset, while the shared encoder is trained to learn transferable features.
- 3.3 Transfer Learning: After pre-training, the decoder is removed and the Med3D encoder is transferred to lung segmentation, pulmonary nodule classification, and liver segmentation.These experiments test transfer across whole-body segmentation, microscopic nodule classification, and liver segmentation.
- 3.3 Transfer Learning: For LiTS liver segmentation, Med3D is used in a two-stage pipeline that first obtains a whole-image liver ROI and then refines the cropped liver contour.The backbone is also embedded in a 3D DenseASPP network to provide dense multi-scale features.
4 Experiments
Experiments evaluate Med3D pretraining across dataset scale, domain variety, transfer tasks, convergence, and LiTS segmentation. Med3D generally outperforms Kinetics initialization and training from scratch, with 94.6% Dice on LiTS without post-processing.
- Experiment design: Med3D was evaluated by varying training-data scale, domain variety, transfer task, initialization, and network architecture.Experiments included 10%–100% training-data fractions, one to eight source domains, lung segmentation, pulmonary nodule classification, and LiTS segmentation.
- Training-data magnitude: Med3D performance improves as the training-data magnitude increases across the evaluated experiments.A ResNet-152 Med3D network was trained with 10%, 20%, 40%, 80%, and 100% of the training data using the same test set.
- Dataset variety: Med3D achieves its highest performance when all eight member datasets are used for training.The experiments compare training on one, two, four, and eight member datasets.
- Transfer learning: Med3D networks outperform Kinetics-initialized networks, while training from scratch performs worst in the lung segmentation comparisons.The authors attribute this pattern to the difference between medical-volume information and 3D information learned from temporal video data.
- Convergence: Med3D converges faster and reaches lower loss than the other initialization strategies in lung segmentation training curves.With sufficient iterations, all three networks reach stable losses, but Med3D pushes the loss lower and converges faster.
- Transfer learning: Med3D outperforms Kinetics and TFS for pulmonary nodule classification, while increasing Med3D depth improves performance.The competing methods show declined performance at higher network complexity, and their long-run training losses become similar to Med3D’s.
- LiTS challenge: 94.6% Dice score is achieved on LiTS without post-processing, approaching state-of-the-art networks that use ensemble techniques.The result also outperforms pure 3D networks with large margins, and ASSD indicates superior segmentation relative to other pure 3D approaches.
5 Conclusion
The work builds 3DSeg-8 and trains Med3D networks to provide transferable 3D medical features for tasks with small training datasets.
- 3DSeg-8 aggregates diverse 3D medical data, while Med3D provides pre-trained features for other tasks with small training datasets.The authors report superior results compared with networks trained on natural videos or from scratch.