Source-linked AI summary
Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation
Kuan-Lun Tseng, Yen-Liang Lin, Winston Hsu, Chung-Yang Huang
TL;DR
The paper addresses limited use of multi-modal and inter-slice information in 3D biomedical segmentation. It introduces cross-modality convolution and convolutional LSTM within an end-to-end encoder-decoder, and reports improved state-of-the-art performance on BRATS-2015.
Problem
Existing approaches often use a single modality or stack modalities as channels, while 2D methods ignore sequential information between consecutive slices.
Method
The method jointly learns modality fusion with cross-modality convolution and slice dependencies with convolutional LSTM in an end-to-end encoder-decoder network.
Results
The method improves state-of-the-art biomedical segmentation methods on the BRATS-2015 dataset.
Takeaways & Limitations
Jointly modeling multi-modal MRI information and consecutive slices provides the paper's supported approach to 3D biomedical segmentation.
Takeaways & Limitations
Weighted loss can overly suppress dominant labels such as normal tissue and cause incorrect predictions.
Abstract
from arXiv · showhide
Deep learning models such as convolutional neural net- work have been widely used in 3D biomedical segmentation and achieve state-of-the-art performance. However, most of them often adapt a single modality or stack multiple modalities as different input channels. To better leverage the multi- modalities, we propose a deep encoder-decoder structure with cross-modality convolution layers to incorporate different modalities of MRI data. In addition, we exploit convolutional LSTM to model a sequence of 2D slices, and jointly learn the multi-modalities and convolutional LSTM in an end-to-end manner. To avoid converging to the certain labels, we adopt a re-weighting scheme and two-phase training to handle the label imbalance. Experimental results on BRATS-2015 show that our method outperforms state-of-the-art biomedical segmentation approaches.
1. Introduction
The paper addresses limitations of 2D and conventional multi-modal approaches by jointly learning cross-modality fusion and slice sequences in an end-to-end 3D segmentation network.
- 2D approaches concatenate slice-wise predictions but ignore sequential information between consecutive slices, while 3D convolution can require many parameters and overfit small datasets.
- The method stacks same-depth slices from Flair, T2, T1, and T1c MRI modalities before encoding them for segmentation.
- The proposed end-to-end encoder-decoder network outperforms state-of-the-art 3D biomedical segmentation methods in experiments.
- Cross-modality convolution aggregates information from multiple MRI modalities through a dedicated fusion operation.
- Convolutional LSTM models spatial and sequential correlations between consecutive slices, and the fusion and sequence-learning components are jointly optimized end to end.
2. Related Work
Related work spans encoder-decoder segmentation, 3D convolution, recurrent slice modeling, and multi-modal imaging; the paper combines these directions with explicit cross-modality fusion.
- Encoder-decoder networks such as SegNet and U-Net use convolutional representations and up-sampling for dense segmentation.
- 3D convolution models correlations between slices but often requires more parameters and is prone to overfitting on small datasets.
- Unlike kU-Net, the proposed method combines cross-modality convolution with recurrent slice modeling and jointly optimizes both components end to end.
- Existing CNN methods commonly treat modalities as separate input channels, whereas this paper targets correlations among modalities and consecutive slices together.
3. Method
The method uses modality-specific encoding, cross-modality convolution, and convolutional LSTM sequence modeling within an encoder-decoder architecture for dense 3D segmentation.
- The architecture contains a multi-modal encoder and decoder, cross-modality convolution, and convolutional LSTM for dense prediction.
- The multi-modal encoder and decoder use a compact SegNet-like structure, with convolution, batch normalization, ReLU, pooling, and transposed-convolution up-sampling.
- Feature multiplication combines encoder and decoder maps without increasing feature-map size or requiring additional weights.
- Cross-Modality Convolution: Cross-modality convolution applies a 4 × 1 × 1 3D kernel across four modality-specific feature maps, assigning modality weights and aggregating responses.
- Slice Sequence Learning: Convolutional LSTM receives consecutive encoded slices, replaces matrix multiplication with convolution, and preserves spatial information while modeling slice dependencies.
- Slice Sequence Learning: Shared convLSTM weights process slice sequences without parameter size increasing linearly with sequence length, after which the decoder upsamples the outputs.
4. Experiments
Experiments evaluate the model on CamVid and BRATS-2015, including multi-resolution fusion, cross-modality convolution, convolutional LSTM, and two-phase training. The proposed components improve segmentation performance and outperform U-Net in reported comparisons.
- Datasets: CamVid experiments use 367 training, 100 validation, and 233 testing images, with Mean IU as the evaluation criterion.The dataset contains 32 semantic classes and images sampled from driving videos.
- Datasets: BRATS-2015 experiments use 30 unseen patients for testing and evaluate Dice, Positive Predicted Value, and Sensitivity.The dataset contains four synchronized MRI modalities and five tissue labels.
- Training: Two-phase training addresses label imbalance by first emphasizing tumor-containing slices and then training with the true distribution at a lower learning rate.Median frequency balancing is used initially, while the second phase uses a learning rate of 10e-6.
- Ablation studies: Cross-modality convolution improves the multi-modal encoder over a basic encoder-decoder by approximately two percent in Mean IU.The authors note that downsampling loses spatial information and suggest higher-resolution feature maps could further improve CMC.
- Results: The final system achieves Mean IU 73.52 on BRATS-2015, while convolutional LSTM consistently improves CamVid configurations and the BRATS slice-to-sequence model outperforms U-Net.The BRATS model also outperforms U-Net across different measurements in five-fold online evaluation, and sequential information improves detailed structures.
5. Conclusions
The paper introduces an encoder-decoder architecture that combines cross-modality convolution with sequence learning for 3D biomedical segmentation. Experiments on BRATS-2015 report improvement over state-of-the-art methods.
- The method introduces a deep encoder-decoder architecture for 3D biomedical segmentation.
- Cross-modality convolution exploits information across multiple MRI modalities.
- Sequence learning integrates information from consecutive image slices.
- The model jointly optimizes sequence learning and cross-modality convolution end-to-end.
- Experiments on BRATS-2015 demonstrate improvement over state-of-the-art methods.