Source-linked AI summary
Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation
Jianxu Chen, Lin Yang, Yizhe Zhang, Mark Alber, Danny Z. Chen
TL;DR
3D biomedical segmentation methods face difficulties handling anisotropic images while exploiting volumetric context efficiently. The paper combines an FCN and an RNN to model intra-slice and inter-slice information through systematic 2D operations. Evaluations on neuron and fungus segmentation report state-of-the-art performance and improvement over known DL schemes using 3D contexts.
Problem
Known 3D segmentation methods struggle with anisotropic biomedical images, while approaches that preserve 3D context can impose high computational costs or lose z-direction information.
Method
The framework combines kU-Net, an FCN for intra-slice feature extraction, with BDC-LSTM, an RNN that integrates contextual information along the z-direction.
Results
The proposed approach achieves state-of-the-art performance across two 3D biomedical segmentation applications and outperforms known DL schemes utilizing 3D contexts.
Takeaways & Limitations
Systematic 2D operations can explicitly leverage 3D image anisotropism while constructing hierarchical features from 3D contexts.
Takeaways & Limitations
Pyramid-LSTM could not be evaluated under its original structure because its Torch7 implementation exceeded available GPU memory, limiting processing to a 40 × 40 × 8 cube.
Abstract
from arXiv · showhide
Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation perfor- mance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a com- bination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches.
1 Introduction
3D biomedical segmentation is difficult because common methods struggle with spatial context, computation, and strongly anisotropic voxel dimensions. The proposed framework combines an FCN for intra-slice information with an RNN for inter-slice context, explicitly leveraging this anisotropy.
- 1 Introduction: 3D biomedical segmentation identifies target objects such as neuronal structures and knee cartilage, often in images whose z-axis voxels are 5∼10 times larger than xy-plane voxels.This anisotropy creates substantially different spatial scales across image dimensions.
- 1 Introduction: Existing methods either concatenate independent 2D slice segmentations, use expensive 3D convolutions, or apply isotropic operations to anisotropic image planes.The paper identifies lost z-direction correlation, high memory and training costs, and geometric mismatch as central issues.
- 1 Introduction: The framework combines an FCN for intra-slice contexts with an RNN for inter-slice contexts while processing biomedical volumes as sequences of 2D slices.This design follows the z-stack representation and uses recurrent processing to propagate contextual information along z.
- 1 Introduction: The FCN component uses a new architecture that compresses intra-slice information into hierarchical features and handles objects with different scales more effectively than known FCNs.Its multi-scale motivation addresses the fixed perception fields of common FCN architectures.
- 1 Introduction: The generalized RNN applies 2D convolutions on the xy plane recurrently, assembling intra-slice contexts into hierarchical 3D features through inter-slice correlations.This avoids the problematic isotropic convolutions used on anisotropic images by some prior 3D segmentation schemes.
- 1 Introduction: The framework explicitly leverages 3D anisotropism and constructs discriminative 3D features using systematic 2D operations.The authors present this as a paradigm for migrating 2D deep architectures to 3D context exploitation.
2 Methodology
The framework combines kU-Net for intra-slice feature extraction with BDC-LSTM for bidirectional inter-slice context, explicitly adapting 2D operations to anisotropic 3D images.
- The method combines an FCN called kU-Net with an RNN called BDC-LSTM to exploit intra-slice and inter-slice contexts, respectively.
- The FCN Component: kU-Net: kU-Net uses multiple U-Net submodules at progressively finer scales, propagating coarse-scale information to assist finer-scale feature extraction.The architecture simulates experts who first locate targets coarsely and then refine their boundaries.
- The FCN Component: kU-Net: The kU-Net submodules process differently scaled inputs while maintaining the same input-window size, so coarser modules view larger regions at lower resolution.The scale hierarchy is created through max-pooling, with U-Net-1 operating at the finest scale and U-Net-k at the coarsest.
- The RNN Component: BDC-LSTM: BDC-LSTM extends convolutional LSTM by stacking two CLSTM layers that process slices in opposite z-directions and concatenate their contextual outputs.Convolutional gates preserve spatial feature maps while recurrent connections propagate information slice by slice.
- Training Strategy: The framework can be trained end-to-end or with separate component training, with decoupling preferred for challenging images requiring very large effective context.Separate training lets each component use larger tiles under the same computing resources.
3 Experiments
The framework was evaluated on neuron and fungus 3D segmentation datasets against representative deep-learning methods. It outperformed known approaches while using inter-slice context to refine FCN predictions and suppress false positives.
- Datasets: Experiments covered neuron-boundary segmentation from the ISBI challenge and tubular-fungus segmentation in in-house electron-microscopy stacks.The neuron stacks measured 512 × 512 × 30 voxels; fungus stacks had voxel-scale ratio x : y : z = 1 : 1 : 3.45.
- Comparisons: Four representative baselines comprised U-Net, 3D-Conv, Tri-planar, and Pyramid-LSTM.These represented linked 2D segmentation, 3D convolutions, orthogonal-plane convolutions, and generalized LSTM approaches.
- Quantitative results: The proposed kU-Net alone substantially improved over U-Net, while the full approach outperformed methods using 3D contexts and stacked BDC-LSTMs.The authors attribute higher-level 3D perception to subsampling layers in the RNN component.
- Computational considerations: Pyramid-LSTM could not produce acceptable fungus results under the tested GPU memory limit, whereas the proposed method used GPU memory more efficiently.The largest feasible Pyramid-LSTM processing cube was 40 × 40 × 8 using the reported workstation.
- Qualitative results: On fungus images, combining FCN and RNN suppressed false positives through inter-slice consistency and strengthened predictions in ambiguous cases.Both FCN-only and combined methods produced nearly no false negatives in the qualitative examples.
4 Conclusions and Future Work
The paper presents a framework combining kU-Net for 2D context with BDC-LSTM for z-direction context. Across two 3D biomedical segmentation applications, it reports state-of-the-art performance and proposes broader future evaluation.
- 4 Conclusions and Future Work: The framework combines kU-Net to exploit 2D contexts with BDC-LSTM to integrate contextual information along the z-direction.This combination is presented as the paper’s framework for 3D image segmentation.
- 4 Conclusions and Future Work: Across two 3D biomedical segmentation applications, the approach achieved state-of-the-art performance and outperformed known deep-learning schemes using 3D contexts.The conclusion frames the method as a paradigm for migrating strong 2D architectures to 3D contexts.
- 4 Conclusions and Future Work: Future work will explore BDC-LSTMs in different deep architectures and evaluate the approach more extensively on datasets such as BraTS and MRBrainS.The authors identify these datasets as targets for broader evaluation.