Source-linked AI summary
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, Olaf Ronneberger
TL;DR
Annotating volumetric biomedical data slice by slice is tedious and inefficient, limiting practical access to rich training labels. This paper presents a 3D U-Net that learns dense segmentations from sparse slice annotations and achieves an average IoU of 0.863 in semi-automated evaluation.
Problem
Slice-by-slice annotation of volumetric biomedical data is tedious and inefficient, limiting the creation of large, richly annotated training sets.
Method
A 3D U-Net uses sparse 2D slice labels with a weighted loss that ignores unlabeled pixels to learn dense volumetric segmentations.
Results
Average IoU reaches 0.863 in 3-fold cross-validation for semi-automated segmentation, while the fully automated setup shows a performance gain over an equivalent 2D implementation.
Takeaways & Limitations
The method can provide full 3D segmentations from a few annotated slices, saving experts from full-volume annotation on Xenopus kidney data.
Takeaways & Limitations
Fully automated segmentation assumes that a representative sparsely annotated training set can be assembled.
Abstract
from arXiv · showhide
This paper introduces a network for volumetric segmentation that learns from sparsely annotated volumetric images. We outline two attractive use cases of this method: (1) In a semi-automated setup, the user annotates some slices in the volume to be segmented. The network learns from these sparse annotations and provides a dense 3D segmentation. (2) In a fully-automated setup, we assume that a representative, sparsely annotated training set exists. Trained on this data set, the network densely segments new volumetric images. The proposed network extends the previous u-net architecture from Ronneberger et al. by replacing all 2D operations with their 3D counterparts. The implementation performs on-the-fly elastic deformations for efficient data augmentation during training. It is trained end-to-end from scratch, i.e., no pre-trained network is required. We test the performance of the proposed method on a complex, highly variable 3D structure, the Xenopus kidney, and achieve good results for both use cases.
1 Introduction
The paper presents a 3D U-Net that produces dense volumetric segmentations from sparse 2D slice annotations. It supports semi-automated densification of annotated volumes and fully automated segmentation of new volumes, demonstrated on Xenopus kidney data.
- Motivation: Slice-by-slice annotation of large volumetric biomedical data is tedious and inefficient because neighboring slices contain similar information.Only 2D slices can be displayed on a computer screen, making full 3D annotation especially costly for learning-based methods requiring substantial labeled data.
- Contribution: The method learns dense volumetric segmentations from only some annotated 2D slices.It is designed for both densifying sparsely annotated data and learning from multiple sparse datasets to generalize to new data.
- Architecture: The proposed network extends U-Net by replacing its 2D operations with 3D convolutions, 3D max pooling, and 3D up-convolutional layers.Its encoder analyzes the whole image and its decoder produces a full-resolution segmentation.
- Training: A weighted loss function and special data augmentation enable training with few manually annotated slices.The paper states that volumetric-image redundancy can allow training on just two volumetric images to generalize to a third one.
- Evaluation: The method is evaluated on difficult confocal microscopy data of the structurally complex Xenopus kidney.The experiments include qualitative densification results and quantitative evaluations, as well as tests of how annotation quantity affects performance.
2 Network Architecture
The network extends u-net with four-step 3D analysis and synthesis paths, batch normalization, and a weighted softmax loss that enables learning from sparse annotations. It processes 3-channel volumetric tiles and predicts a lower-resolution segmentation with sufficient spatial context.
- 2 Network Architecture: The architecture has four resolution steps in both analysis and synthesis paths, using 3D convolutions, ReLUs, max pooling, and upconvolutions.Analysis layers use two 3 × 3 × 3 convolutions followed by 2 × 2 × 2 max pooling; synthesis layers use 2 × 2 × 2 upconvolutions followed by two 3 × 3 × 3 convolutions.
- 2 Network Architecture: The final 1×1×1 convolution produces 3 output channels, and the complete architecture contains 19069955 parameters.The network doubles channels before max pooling and applies the same scheme in the synthesis path to avoid bottlenecks.
- 2 Network Architecture: The network takes a 132 × 132 × 116 voxel tile with 3 channels and outputs a 44×44×28 voxel prediction.With 1.76×1.76×2.04µm3 voxels, each predicted voxel has an approximate receptive field of 155 × 155 × 180µm3.
- 2 Network Architecture: Batch normalization is introduced before each ReLU, with the design motivated by the batch size of one and few available samples.The passage describes using current statistics at test time in such applications, but its final sentence is incomplete.
- 2 Network Architecture: A weighted softmax loss assigns zero weight to unlabeled pixels, allowing training from labeled slices while generalizing to the whole volume.This loss is the architecture’s key mechanism for learning from sparse annotations.
3 Implementation Details
Implementation uses three Xenopus kidney embryo samples acquired as tiled, multichannel 3D microscopy volumes, with augmentation based on smooth deformations and a weighted cross-entropy training objective.
- Data acquisition: The data comprise three Xenopus kidney embryo samples at Nieuwkoop-Faber stage 36-37, recorded as four tiles with three channels at 0.88×0.88×1.02µm3 voxel size.Imaging used a Zeiss LSM 510 DUO inverted confocal microscope with a Plan-Apochromat 40x/1.3 oil immersion objective lens.
- Data augmentation: Besides rotation, scaling, and gray-value augmentation, training applies smooth dense deformation fields to data and ground-truth labels using B-spline interpolation.Random vectors are sampled from a normal distribution with standard deviation 4 on grids spaced 32 voxels in each direction.
- Training objective: The network output and ground-truth labels are compared with softmax and weighted cross-entropy loss, reducing weights for frequently seen background.The passage states that weights for frequently seen background are reduced and begins describing increased weights for other classes.
4 Experiments
The experiments evaluate semi-automated segmentation from a few annotated slices and fully automated segmentation using representative partially annotated training volumes. Results include qualitative dense reconstructions, cross-validation comparisons, and analyses of annotation sparsity and batch normalization.
- Semi-automated segmentation: In the semi-automated setup, users annotate a few slices from each volume, and the network produces a dense volumetric segmentation.This approach is intended to avoid full-volume annotation by experts.
- Semi-automated segmentation: Qualitative results show that the network recovered the whole 3D segmentation of a Xenopus kidney from a few annotated slices.The illustrated result used the proposed 3D u-net with batch normalization.
- Semi-automated segmentation: The semi-automated quantitative evaluation used 3-fold cross validation over 77 manually annotated slices, with and without batch normalization, and compared against a pure 2D implementation.Test slices were removed and left unlabeled to simulate even sparser user annotation.
- Semi-automated segmentation: Increasing the number of annotated slices produced high performance gains for samples S1, S2, and S3 in the semi-automated setup.The experiment began with 1 annotated slice in each orthogonal direction and increased the number gradually; results were reported in Table 2.
- Fully automated segmentation: In the fully automated setup, the network was trained on two partially annotated kidney volumes and tested on the third across all 3 train-test combinations.Batch normalization improved results except in the third setting, where it was counterproductive.
5 Conclusion
The paper introduces an end-to-end method for semi-automatic and fully automatic 3D volume segmentation from sparse annotations. It achieves accurate segmentation of highly variable Xenopus kidney structures, including an average IoU of 0.863 for the semi-automated setup.
- 5 Conclusion: The method segments 3D volumes semi-automatically and fully automatically from sparse annotations.It is described as an end-to-end learning method.
- 5 Conclusion: The method provides accurate segmentation for the highly variable structures of the Xenopus kidney.
- 5 Conclusion: 0.863 average IoU is achieved in 3-fold cross-validation experiments for the semi-automated setup.
- 5 Conclusion: The fully automated setup demonstrates a performance gain from the 3D architecture over an equivalent 2D implementation.
- 5 Conclusion: The network is trained from scratch.