PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation

Nabil Ibtehaz, M. Sohel Rahman

arXiv:1902.04049v1cs.CV

TL;DR

U-Net may lack features needed for challenging multimodal biomedical image segmentation. The paper introduces MultiResUNet, which substantially improves performance on difficult images, including a 10.15% relative improvement over U-Net on one dataset.

  • Problem

    The classical U-Net architecture may lack criteria needed for challenging multimodal biomedical image segmentation.

  • Method

    MultiResUNet enhances U-Net with residual paths and lightweight MultiRes blocks for feature reconciliation and multi-resolution analysis.

  • Results

    10.15% relative improvement over U-Net was observed on one dataset, with larger gains on challenging images than near-perfect images.

  • Takeaways & Limitations

    MultiResUNet is presented as a potential successor to classical U-Net for general multimodal biomedical image segmentation.

  • Takeaways & Limitations

    Jaccard Index may be inflated by majority-class segmentation and therefore may not properly represent accuracy on that dataset.

Abstract

from arXiv · show

In recent years Deep Learning has brought about a breakthrough in Medical Image Segmentation. U-Net is the most prominent deep network in this regard, which has been the most popular architecture in the medical imaging community. Despite outstanding overall performance in segmenting multimodal medical images, from extensive experimentations on challenging datasets, we found out that the classical U-Net architecture seems to be lacking in certain aspects. Therefore, we propose some modifications to improve upon the already state-of-the-art U-Net model. Hence, following the modifications we develop a novel architecture MultiResUNet as the potential successor to the successful U-Net architecture. We have compared our proposed architecture MultiResUNet with the classical U-Net on a vast repertoire of multimodal medical images. Albeit slight improvements in the cases of ideal images, a remarkable gain in performance has been attained for challenging images. We have evaluated our model on five different datasets, each with their own unique challenges, and have obtained a relative improvement in performance of 10.15%, 5.07%, 2.63%, 1.41%, and 0.62% respectively.

1 Introduction

The paper examines limitations of the widely successful U-Net architecture for biomedical image segmentation and proposes MultiResUNet as an enhanced alternative. Experiments across multimodal medical-image datasets, including challenging cases and a 3D variant, report superior performance over standard U-Net models.

  • Motivation: Manual medical-image segmentation is complex, motivating automated methods that can operate robustly across diverse data.Earlier rule-based approaches failed to maintain robustness across a huge variety of data.
  • U-Net background: U-Net became the de-facto standard for medical-image segmentation because it performs impressively even with scarce labeled training data.Its encoder and decoder layers are connected by skip connections around pooling and deconvolution operations.
  • Contributions: The authors analyze U-Net in depth, identify potential opportunities for enhancement, and propose MultiResUNet as an enhanced version of the standard architecture.The proposed alterations are motivated by hypothesized shortcomings in U-Net and contemporary advances in deep computer vision.
  • Evaluation: Experiments on public medical-image datasets spanning different modalities show that MultiResUNet achieves superior accuracy.The paper evaluates the model across different public datasets and modalities.
  • Evaluation: The 3D version of MultiResUNet outperforms the standard 3D U-Net.This extends the evaluation beyond the two-dimensional architecture.
  • Results: Challenging images show significant improvement with MultiResUNet over U-Net.The paper specifically examines difficult cases where the proposed model improves on the standard architecture.

2 Overview of the UNet Architecture

U-Net is a symmetric, convolutional encoder–decoder architecture for semantic segmentation. Its skip connections restore spatial information lost during pooling, and a 3D variant adapts the operations for volumetric segmentation while reducing parameters.

  • U-Net architecture: U-Net uses an entirely convolutional, symmetric architecture with an encoder for spatial feature extraction and a decoder for segmentation-map construction.The encoder repeats convolution and downsampling stages, while the decoder reconstructs the output through corresponding upsampling stages.
  • Encoder: The encoder repeats two 3 × 3 convolutions and downsampling four times, doubling the number of filters after each downsampling.Two 3 × 3 convolutions connect the encoder to the decoder.
  • Skip connections: Skip connections concatenate encoder feature maps with upsampled decoder maps, enabling recovery of spatial information lost during pooling.Connections occur at all four levels, transferring outputs from encoder convolutional layers before pooling to the decoder.
  • 3D U-Net: 3D U-Net replaces two-dimensional convolutions, max pooling, and transposed convolutions with three-dimensional counterparts for volumetric segmentation.To limit parameters, its depth is reduced by one, and filters are doubled before pooling layers to avoid bottlenecks.

3 Motivations and High Level Considerations

The paper identifies two limitations in U-Net for medical image segmentation: objects vary substantially in scale, and skip connections may create a semantic gap between encoder and decoder features. It motivates MultiRes blocks and residual convolutional paths to address these issues while controlling memory requirements.

  • Multiresolution analysis: Medical segmentation targets such as cells, organs, and tumors often have irregular shapes and different scales, requiring a network robust to multiscale objects.Dermoscopy lesions are presented as examples ranging from small to medium and large sizes.
  • Multiresolution analysis: U-Net’s successive 3 × 3 convolutions resemble 5 × 5 convolution, motivating parallel 3 × 3, 5 × 5, and 7 × 7 operations for multiresolution analysis.The parallel operations follow the approach of Inception networks.
  • Multiresolution analysis: The proposed MultiRes block factorizes larger 5 × 5 and 7 × 7 filters into sequences of smaller 3 × 3 blocks and concatenates their outputs.The second and third 3 × 3 blocks effectively approximate 5 × 5 and 7 × 7 convolutions, respectively.
  • Memory-efficient design: To reduce memory requirements, the three consecutive convolutional layers gradually increase their filter counts from 1 to 3 rather than using equal filter counts.The motivation is that successive convolutional layers can make the first layer’s filter count have a quadratic effect on memory.
  • Semantic-gap reduction: U-Net skip connections may create a semantic gap because encoder features are lower-level while decoder features are higher-level and more heavily processed.The paper proposes convolutional layers with residual connections along shortcut paths to reduce this disparity and facilitate learning.

4 Proposed Architecture

MultiResUNet replaces U-Net’s paired convolutional layers and ordinary shortcut connections with MultiRes blocks and Res paths. Its filter allocation is designed to keep parameter counts comparable while progressively increasing capacity within blocks and across network levels.

  • MultiRes blocks: MultiResUNet replaces each sequence of two convolutional layers with a MultiRes block whose filter count is controlled by parameter W.W is computed to maintain a comparable parameter relationship with the original U-Net.
  • Filter allocation: 1.67 is the selected α coefficient, keeping MultiResUNet’s parameter count slightly below the U-Net baseline.The comparison U-Net uses #filters = [32, 64, 128, 256, 512] across levels, which are also the U values in MultiResUNet.
  • Filter allocation: MultiRes blocks progressively increase filters across their three successive convolutional layers, using the experimentally best-performing combination.The value of W doubles after each pooling or deconvolution operation, as in U-Net.
  • Res paths: MultiResUNet replaces ordinary shortcut connections with Res paths that apply convolutions to encoder-to-decoder feature maps.The number of convolutional filters is gradually reduced toward inner shortcut paths to address the decreasing semantic gap.
  • Activation and normalization: All convolutional layers except the output use ReLU activation and batch normalization, while the output layer uses Sigmoid activation.The architecture is presented in Fig. 5, with implementation details listed in Table 1.

5 Datasets

The experiments used diverse 2D and 3D medical imaging datasets, each presenting distinct acquisition, annotation, imaging-quality, or computational challenges. The collection included fluorescence microscopy, electron microscopy, dermoscopy, endoscopy, and multimodal MRI data.

  • Dataset selection: The datasets were selected because public medical imaging benchmarks are scarce, typically small, and difficult to curate due to equipment, acquisition, annotation, and privacy requirements.Each dataset also posed a unique challenge, with further details referenced in Sections 7 and 8.
  • Fluorescence microscopy: 97 fluorescence microscopy images containing 4009 cells were manually annotated for nuclei segmentation, with irregular brightness and noticeable debris creating a challenging dataset.The dataset comprised U2OS and NIH3T3 cells.
  • Electron microscopy: 30 ssTEM images from the ISBI-2012 2D EM challenge contained slight alignment errors and noise, and were resized from 512×512 to 256 × 256.The images depicted the Drosophila first instar larva ventral nerve cord.
  • Dermoscopy: 2594 expert-annotated dermoscopy images from ISIC-2018 represented different skin-lesion types and were resized to 256 × 192 while maintaining the average aspect ratio.The data originated from the ISIC-2017 and HAM10000 datasets.
  • Endoscopy: 612 polyp-containing colonoscopy images from 29 video sequences formed the CVC-ClinicDB endoscopy dataset and were resized from 384 × 288 to 256 × 192.Only images containing polyps were considered.
  • Multimodal MRI: 285 multimodal BraTS17 MRI scans comprised 210 HGG and 75 LGG cases, using T1, T1Gd, T2, and FLAIR volumes as four channels for 3D evaluation.The dataset enabled assessment on 3D medical images.

6 Experiments

The experiments evaluate MultiResUNet against parameter-matched 2D and 3D U-Net baselines for medical image segmentation. They use minimal general preprocessing, binary cross-entropy with Adam, Jaccard Index evaluation, and k-fold cross-validation.

  • Experimental models: MultiResUNet was compared with a five-layer U-Net baseline using filter counts of 32, 64, 128, 256, and 512 to keep parameters comparable.For 3D segmentation, the baseline was the original paper’s 3D U-Net.
  • Experimental models: The 3D MultiResUNet was created by replacing 2D convolution, pooling, and transposed-convolution layers with their 3D counterparts.No further architectural alterations were made.
  • Data processing: Input images were only resized to fit GPU memory and normalized by dividing pixel values by 255, with no domain-specific preprocessing or application-specific post-processing.The experiments were designed to investigate MultiResUNet as a general model.
  • Training: The models were trained by minimizing binary cross-entropy with the Adam optimizer.Binary cross-entropy was used because semantic segmentation was treated as pixel-wise binary classification.
  • Evaluation: Jaccard Index was used to evaluate segmentation because it emphasizes precise segmentation and penalizes under-segmentation and over-segmentation.The authors note that precision and recall can be inadequate when points of interest occupy only a small image region.
  • Evaluation: In k-fold cross-validation, the dataset was split into k mutually exclusive subsets, and the best validation result across 150 epochs was recorded for each run before combining all runs.Each run used one subset for validation and the remaining subsets for training.

7 Results

Across five-fold cross-validation, MultiResUNet outperformed U-Net across the evaluated medical-image datasets, with especially notable gains on challenging images. It also converged faster and handled ambiguous boundaries, perturbations, and narrow separations more effectively, although Jaccard Index could obscure differences for majority-class segmentation.

  • Cross-validation results: MultiResUNet outperformed U-Net on all evaluated medical-image types in five-fold cross-validation, with the most notable improvements for Dermoscopy and Endoscopy images.The experiments selected the best validation results across 150 epochs in each of five runs.
  • Cross-validation results: 1.4104% relative improvement was obtained for MRI images, while Electron Microscopy achieved a 0.6161% relative improvement over U-Net.For Electron Microscopy, U-Net was described as nearly on par, but MultiResUNet performed slightly better.
  • Convergence: MultiResUNet attained convergence much faster than U-Net across all datasets, suggesting comparable performance could be reached in fewer training epochs.The reported validation-performance bands also showed smaller standard deviation for MultiResUNet.
  • Qualitative comparison: On challenging images with vague boundaries, MultiResUNet avoided U-Net’s under-segmentation and over-segmentation and delineated boundaries better, including in dermoscopic images.MultiResUNet was also superior in cases where both models faced complications.
  • Robustness to perturbations: MultiResUNet rejected small outlier particles that U-Net mistakenly classified as cell nuclei, while Jaccard Index understated this qualitative difference because the outliers were tiny.The passage reports a U-Net Jaccard Index of 0.9317 before the excerpt ends.
  • Electron Microscopy: MultiResUNet preserved narrow background separations in Electron Microscopy images that U-Net often missed, despite respective Jaccard Index values of 0.8914 and 0.8841.Because the foreground occupied most of the image, Jaccard Index was inflated by majority-class segmentation and was not fully representative of accuracy.

8 Conclusion

MultiResUNet modifies U-Net with Res paths to reconcile encoder and decoder features, and shows increasingly strong gains over U-Net on challenging images. The authors therefore propose it as a potential successor while identifying domain-specific knowledge and post-processing as future directions.

  • Architecture: Res paths add processing intended to make encoder and decoder feature maps more homogeneous.The modification addresses a discrepancy between features passed from the encoder and features propagating through the decoder.
  • Performance: 10.15%, 5.07%, 2.63%, 1.41%, and 0.62% relative improvement in performance were observed across five datasets using MultiResUNet.The evaluated datasets were selected because they differed substantially and posed distinct segmentation challenges.
  • Performance: MultiResUNet performs only slightly better than U-Net on perfect or near-perfect images but gains dramatically on intricate images with noise, perturbations, or unclear boundaries.The conclusion contrasts straightforward segmentation cases with challenging images.
  • Implications: The authors believe MultiResUNet can be a potential successor to the classical U-Net architecture.This conclusion follows the reported performance across five diverse biomedical image datasets.
  • Future work: Future work includes fusing the model with domain-specific expert-knowledge pipelines and coupling it with proper post-processing stages.These directions are intended to improve performance and support better segmentation methods for diversified applications.
Loading 1902.04049v1…