PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation

Jose Dolz, Karthik Gopinath, Jing Yuan, Herve Lombaert, Christian Desrosiers, Ismail Ben Ayed

arXiv:1804.02967v2cs.CV

TL;DR

Multi-modal brain MRI segmentation requires combining complementary image information, but conventional CNNs typically fuse modalities at only one network level. HyperDenseNet uses modality-specific paths with dense connections within and across paths, and it achieved state-of-the-art performance on both infant and adult brain-tissue segmentation benchmarks.

  • Problem

    Conventional multi-modal CNN segmentation methods typically fuse modalities at the input or output, limiting how relationships across modalities and abstraction levels are modeled.

  • Method

    HyperDenseNet is a 3D fully convolutional network with one path per imaging modality and dense connections within and across modality paths.

  • Results

    HyperDenseNet achieved state-of-the-art performance on the iSEG 2017 infant and MRBrainS 2013 adult brain-tissue segmentation benchmarks.

  • Takeaways & Limitations

    Hyper-dense connections allow the network to explore complex combinations of modalities across levels of abstraction in multi-modal volumetric segmentation.

Abstract

from arXiv · show

Recently, dense connections have attracted substantial attention in computer vision because they facilitate gradient flow and implicit deep supervision during training. Particularly, DenseNet, which connects each layer to every other layer in a feed-forward fashion, has shown impressive performances in natural image classification tasks. We propose HyperDenseNet, a 3D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems. Each imaging modality has a path, and dense connections occur not only between the pairs of layers within the same path, but also between those across different paths. This contrasts with the existing multi-modal CNN approaches, in which modeling several modalities relies entirely on a single joint layer (or level of abstraction) for fusion, typically either at the input or at the output of the network. Therefore, the proposed network has total freedom to learn more complex combinations between the modalities, within and in-between all the levels of abstraction, which increases significantly the learning representation. We report extensive evaluations over two different and highly competitive multi-modal brain tissue segmentation challenges, iSEG 2017 and MRBrainS 2013, with the former focusing on 6-month infant data and the latter on adult images. HyperDenseNet yielded significant improvements over many state-of-the-art segmentation networks, ranking at the top on both benchmarks. We further provide a comprehensive experimental analysis of features re-use, which confirms the importance of hyper-dense connections in multi-modal representation learning. Our code is publicly available at https://www.github.com/josedolz/HyperDenseNet.

1 INTRODUCTION

Multi-modal MRI improves brain-tissue characterization, but its data volume makes manual segmentation impractical and existing CNN fusion strategies limit cross-modality representation learning. HyperDenseNet addresses this gap by connecting modality-specific paths densely within and across abstraction levels.

  • Motivation: Combining MRI modalities improves tissue characterization because different sequences provide complementary contrasts and reveal distinct abnormalities.T1 distinguishes gray and white matter, while T2, proton density, and FLAIR highlight abnormalities such as lesions.
  • Motivation: Automatic multi-modal segmentation is needed because manual inspection of large medical-image collections is time-consuming, error-prone, and unsuitable for large studies.
  • Limitations of prior work: Existing multi-modal CNNs commonly fuse modalities at a single layer, while early fusion may struggle to learn highly nonlinear relationships between differing low-level feature distributions.
  • Limitations of prior work: Dense connectivity has supported gradient propagation and deeper networks, but its use across multiple modality paths for segmentation had remained unexplored.
  • Contribution: HyperDenseNet assigns each modality a path and connects layers densely both within individual paths and across paths, rather than relying on only early or late fusion.The architecture is designed to learn combinations between modalities across levels of abstraction.
  • Contribution: The journal study broadens the earlier conference work with a more rigorous multi-modal segmentation treatment, additional baselines and benchmarks, and experiments on hyper-dense connections.

2 METHODS AND MATERIALS

The paper frames CNN segmentation as learning hierarchical image representations from data, while noting benchmark differences between infant and adult brain MRI datasets. The supplied methods passages also identify sliding-window processing as an early CNN segmentation strategy.

  • CNN segmentation: CNNs learn feature representations automatically through layers that process imaging data at different abstraction levels.
  • CNN segmentation: Early CNN segmentation approaches used a sliding-window strategy in which local image regions define the processing inputs.
  • Datasets: The iSEG 2017 and MRBrainS 2013 benchmarks differ substantially in image characteristics, including voxel spacing and the number of available modalities.iSEG 2017 focuses on 6-month infant data, whereas MRBrainS 2013 uses adult data.

1. iSEG

HyperDenseNet extends dense connectivity to multi-modal 3D segmentation by linking features within and across modality-specific paths. The architecture uses sub-volumes without pooling and is evaluated against several dense-fusion baselines.

  • 2.1 The proposed Hyper-Dense network: Dense connections concatenate outputs from all preceding layers, improving information and gradient flow while providing implicit deep supervision and regularization.These properties motivate extending DenseNet-style connectivity to multi-modal segmentation.
  • 2.1 The proposed Hyper-Dense network: HyperDenseNet assigns each imaging modality a path and connects layers densely both within and across paths.This connectivity is intended to learn relationships between modalities across multiple abstraction levels.
  • 2.3 Network architecture: The baselines compare single-path early fusion, dual-path disentangled modalities, and early fusion after the first convolutional layer.These alternatives correspond to different choices for when modality information is combined.
  • 2.1 The proposed Hyper-Dense network: The architecture uses modality-specific processing functions and shuffling of inputs, distinguishing it from early-fusion baselines.The authors state that experiments demonstrate advantages over the early-fusion comparison model.
  • 2.3 Network architecture: Sub-volumes of size 27×27×27 are used for training and 35 × 35 × 35 non-overlapping sub-volumes for inference, avoiding spatial pooling.This reduces memory requirements, increases the number of training examples, and removes the need for data augmentation.
  • 2.3 Network architecture: Training optimizes convolution weights, biases, and parametric-rectifier parameters with RMSprop and cross-entropy.The network uses a 0.001 initial learning rate, momentum 0.6, and 30 training epochs.

3 EXPERIMENTS AND RESULTS

HyperDenseNet was evaluated on challenging multi-modal brain tissue segmentation tasks using publicly available infant iSEG and adult MRBrainS3 challenge data, with quantitative comparisons against state-of-the-art methods.

  • The evaluation covered infant brain tissue segmentation in iSEG and adult brain tissue segmentation in MRBrainS3.
  • Quantitative evaluations were conducted for each application.
  • HyperDenseNet was compared with state-of-the-art methods on both segmentation tasks.

3.1 iSEG Challenge

On iSEG-2017 infant brain tissue segmentation, HyperDenseNet outperformed the investigated baselines and ranked among the top methods in challenge evaluations. Its hyper-dense multi-path design improved performance without requiring more parameters, while preserving clinically acceptable inference time.

  • Results: Dense connectivity across modality paths outperformed early fusion, late fusion, and fusion after the first convolutional block.
  • Results: Increasing learned parameters did not provide an important performance boost, and widening some architectures slightly decreased CSF performance.
  • Results: HyperDenseNet outperformed baselines during both training and validation, including architectures with similar parameter counts.Figures 4 and 5 evaluated mean DSC for CSF, GM, and WM.
  • Computational efficiency: Less than 2 minutes were required to segment a whole 3D brain with HyperDenseNet, compared with around 45 seconds for the lightest architecture.
  • Challenge results: HyperDenseNet ranked among the top-3 methods in 6 of 9 metrics across the first and second iSEG submission rounds.
  • Challenge results: All methods performed worse on GM and WM, which had lower DSC and larger ASD values than easier tissues.The passage attributes this challenge to unclear boundaries between GM and WM.

3.2 MRBrainS Challenge

The MRBrainS evaluation compared HyperDenseNet with established 3D segmentation networks using adult brain MRI data and multiple evaluation measures. Results showed that UNet3D and DeepMedic outperformed FCN Res3D, while HyperDenseNet performance depended on modality combination.

  • Evaluation setup: MRBrainS evaluates adult brain tissue segmentation from T1, T1 IR, and FLAIR MRI using DSC, MHD, and absolute volume difference.The validation protocol used leave-one-out cross-validation over available datasets and averaged accuracy across three folds.
  • Modality experiments: Two-modality HyperDenseNet variants were competitive but varied substantially across modality combinations.The T1-and-FLAIR configuration ranked first for GM and WM DSC measures and second for CSF despite competing methods using all three modalities.
  • Baseline comparison: FCN Res3D achieved the lowest mean DSC among the tested models.The authors associate this result with transpose convolutions and downsampling or upsampling that may cause voxel misclassification and loss of image details.
  • Baseline comparison: UNet3D clearly outperformed FCN Res3D across all reported metrics.The authors relate this difference to skip connections that propagate information across abstraction levels between encoding and decoding paths.
  • Baseline comparison: DeepMedic performed better than its competitors and approached the performance of HyperDenseNet’s two-modality configurations.Its dual multiscale path enlarges the receptive field, while removing pooling operations may improve performance relative to FCN Res3D.

UNet3D.

HyperDenseNet performance improved with broader modality input in the reported MRBrainS comparisons. The three-modality model handled segmentation regions, particularly thin regions, better than two-modality versions and ranked first in the challenge.

  • Modality comparison: The three-modality HyperDenseNet achieved the highest mean DSC values for GM, WM, and CSF.The authors report significantly better segmentations than the two-modality configurations.
  • Modality comparison: The two-modality configurations showed competitive performance but significant variability across the three possible combinations.The T1-and-FLAIR configuration performed especially well for GM and WM DSC measures.
  • Challenge ranking: HyperDenseNet ranked first among competing MRBrainS methods, with the highest DSC and HD for GM and WM.The comparison covered the challenge ranking of 47 international teams and the top-10 methods.
  • Qualitative comparison: Three-modality HyperDenseNet handled thin regions better than its two-modality versions.The figure comparison identifies boundary regions between GM and WM as a common source of errors.

3.3 Analysis of features re-use

The feature-reuse analysis measured connection-weight dependence across HyperDenseNet layers and modalities. Strong reuse was distributed across many preceding layers, including layers from other modality streams.

  • Analysis method: Average L1-norms of connection weights served as a surrogate for each layer’s dependence on preceding layers.Values were normalized between 0 and 1 for visualization.
  • Plot interpretation: Dark squares in the connection plots indicate strong use of features from the source layer by the target layer.The target layer is shown on the x-axis and the source layer on the y-axis.
  • Feature reuse: Most layers distributed connection importance across many previous layers within and across modality paths.This pattern indicates that shallow features are directly reused by deeper layers from both streams.
  • Dataset-specific patterns: In iSEG experiments, immediate previous layers often had higher impact, while T2-connected features typically received the strongest values.The authors suggest T2 may be more discriminative than T1 in this particular setting.
  • Dataset-specific patterns: MRBrainS two-modality models also showed high-impact immediate connections within and between paths, with stronger reuse of low-level features by deeper layers.The reported behavior was especially evident for the T1-IR and FLAIR configuration.

4 CONCLUSION

The study introduced HyperDenseNet, a hyper-densely connected 3D fully convolutional network for multi-modal brain tissue segmentation. Its cross-stream connectivity supported complex feature combinations across abstraction levels, and the model achieved state-of-the-art performance on two benchmarks.

  • Contribution: HyperDenseNet extends dense connectivity to multi-path 3D CNNs for multi-modal brain tissue segmentation.Connections occur both within each modality stream and across different streams.
  • Contribution: Cross-stream dense connections give the network freedom to explore complex combinations between modalities across abstraction levels.The conclusion frames this design as particularly relevant to multi-modal volumetric medical-image segmentation.
  • Results: HyperDenseNet achieved state-of-the-art performance on the iSEG-2017 infant and MRBrainS adult segmentation benchmarks.The experiments also provided insights into shortcut connections for medical-image segmentation in multi-modal settings.

Supplemental Materials

The supplemental materials describe infant and older-adult imaging cohorts, tissue-label preparation, and overlap-based segmentation evaluation.

  • Imaging cohorts: Infant images came from the Baby Connectome Project pilot study and were acquired on a Siemens head-only 3T scanner during unsedated sleep.The protocol used ear protection and vacuum fixation; T1-weighted images had 1×1×1 mm3 resolution.
  • Imaging cohorts: The older-adult cohort included 20 functionally independent subjects aged 65–80 years without invalidating stroke or other brain diseases.The sample had a mean age of 71 ± 4 years, with 10 male and 10 female participants.
  • Segmentation labels: Eight structures were segmented, then merged into gray matter, white matter, and cerebrospinal fluid for evaluation, excluding the cerebellum and brainstem.The structures included cortical gray matter, basal ganglia, white matter, white matter lesions, peripheral cerebrospinal fluid, and lateral ventricles.
  • Segmentation labels: Manual segmentations were drawn on 3mm slice-thickness scans with an in-house tool based on MeVisLab contour segmentation objects.Annotation proceeded from the innermost structures.
  • Evaluation metrics: The Dice similarity coefficient compares reference and automatic segmentations using twice their intersection divided by the sum of their volumes.DSC ranges from 0 to 1, with 1 indicating perfect overlap and 0 indicating total mismatch.

Modified Hausdorff distance (MHD)

Modified Hausdorff distance measures boundary similarity between reference and automatic segmentations using point-to-set distances; lower values indicate closer boundaries.

  • Definition: MHD compares the reference and automatic segmentation boundaries represented as voxel point sets.Pref and Pauto denote the voxels within the respective segmentation boundaries.
  • Distance calculation: Point-to-set distance is the minimum Euclidean distance from a point to any point in the comparison set.This is expressed as d(q, P) = minp∈P ∥q − p∥.
  • Interpretation: Low MHD values indicate high boundary similarity between the reference and automatic segmentations.Distance-based metrics interpret smaller values as greater proximity and better segmentation.
  • Related metric: ASD is presented using the same Hausdorff-distance notation and belongs to the distance-based metrics whose smaller values indicate better segmentation.The supplied definition specifies the notation and interpretation but does not provide the complete ASD expression.

Implementation details

The implementation details cover HyperDenseNet training costs and architectural comparisons with DeepMedic, FCN Res3D, and UNet3D.

  • Training and inference: Training HyperDenseNet took around 70 minutes per epoch and 35 hours total for the two-modality version.With three modalities, each epoch took nearly 3 hours on an NVIDIA Tesla P100 GPU with 16 GB of RAM.
  • Training and inference: Inference on a whole 3D MR scan took on average from 70–80 to 250…The supplied passage truncates the upper endpoint of the reported inference range.
  • Architecture specifications: Table 8 details the number of kernels in convolutional and fully connected layers for the baselines and proposed architecture.Two-path architectures use the same number of kernels in both paths for each corresponding convolutional block.
  • Baseline architectures: DeepMedic uses two paths with eight convolutional blocks and adds two 1×1×1 fully connected convolutional layers before classification.Its second path receives a low-resolution input to provide a larger receptive field.
  • Baseline architectures: FCN Res3D uses residual encoder blocks and four convolutional upsampling blocks, while replacing max pooling with stride-2 layers.Batch normalization and Leaky ReLU activation with leakage 0.1 precede convolutions in residual blocks.
  • Baseline architectures: UNet3D has four encoding and five decoding blocks, with skip connections added at convolutional blocks of the…The supplied passage truncates the final description of the skip connections.
Loading 1804.02967v2…