PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation

Holger R. Roth, Le Lu, Nathan Lay, Adam P. Harrison, Amal Farag, Andrew Sohn, Ronald M. Summers

arXiv:1702.00045v1cs.CV

TL;DR

Pancreas segmentation in CT is difficult because of substantial anatomical variability and the importance of modeling both organ interiors and boundaries. This paper uses multi-view HNNs for localization and segmentation, achieving an average DSC of 81.27%±6.27% in 4-fold cross-validation.

  • Problem

    Pancreas segmentation in CT remains challenging because of high anatomical variability and the need to model boundaries, appearance consistency, and spatial consistency.

  • Method

    A cascaded deep-learning system uses HNNs across axial, sagittal, and coronal views for high-recall localization, then combines interior and boundary cues for segmentation.

  • Results

    81.27%±6.27% average DSC is achieved in 4-fold cross-validation, with HNN-RF significantly improving worst-case Hausdorff distance accuracy.

  • Takeaways & Limitations

    Multi-view pooling and spatial aggregation advance reported pancreas-segmentation performance, while the approach is suggested as generalizable to organs and tumors with large variations and pathologies.

Abstract

from arXiv · show

Accurate and automatic organ segmentation from 3D radiological scans is an important yet challenging problem for medical image analysis. Specifically, the pancreas demonstrates very high inter-patient anatomical variability in both its shape and volume. In this paper, we present an automated system using 3D computed tomography (CT) volumes via a two-stage cascaded approach: pancreas localization and segmentation. For the first step, we localize the pancreas from the entire 3D CT scan, providing a reliable bounding box for the more refined segmentation step. We introduce a fully deep-learning approach, based on an efficient application of holistically-nested convolutional networks (HNNs) on the three orthogonal axial, sagittal, and coronal views. The resulting HNN per-pixel probability maps are then fused using pooling to reliably produce a 3D bounding box of the pancreas that maximizes the recall. We show that our introduced localizer compares favorably to both a conventional non-deep-learning method and a recent hybrid approach based on spatial aggregation of superpixels using random forest classification. The second, segmentation, phase operates within the computed bounding box and integrates semantic mid-level cues of deeply-learned organ interior and boundary maps, obtained by two additional and separate realizations of HNNs. By integrating these two mid-level cues, our method is capable of generating boundary-preserving pixel-wise class label maps that result in the final pancreas segmentation. Quantitative evaluation is performed on a publicly available dataset of 82 patient CT scans using 4-fold cross-validation (CV). We achieve a Dice similarity coefficient (DSC) of 81.27+/-6.27% in validation, which significantly outperforms previous state-of-the art methods that report DSCs of 71.80+/-10.70% and 78.01+/-8.20%, respectively, using the same dataset.

I. INTRODUCTION

Pancreas segmentation remains difficult because its anatomy varies substantially across patients, limiting conventional methods. The paper addresses this challenge with a two-stage, multi-view HNN system that aggregates interior and boundary cues for localization and segmentation.

  • Pancreas shape, size, and location variability limits reported segmentation accuracy to below 73% DSC.
  • Earlier pancreas methods primarily used volumetric multiple-atlas registration and robust label fusion for per-pixel organ labeling.
  • Deep CNN approaches improved performance stability, with smaller standard deviations than earlier top-down and bottom-up pancreas segmentation methods.
  • Mid-level visual cues from organ interiors and boundaries can complement pixel-level features by improving spatial, appearance, and boundary consistency.
  • The proposed system aggregates HNN probability maps from axial, sagittal, and coronal views for pancreas localization and segmentation.
  • The improved localizer replaces an earlier superpixel-based approach with deep learning designed to maximize spatial recall while reducing non-pancreas volume.

II. METHODS

The method uses a two-stage framework that localizes the pancreas before segmentation, combining HNN-derived interior and boundary cues with spatial aggregation. HNNs are trained across multiple scales and side outputs to produce fused predictions for pancreas labeling.

  • Two-stage framework: The framework first computes a pancreas bounding box, then performs segmentation within the localized spatial extent.Localization aims to cover the entire pancreas while pruning most of the CT volume; segmentation integrates interior and boundary cues.
  • Two-stage framework: The system cascades and aggregates six HNN outputs trained at three orthogonal views and two spatial scales.The configuration avoids multi-atlas registration and multi-label fusion techniques.
  • HNN mid-level cues: HNNs learn pancreas interior and boundary image-labeling maps that serve as mid-level visual cues for localization and segmentation.The architecture was originally introduced for holistically-nested edge detection and is applied here to organ-specific cues.
  • HNN mid-level cues: The HNN architecture uses five stages with strides of 1, 2, 4, 8, and 16 to capture multiple feature scales.The networks are initialized from pretrained models and fine-tuned on pancreas training data with a learning rate of 10^-6.
  • HNN mid-level cues: Deep supervision applies losses at side-output layers, while a learned weighted-fusion layer combines multi-scale predictions.Smaller side outputs have larger receptive fields, supporting multi-scale and multi-level feature integration.

B. Pancreas Localization

Pancreas localization prunes irrelevant CT regions before segmentation, but conventional regression methods may not ensure extremely high pixel-level organ coverage.

  • Localization removes irrelevant CT volume before the more detailed pancreas segmentation stage.
  • Random forest regression may fail to guarantee that the predicted bounding box contains the entire pancreas with extremely high pixel-level sensitivity.
  • A superpixel-based approach using hand-crafted and CNN features can provide improved localization performance over conventional regression methods.

1) Regression Forest:

Regression forests predict pancreas bounding-box parameters from image patches and aggregate multiple candidate predictions into a final localization.

  • A regression forest predicts pancreas center and lower and upper bounding-box offsets from image patches centered at sampled 3D locations.
  • Candidate predictions are aggregated using methods such as non-maximum suppression, mean aggregation, cluster medoids, or discriminative acceptance.
  • The predicted bounding box is represented by its center and offsets to the lower and upper corners, which need not be symmetric around the object center.

2) Random Forest on Superpixels:

The superpixel-based initialization combines hand-crafted and deep CNN features with random-forest labeling to generate candidate pancreas regions.

  • Random-forest classification uses hand-crafted and deep CNN-derived image features to compute candidate bounding-box regions.
  • Applying the random forest at a probability threshold above 0.5 rejects much of the non-pancreas volume in CT scans.
  • The initial candidate generation nearly surrounds the pancreas in all patient cases with approximately 97% recall.
  • The approach is described as a hybrid of deep-learning and non-deep-learning techniques.

3) Multi-view Aggregated HNNs:

Multi-view HNN-I probability maps are thresholded and spatially cleaned to produce conservative, high-recall pancreas candidate regions for bounding-box generation.

  • HNN-I processes axial, sagittal, and coronal views to identify pancreas-probability regions while discarding large negative CT volumes.
  • A threshold of >=0.5 is used conservatively to retain high sensitivity and recall exceeding 99%.
  • Erosion, largest-connected-component selection, and dilation remove small incorrect connections and outlier high-probability blobs.
  • Figure 3 depicts the candidate bounding-box region-generation pipeline from left to right, with the gold-standard pancreas shown in red.

C. Pancreas Segmentation

The segmentation pipeline combines HNN-derived interior and boundary cues with superpixel generation and spatial aggregation to produce pancreas segmentations. Multiscale combinatorial grouping organizes boundary maps into object proposals for this process.

  • The pipeline uses HNN probability maps to generate boundary and interior cues, then produces superpixels and aggregates them into a final segmentation with random forest classification.
  • MCG extracts organ-specific superpixels from learned HNN-B boundary maps at three scales.
  • MCG builds hierarchical superpixel partitions and merges them across scales to explore combinatorial object proposals.
  • The first two MCG proposal levels achieve approximately 88% DSC with optimally computed superpixel labels.

3) Spatial Aggregation with Random Forest:

The random-forest spatial aggregation stage classifies previously generated superpixels using intensity, multi-view HNN, and normalized spatial features. Experiments use 82 contrast-enhanced CT volumes with 4-fold cross-validation.

  • Random forest classification aggregates superpixel features derived from CT intensity statistics, multi-view HNN-I and HNN-B pooling, and normalized 3D coordinates.
  • The feature set includes first- through fourth-order intensity moments, eight intensity percentiles, pooled HNN outputs, and normalized x, y, and z coordinates.
  • The experiments use manual pancreas tracings from 82 contrast-enhanced abdominal CT volumes with random approximately 60-patient training and 20-patient testing splits in 4-fold cross-validation.
  • The study reports extensive quantitative evaluation across method configurations and comparisons with previous state-of-the-art work.

1) Localization:

The localization stage evaluates candidate bounding boxes for pancreas coverage and background reduction. Multi-view max-pooled HNN-I achieves nearly complete recall, whereas regression-forest localization has several low-recall cases.

  • Candidate regions reduce absolute CT volume by 90.36% on average while retaining 99.93% mean pancreas recall, ranging from 94.54% to 100.00%.
  • Regression-forest localization produces 16 of 82 bounding boxes below 60% pixel-to-pixel recall against ground-truth pancreas masks.
  • Multi-view max-pooled HNN-I produces 100% recalls except for two cases with recalls of at least 94.54%.
  • Table I evaluates regression-forest localization using Dice and average Euclidean distance against gold-standard bounding boxes under 4-fold cross-validation.

2) HNN Spatial Aggregation for Pancreas Segmentation:

Multi-view pooling improves HNN-based pancreas segmentation after localization, with meanmax pooling performing best among tested pooling functions. Adding random-forest spatial aggregation yields a small, statistically nonsignificant improvement, while cross-dataset results indicate transfer without retraining.

  • Element-wise multi-view mean or max pooling generally outperforms single-view HNN-I maps, and meanmax pooling provides the strongest mean DSC in the candidate region stage.
  • Meanmax pooling reaches 81.14% mean DSC in the second cascaded stage, increasing from 76.79% in the candidate region stage.
  • Random-forest spatial aggregation increases average DSC from 81.14% to 81.27%, but the improvement is not statistically significant with p > 0.05.
  • Both HNNmeanmax and HNN-RF outperform previous methods reporting mean DSC scores of 71.4% and 78.01% on the compared metrics.
  • On a different 30-patient CT dataset without retraining, HNN-I achieves 62.26% mean DSC, increasing to 65.66% when averaging four cross-validation models.
  • On 19 additional unseen CT scans collected under the same protocol, HNNmeanmax achieves a mean DSC of 81.2%.

IV. DISCUSSION & CONCLUSION

The multi-view HNN-based approach advances pancreas segmentation performance while maintaining statistical stability and computational efficiency. Boundary-aware aggregation particularly improves worst-case accuracy, although direct comparisons across datasets remain limited.

  • Quantitative performance: 81.27%±6.27% DSC is reported as the highest average testing performance under 4-fold cross-validation.The authors note that strict comparison with most other methods is not directly possible because different datasets were used.
  • Statistical stability: The method achieves DSC standard deviations of 7.3% or 6.27%, compared with 18.6% and 15.3% for cited prior methods.The reported minimum DSC values are 44.69% for HNNmeanmax and 50.69% for HNN-RF.
  • Quantitative performance: 80% of cases achieve DSC 78.05%, while 90% achieve DSC 74.22% or higher after RF aggregation.RF aggregation also makes DSC performance more stable across pancreas-probability thresholds.
  • Methodological implications: Multi-view spatial pooling and interior-boundary cue fusion assemble 3D information from orthogonal 2D views.This design avoids the spatial discontinuity or inconsistency that can arise at overlapping windows in padded 3D CNN processing.
  • Quantitative performance: HNN-RF improves worst-case pancreas segmentation accuracy in Hausdorff distance compared with HNNmeanmax.The improvement is statistically significant with p<0.001.
  • Methodological implications: Inference takes approximately 2–3 minutes rather than the hours reported for several earlier methods.The comparison concerns computational cost for the pancreas segmentation system.
Loading 1702.00045v1…