PaperlayerView exact revision on arXiv

Source-linked AI summary

Inter-slice Context Residual Learning for 3D Medical Image Segmentation

Jianpeng Zhang, Yutong Xie, Yan Wang, Yong Xia

arXiv:2011.14155v1eess.IVcs.CV

TL;DR

3D medical-image segmentation remains challenging, in part because DCNNs have limited ability to perceive 3D context and inter-slice tumor differences are tiny. ConResNet jointly predicts segmentation and context-residual masks, using residual and attention mappings between two decoders. It outperforms state-of-the-art methods on both brain-tumor and pancreas segmentation tasks.

  • Problem

    Deep convolutional neural networks have limited ability to perceive 3D context, while inter-slice tumor differences are usually tiny and had not been characterized directly.

    Source: §I · pp. 1–2

  • Method

    ConResNet uses a shared encoder and segmentation and context-residual decoders, bridged by residual and attention mappings to learn and use inter-slice context.

    Source: §I · p. 2, §III · p. 3

  • Results

    ConResNet outperforms state-of-the-art methods on both brain-tumor segmentation and pancreas segmentation tasks.

    Source: §I · p. 2, §VI · p. 11

  • Takeaways & Limitations

    Ablation results indicate that both context residual mapping and context attention mapping contribute to the proposed context-learning scheme’s effectiveness.

    Source: A. Ablation Analysis · p. 9, §VI · p. 11

Abstract

from arXiv · show

Automated and accurate 3D medical image segmentation plays an essential role in assisting medical professionals to evaluate disease progresses and make fast therapeutic schedules. Although deep convolutional neural networks (DCNNs) have widely applied to this task, the accuracy of these models still need to be further improved mainly due to their limited ability to 3D context perception. In this paper, we propose the 3D context residual network (ConResNet) for the accurate segmentation of 3D medical images. This model consists of an encoder, a segmentation decoder, and a context residual decoder. We design the context residual module and use it to bridge both decoders at each scale. Each context residual module contains both context residual mapping and context attention mapping, the formal aims to explicitly learn the inter-slice context information and the latter uses such context as a kind of attention to boost the segmentation accuracy. We evaluated this model on the MICCAI 2018 Brain Tumor Segmentation (BraTS) dataset and NIH Pancreas Segmentation (Pancreas-CT) dataset. Our results not only demonstrate the effectiveness of the proposed 3D context residual learning scheme but also indicate that the proposed ConResNet is more accurate than six top-ranking methods in brain tumor segmentation and seven top-ranking methods in pancreas segmentation. Code is available at https://git.io/ConResNet

I. INTRODUCTION

The introduction identifies limited 3D context perception as a barrier to accurate medical-image segmentation, especially because inter-slice tumor differences are small. ConResNet explicitly learns and uses those differences, and the paper reports improved performance on brain-tumor and pancreas segmentation.

I. INTRODUCTION

  • 3D medical-image segmentation is difficult because organs and tumors vary in shape, size, and location, soft-tissue contrast is low, and annotated data are scarce.Accurate segmentation supports diagnosis and assessment of disease severity and prognosis.Source: §I · p. 1
  • Inter-slice context residuals capture voxel-wise changes between adjacent tumor masks, but are usually tiny and had not been characterized directly.The residuals occur on or near the tumor surface and, together with a slice’s tumor region, can reconstruct the 3D tumor shape.Source: §I · p. 2
  • ConResNet uses a shared encoder and two decoders to predict segmentation masks and inter-slice context residuals, with the residual decoder providing attention guidance.Its context residual module bridges the decoders through residual mapping and context attention mapping.Source: §I · p. 2
  • ConResNet improves performance over state-of-the-art methods on both brain-tumor and pancreas segmentation tasks.The model adds a context residual decoder and uses context residual and context attention mappings.Source: §I · p. 2

II. RELATED WORK

The related work reviews context-learning and residual-learning approaches, then motivates adding explicit inter-slice residual modeling to existing 3D medical-image segmentation networks.

A. Context learning

  • Existing context strategies aggregate multi-scale information, expand receptive fields, or filter features with attention, but do not directly characterize inter-slice context residuals.The reviewed strategies include spatial-pyramid pooling, atrous convolution, large kernels, and attention mechanisms.Source: A. Context learning · p. 2

B. Residual learning

  • Residual learning uses skip connections to learn feature differences between inputs and outputs; ConResNet instead targets feature residuals between adjacent slices to use 3D context.The passage links skip connections to avoiding degradation and training very deep networks.Source: B. Residual learning · p. 2

C. Medical image segmentation

  • Because slice-by-slice 2D models cannot capture inter-slice context, the paper proposes adding a ConRes decoder to existing 3D DCNNs rather than designing a new segmentation model.Prior 3D methods use volumetric convolutions, multi-scale context, or contour and distance-based losses.Source: )"*+,Context Residual Module · p. 3

III. METHOD

ConResNet jointly predicts a voxel-wise segmentation mask and a context residual mask that marks category changes between neighboring slices. A shared encoder and two decoders are bridged by mappings that pass residual features to the residual decoder and attention guidance back to the segmentation decoder.

III. METHOD

  • The context residual mask marks locations where corresponding voxels on adjacent slices belong to different semantic categories.A value of 0 indicates matching foreground/background categories; a value of 1 indicates one foreground and one background voxel.Source: §III · p. 3
  • ConResNet predicts the segmentation mask and context residual mask simultaneously from the 3D medical image.The model has a shared feature-extraction encoder and separate decoders for the two predictions.Source: §III · p. 3
  • The context residual module bridges the decoders: segmentation features are mapped into residual features, while context attention mapping sends attention guidance back to segmentation.The module is embedded between paired layers of the dual decoders.Source: §III · p. 3

A. Shared Encoder

ConResNet combines a shared encoder and two decoders, linked at each scale by context residual modules that learn inter-slice differences and use them as attention guidance.

A. Shared Encoder

  • The shared encoder uses nine residual blocks across five stages, down-sampling feature maps to one-eighth resolution while increasing channels from 32 to 256.The final stage uses dilated convolutions to expand the receptive field while retaining feature resolution.Source: A. Shared Encoder · p. 4

B. Dual Decoders

  • The segmentation decoder fuses up-sampled features with encoder skip features, while the ConRes decoder predicts corresponding inter-slice context residual masks and feeds them back as attention guidance.The two decoders are connected at each of three stages.Source: B. Dual Decoders · p. 4

C. Context Residual Module

  • Each context residual module computes position-wise absolute differences between adjacent-slice features, then uses the residual path output as sigmoid attention to weight segmentation features.Because attention highlights residual regions, the segmentation output uses 1 + σ(Ores) to avoid attenuating both foreground and background.Source: C. Context Residual Module · pp. 4–5

D. Network Optimization

The network is trained end-to-end with segmentation and auxiliary context-residual supervision. The segmentation decoder uses a combined loss, while residual prediction uses weighted binary cross entropy with deep supervision.

D. Network Optimization

  • The segmentation decoder is trained with a combination of cross-entropy and Dice losses.The passage specifies binary cross entropy and predicted and ground-truth segmentation labels, with a smooth factor in the Dice loss.Source: D. Network Optimization · p. 5
  • The ConRes decoder is supervised by ground-truth context residual masks using foreground-weighted binary cross entropy and deep supervision.The loss combines predictions at multiple supervision depths, with the trade-off parameter λ empirically set to 0.5.Source: D. Network Optimization · p. 5
  • ConResNet is jointly optimized end-to-end by minimizing the segmentation and context-residual losses together.The joint objective is L = Lseg + Lres.Source: D. Network Optimization · p. 5

A. Datasets

Experiments use the BraTS 2018 brain-tumor dataset and the NIH Pancreas-CT dataset. Implementation details describe preprocessing, training inputs, and segmentation metrics for the two tasks.

A. Datasets

  • BraTS provides 285 training cases and 66 online-test cases, each with four registered MR sequences; Pancreas-CT contains 82 contrasted-enhanced scans evaluated with four-fold cross-validation.BraTS labels cover enhancing tumor, tumor core, and whole tumor; Pancreas-CT reports average performance over all cases.Source: A. Datasets · p. 5

B. Implementation Details

  • BraTS volumes were normalized and cropped into 80×160×160 training sub-volumes, while Pancreas-CT scans were intensity-truncated, mapped to [0, 1], and cropped into 64×120×120 patches.BraTS was treated as three binary segmentation tasks with outputs for ET, WT, and TC.Source: B. Implementation Details · pp. 5–6
  • Evaluation uses Dice coefficient and Hausdorff distance on BraTS, and mean, maximum, and minimum Dice scores on Pancreas-CT.The supplied table captions identify comparisons of ConResNet with state-of-the-art methods on the BraTS 2018 online test set and Pancreas-CT, but provide no result values.Source: B. Implementation Details · p. 6, Table I · p. 6, Table II · p. 6

C. Results

ConResNet has the smallest overall Sum-Score among the compared BraTS methods, while on Pancreas-CT it leads the reported Dice metrics. Qualitative comparisons also favor ConResNet over the baseline and show attention emphasizing tumor boundaries.

Comparison on the BraTS dataset

  • ConResNet outperforms the six compared BraTS methods on overall Sum-Score, although nnUnet comes close and uses additional co-training data.The comparison aggregates ranks across Dice and Hausdorff metrics for ET, WT, and TC; all competing methods use ensembles, while ConResNet uses three independently trained models.Source: C. Results · pp. 6–7

Qualitative comparison

  • In CT-slice overlays and 3D visualizations, ConResNet produces pancreas segmentations more similar to ground truth than the baseline.The pancreas is highlighted in red in Fig. 4.Source: Figure 4 · p. 7, C. Results · p. 7

Comparison on the Pancreas-CT dataset

  • ConResNet achieves the highest mean Dice (86.06%), highest max Dice (92.00%), and highest min Dice (73.40%) on Pancreas-CT.It outperforms the baseline and other competitive methods on all three reported metrics.Source: C. Results · p. 7

Visualization of the learned context residual attention maps

  • Learned context residual attention maps highlight tumor-subregion residual positions, and attention-enhanced feature maps better highlight their boundaries.These visualizations compare feature maps obtained with and without context attention.Source: C. Results · pp. 7–8

A. Ablation Analysis

Ablations show gains from context residual mapping and further gains from context attention, while additional experiments examine deep supervision and residual learning across views.

A. Ablation Analysis

  • Against baseline, ConResNet raises ET/WT/TC Dice by 1.66%/0.83%/2.27% and lowers Hausdorff distance by 1.76/0.41/1.44, respectively.The comparison held network width, depth, and training strategies fixed; context residual mapping improved metrics, and context attention further improved them.Source: A. Ablation Analysis · p. 9
  • With both mappings, feature maps focus more on target regions than those from the baseline or residual-only model.The learned attention maps highlight context residual positions.Source: A. Ablation Analysis · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

B. Deep Supervision

  • λ=0.5 produced the lowest training loss and best local-validation performance, so the authors used it in their experiments.The study compared λ values of 0, 0.1, 0.5, and 1; λ=0 means no deep supervision.Source: B. Deep Supervision · p. 9, Figure 8 · p. 10

C. Learning Context Residual along Other Views

  • Axial, sagittal, and coronal residual learning achieved comparable performance, and each view outperformed the baseline on all evaluation metrics.For the sagittal view, the method computes position-wise absolute differences between adjacent features along the longitudinal axis.Source: C. Learning Context Residual along… · pp. 9–10, Table IV · p. 10

D. Applying to Multi-class Problems

The paper evaluates ConResNet in binary and multi-class tumor segmentation settings and compares context residual learning with boundary-loss methods. Binary mode performs better than multi-class mode, while both outperform the baseline.

D. Applying to Multi-class Problems

  • Binary ConResNet performs better than multi-class ConResNet, but both settings outperform the baseline on the local validation set.The BraTS task is framed as three binary region tasks or one four-class task; the authors suggest direct optimization of the three sub-regions may explain the difference.Source: D. Applying to Multi-class Problems · p. 10, Table V · p. 10

E. Comparing to Boundary Loss-based Methods

  • Context residual learning differs from boundary loss, yet each helps the model attend to boundaries; combining them can produce mutual promotion.With boundary loss, ConResNet still outperforms the baseline model using boundary loss.Source: E. Comparing to Boundary Loss-based… · p. 10

F. Efficiency Analysis

ConResNet adds complexity over the baseline, while the paper reports state-of-the-art performance on brain tumor and pancreas segmentation and nearly real-time brain tumor segmentation. The conclusion also reports effective context learning and outlines future research directions.

F. Efficiency Analysis

  • 2.66% more parameters, 18.64% more computations, and 2.53% larger model size than the baseline accompany ConResNet.GFLOPs were calculated for an input volume of 80×160×160.Source: Table VII · p. 11, F. Efficiency Analysis · p. 11
  • Despite its higher complexity and slower speed than the baseline, ConResNet performs brain tumor segmentation almost in real time and has potential for clinical practice.Source: F. Efficiency Analysis · p. 11

VI. CONCLUSION

  • ConResNet outperforms state-of-the-art methods on both brain tumor and pancreas segmentation tasks.The ablation study also demonstrates the effectiveness of context residual mapping and context attention mapping.Source: §VI · p. 11
  • Future work proposes using predicted residual masks as priors to highlight error-prone regions and refining segmentation with another network.The authors also plan to combine 3D context learning with self-supervised learning for semi-supervised segmentation.Source: §VI · p. 11

Related papers

Loading 2011.14155v1…