Source-linked AI summary
AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy
Wentao Zhu, Yufang Huang, Liang Zeng, Xuming Chen, Yong Liu, Zhen Qian, Nan Du, Wei Fan, Xiaohui Xie
TL;DR
Accurate head-and-neck organ-at-risk segmentation is laborious but important for radiation-therapy planning. AnatomyNet addresses this with an end-to-end whole-volume CT model, achieving higher benchmark Dice performance while remaining subject to spatial-modeling limitations.
Problem
Manual delineation of head-and-neck organs-at-risk is laborious, motivating accurate automated segmentation for radiation-therapy planning.
Method
AnatomyNet is a single end-to-end network that segments all organs-at-risk from whole-volume CT images, extending 3D U-Net with squeeze-and-excitation residual blocks and specialized losses.
Results
79.25 average Dice coefficient, 3.3% better than the best MICCAI 2015 Challenge result despite the more challenging ten-class formulation.
Takeaways & Limitations
The atlas-free, fully automated model segments all target organs in one pass with minimal pre- and post-processing.
Takeaways & Limitations
AnatomyNet treats voxels equally and therefore cannot effectively model shape priors and connectivity patterns, sometimes misclassifying small background regions as organs-at-risk.
Abstract
from arXiv · showhide
Methods: Our deep learning model, called AnatomyNet, segments OARs from head and neck CT images in an end-to-end fashion, receiving whole-volume HaN CT images as input and generating masks of all OARs of interest in one shot. AnatomyNet is built upon the popular 3D U-net architecture, but extends it in three important ways: 1) a new encoding scheme to allow auto-segmentation on whole-volume CT images instead of local patches or subsets of slices, 2) incorporating 3D squeeze-and-excitation residual blocks in encoding layers for better feature representation, and 3) a new loss function combining Dice scores and focal loss to facilitate the training of the neural model. These features are designed to address two main challenges in deep-learning-based HaN segmentation: a) segmenting small anatomies (i.e., optic chiasm and optic nerves) occupying only a few slices, and b) training with inconsistent data annotations with missing ground truth for some anatomical structures. Results: We collected 261 HaN CT images to train AnatomyNet, and used MICCAI Head and Neck Auto Segmentation Challenge 2015 as a benchmark dataset to evaluate the performance of AnatomyNet. The objective is to segment nine anatomies: brain stem, chiasm, mandible, optic nerve left, optic nerve right, parotid gland left, parotid gland right, submandibular gland left, and submandibular gland right. Compared to previous state-of-the-art results from the MICCAI 2015 competition, AnatomyNet increases Dice similarity coefficient by 3.3% on average. AnatomyNet takes about 0.12 seconds to fully segment a head and neck CT image of dimension 178 x 302 x 225, significantly faster than previous methods. In addition, the model is able to process whole-volume CT images and delineate all OARs in one pass, requiring little pre- or post-processing. https://github.com/wentaozhu/AnatomyNet-for-anatomical-segmentation.git.
I. INTRODUCTION
AnatomyNet addresses the limitations of manual, atlas-based, and earlier learning-based head-and-neck OAR segmentation with an end-to-end whole-volume model. Its design targets small anatomies, annotation gaps, computational cost, and segmentation accuracy.
- Manual OAR delineation requires examining hundreds of CT slices individually, making radiation-therapy planning laborious.
- Atlas-based methods are computationally intensive and struggle with patient anatomy variation because they rely on fixed exemplars.
- Earlier learning-based methods often require extensive preprocessing or hand-crafted features, while deep models may use local patches or atlas registration.
- Its architecture and hybrid loss address GPU constraints, very small OARs, and missing anatomical annotations during training.
- AnatomyNet jointly segments all OARs from raw whole-volume CT images in one end-to-end pass with minimal pre- and post-processing.
- 3.3% better average Dice coefficient than the best MICCAI 2015 result was achieved across nine anatomical structures.
II. MATERIALS AND METHODS
The study trains and evaluates AnatomyNet on curated whole-volume head-and-neck CT data. The model modifies 3D U-Net with SE residual blocks, reduced down-sampling, and whole-volume multi-organ output.
- Model: AnatomyNet receives a cropped whole-volume head-and-neck CT image and outputs probability maps for nine OARs plus background.
- Data: The dataset combines whole-volume CT images and manual masks from four public sources, yielding 261 training samples and a separate MICCAI 2015 test set.
- Network architecture: AnatomyNet is a 3D U-Net variant that replaces standard convolutional layers with three-dimensional squeeze-and-excitation residual blocks.
- Network architecture: Only the first encoder block down-samples, while later down-sampling layers are removed to preserve performance on small anatomies and reduce memory demands.
- SE residual blocks: SE blocks use global average pooling and learned channel-wise scaling to calibrate residual feature maps.
- Output: The output concatenates the input with transposed-convolution features before producing ten-channel softmax segmentation maps.
C. Loss function
The loss-function design targets severe voxel imbalance in small-organ segmentation. AnatomyNet combines Dice loss, which addresses class distribution, with focal loss, which emphasizes poorly classified voxels.
- Small organs such as the optic chiasm occupy about 1/100,000 of the whole-volume CT image, creating severe voxel imbalance.
- Generalized Dice loss can make optimization unstable in extremely unbalanced segmentation, motivating alternative loss designs.
- The hybrid loss combines Dice loss for class distribution with focal loss for improving learning on poorly classified voxels.
- The loss includes trade-offs between Dice and focal components, plus penalties for false negatives and false positives.
D. Handling missing annotations
AnatomyNet addresses inconsistent ground-truth coverage by masking unannotated anatomies and weighting classes according to annotation frequency during hybrid-loss training.
- 261 head and neck CT images were collected with ground truths whose nine anatomical annotations varied substantially across cases.
- The training objective combines Dice and focal losses, with focal loss targeting difficult or poorly classified voxels.The hybrid objective is illustrated as Dice loss plus λ-weighted focal loss.
- The voxel distribution is highly imbalanced: background occupies 98.18% of voxels, while chiasm occupies about 1/100,000 of the whole volume.Chiasm represents 0.35% of foreground voxels, illustrating the small-organ imbalance motivating the loss design.
- The loss masks out missed anatomies so absent annotations are not treated as negative labels during training.The background is also masked when not all anatomies are annotated.
- Class weights are set inversely to the number of annotations for each anatomy to balance updates across structures.The weighting scheme is intended to equalize the contribution of differently annotated anatomies.
E. Implementation details and performance evaluation
AnatomyNet training uses a two-stage optimization schedule with on-the-fly augmentation, and performance is evaluated using the Dice coefficient.
- Training uses RMSprop for 150 epochs followed by stochastic gradient descent with momentum 0.9 for 50 epochs.The learning rates are 0.002 for RMSprop and 0.001 for stochastic gradient descent.
- A batch size of 1 accommodates variation in whole-volume CT image sizes.
- Affine transformation and elastic deformation are applied online for data augmentation during training.
- Performance is measured with the Dice coefficient, defined as 2TP/(2TP + FN + FP).TP, FN, and FP denote true positives, false negatives, and false positives, respectively.
III. RESULTS
The results section evaluates AnatomyNet after training on 261 samples, using the MICCAI 2015 challenge test set and comparisons with prior methods.
- AnatomyNet is trained on 261 samples and evaluated on 10 MICCAI Head and Neck Segmentation Challenge 2015 test samples.The evaluation compares its performance with previous methods benchmarked on the same test dataset.
A. Determining down-sampling scheme
The study selects a single down-sampling layer and compares architectural feature-combination and feature-learning choices to determine the network structure.
- A. Determining down-sampling scheme: The one-down-sampling design is motivated by preserving resolution for small OARs that occupy only a few CT slices.Additional down-sampling can reduce resolution and make small-organ features easier to miss.
- A. Determining down-sampling scheme: One down-sampling block achieves the best average performance, outperforming alternative schemes in 6 of 9 anatomies.The largest performance gaps occur for the left optic nerve, right optic nerve, and optic chiasm.
- B. Choosing network structures: The architecture study compares concatenation versus summation for horizontal features and convolutional, residual, and SE-residual learning blocks.The six compared models include 3D SE Res U-Net, 3D Res U-Net, vanilla U-Net, and summation variants.
- B. Choosing network structures: Feature concatenation consistently outperforms summation, and 3D SE residual U-Net with concatenation performs best in the study.The authors associate this result with concatenation's greater flexibility and SE channel-wise calibration.
- B. Choosing network structures: SE residual learning improves DSC by 2–3% over traditional U-Net and outperforms it in 6 of 9 anatomies.
C. Choosing loss functions
The study compared loss functions and selected a Dice–focal hybrid for AnatomyNet, then benchmarked the resulting model against prior methods on the MICCAI 2015 test set.
- Hybrid losses outperformed Dice or exponential logarithmic loss in 8 of 9 anatomies, indicating benefits from incorporating voxel-level loss.
- Dice combined with focal loss performed better than the other hybrid loss, improving optic nerve right and optic chiasm segmentation by 2–3%.
- The selected Dice–focal hybrid loss was used to train AnatomyNet for benchmarking against previous methods.
- 79.25 average Dice coefficient was achieved on the MICCAI 2015 test set, 3.3% better than its best challenge result.
- AnatomyNet segmented a 178×302×225 head-and-neck CT image in about 0.12 seconds, whereas atlas-based methods could take a dozen minutes.
E. Visualizations on MICCAI 2015 test
The figures visualize AnatomyNet predictions against ground truth on MICCAI 2015 cases and on an independent internal dataset, including examples where performance was weaker.
- Green denotes ground truth, red denotes predicted segmentation, and yellow denotes overlap between them.
- Each column represents one sample, while each row represents one left-and-right anatomy or a 3D reconstructed anatomy.
- The last two MICCAI test-set columns show cases where AnatomyNet did not perform well.
- Visual inspection on the independent internal dataset suggested good performance overall, with larger anatomies generally outperforming small ones such as the optic chiasm.
A. Impacts of training datasets
Larger and combined training datasets generally improve performance, but annotation inconsistencies materially affect organ-specific results and cross-dataset generalization. Evaluation also highlights limitations of Dice-based assessment and whole-volume predictions for surface-distance metrics.
- Dataset size and annotation quality: A larger combined dataset improved overall performance by 2.5% over the smaller dataset.The authors report that the larger sample size improved average performance, although some organs favored the smaller dataset.
- Dataset size and annotation quality: Annotation inconsistencies between datasets affected model performance, with smaller-data training performing better on mandible and optic nerves.The authors identify annotation quality and data size as important factors in deep-learning training.
- Dataset size and annotation quality: Training on DATASET 1 alone produced generally poor performance when evaluated on DATASETS 2–3, indicating limited cross-dataset generalization.The datasets came from different sources and physician annotation criteria.
- Evaluation considerations: Dice coefficient was the primary evaluation metric, although the authors note it may not be the most clinically relevant measure.They also evaluated average 95th percentile Hausdorff distance in millimeters.
- Evaluation considerations: Whole-volume predictions can produce outliers that substantially worsen Hausdorff distance compared with local-patch methods.Despite this challenge, AnatomyNet was roughly within the range of the best MICCAI 2015 results for six of nine anatomies.
V. CONCLUSION
The paper proposes an end-to-end, atlas-free deep-learning model for automated anatomy segmentation. Its conclusion emphasizes the model’s fully automated segmentation framework.
- The paper proposes an end-to-end, atlas-free, fully automated deep-learning model for anatomy segmentation.