Source-linked AI summary
3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes
Siqi Liu, Daguang Xu, S. Kevin Zhou, Thomas Mertelmeier, Julia Wicklein, Anna Jerebko, Sasa Grbic, Olivier Pauly, Weidong Cai, Dorin Comaniciu
TL;DR
3D anisotropic volumes are difficult to model because their within-slice and between-slice resolutions differ sharply, while 3D datasets and compute are limited. AH-Net transfers pretrained 2D convolutional features into a 3D network with anisotropic convolutions, preserving within-slice generalization while using 3D context. Evaluations on DBT lesion detection and CT liver and tumor segmentation report state-of-the-art results, with faster GPU inference than stacked 2D processing.
Problem
Anisotropic 3D volumes combine much higher within-slice than between-slice resolution with limited 3D data, challenging direct 3D CNN training and 2D methods that omit between-slice information.
Method
AH-Net transforms a pretrained 2D encoder into 3D and uses anisotropic decoder convolutions to extract 3D context from anisotropic volumes.
Results
AH-Net obtains state-of-the-art results across DBT lesion detection and CT liver and liver tumor segmentation, while running faster on GPU than stacked 2D processing.
Takeaways & Limitations
Transferred 2D features and anisotropic convolution blocks support better generalization than traditional 3D networks while exploiting volumetric context.
Abstract
from arXiv · showhide
While deep convolutional neural networks (CNN) have been successfully applied for 2D image analysis, it is still challenging to apply them to 3D anisotropic volumes, especially when the within-slice resolution is much higher than the between-slice resolution and when the amount of 3D volumes is relatively small. On one hand, direct learning of CNN with 3D convolution kernels suffers from the lack of data and likely ends up with poor generalization; insufficient GPU memory limits the model size or representational power. On the other hand, applying 2D CNN with generalizable features to 2D slices ignores between-slice information. Coupling 2D network with LSTM to further handle the between-slice information is not optimal due to the difficulty in LSTM learning. To overcome the above challenges, we propose a 3D Anisotropic Hybrid Network (AH-Net) that transfers convolutional features learned from 2D images to 3D anisotropic volumes. Such a transfer inherits the desired strong generalization capability for within-slice information while naturally exploiting between-slice information for more effective modelling. The focal loss is further utilized for more effective end-to-end learning. We experiment with the proposed 3D AH-Net on two different medical image analysis tasks, namely lesion detection from a Digital Breast Tomosynthesis volume, and liver and liver tumor segmentation from a Computed Tomography volume and obtain the state-of-the-art results.
1. Introduction
3D anisotropic volumes preserve rich within-slice information while retaining meaningful between-slice context, but their unequal resolutions and limited 3D training data challenge conventional CNNs. AH-Net transfers pretrained 2D features into a 3D architecture with anisotropic convolutions to model both information sources.
- Anisotropic DBT and CT volumes have within-slice resolution more than ten times higher than between-slice resolution, although between-slice context remains meaningful.
- Direct 3D CNNs face mismatched information density, greater feature and memory demands, limited pretrained 3D data, and high data variation.
- AH-Net transforms a pretrained 2D encoder into 3D and adds anisotropic 3 × 3 × 1 and 1 × 1 × 3 decoder convolutions, dense connections, skip connections, and pyramid volumetric pooling.
- Pretraining on large 2D datasets supports easier training and generalization, while anisotropic blocks exploit 3D context.
- End-to-end 3D inference makes AH-Net faster on GPU than conventional multi-channel 2D processing for each volume.
2. Related Work
Prior work addresses anisotropic volumes through resampling, mixed 2D/3D networks, recurrent slice modeling, or anisotropic kernels, but AH-Net combines transferred 2D features with anisotropic 3D decoding.
- Resampling to isotropic resolution can add redundant voxels and computational cost when z resolution is much smaller than xy resolution.
- Mixed 2D/3D approaches process slices with 2D networks before integrating outputs volumetrically, but HDenseNet lacks shared convolutions between its 2D and 3D networks.
- BDC-LSTM combines slice-wise convolutions with recurrent propagation to model 3D context along the z-direction.
- Anisotropic-kernel methods distribute capacity toward the xy plane, but large kernels increase computational cost and small-kernel decompositions approximate them.
- The MC-GCN pretraining architecture uses a ResNet50 initialized with ImageNet weights, global convolutional modules, refinement modules, and upsampling.
- AH-Net uses anisotropic convolutions only in the decoder and locks the encoder with transferred 2D pretrained weights, allowing initialization from large-scale 2D datasets.
3. Anisotropic Hybrid Network
AH-Net transfers a learned 2D encoder into a 3D network, then adds anisotropic decoder blocks to model volumetric context while accommodating limited z-resolution and memory.
- Architecture: The AH-Net uses a transferred 2D feature encoder and a newly designed 3D decoder to combine within-slice representations with volumetric context.The encoder is learned first, after which the decoder is added and fine-tuned with the encoder.
- 2D feature transfer: The 2D MC-GCN encoder is initialized from ImageNet-pretrained ResNet50 weights and fine-tuned on slices extracted from 3D volumes.After convergence, its encoder parameters are transferred to the corresponding AH-Net encoder layers.
- 2D-to-3D transformation: 2D convolutional tensors are transformed into 3D tensors by adding an extra dimension, allowing encoder convolutions to operate slice by slice.The first layer is rearranged to form a kernel spanning three neighboring slices, while other kernels receive an identity depth dimension.
- Input layer transform: The input transform uses Conv 7 × 7 × 3/(2, 2, 1), while MaxPool 1 × 1 × 2/(1, 1, 2) downsamples the z dimension by fusing neighboring slices.An additional MaxPool 3×3×3/(2, 2, 2) maintains feature resolution consistent with the 2D network.
- ResNet block transform: ResNet encoder blocks retain slice-wise xy processing, while z downsampling uses Conv 1 × 1 × 1/(2, 2, 1) followed by MaxPool 1 × 1 × 2/(1, 1, 2).This avoids skipping slices when the volume contains few slices along z.
- 3D decoder: The decoder uses Conv 1 × 1 × 1, Conv 3 × 3 × 1, and Conv 1 × 1 × 3 blocks, with dense connections, upsampling, feature summation, and pyramid volumetric pooling.The design separately processes xy and z context and extracts multiscale features at the decoder output.
- Training: Training first locks transferred parameters and fine-tunes the randomly initialized decoder, then jointly fine-tunes the entire AH-Net.This staged optimization avoids using the same learning rate for pretrained encoder and randomly initialized decoder parameters.
- Focal loss: Focal loss shifts training toward hard voxels after the L2 loss plateaus, allowing the loss to keep descending while output details progressively improve.A larger focusing parameter γ emphasizes large numeric errors from hard voxels.
4. Experimental Results
The experiments evaluate AH-Net on DBT lesion detection and CT liver-lesion segmentation, using transferred 2D features, 3D anisotropic decoding, and comparisons with baseline networks and challenge submissions.
- Breast lesion detection from DBT: The DBT study uses 2809 volumes from 12 sites, with lesions annotated by experienced radiologists using approximate 3D bounding boxes.The DBT images have 0.085mm × 0.085mm in-plane resolution and 1mm slice resolution.
- Breast lesion detection from DBT: FROC evaluation measures lesion TPR against false positives allowed per volume, with AH-Net outperforming both 2D and 3D U-Nets by large margins.AH-Net achieves nearly 50% TPR when only 0.01 false-positive findings per volume are allowed.
- Breast lesion detection from DBT: Visual DBT comparisons show AH-Net detecting a lesion missed by 2D GCN and removing false positives in another volume.The comparisons use maximum-intensity projections in the xy, xz, and yz planes.
- Breast lesion detection from DBT: AH-Net training transfers encoder weights from a converged 2D MC-GCN and fine-tunes a 3D decoder on volumetric patches.The compared networks include 2D and 3D U-Nets, while focal loss is used for hard-voxel mining after convergence.
- Liver and liver tumor segmentation from CT: The CT evaluation uses the LITS challenge dataset and compares AH-Net with six top-ranked submissions using Dice Global and Dice per Case metrics.AH-Net achieves state-of-the-art performance in three of four metrics, including lesion Dice Global and liver Dice Global and Dice per Case.
5. Conclusion
The paper concludes that AH-Net transfers 2D convolutional features into 3D anisotropic volumes while exploiting 3D context through anisotropic convolution blocks. Across DBT detection and CT segmentation, it reports state-of-the-art results and faster GPU inference than assembling outputs from a 2D network.
- Conclusion: AH-Net transfers convolutional features from 2D images to 3D volumes with anisotropic resolution.The network combines a transferred encoder with anisotropic convolution blocks.
- Conclusion: The paper reports state-of-the-art results on both a large-scale in-house DBT dataset and a competitive open CT segmentation challenge.The conclusion attributes better generalization than traditional 3D networks to transferred 2D features and anisotropic convolution blocks.
- Conclusion: AH-Net has faster GPU inference than piling results from a 2D network.The paper also states that the architecture may apply to volumes approaching isotropic resolution.
A. Visual Cases of the DBT dataset
The DBT examples show that both MC-GCN and 3D AH-Net detect lesions across varied appearances, while 3D AH-Net additionally detects cases missed by MC-GCN using between-slice context.
- Both MC-GCN and 3D AH-Net detect lesions varying in size and appearance across five test DBT volumes.The 3D AH-Net confidence is usually higher than MC-GCN's in these examples.
- 3D AH-Net detects lesions that MC-GCN misses when single-slice information cannot distinguish them from surrounding breast tissues.The examples attribute this advantage to using 3D context between slices.
- Some lesions remain undetected by both networks when lesion-to-tissue contrast is low or lesions resemble lymph nodes or skin moles.These failures occur particularly for lesions in dense breast tissues.
- The visual cases include lesions blended in dense breast tissue but detectable through boundary speculations or architectural distortion.
B. Liver Tumor Segmentation Challenge
The LITS examples illustrate segmentation across substantial liver and lesion variation, while showing that 3D AH-Net uses inter-slice consistency to improve boundaries over MC-GCN; evaluation is limited by missing test ground truth.
- Visual variation: Livers and liver lesions vary substantially in size, morphology, shape, and intensity across the LITS test volumes.Liver lesions are highly sparse and can be small, making segmentation challenging.
- Evaluation boundary: The test-volume examples cannot provide ground-truth comparisons because the LITS testing images lack ground-truth label maps.
- Segmentation behavior: Without sufficient 3D context, MC-GCN produces false-positive regions at structure boundaries, especially under low image contrast.It also fails to generate correct boundaries near the top or bottom of lesions.
- Segmentation behavior: 3D AH-Net segments structures correctly in 3D by considering consistency between slices despite transferring its feature-extraction network from a 2D network.
- Segmentation behavior: The sagittal and coronal boundaries can appear jagged because the z direction has low resolution.