Source-linked AI summary
CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation
Ran Gu, Guotai Wang, Tao Song, Rui Huang, Michael Aertsen, Jan Deprest, Sébastien Ourselin, Tom Vercauteren, Shaoting Zhang
TL;DR
Medical image segmentation must handle targets with substantial positional, shape, and scale variation, while CNN decisions remain difficult to explain. CA-Net combines spatial, channel, and scale attention in a CNN architecture, improving segmentation against U-Net and offering comparable or better accuracy than DeepLabv3+ with a much smaller model.
Problem
Medical image segmentation remains challenging for CNNs when targets vary in position, shape, and scale, and existing networks have poor explainability for clinical use.
Method
CA-Net extends a U-Net backbone with specialized spatial, channel, and scale attention modules that jointly guide feature processing.
Results
CA-Net improved Dice from 87.77% to 92.08% for skin lesions and from 84.79% to 87.08% for placenta segmentation versus U-Net, while using around 15 times fewer parameters than DeepLabv3+ with close accuracy.
Takeaways & Limitations
Visualized attention maps provide a single-forward-pass interpretation of spatial positions, feature channels, and scales used for segmentation.
Abstract
from arXiv · showhide
Accurate medical image segmentation is essential for diagnosis and treatment planning of diseases. Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they are still challenged by complicated conditions where the segmentation target has large variations of position, shape and scale, and existing CNNs have a poor explainability that limits their application to clinical decisions. In this work, we make extensive use of multiple attentions in a CNN architecture and propose a comprehensive attention-based CNN (CA-Net) for more accurate and explainable medical image segmentation that is aware of the most important spatial positions, channels and scales at the same time. In particular, we first propose a joint spatial attention module to make the network focus more on the foreground region. Then, a novel channel attention module is proposed to adaptively recalibrate channel-wise feature responses and highlight the most relevant feature channels. Also, we propose a scale attention module implicitly emphasizing the most salient feature maps among multiple scales so that the CNN is adaptive to the size of an object. Extensive experiments on skin lesion segmentation from ISIC 2018 and multi-class segmentation of fetal MRI found that our proposed CA-Net significantly improved the average segmentation Dice score from 87.77% to 92.08% for skin lesion, 84.79% to 87.08% for the placenta and 93.20% to 95.88% for the fetal brain respectively compared with U-Net. It reduced the model size to around 15 times smaller with close or even better accuracy compared with state-of-the-art DeepLabv3+. In addition, it has a much higher explainability than existing networks by visualizing the attention weight maps. Our code is available at https://github.com/HiLab-git/CA-Net
I. INTRODUCTION
Medical image segmentation supports pathology assessment, treatment planning, and disease monitoring but remains difficult because targets vary in appearance, position, shape, and scale. CA-Net addresses these challenges by combining spatial, channel, and scale attentions while improving explainability.
- Medical image segmentation facilitates quantitative pathology assessment, treatment planning, and monitoring disease progression.
- Low contrast, inhomogeneous appearances, and variable target scales, shapes, positions, and orientations complicate accurate segmentation.
- Existing CNNs can lack spatial awareness because convolutional layers use shared weights across spatial positions.
- Attention mechanisms focus CNNs on relevant spatial regions, feature channels, and scales while suppressing irrelevant information.
- CA-Net comprehensively attends to spatial positions, channels, and scales through specialized building blocks for medical image segmentation.
- CA-Net was validated on skin-lesion segmentation and multi-class fetal-MRI segmentation, with attention maps used to visualize how the network works.
II. RELATED WORKS
Prior segmentation networks often provide limited interpretability and use position-invariant kernels, while existing medical-image attention methods typically cover only one or two attention types. CA-Net extends a U-Net backbone with simultaneous spatial, channel, and scale attention.
- CNNs for Image Segmentation: DeepLab, SegNet, and DeepLabv3+ established encoder-decoder frameworks for dense semantic segmentation, with DeepLabv3+ adding a decoder and depth-wise separable convolution.
- CNNs for Image Segmentation: Medical segmentation networks such as U-Net often use position-invariant kernels and provide little mechanism for interpreting decisions.
- Attention Mechanism: Existing attention methods include spatial attention, channel-wise Squeeze-and-Excitation, and attention over feature maps with different receptive fields or sizes.
- Attention Mechanism: Most medical-image attention methods focus on only one or two mechanisms and seldom attend to feature maps at different scales.
- Comprehensive-Attention CNN: CA-Net adds specialized blocks to a U-Net backbone for simultaneous attention to feature-map space, channels, and scales.
- Comprehensive-Attention CNN: Its four spatial modules strengthen regions of interest, four channel modules recalibrate feature channels, and one scale module highlights relevant scales.
1) Joint Spatial Attention Modules:
The joint spatial attention design combines a non-local block at the lowest resolution with dual-pathway attention blocks at higher resolutions. It uses multi-scale spatial interactions to emphasize relevant regions while limiting noise and memory costs.
- Joint spatial attention: The lowest-resolution SA1 block uses non-local attention to capture interactions between all pixels and improve global contextual awareness.Input features are compressed with parallel 1×1 convolutions before generating spatial attention coefficients.
- Joint spatial attention: SA1 expands the calibrated reduced feature map back to the input channel dimension and adds a residual connection for information propagation during training.A 1×1 convolution with batch normalization produces 256 output channels before the residual addition.
- Joint spatial attention: Figure 2 organizes the spatial modules as non-local SA1 at the lowest resolution, a single-pathway block, and dual-pathway SA2−4 at higher resolutions.The high-level query feature xh calibrates the low-level key feature xl, and δ denotes the Sigmoid function.
- Joint spatial attention: At higher resolutions, SA2−4 replace non-local attention with dual-pathway attention because non-local blocks have increased memory consumption on larger feature maps.The dual-pathway design uses two attention gates in parallel to strengthen region-of-interest attention and reduce map noise.
- Joint spatial attention: The higher-resolution attention outputs use channel counts C=64, 32, and 16 for SA2, SA3, and SA4, respectively.
2) Channel Attention Modules:
The channel attention modules recalibrate concatenated encoder–decoder features by emphasizing channels most relevant to segmentation. They augment average-pooled channel information with max-pooled features and use residual connections.
- Channel attention: Channel attention is applied after concatenating encoder low-level features with decoder semantic features, whose channels may have different importance for segmentation.
- Channel attention: The proposed channel attention module uses both global average pooling and global max pooling to retain more channel information than average pooling alone.The pooled outputs represent one global descriptor per input channel before further channel recalibration.
- Channel attention: Four channel attention modules, CA1−4, are used, with one module for each concatenated feature.Residual connections are used to benefit training.
3) Scale Attention Module:
The scale attention module combines decoder features from multiple resolutions and learns image-specific, pixel-aware scale weights. This allows the network to select scale information according to object size.
- Scale attention: The scale attention module learns image-specific weights for each scale to calibrate features at different scales before final prediction.Its purpose is to adapt the CNN to the most suitable scale for a given input object.
- Scale attention: Decoder feature maps at scales s=1, 2, 3, 4 are bilinearly interpolated to the original image size, compressed to four channels, and concatenated.
- Scale attention: The module combines average-pooled and max-pooled information with an MLP to obtain a coefficient for each scale.The resulting scale coefficient attention vector is denoted by γ.
- Scale attention: The scale-attention output uses residual connections and combines scale-wise coefficient γ with pixel-wise coefficient γ∗.The pixel-wise coefficient is obtained through an additional spatial attention block LA∗.
- Evaluation: The framework was validated on binary skin-lesion segmentation and multi-class fetal-MRI segmentation of the fetal brain and placenta.Both applications included ablation studies and comparisons with state-of-the-art networks.
- Evaluation: Training used Adam with initial learning rate 10^-4, weight decay 10^-8, batch size 16, and 300 epochs.The learning rate was decayed by 0.5 every 256 epochs, and final evaluation used 5-fold cross-validation.
- Evaluation: Segmentation accuracy was evaluated using Dice score and average symmetric surface distance (ASSD).Dice compares segmented and ground-truth regions, while ASSD evaluates distances between their boundary-point sets.
B. Lesion Segmentation from Dermoscopic Images
CA-Net addresses skin-lesion variability in scale, shape, and color using spatial attention, with experiments showing improved segmentation and more focused attention maps. The joint spatial attention performs best among compared spatial variants, while annotation boundaries can be subjective.
- Skin lesions have varied scales, shapes, and colors, making robust automatic boundary segmentation challenging.
- The study evaluates spatial-attention variants on 2,594 ISIC 2018 images split into training, validation, and testing sets.Images were resized to 256 × 342 and augmented through cropping, flipping, and rotation.
- All spatial-attention variants improve over U-Net, while joint spatial attention raises average Dice from 88.46% to 90.83% versus standard AG.
- Joint spatial attention concentrates more closely on the lesion than single- and dual-pathway attention, whose maps are respectively dispersive and insufficiently adaptive.
- Joint spatial attention produces better qualitative results, including for lesions with complex shapes and blurry boundaries.
- ISIC 2018 annotations may diverge from non-expert perception because low contrast makes exact lesion boundaries require expertise.
3) Comparison of Channel Attention Methods:
Channel and scale attention are evaluated through placement, pooling, and multi-scale feature-fusion variants. Decoder channel attention and four-scale fusion provide the strongest reported configurations, while attention coefficients vary with lesion size.
- Channel attention: Channel-attention blocks are tested in the encoder, decoder, or both, with and without additional max pooling.
- Channel attention: 91.68% average Dice is achieved by C-A (Dec), outperforming the other channel-attention variants.
- Channel attention: Additional max-pooled information generally improves channel attention, and decoder placement performs better than encoder or dual placement.
- Scale attention: Multi-scale scale attention outperforms the baseline, with scales 1–4 achieving 91.58% Dice and 0.66 pixels ASSD.
- Scale attention: Using all five scales decreases accuracy, suggesting the lowest-resolution feature maps are unsuitable for detailed pixel-wise prediction.
- Scale attention: Scale attention shifts toward scale 2 for a large lesion and scale 1 for a small lesion, indicating size-dependent feature emphasis.
5) Comparison of Partial and Comprehensive Attention:
The partial-attention comparison shows that spatial, channel, and scale modules each improve U-Net, combinations improve further, and CA-Net performs best overall on skin-lesion segmentation.
- Each spatial, channel, and scale attention module improves performance over baseline U-Net.
- Combining two attention types outperforms methods using a single attention type.
- CA-Net achieves the best Dice score and ASSD among the compared variants, reaching 92.08% and 0.58 pixels, respectively.
6) Comparison with the State-of-the-Art Frameworks:
CA-Net is compared with established segmentation frameworks on skin lesions and with spatial-attention variants for fetal MRI organs. It combines competitive accuracy with a smaller model and explainability advantages on skin lesions.
- Skin lesion segmentation: The skin-lesion comparison includes DenseASPP, RefineNet, and DeepLabv3+ variants using retrained, non-pretrained networks.
- Skin lesion segmentation: 92.08% Dice is obtained by CA-Net versus 87.77% for U-Net on skin-lesion segmentation.
- Skin lesion segmentation: CA-Net has slightly lower but not significantly different performance from DeepLabv3+ (p-value=0.46 > 0.05), while using a model size 15 times smaller and offering better explainability.
- Skin lesion segmentation: CA-Net inference takes 2.1ms per skin-lesion image, compared with 4.0ms for DeepLabv3+ and 3.4ms for RefineNet.
- Fetal MRI segmentation: Spatial-attention variants improve Dice and ASSD for placenta and fetal-brain segmentation.
- Fetal MRI segmentation: Fetal MRI experiments jointly segment the placenta and fetal brain, using multi-view T2-weighted scans and patient-level data splits.
2) Comparison of Spatial Attention Methods:
Spatial-attention variants improved fetal MRI segmentation, with the proposed joint spatial attention achieving the strongest reported scores. Multi-scale feature combinations also improved performance, with scales 1–4 performing best in the reported comparison.
- Spatial attention: All evaluated spatial-attention variants improved Dice and ASSD over the baseline.
- Spatial attention: 95.47% Dice and 0.30 pixels ASSD were achieved by the proposed Js-A module, the highest scores among the spatial-attention variants.
- Spatial attention: 85.65% Dice and 0.58 pixels ASSD were maintained for placenta segmentation despite its fuzzy tissue boundary.
- Scale attention: Combining features from scales 1 to 4 produced the best multi-scale results, with Dice values of 86.21% for placenta and 95.18% for fetal brain.
- Scale attention: Adding the lowest-resolution feature scale reduced Dice scores slightly, while visualizations showed attention shifting toward finer scales for smaller fetal brains.
5) Comparison of Partial and Comprehensive Attention:
Combining spatial, channel and scale attention generally outperformed single-attention variants, while CA-Net achieved the strongest fetal MRI results across the compared views and offered visualized attention-based explanations.
- Partial versus comprehensive attention: Models combining two attention mechanisms generally outperformed models using a single mechanism.
- Partial versus comprehensive attention: SA + CA achieved Dice scores of 86.68% for placenta and 95.42% for fetal brain among binary-attention methods.
- Partial versus comprehensive attention: CA-Net achieved Dice scores of 87.08% for placenta and 95.88% for fetal brain, with lower ASSD than the other methods.
- State-of-the-art comparison: Across axial, sagittal and coronal views, CA-Net improved Dice over U-Net for placenta by 2.35%, 1.78% and 2.60%, respectively.For fetal brain, the corresponding improvements were 3.75%, 0.85% and 2.84%.
- Explainability: CA-Net’s attention maps help interpret its decisions, unlike the compared DeepLabv3+, RefineNet and DenseASPP networks.
V. DISCUSSION AND CONCLUSION
CA-Net integrates spatial, channel and scale attention to improve medical image segmentation while targeting efficiency and explainability. Experiments report competitive or better accuracy than comparison networks, substantially fewer parameters, and attention maps obtained during the forward pass.
- Explainability: The model provides comprehensive interpretation of spatial, channel and scale usage in a single forward pass without additional computations.
- Experimental scope: Across RGB skin-lesion images and fetal MRI, CA-Net showed considerable segmentation improvement compared with U-Net.
- Future directions: The authors identify extension to other modalities, anatomies and 3D images as future directions.
- Efficiency and accuracy: CA-Net achieved comparable or higher accuracy than DeepLabv3+ with around 15 times fewer parameters.
- Contribution: CA-Net simultaneously attends to spatial positions, feature channels and object scales for segmentation.