Source-linked AI summary
Boundary-aware Transformers for Skin Lesion Segmentation
Jiacheng Wang, Lan Wei, Liansheng Wang, Qichao Zhou, Lei Zhu, Jing Qin
TL;DR
Automatic skin lesion segmentation is important for skin cancer analysis but is challenged by lesion variability, ambiguous boundaries, limited global context, and small training datasets. The proposed boundary-aware transformer combines transformer-based long-range modeling with a boundary-wise attention gate and auxiliary supervision. BAT consistently outperformed state-of-the-art methods across two datasets, including reported gains over TransUNet.
Problem
Skin lesion segmentation is difficult because lesions vary substantially, boundaries are ambiguous, and existing methods inadequately capture global context under limited training data.
Method
BAT integrates a boundary-wise attention gate into a transformer to use boundary prior knowledge, capture local details, and assist position-embedding learning through auxiliary supervision.
Results
0.920 Dice and 0.858 IoU on ISIC 2016 + PH2; BAT outperformed Lee et al by 0.2% Dice and 1.5% IoU, and improved over TransUNet by 1.8% Dice and 2.1% IoU.
Takeaways & Limitations
Experiments on two public datasets confirm BAT’s effectiveness, including improved segmentation of lesions with ambiguous boundaries.
Takeaways & Limitations
Transformer-based segmentation remains constrained by the small size of medical datasets, which makes position-embedding learning difficult.
Abstract
from arXiv · showhide
Skin lesion segmentation from dermoscopy images is of great importance for improving the quantitative analysis of skin cancer. However, the automatic segmentation of melanoma is a very challenging task owing to the large variation of melanoma and ambiguous boundaries of lesion areas. While convolutional neutral networks (CNNs) have achieved remarkable progress in this task, most of existing solutions are still incapable of effectively capturing global dependencies to counteract the inductive bias caused by limited receptive fields. Recently, transformers have been proposed as a promising tool for global context modeling by employing a powerful global attention mechanism, but one of their main shortcomings when applied to segmentation tasks is that they cannot effectively extract sufficient local details to tackle ambiguous boundaries. We propose a novel boundary-aware transformer (BAT) to comprehensively address the challenges of automatic skin lesion segmentation. Specifically, we integrate a new boundary-wise attention gate (BAG) into transformers to enable the whole network to not only effectively model global long-range dependencies via transformers but also, simultaneously, capture more local details by making full use of boundary-wise prior knowledge. Particularly, the auxiliary supervision of BAG is capable of assisting transformers to learn position embedding as it provides much spatial information. We conducted extensive experiments to evaluate the proposed BAT and experiments corroborate its effectiveness, consistently outperforming state-of-the-art methods in two famous datasets.
1 Introduction
Automatic skin lesion segmentation matters for skin cancer analysis but remains difficult because lesions vary widely, hair can occlude them, boundaries can be ambiguous, and training data are limited. BAT addresses these challenges by combining transformer-based global context modeling with boundary-wise prior knowledge and auxiliary supervision.
- Manual lesion delineation is tedious, time-consuming, and error-prone, motivating automated segmentation for clinical analysis.
- Lesion variation, hair occlusion, low contrast, ambiguous boundaries, and limited training data make automatic segmentation challenging.
- Existing solutions often fail to capture sufficient global context, while limited medical datasets hinder transformers’ position embedding and long-range interaction modeling.
- BAT integrates a boundary-wise attention gate into a transformer to identify ambiguous-boundary patches and guide feature refinement.The gate uses boundary supervision to provide spatial information and assist position-embedding learning on relatively small datasets.
2 Method
The method combines CNN locality with transformer long-range modeling, then adds boundary-aware attention and auxiliary supervision to refine ambiguous lesion boundaries.
- Basic Transformer for Segmentation: The basic transformer uses CNN features for image sequentialization, transformer layers for long-range context, and atrous prediction for multi-scale lesion segmentation.ResNet50 is the default CNN backbone; transformed features are reshaped for dense prediction.
- Basic Transformer for Segmentation: Image sequentialization flattens CNN feature patches into one-dimensional embeddings and adds learnable positional embeddings to compensate for lost spatial information.The input feature map is partitioned into 16 × 16 patches before flattening.
- Basic Transformer for Segmentation: Atrous prediction concatenates dilated convolutions with different receptive fields and projects them into segmentation-map space.The described dilation rates are 1, 3, and 6.
- Boundary-Aware Transformer: Boundary-wise attention gates are inserted after transformer encoder layers to generate patch-wise attention maps and preserve boundary information through residual attention.A query-embedding BAG after the encoder identifies ambiguous-boundary patches by comparing a learnable boundary prototype with patch embeddings.
- Boundary-Aware Transformer: Boundary supervision derives patch labels from edge-detected points scored by local lesion proportions, then maps filtered point locations to patch coordinates.The default circle radius is 10 and non-maximum suppression retains up to 30 neighboring points.
- Objective Function: Training combines Dice loss for segmentation with cross-entropy loss for predicted key-patch maps across the transformer encoder layers.The number of encoder layers is set to 4 by default.
3 Experimental Results
Experiments on ISBI 2016 and 2018 skin-lesion datasets show that BAT achieves strong segmentation performance, with boundary-aware guidance complementing transformer-based global context.
- Datasets: BAT was evaluated on ISBI 2016 and 2018 datasets collected from multiple treatment centers and archived by ISIC.ISIC 2016 contains 900 training samples and 379 testing images.
- Comparison with State-of-the-Arts: BAT achieves 0.920 Dice and 0.858 IoU on ISIC 2016 + PH2, outperforming Lee et al by 0.2% and 1.5%, respectively.The comparison includes five state-of-the-art methods.
- Comparison with State-of-the-Arts: BAT consistently and significantly improves both metrics over other state-of-the-art segmentation models.The reported comparisons include convolutional and transformer-based methods.
- Comparison with State-of-the-Arts: Compared with TransUNet, BAT improves Dice by 1.8% and IoU by 2.1% through combining transformer architecture with boundary-prior knowledge.The authors attribute the improvement to using boundary information alongside transformer-based modeling.
- Visual Comparison: Visualized challenging cases show BAT closest to ground truth across varied lesion color, size, shape, and ambiguous small boundaries.The authors associate stable performance on varied lesions with global context and ambiguous-boundary handling with boundary-wise prior knowledge.
- Ablation Study: Ablations show that self-attention substantially increases IoU, while BAG further improves performance on lesions with ambiguous boundaries.The findings support complementary contributions from global context and boundary-wise prior knowledge.
4 Conclusion
The paper concludes that BAT is an efficient context-aware network for skin-lesion segmentation. Experiments on two public datasets confirm its effectiveness and report strong performance on lesions with ambiguous boundaries.
- BAT is presented as an efficient context-aware network for accurate skin-lesion segmentation from dermoscopy images.
- Experiments on two public datasets confirm BAT’s effectiveness and report a large-margin advantage over state-of-the-art models.Visualizations also show satisfactory performance on lesions with ambiguous boundaries.