Source-linked AI summary
DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation
Feilong Tang, Qiming Huang, Jinfeng Wang, Xianxu Hou, Jionglong Su, Jingxin Liu
TL;DR
Transformer-based segmentation models capture global representations but can lose local details needed for small objects and boundaries. DuAT combines global-to-local spatial aggregation with selective boundary aggregation, and experiments report stronger performance across polyp and skin-lesion benchmarks, including challenging unseen datasets.
Problem
Transformer-based models can lose local details such as small objects and boundaries while modeling long-range dependencies for medical image segmentation.
Method
DuAT uses a pyramid vision transformer with GLSA to combine global and local features and SBA to enhance boundary information and recalibrate objects.
Results
DuAT advances state-of-the-art performance across polyp, skin-lesion, and Data Science Bowl datasets, outperforming baselines on unseen datasets and reaching 82.2% average Dice on ETIS.
Takeaways & Limitations
DuAT provides strong learning and generalization ability with lightweight segmentation efficiency across the evaluated medical-image tasks.
Abstract
from arXiv · showhide
Transformer-based models have been widely demonstrated to be successful in computer vision tasks by modelling long-range dependencies and capturing global representations. However, they are often dominated by features of large patterns leading to the loss of local details (e.g., boundaries and small objects), which are critical in medical image segmentation. To alleviate this problem, we propose a Dual-Aggregation Transformer Network called DuAT, which is characterized by two innovative designs, namely, the Global-to-Local Spatial Aggregation (GLSA) and Selective Boundary Aggregation (SBA) modules. The GLSA has the ability to aggregate and represent both global and local spatial features, which are beneficial for locating large and small objects, respectively. The SBA module is used to aggregate the boundary characteristic from low-level features and semantic information from high-level features for better preserving boundary details and locating the re-calibration objects. Extensive experiments in six benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods in the segmentation of skin lesion images, and polyps in colonoscopy images. In addition, our approach is more robust than existing methods in various challenging situations such as small object segmentation and ambiguous object boundaries.
1. Introduction
Medical image segmentation must preserve both global context and local details because small objects and ambiguous boundaries remain difficult. DuAT addresses these challenges with dual aggregation modules and reports improved performance across polyp and skin-lesion benchmarks.
- Motivation: Medical image segmentation extracts regions of interest such as tissues, lesions, and organs to support more efficient and precise diagnosis and treatment.Colonoscopy polyp localization and skin-lesion segmentation are highlighted as clinically relevant applications.
- Challenges: Small objects are difficult to segment because of low contrast and strong camouflage, while acquisition effects make medical-image boundaries ambiguous.
- Challenges: Transformer models capture long-range dependencies but can lose local features, producing overly smooth small-object predictions and blurred boundaries.
- DuAT: DuAT uses a pyramid vision transformer encoder to extract more robust features than existing CNN-based methods.
- DuAT: GLSA combines local spatial details with global semantic information, while SBA aggregates boundary and semantic information to fine-tune object boundaries.GLSA targets large and small object localization; SBA addresses ambiguous boundaries.
- Results: Extensive experiments across polyp, skin-lesion, and Data Science Bowl datasets report state-of-the-art performance for DuAT.
2. Related Work
Prior work established Transformers as effective global-context models for vision, while pyramid structures and boundary-aware methods addressed segmentation efficiency and detail preservation. DuAT builds on these directions with a new aggregation strategy for localizing objects and delineating boundaries.
- Vision Transformers: Vision Transformers use patch sequences, multi-head self-attention, and multilayer perceptrons to model long-distance feature dependencies.
- Pyramid Transformers: Pyramid structures make Transformers more suitable for downstream tasks, and PVT requires less computation than ViT for semantic segmentation.
- Medical Segmentation: Medical segmentation methods such as TransUNet, TransFuse, and Polyp-PVT use Transformers or hybrid designs to improve encoding, efficiency, or small-object localization.
- Boundary Modeling: Boundary-aware methods fuse low-level edge information with high-level semantic features or model boundary detection explicitly.
- DuAT: DuAT proposes a distinct aggregation method intended to improve object localization and boundary delineation.
3. Method
DuAT combines a pyramid vision transformer encoder with Selective Boundary Aggregation and Global-to-Local Spatial Aggregation to preserve boundary, semantic, global, and local information. The method evaluates these designs on seen and unseen medical-image segmentation datasets.
- Overall architecture: DuAT uses a pyramid vision transformer encoder, an SBA module, and a GLSA module.The encoder extracts multi-level feature maps, including detailed boundary information in F1 and higher-level semantic features in F2, F3, and F4.
- Selective Boundary Aggregation: SBA selectively combines shallow boundary details with deep semantic information before fusion.Its re-calibration units adaptively select mutual representations, and the outputs are concatenated after a 3 × 3 convolution.
- Global-to-Local Spatial Aggregation: GLSA splits feature channels between global spatial attention and local spatial attention, then concatenates their outputs through a 1 × 1 convolution.The dual-stream design preserves non-local modeling and local spatial information while reducing parameters through channel separation and depth-wise convolutions.
- Global-to-Local Spatial Aggregation: Global spatial attention models long-range pixel relationships, while local spatial attention extracts features from regions of interest such as small objects.The global attention uses attention operations and matrix multiplication; the local branch uses cascaded 1 × 1 and 3 × 3 depth-wise convolutions.
- Evaluation: Table 1 compares methods on Kvasir, ClinicDB, ISIC-2018, and 2018-DSB, while Table 2 evaluates generalization on ColonDB, ETIS, and EndoScene.Both tables mark whether higher or lower metric values are preferred.
- Loss function: The method uses binary cross-entropy and IoU losses with weighting coefficients to supervise two side outputs against the ground truth.The total loss is defined from the weighted loss terms used for supervision.
4. Experiments
Experiments evaluate DuAT across colonoscopy, dermoscopic, and microscopy datasets, measuring segmentation accuracy, generalization, efficiency, small-object handling, and component effectiveness. DuAT achieves strong benchmark performance, improves results on challenging unseen and small-object cases, and benefits from both GLSA and SBA.
- Datasets and metrics: DuAT is evaluated on five polyp datasets, ISIC-2018, and 2018-DSB using mDice, mIoU, and MAE.The experiments cover colonoscopy, dermoscopic, and microscopy image sets with standard segmentation metrics.
- Learning ability: DuAT achieves 0.924 and 0.948 mDice on Kvasir and ClinicDB, respectively, and improves ISIC-2018 over the SOTA method by 1.0% mDice and 1.5% mIoU.On 2018-DSB, it reports 0.87 mIoU, 0.926 mDice, and 0.027 MAE.
- Generalization capabilities: On unseen datasets, DuAT outperforms existing baselines across all metrics and reaches 82.2% average Dice on ETIS, 3.5% higher than Polyp-PVT.The unseen evaluation covers ETIS, ColonDB, and EndoScene, which present distinct dataset challenges.
- Visual results: Qualitative results show that DuAT captures boundaries and detects small polyps on ETIS, closely matches lesion ground truth on ISIC-2018, and better segments nuclei on 2018-DSB.These comparisons are reported across five benchmark visual evaluations.
- Computational efficiency: DuAT uses 24.92M parameters and 9.88G FLOPs, remaining more lightweight and compact than the compared CNN-based and Transformer-based methods.Its model size is similar to Polyp-PVT because both use the same backbone.
- Small-object segmentation: For polyp areas below 5% of an image, DuAT has higher overall accuracy than TransUnet and Polyp-PVT across the five polyp datasets.The analysis groups test samples by object-size intervals using mean Dice.
- Ablation study: Removing SBA reduces ETIS mDice from 0.822 to 0.814, while the complete model improves ISIC-2018 mDice and mIoU by 2.2% and 6.7% over the w/o GLSA variant.Adding both SBA and GLSA further improves performance, supporting the effectiveness of the dual-aggregation design.
- Ablation study: On ETIS, GLSA outperforms GSA, LSA, GSA + LSA (Serial), and LSA + GSA (Serial) by 4.2%, 2.9%, 2.8%, and 1.5% in mean Dice.All tested arrangements improve the baseline, while GLSA provides the most accurate and reliable results.
5. Conclusions
DuAT addresses medical image segmentation with GLSA and SBA modules that combine global, local, semantic, and boundary information. The model achieves strong segmentation performance, generalization, and lightweight efficiency, while future work will extend it to 3D medical image segmentation.
- DuAT introduces GLSA and SBA to address issues in medical image segmentation.GLSA extracts global and local spatial features, while SBA alleviates unclear high-level feature boundaries.
- GLSA extracts global and local spatial features that help locate large and small objects.
- SBA alleviates unclear high-level feature boundaries and further improves segmentation performance.
- DuAT achieves strong learning, generalization ability, and lightweight segmentation efficiency.
- Qualitative and quantitative results demonstrate DuAT's superiority over competing methods.
- Future work will extend DuAT to 3D medical image segmentation.