DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

Guanqun Sun; Yizhi Pan; Weikun Kong; Zichang Xu; Jianhua Ma; Teeradaj Racharak; Le-Minh Nguyen; Junyi Xin

Source-linked AI summary

DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

Guanqun Sun, Yizhi Pan, Weikun Kong, Zichang Xu, Jianhua Ma, Teeradaj Racharak, Le-Minh Nguyen, Junyi Xin

arXiv:2310.12570v2Oct 19, 2023eess.IVcs.CVcs.GRcs.LG

TL;DR

Medical image segmentation needs accurate automated delineation, but existing Transformer–U-Net models do not explicitly capture image-specific positional and channel features and can incur high complexity. DA-TransUNet adds dual-attention blocks before the Transformer and in skip connections, achieving reported state-of-the-art performance across multiple datasets. The authors identify added computational complexity and an unoptimized decoder as limitations.

Problem
Existing Transformer–U-Net models lack built-in mechanisms for image-specific positional and channel features, while extensive Transformer use can increase parameters and computational complexity.
Method
DA-TransUNet integrates DA-Blocks before the Transformer and within skip connections to extract positional and channel features and refine features passed to the decoder.
Results
DA-TransUNet achieves state-of-the-art performance across multiple medical imaging datasets, with improvements reported across various datasets and particularly on Synapse.
Takeaways & Limitations
Integrating dual attention with the Transformer and skip connections provides an effective alternative for medical image segmentation within the reported evaluation scope.
Takeaways & Limitations
DA-Blocks increase computational complexity, and the decoder retains the original U-Net design rather than being specifically optimized for this application.

Abstract

from arXiv · show

Accurate medical image segmentation is critical for disease quantification and treatment evaluation. While traditional Unet architectures and their transformer-integrated variants excel in automated segmentation tasks. However, they lack the ability to harness the intrinsic position and channel features of image. Existing models also struggle with parameter efficiency and computational complexity, often due to the extensive use of Transformers. To address these issues, this study proposes a novel deep medical image segmentation framework, called DA-TransUNet, aiming to integrate the Transformer and dual attention block(DA-Block) into the traditional U-shaped architecture. Unlike earlier transformer-based U-net models, DA-TransUNet utilizes Transformers and DA-Block to integrate not only global and local features, but also image-specific positional and channel features, improving the performance of medical image segmentation. By incorporating a DA-Block at the embedding layer and within each skip connection layer, we substantially enhance feature extraction capabilities and improve the efficiency of the encoder-decoder structure. DA-TransUNet demonstrates superior performance in medical image segmentation tasks, consistently outperforming state-of-the-art techniques across multiple datasets. In summary, DA-TransUNet offers a significant advancement in medical image segmentation, providing an effective and powerful alternative to existing techniques. Our architecture stands out for its ability to improve segmentation accuracy, thereby advancing the field of automated medical image diagnostics. The codes and parameters of our model will be publicly available at https://github.com/SUN-1024/DA-TransUnet.

1 Introduction

DA-TransUNet addresses limitations in transformer-based U-Net segmentation by extracting image-specific positional and channel features while improving skip connections. The proposed architecture targets accurate, efficient segmentation and reports state-of-the-art performance across multiple medical imaging datasets.

Motivation: Automated medical image segmentation supports diagnosis, disease quantification, prognosis assessment, treatment evaluation, and more consistent clinical workflows.It can improve efficiency while reducing labor and costs without sacrificing accuracy.
Research gap: Transformer-integrated U-Net models lack built-in mechanisms for image-specific positional and channel features, while extensive Transformer stacking can increase parameters and computational complexity.These limitations motivate further investigation of Transformer–U-Net integration.
Proposed approach: DA-TransUNet integrates dual attention mechanisms into a Transformer U-Net to process positional and channel information.The DA-Block is designed to extract image-specific features and improve parameter efficiency and segmentation performance.
Architecture: DA-Blocks are placed before the Transformer in the encoder and within each skip-connection layer to refine extracted and transmitted features.The encoder placement enriches feature extraction, while skip-connection placement improves feature delivery to the decoder; both designs are supported by ablation studies.
Results: DA-TransUNet achieves state-of-the-art performance across multiple medical imaging datasets.The study evaluates Synapse, CVC-ClinicDB, ISIC2018, kvasir-seg, Kvasir-Instrument, and Chest X-ray datasets.

2 Related Work

Related work combines attention, U-Net skip-connection modifications, and Transformers to improve medical image segmentation. DA-TransUNet follows these directions by applying dual attention within skip connections and the broader U-Net–Transformer design.

Attention-based U-Nets: Attention U-Net, DAResUnet, Attention Res-UNet, and SA-UNet apply attention mechanisms to improve localization, segmentation, or feature processing in U-Net architectures.These methods use different attention or residual designs, including spatial and soft-attention mechanisms.
Skip connections: U-Net skip connections bridge encoder–decoder semantic gaps and help recover fine-grained object details.Prior modifications increase skip-connection complexity, add attention-aware extraction, or combine encoder and decoder feature maps.
Architecture overview: The proposed architecture combines an encoder, decoder, and skip connections, with Transformer and DA-Block components in the encoder.The figure describes three feature scales being purified before fusion with the decoder and CNN-based up-sampling.
DA-TransUNet: DA-TransUNet integrates Dual Attention Blocks into each skip-connection layer to enhance decoder feature extraction and segmentation accuracy.This follows prior skip-connection enhancement strategies while focusing on positional and channel features.

3 Method

DA-TransUNet combines convolutional and Transformer components with dual attention to extract global, positional, and channel features. DA-Blocks refine encoder features and skip connections by fusing position- and channel-aware representations.

DA-Block: The DA-Block processes the same feature map through position and channel extraction branches, then fuses their outputs.Its two attention components are dominated by PAM and CAM, respectively.
Position Attention: PAM computes spatial dependencies between feature-map positions and updates each position using a weighted sum of position features.The spatial attention map is derived from feature similarity, and the output combines attended features with the original features.
Channel Attention: CAM computes dependencies between channels and produces each channel feature as a weighted sum of all channels and the original features.The channel attention map has dimensions R^C×C, and its learned scale parameter β controls the residual aggregation.
Architecture: DA-TransUNet combines a CNN, Transformer layer, DA-Block, embedding layers, and three-layer skip connections within a U-shaped architecture.The decoder primarily uses conventional convolutional mechanisms, while DA-Blocks are inserted before Transformer processing and in skip connections.
Feature Integration: DA-Blocks before Transformer layers and within skip connections combine image-specific spatial and channel characteristics with global contextual features.In skip connections, they filter redundant or irrelevant features to improve information delivered to the decoder.

4 Experiments

DA-TransUNet was evaluated on six medical image segmentation datasets, and the reported experiments found that it outperformed existing methods across all six.

Overall Results: DA-TransUNet outperformed existing methods across all six evaluated datasets.The evaluation covered Synapse, CVC-ClinicDB, Chest X-ray, Kvasir SEG, Kvasir-Instrument, and ISIC 2018.

4.1 Datasets

The experiments use datasets spanning abdominal CT, colonoscopy polyps, chest X-rays, gastrointestinal images and instruments, and dermatoscopic skin lesions.

Synapse: Synapse contains 30 scans covering eight abdominal organs and 3779 axial contrast-enhanced clinical CT images.The organs include the kidneys, aorta, spleen, gallbladder, liver, stomach, and pancreas.
CVC-ClinicDB: CVC-ClinicDB contains 612 colonoscopy still images from 29 sequences, each with a manually annotated polyp ground truth.The frames come from colonoscopy videos in the Endoscopic Vision Challenge.
Chest X-ray: The Chest X-ray dataset provides de-identified DICOM radiographs with corresponding masks and labels, including normal and tuberculosis-abnormal cases.The passages report 138 posterior-anterior radiographs, including 80 normal and 58 tuberculosis-abnormal images.
Kvasir SEG: Kvasir SEG contains 1000 gastrointestinal polyp images with corresponding masks manually annotated and verified by an experienced gastroenterologist.Image resolutions range from 332x487 to 1920x1072 pixels.
Kvasir-Instrument: Kvasir-Instrument contains 590 annotated endoscopic tool images with ground-truth masks.The tools include snares, balloons, and biopsy forceps.
ISIC 2018: The 2018 ISIC dataset contains 2512 JPG lesion images collected using dermatoscopic techniques from multiple institutions.Each image contains one primary lesion, excluding mucous membranes and nails.

4.2 Implementation Settings

The study evaluates DA-TransUNet against established segmentation models using multiple datasets, metrics, and training configurations. Experiments cover Synapse and five additional datasets, with Dice, IoU, and Hausdorff Distance used to assess performance.

Baselines: Experiments benchmark DA-TransUNet against U-Net, UNet++, DA-Unet, Attention U-Net, and TransUNet.Additional comparisons include Res-Unet, TransNorm, UCTransNet, MultiResUNet, Swin-Unet, and MIM.
Training settings: Training on five datasets uses an end-to-end objective combining weighted binary cross-entropy and Dice coefficient losses.The final objective is referred to as Loss.
Training settings: Synapse experiments use the pre-trained R50-ViT model with 224x224 inputs, patch size 16, SGD optimization, and batch size 24.The stated SGD settings include a learning rate of 0.01, momentum of 0.9, and weight decay of 1e-4.
Data split: Most datasets use a 3:1 split, with 75% for training and 25% for testing.The split is described as supporting adequate training.
Evaluation metrics: Performance is assessed with IoU, Dice Coefficient (DSC), and Hausdorff Distance (HD), using Dice and HD for Synapse and Dice and IOU elsewhere.These metrics provide complementary assessments of segmentation overlap and boundary differences.

4.3 Comparison to the State-of-the-Art Methods

DA-TransUNet is compared with state-of-the-art and classical segmentation methods across Synapse and five additional datasets. The reported results show stronger overall performance, while qualitative comparisons and placement studies further examine the model and its DA-Blocks.

Baselines: DA-TransUNet is compared with U-net, Res-Unet, TransUNet, U-Net++, Att-Unet, TransNorm, UCTransNet, MultiResUNet, Swin-Unet, and MIM.The Synapse experiment compares the proposed model with 11 state-of-the-art models.
Synapse results: 79.80% average DSC and 23.48 mm average HD improve by 2.32% and 8.21 mm, respectively, versus TransUNet on Synapse.The paper reports improved overall segmentation and organ-edge prediction, while segmentation times are 35.98 ms for DA-TransUNet and 33.58 ms for TransUNet.
Synapse results: DA-TransUNet exceeds TransUNet on five organ-specific datasets, with reported gains for the gallbladder, right kidney, liver, spleen, stomach, and pancreas.The reported gains are 2.14%, 3.43%, 0.48%, 3.45%, 4.11%, and 5.73%, respectively.
Cross-dataset results: DA-TransUNet has higher IoU and Dice values than TransUNet on CVC-ClinicDB, Chest Xray Masks and Labels, ISIC2018-Task, Kvasir-Instrument, and Kvasir-Seg.It achieves the best dataset segmentation in four of the five datasets.
Qualitative results: Qualitative visualizations report better segmentation for DA-TransUNet than TransUNet and improvements over U-net, Unet++, Attn-Unet, and Res-Unet across five datasets.The visual comparisons span both 3D and 2D medical image segmentation.
Ablation studies: Tables 3 and 4 examine how DA-Block placement in the encoder and skip connections affects performance metrics.These studies evaluate combinatorial placement and different layer placements.

4.4 Ablation Study

The ablation study evaluates DA-Block placement in the encoder and skip connections, showing improved segmentation performance when these components are incorporated.

DA-Block placement: DA-Blocks were evaluated in the encoder and across skip-connection layers to measure their effects on segmentation performance.The study compares encoder placement with skip-connection placement and tests configurations across three skip-connection layers.
Skip connections: 78.28% DSC and 29.09mm HD were achieved after adding DA-Blocks to each skip-connection layer, compared with 77.48% and 31.69mm at baseline.The results indicate refined decoder features and reduced feature loss during upsampling.
Skip-connection configurations: 79.80% DSC and 23.48mm HD were achieved when DA-Blocks were integrated across all skip-connection layers.This configuration outperformed adding DA-Blocks to only the first layer, which produced 79.36% DSC and 25.80mm HD.

5 Discussion

The discussion attributes DA-TransUNet’s performance to placing DA-Blocks before the Transformer and within skip connections, while acknowledging increased computational complexity and an unoptimized decoder.

Overall performance: Encouraging results were consistently achieved across all six experimental datasets.The reported findings concern the integration of DA-Blocks with the Transformer and skip connections.
Encoder integration: DA-Blocks before the Transformer extract and filter image-specific positional and channel features before global feature extraction.The refined features are then processed by the Transformer to improve feature learning and segmentation performance.
Skip connections: DA-Blocks in skip connections improve semantic continuity and the decoder’s reconstruction of accurate feature maps.They focus on relevant features and filter extraneous information during image reconstruction.
Limitations: DA-Blocks increase computational complexity, while the decoder retains the original U-Net architecture rather than being specifically optimized.The authors identify these factors as limitations for real-time or resource-constrained applications and future decoder research.

6 Conclusion

The paper concludes that integrating DA-Blocks with the Transformer and skip connections improves feature extraction and segmentation performance across datasets, while leaving decoder optimization and computational efficiency for future work.

Conclusion: DA-TransUNet integrates DA-Blocks with the Transformer and skip connections to focus on image-specific position and channel features.The approach was evaluated through an extensive ablation study across various datasets, particularly Synapse.
Conclusion: DA-Blocks enhance the Transformer’s feature extraction and global information retention without creating redundancy.The conclusion identifies this integration as a contributor to improved model performance.
Future work: Future work may optimize the decoder and reduce DA-Block computational complexity without compromising model performance.The authors present these as directions for extending the architecture.