EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation

Md Mostafijur Rahman; Mustafa Munir; Radu Marculescu

Source-linked AI summary

EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation

Md Mostafijur Rahman, Mustafa Munir, Radu Marculescu

arXiv:2405.06880v1May 11, 2024eess.IVcs.CV

TL;DR

Medical image segmentation decoders can achieve strong results but remain computationally expensive, motivating more efficient designs. EMCAD uses multi-scale depth-wise convolution and attention-based decoding, achieving strong performance across 12 datasets while reducing decoder cost relative to CASCADE.

Problem
Attention-based medical image segmentation models can improve feature refinement, but their convolutional decoder blocks are often computationally expensive.
Method
EMCAD combines multi-scale depth-wise convolutions with channel, spatial, and grouped large-kernel gated attention to refine and fuse hierarchical encoder features.
Results
EMCAD achieves superior performance compared with SOTA methods across 12 public datasets while using 79.4% fewer parameters and 80.3% less FLOPs than CASCADE.
Takeaways & Limitations
EMCAD is adaptable to hierarchical encoders and supports efficient medical image segmentation across multiple segmentation tasks.
Takeaways & Limitations
CASCADE remains computationally inefficient because it uses triple 3×3 convolution layers at each decoder stage and single-scale convolutions during decoding.

Abstract

from arXiv · show

An efficient and effective decoding mechanism is crucial in medical image segmentation, especially in scenarios with limited computational resources. However, these decoding mechanisms usually come with high computational costs. To address this concern, we introduce EMCAD, a new efficient multi-scale convolutional attention decoder, designed to optimize both performance and computational efficiency. EMCAD leverages a unique multi-scale depth-wise convolution block, significantly enhancing feature maps through multi-scale convolutions. EMCAD also employs channel, spatial, and grouped (large-kernel) gated attention mechanisms, which are highly effective at capturing intricate spatial relationships while focusing on salient regions. By employing group and depth-wise convolution, EMCAD is very efficient and scales well (e.g., only 1.91M parameters and 0.381G FLOPs are needed when using a standard encoder). Our rigorous evaluations across 12 datasets that belong to six medical image segmentation tasks reveal that EMCAD achieves state-of-the-art (SOTA) performance with 79.4% and 80.3% reduction in #Params and #FLOPs, respectively. Moreover, EMCAD's adaptability to different encoders and versatility across segmentation tasks further establish EMCAD as a promising tool, advancing the field towards more efficient and accurate medical image analysis. Our implementation is available at https://github.com/SLDGroup/EMCAD.

1. Introduction

Medical image segmentation benefits from attention and hierarchical encoders, but existing attention-based decoders can remain computationally expensive. EMCAD addresses this gap with efficient multi-scale convolutional attention while improving segmentation performance at low cost.

Attention-based segmentation models improve feature maps and pixel classification but often rely on computationally expensive convolutional blocks.
Vision transformers capture long-range dependencies, yet their self-attention is less effective at local spatial context and convolutional decoder components can be costly.
EMCAD combines multi-scale depth-wise convolutions with channel, spatial, and grouped gated attention to refine multi-scale spatial representations efficiently.
Across 10 binary medical image segmentation datasets, PVT-EMCAD-B0 and PVT-EMCAD-B2 achieve the lowest #FLOPs and highest DICE scores.
1.91M parameters and 0.381G FLOPs are required by EMCAD with a standard encoder using channels [64, 128, 320, 512].

2. Related Work

Related work spans convolutional, transformer, U-shaped, and attention-based architectures for medical image segmentation. These approaches improve multi-scale, local, or global feature modeling but leave computational efficiency as an important concern.

CNN encoders progressively extract spatial features, while architectures such as Inception and MobileNet improve multi-scale or lightweight computation.
Vision transformers model long-range pixel relationships using self-attention, with Swin, SegFormer, PVT, and PVTv2 extending transformer-based designs.
U-shaped networks use encoder-decoder structures and skip connections to fuse features across resolutions for medical image segmentation.
Transformer-based segmentation models combine CNNs and transformers or use hierarchical multi-scale designs to capture local and global information.
Attention-based decoders refine features at multiple stages, but CASCADE is computationally inefficient because it uses triple 3×3 convolutions and single-scale decoding.

3. Methodology

EMCAD processes hierarchical encoder features through efficient multi-scale attention, gated skip-connection fusion, upsampling, and segmentation heads. Its depth-wise and grouped convolutions target multi-scale contextual refinement with reduced computation.

EMCAD decoder: EMCAD combines MSCAMs, LGAGs, EUCBs, and segmentation heads to refine, fuse, upsample, and decode multi-stage encoder features.
EMCAD decoder: Four MSCAMs refine encoder features, intermediate heads generate stage maps, EUCBs upsample features, and four segmentation maps are summed for final output.
Large-kernel grouped attention gate (LGAG): LGAG applies separate 3×3 group convolutions to gating and upsampled features before learning attention coefficients for feature fusion.
Large-kernel grouped attention gate (LGAG): 3×3 group convolutions allow LGAG to capture comparatively larger spatial contexts with less computational cost.
Multi-scale convolutional attention module (MSCAM): MSCAM sequentially applies channel attention, spatial attention, and multi-scale convolution to emphasize relevant channels, local context, and contextual relationships.
Multi-scale convolutional attention module (MSCAM): Depth-wise convolutions at multiple scales make MSCAM more effective with significantly lower computational cost than the CAM module.
Multi-scale convolution block (MSCB): MSCB expands channels, applies parallel or sequential multi-scale depth-wise convolutions, shuffles channels, and projects features back to the original width.
Attention blocks: CAB uses pooled feature statistics and learned channel weights, while SAB pools maximum and average spatial responses to focus attention spatially.

4. Experiments

Experiments compare PVT-EMCAD models with CNN- and transformer-based SOTA methods across 12 datasets spanning six medical image segmentation tasks. EMCAD achieves strong segmentation performance across binary, abdominal, and cardiac settings while retaining low computational cost.

Experimental setup: The evaluation compares PVT-EMCAD-B0 and PVT-EMCAD-B2 with SOTA CNN and transformer-based methods across 12 datasets covering six segmentation tasks.Experiments use ImageNet-pretrained PVTv2-b0 and PVTv2-b2 encoders with task-specific image resizing and augmentation settings.
Binary segmentation: 91.10% average DICE, 26.76M parameters, and 5.6G FLOPs are achieved by PVT-EMCAD-B2 across 10 binary medical image segmentation datasets.The reported binary tasks include polyp, skin lesion, cell, and breast cancer segmentation.
Binary segmentation: PVT-EMCAD-B2 surpasses all SOTA methods across five polyp datasets, while the 3.92M-parameter, 0.84G-FLOP model also generally outperforms competing methods.Exceptions for the smaller model are PVT-CASCADE on Kvasir and BKAI-IGH and SSFormer-L on ColonDB.
Ablation studies: Ablations on Synapse show that the cascaded decoder, LGAG, and MSCAM each improve performance, with their combination producing the best DICE score of 83.63%.The experiments isolate decoder components and compare their effects using the PVTv2-b2 encoder.
Binary segmentation: 80.25% DICE on BUSI establishes the breast-cancer segmentation SOTA, while PVT-EMCAD-B0 exceeds computationally similar UNeXt by 5.54%.On DSB18, PVT-EMCAD-B2 sets a SOTA cell-nuclei result; on EM, it obtains 95.53% DICE, close to AttnUNet's 95.55% at lower computational cost.
Multi-organ and cardiac segmentation: 83.63% average DICE on Synapse and 92.12% on ACDC are the highest reported averages, with PVT-EMCAD-B2 surpassing competing methods in abdominal and cardiac segmentation.On Synapse, it improves over PVT-CASCADE by 2.57% DICE and 4.55 HD95; on ACDC, it improves over Cascaded MERIT by about 0.27%.

5. Ablation Studies

The ablations show that cascaded decoding, LGAG, and MSCAM improve performance, with MSCAM providing the strongest component-level benefit. A 5×5 kernel combined with 1×1 and 3×3 kernels gives the best multi-scale convolution results, while EMCAD substantially reduces complexity versus CASCADE.

Component ablations: Cascaded decoding improves performance over a non-cascaded decoder, while LGAG and MSCAM further improve results.MSCAM is more effective than LGAG in the reported component ablation.
Multi-scale kernels: 5×5 combined with 1×1 and 3×3 kernels achieves the best results on both Synapse multi-organ and ClinicDB datasets.Adding 7×7 or 9×9 kernels reduces performance.
Computational complexity: 80.3% fewer FLOPs and 79.4% fewer parameters let EMCAD with PVTv2-b2 outperform CASCADE by 0.85%.With PVTv2-B0, EMCAD achieves a 1.43% better DICE score with 78.1% fewer parameters and 74.9% fewer FLOPs.

6. Conclusions

EMCAD is an efficient multi-scale convolutional attention decoder for multi-stage feature aggregation and refinement in medical image segmentation. Across 12 public datasets spanning six 2D tasks, it surpasses CASCADE and recent SOTA methods while retaining compatibility with smaller encoders.

Conclusions: EMCAD uses multi-scale depth-wise convolution to capture diverse scale information while reducing computational cost relative to standard 3×3 convolution blocks.Its decoder is designed for multi-stage feature aggregation and refinement.
Conclusions: 79.4% fewer parameters and 80.3% fewer FLOPs accompany higher DICE scores than the CASCADE decoder.The comparison is reported as an overall conclusion for EMCAD versus CASCADE.
Conclusions: EMCAD achieves superior performance to SOTA methods across 12 public datasets covering six different 2D medical image segmentation tasks.The conclusion also identifies compatibility with smaller encoders as suitable for point-of-care applications.

7. Experimental Details

The experiments evaluate EMCAD across 12 datasets from six medical image segmentation tasks using DICE throughout, with HD95 and mIoU additionally used for Synapse multi-organ segmentation. Qualitative comparisons assess organ and polyp masks against existing methods and ground truth.

Datasets: Experiments cover 12 datasets belonging to six medical image segmentation tasks, including polyp, cardiac organ, skin lesion, and breast cancer segmentation.The listed polyp datasets are Kvasir, ClinicDB, ColonDB, ETIS, and BKAI; ACDC is used for cardiac organ segmentation.
Experimental setup: An 80:10:10 train-validation-test split is used for nine datasets, while ISIC17 uses the competition organizer’s official splits.The split policy is specified for ClinicDB, Kvasir, ColonDB, ETIS, BKAI, ISIC18, DSB18, EM, and BUSI.
Evaluation metrics: DICE evaluates all datasets, while Synapse multi-organ additionally uses HD95 and mIoU.The evaluation section defines segmentation maps Y and P as ground truth and predicted maps, respectively.
Qualitative results: On Synapse, PVT-EMCAD-B0 and PVT-EMCAD-B2 segment difficult kidney and pancreas regions more accurately than most methods at lower computational cost.For ClinicDB polyp segmentation, both models show strong overlap with ground truth while existing SOTA methods produce false polyp segmentation.

8. Additional Ablation Study

Additional ablations examine convolution ordering, attention gates, transfer learning, deep supervision, and input resolution. They favor parallel multi-scale convolutions, LGAG, ImageNet pretraining, deep supervision, and higher input resolutions, with higher resolution increasing FLOPs.

Convolution arrangement: Parallel multi-scale depth-wise convolutions provide a slight 0.03% to 0.15% performance improvement over sequential convolutions.Sequential convolutions also show higher standard deviations, so parallel convolutions are used in experiments.
Attention gates: LGAG improves DICE while reducing parameters by 82.57% for PVT-EMCAD-B0 and 91.17% for PVT-EMCAD-B2 versus AG.It also reduces FLOPs by 67.06% and 83.03%, respectively, with larger reductions for larger models.
Transfer learning: ImageNet-pretrained PVT-v2 encoders improve PVT-EMCAD-B0 DICE by 4.5%, mIoU by 5.92%, and HD95 by 2.54.Transfer learning has a comparatively greater impact on the smaller B0 model and improves all listed organs except the gallbladder.
Deep supervision: Deep supervision produces slightly better DICE scores on six of seven datasets, with its largest impact on Synapse multi-organ segmentation.The result comes from the PVT-EMCAD-B2 ablation.
Input resolution: Higher input resolution improves DICE but increases FLOPs; PVT-EMCAD-B0 reaches 85.52% DICE at 3.36G FLOPs with 512×512 inputs.PVT-EMCAD-B2 reaches 86.53% DICE at 22.39G FLOPs under the reported 512×512 setting.