PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention

Ju-Hyeon Nam, Nur Suriza Syazwany, Su Jung Kim, Sang-Chul Lee

arXiv:2405.06284v1eess.IVcs.CVcs.LG

TL;DR

Medical image segmentation models can struggle to generalize across modalities and unseen clinical settings, while deep supervision can lose information during upsampling. MADGNet combines multi-frequency and multi-scale attention with ensemble sub-decoding, and the paper reports strong segmentation performance across six modalities and fifteen datasets.

  • Problem

    Medical image segmentation models often lack generalizability across modalities and unseen clinical settings, and deep-supervised multitask learning can lose information during upsampling.

    Source: §1 · p. 2

  • Method

    MADGNet combines MFMSA, which uses multi-frequency channel and multi-scale spatial attention, with E-SDM, which supplements core predictions with sub-task predictions.

    Source: §3.1 · p. 4, §3.2 · p. 5

  • Results

    Across fifteen datasets and six modalities, MADGNet achieved the highest segmentation performance on various datasets and clinical settings, including OD and OC binary and multi-label segmentation.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper concludes that multi-scale and multi-frequency features are pivotal for segmentation and that E-SDM compensates for upsampling-related information loss in deep-supervised multitask learning.

    Source: §5 · p. 9

    The conclusion identifies increasing memory efficiency for real clinical use as future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generalizability in deep neural networks plays a pivotal role in medical image segmentation. However, deep learning-based medical image analyses tend to overlook the importance of frequency variance, which is critical element for achieving a model that is both modality-agnostic and domain-generalizable. Additionally, various models fail to account for the potential information loss that can arise from multi-task learning under deep supervision, a factor that can impair the model representation ability. To address these challenges, we propose a Modality-agnostic Domain Generalizable Network (MADGNet) for medical image segmentation, which comprises two key components: a Multi-Frequency in Multi-Scale Attention (MFMSA) block and Ensemble Sub-Decoding Module (E-SDM). The MFMSA block refines the process of spatial feature extraction, particularly in capturing boundary features, by incorporating multi-frequency and multi-scale features, thereby offering informative cues for tissue outline and anatomical structures. Moreover, we propose E-SDM to mitigate information loss in multi-task learning with deep supervision, especially during substantial upsampling from low resolution. We evaluate the segmentation performance of MADGNet across six modalities and fifteen datasets. Through extensive experiments, we demonstrate that MADGNet consistently outperforms state-of-the-art models across various modalities, showcasing superior segmentation performance. This affirms MADGNet as a robust solution for medical image segmentation that excels in diverse imaging scenarios. Our MADGNet code is available in GitHub Link.

1. Introduction

The introduction frames medical image segmentation around generalization across modalities and unseen clinical settings, motivating the use of complementary frequency and scale features and ensemble deep supervision.

1. Introduction

  • Existing segmentation methods often lack generalizability to unseen clinical settings amid modality differences and limited patient-accessible datasets.Traditional algorithms also struggle with intensity variation, artifacts, and noise.Source: §1 · pp. 1–2
  • Frequency varies more than lesion scale across medical images, motivating the use of both dimensions as distinct, complementary cues.The figure defines scale by foreground-pixel proportion and frequency by the high-frequency to full-frequency power-spectrum ratio.Source: Figure 1 · p. 1, §1 · p. 2
  • MFMSA applies DCT-based multi-frequency channel attention to suppress noisy-channel influence, then multi-scale spatial attention to extract boundary features.It combines frequency-band information with multi-scale spatial features.Source: §1 · p. 2
  • MADGNet combines MFMSA and E-SDM and achieves the highest segmentation performance across datasets, modalities, and clinical settings.The paper evaluates it on fifteen datasets spanning six modalities and reports multi-label segmentation use.Source: §1 · p. 2, §2 · p. 3

2. Related Works

  • Prior work has often focused on multi-scale or multi-frequency information separately; this paper integrates both to detect subtle lesion variations.Related work describes multi-scale methods using varied receptive fields and frequency methods using transformations such as 2D DCT.Source: §2 · pp. 2–3
  • Upsampling low-resolution predictions during multi-task deep supervision can lose information, so E-SDM supplements the core prediction with sub-task predictions.The method targets information loss that can affect detailed boundary prediction.Source: §1 · p. 2, §2 · p. 3

3. Method

MADGNet’s method combines MFMSA, which processes features across frequency and scale, with E-SDM, which supplements upsampled sub-task predictions into the core prediction. These components address feature extraction and information loss in deep-supervised multi-task learning.

3.1. Multi-Frequency in Multi-Scale Attention Block

  • MFMSA decomposes features into multiple scale branches, then applies multi-frequency channel attention and multi-scale spatial attention.The block uses 2D DCT frequency statistics for channel attention and spatial attention to extract boundary cues across scales.Source: §3.1 · pp. 3–4
  • Each MFMSA scale branch reduces feature resolution and channel count using a reduction ratio, with minimum dimensions constraining the reductions.The branches are upsampled to match resolutions before their refined features are aggregated.Source: §3.1 · p. 4

3.2. Ensemble Multi-task Learning with Deep Supervision

  • E-SDM cascades upsampled sub-task predictions into the core prediction to supplement information loss during multi-task learning with deep supervision.The method describes E-SDM as an ensemble of predictions across tasks for the same lesion.Source: §3.2 · p. 5
  • Deep supervision uses a core region task and boundary and distance-map sub-tasks, with task-specific losses combined for training.The supplied method passage identifies weighted IoU and BCE for region prediction, and BCE and MSE for boundary and distance-map losses.Source: §3.2 · p. 5

4. Experiment Results

MADGNet is evaluated across modalities and clinical settings, with comparisons showing strong segmentation performance and ablations examining its multi-frequency, multi-scale attention and ensemble decoding. The results include binary and multi-label segmentation, as well as seen and unseen clinical settings.

4.3. Comparison with State-of-the-art models

  • MADGNet improved average DSC and mIoU by 1.1% and 1.0%, respectively, versus M2SNet across seen settings, and outperformed it on most unseen datasets.The unseen-setting exception was CVC-300; on STU, MADGNet improved DSC by 1.9% and mIoU by 2.7% versus FRCUNet.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7

4.1. Experiment Settings

  • The evaluation assessed domain generalizability on eight external datasets spanning dermoscopy, radiology, ultrasound, microscopy, colonoscopy, and fundus imaging, using DSC and mIoU and reporting mean performance across three trials.Source: §4.1 · p. 6

4.2. Implementation Details

  • Training resized images to 352 × 352 and used three scales and 16 frequency settings for MADGNet across datasets.The aggregation function averaged refined feature maps from each scale branch.Source: §4.2 · p. 6

4.3. Comparison with State-of-the-art models

  • Qualitative comparisons report detailed MADGNet boundaries across modalities despite severe noise and lesions of varying sizes.Figure 5 compares predictions with ground-truth boundaries and predictions from the other methods.Source: §4.3 · p. 7, Figure 5 · p. 8
  • For multi-label fundus segmentation, MADGNet improved average DSC over CENet by 2.9% for OD and 1.9% for OC.The evaluation used REFUGE as seen data and Drishti-GS as unseen data.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Ablation Study on MADGNet

  • MADGNet combines multi-scale and multi-frequency features; the ablation reports improved feature representation and boundary features when both are used.The study also reports that variants using either feature separately achieved performance gains.Source: §4.4 · pp. 7–8, Figure 6 · p. 8
  • E-SDM improved DSC by 1.2% on seen and 7.2% on unseen Colonoscopy datasets versus parallel decoding with deep supervision.The authors attribute the degradation in parallel decoding to information loss during drastic upsampling.Source: Table 4 · p. 8, §4.4 · p. 8

5. Conclusion

MADGNet combines MFMSA and E-SDM and is reported to outperform other state-of-the-art options across modalities and clinical settings. The conclusion also identifies memory efficiency for real clinical use as future work.

5. Conclusion

  • MADGNet is reported to surpass other state-of-the-art options in segmentation performance across modalities and clinical settings.Source: §5 · p. 9
  • The conclusion identifies multi-scale and multi-frequency features, MFCA-to-MSSA boundary cues, and E-SDM task ensembling as key aspects of the model.Source: §5 · p. 9
  • MADGNet demonstrated performance in both binary and multi-label segmentation of optic disc and optic cup.Source: §5 · p. 9
  • Increasing memory efficiency for real clinical use is identified as future work.Source: §5 · p. 9

6. Dataset Descriptions

The experiments cover breast ultrasound, skin lesions, lung infections, cells, polyps, and fundus images; the section also distinguishes seen from unseen clinical settings.

6. Dataset Descriptions

  • Tables 5 and 6 describe the seen and unseen clinical settings used in the experiments, respectively.Source: Table 5 · p. 13, Table 6 · p. 13
  • Breast ultrasound experiments use BUSI (780 images from 600 patients) and STU (42 images), with STU reserved for cross-dataset generalizability evaluation.Source: §6 · p. 13
  • Skin-lesion training, validation, and testing use 1,868, 465, and 261 ISIC 2018 images; PH2 evaluates generalizability to an unseen clinical setting.Source: §6 · p. 13
  • COVID19-1 supplies 1,277 CT images split into 643 training, 251 validation, and 383 test images; COVID19-2 is used only for testing.Source: §6 · p. 13
  • Cell segmentation uses 670 Data Science Bowl images split into 483 training, 120 validation, and 67 test images; MonuSeg2018 is used only for testing.Source: §6 · p. 13
  • Polyp experiments train on 900 Kvasir and 550 CVC-ClinicDB samples, using the remaining images and CVC-ColonDB, ETIS, and CVC-300 only for testing.Source: §6 · p. 13
  • Fundus multi-label segmentation trains and tests on 280 and 80 REFUGE images, respectively, and tests on 50 Drishti-GS images.Source: §6 · p. 13

7. Efficiency Analysis

The efficiency analysis reports parameter counts and inference speeds for different models, and describes parameter-saving choices in MFMSA's scale branches and attention components.

7. Efficiency Analysis

  • Table 7 reports parameter counts in millions and inference speed in milliseconds for different models; its caption provides no comparative values.Source: Table 7 · p. 13

7.1. Parameter Efficiency Proof

  • Figure 8 presents training results for αs 3 and βs 3 across five modalities; Figure 9 identifies Top, Bot, and Low frequency-selection strategies.Source: Figure 8 · p. 14, Figure 9 · p. 14
  • Scale decomposition replaces a kernel of size 2s + 1 with a size-3 dilated convolution using dilation s in each scale branch.The passage states this replacement reduces parameters in each branch by 9/(2s + 1)2.Source: §7.1 · p. 14
  • MSSA uses a size-1 convolution to extract an attention map and restores channels to C when aggregating refined features across scales.Source: §7.1 · p. 14

8. Ensemble Sub-Decoding Module for Multi-label Segmentation

E-SDM produces core and sub-task pseudo predictions during a forward stream, then a backward stream yields a final core prediction. Its task predictions are interpreted as an ensemble for each label.

8. Ensemble Sub-Decoding Module for Multi-label Segmentation

  • E-SDM is described as applicable to segmentation datasets with M labels and produces core and sub-task pseudo predictions at decoder stages.Source: §8 · p. 15
  • The forward stream lets each subsequent sub-task prediction focus cascadingly on a region through spatial attention, starting from the core pseudo prediction.Source: §8 · p. 15
  • After producing the L-th sub-task pseudo prediction, the backward stream produces the final core task prediction.Source: §8 · p. 15
  • E-SDM can be interpreted as an ensemble of predictions from different tasks describing the same lesion for each label.Source: §8 · p. 15

9. More Detailed Ablation Study on MADGNet

The ablation study evaluates MADGNet across backbone choices and hyperparameter settings for scale, frequency, and frequency-selection strategy, reporting mean performance on seen and unseen datasets.

9.1. Ablation on Backbone Model

  • Backbone performance is compared across ResNet50, Res2Net50, ViT-B-16, and ResNeSt50, with mean results reported for seen and unseen datasets.Table 8 gives quantitative results by dataset and backbone.Source: §9.1 · p. 15, Table 8 · p. 15

9.2. Hyperparameter on MADGNet

  • Scale-count comparisons evaluate S ∈ {1, 2, 3, 4, 5, 6} with F = 16, reporting mean performance for seen and unseen datasets.Table 9 and Figure 10 present the results; Figure 10 also compares parameter count with DSC.Source: §9.2 · p. 16, Table 9 · p. 16, Figure 10 · p. 16
  • Frequency-count comparisons evaluate F ∈ {1, 2, 4, 8, 16, 32} with S = 3, reporting mean performance for seen and unseen datasets.Table 10 and Figure 11 present the results; Figure 11 plots frequency count against DSC.Source: §9.2 · p. 16, Table 10 · p. 16, Figure 11 · p. 16
  • Frequency-selection strategies Top, Bot, and Low are compared with S = 3 and F = 16 using mean seen- and unseen-dataset performance.Table 11 reports the comparison, and Figure 9 shows the corresponding DCT basis images.Source: §9.2 · p. 16, Table 11 · p. 16

10. Technical Innovation, Design Principle and Interpretability of MADGNet

MADGNet combines frequency- and spatial-domain attention to extract discriminative features. MFCA emphasizes salient frequencies while MSSA captures boundary cues across lesion sizes.

10. Technical Innovation, Design Principle and Interpretability of MADGNet

  • MFCA uses multiple frequency statistics, while MSSA uses multi-scale attention and learnable information-flow parameters to extract features in frequency and spatial domains.The design is motivated by prior work.Source: §10 · p. 16
  • MFCA emphasizes salient features and reduces noisy-feature influence by focusing on frequencies of interest, which have high variance in medical images.The discussion points to Fig. 1 for frequency variance.Source: §10 · p. 16
  • MSSA combines foreground and background attention with multi-scale attention and information-flow parameters to capture boundary cues for lesions of varied sizes.The approach integrates both attentions with dilated convolution and downsampling.Source: §10 · pp. 16–17

11. Metrics Descriptions

The section describes six segmentation metrics: DSC, IoU, weighted F-measure, S-measure, E-measure, and MAE. Higher values are preferred for all except MAE.

12. Metrics Descriptions

  • Mean Dice Similarity Coefficient (DSC) measures similarity between two samples and is used to assess segmentation performance; higher is better.Source: §11 · p. 17
  • Mean Intersection over Union (IoU) measures the intersection-to-union ratio of predicted and ground-truth masks; higher is better.Source: §11 · p. 17
  • Mean Weighted F-Measure (F w β) combines precision and recall using their harmonic mean; higher is better.Source: §11 · p. 17
  • Mean S-Measure (Sα) evaluates structural similarity between predicted and ground-truth segmentations; higher is better.Source: §11 · p. 17
  • Mean E-Measure (Emax ϕ) assesses edge accuracy by evaluating predicted-edge alignment with ground truth; higher is better.Source: §11 · p. 17
  • Mean Absolute Error (MAE) is the average absolute difference between predicted and ground-truth values; lower is better.Source: §11 · p. 17

12. More Qualtative and Quantitative Results

The section presents quantitative results across five segmentation tasks and accompanying qualitative comparisons. Results are averaged over three trials, and the figures compare predictions with ground-truth boundaries.

12. More Qualtative and Quantitative Results

  • Quantitative results are provided in Tables 12–16, with means over three trials and standard deviations; Figures 12–16 provide further qualitative results.Red and blue mark the first- and second-best results, respectively.Source: §12 · p. 17
  • Skin-lesion results compare models trained on ISIC2018 with evaluations on ISIC2018 and PH2; Figure 12 shows predictions against ground-truth boundaries.Source: Table 12 · p. 17, Figure 12 · p. 18
  • COVID-19 infection results compare training on COVID19-1 with evaluation on COVID19-1 and COVID19-2; Figure 13 shows predictions against ground-truth boundaries.Source: Table 13 · p. 17, Figure 13 · p. 19
  • Breast-tumor results compare training and testing on BUSI with testing on STU; Figure 14 shows predictions against ground-truth boundaries.Source: Figure 14 · p. 20, Table 14 · p. 23
  • Cell-segmentation results compare training and testing on DSB2018 with testing on MonuSeg2018; Figure 15 shows predictions against ground-truth boundaries.Source: Figure 15 · p. 21, Table 15 · p. 23
  • Polyp results train on CVC-ClinicDB and Kvasir-SEG, then evaluate on those datasets and CVC-300, CVC-ColonDB, and ETIS; Figure 16 shows prediction and ground-truth boundaries.Source: Figure 16 · p. 22, Table 16 · p. 23

Related papers

Loading 2405.06284v1…