PaperlayerView exact revision on arXiv

Source-linked AI summary

MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation

Abhishek Srivastava, Debesh Jha, Sukalpa Chanda, Umapada Pal, Håvard D. Johansen, Dag Johansen, Michael A. Riegler, Sharib Ali, Pål Halvorsen

arXiv:2105.07451v2eess.IVcs.CV

TL;DR

Biomedical segmentation must handle variable-sized objects despite costly, inconsistent annotation and limited datasets. MSRF-Net addresses this with dual-scale residual feature fusion and outperforms several state-of-the-art methods across four biomedical datasets, while achieving competitive cross-dataset results.

  • Problem

    Biomedical segmentation is challenged by costly, time-consuming annotation and limited high-quality labeled datasets, motivating robust automated methods.

  • Method

    MSRF-Net maintains high-resolution representations while using sequential Dual-Scale Dense Fusion blocks to exchange and fuse features across scales, supplemented by a gated shape stream.

  • Results

    MSRF-Net outperformed several state-of-the-art methods on four independent biomedical datasets and achieved cross-dataset DSC values of 0.7921 and 0.7575.

  • Takeaways & Limitations

    MSRF-Net can capture target-object boundaries across variable sizes and produces competitive results in cross-dataset generalizability tests.

Abstract

from arXiv · show

Methods based on convolutional neural networks have improved the performance of biomedical image segmentation. However, most of these methods cannot efficiently segment objects of variable sizes and train on small and biased datasets, which are common for biomedical use cases. While methods exist that incorporate multi-scale fusion approaches to address the challenges arising with variable sizes, they usually use complex models that are more suitable for general semantic segmentation problems. In this paper, we propose a novel architecture called Multi-Scale Residual Fusion Network (MSRF-Net), which is specially designed for medical image segmentation. The proposed MSRF-Net is able to exchange multi-scale features of varying receptive fields using a Dual-Scale Dense Fusion (DSDF) block. Our DSDF block can exchange information rigorously across two different resolution scales, and our MSRF sub-network uses multiple DSDF blocks in sequence to perform multi-scale fusion. This allows the preservation of resolution, improved information flow and propagation of both high- and low-level features to obtain accurate segmentation maps. The proposed MSRF-Net allows to capture object variabilities and provides improved results on different biomedical datasets. Extensive experiments on MSRF-Net demonstrate that the proposed method outperforms the cutting-edge medical image segmentation methods on four publicly available datasets. We achieve the dice coefficient of 0.9217, 0.9420, and 0.9224, 0.8824 on Kvasir-SEG, CVC-ClinicDB, 2018 Data Science Bowl dataset, and ISIC-2018 skin lesion segmentation challenge dataset respectively. We further conducted generalizability tests and achieved a dice coefficient of 0.7921 and 0.7575 on CVC-ClinicDB and Kvasir-SEG, respectively.

I. INTRODUCTION

Medical image segmentation supports lesion assessment and treatment planning but is limited by costly, inconsistent annotation and architectures that do not adequately handle hierarchical or variable-scale features. MSRF-Net addresses these challenges with high-resolution processing, dual-scale dense fusion, and a gated shape stream, achieving broad evaluation across biomedical datasets.

  • Motivation: Medical image segmentation can identify lesions, help detect missed abnormalities, prevent disease, and improve therapy planning and treatment.The introduction highlights lesion assessment, including determining whether polyps are cancerous and removing them when necessary.
  • Motivation: High-quality labeled datasets are difficult to obtain because pixel-wise annotation is time-consuming, costly, expert-dependent, and potentially inconsistent.Annotation guidelines may differ among experts, producing discrepancies such as whether a lesion area is cancerous or non-cancerous.
  • Limitations: Existing CNN segmentation methods can use restrictive same-scale skip connections and fail to exploit hierarchical features, while network depth and dataset size constrain model complexity.These limitations affect the suitability of architectures across biomedical applications.
  • Proposed Method: MSRF-Net maintains high-resolution representations and uses a Dual-Scale Dense Fusion block plus a multi-scale exchange sub-network to propagate features across resolutions.The DSDF block performs dual-scale feature exchange, while residual dense connections support propagation of both high- and low-resolution features.
  • Evaluation: MSRF-Net adds a gated shape stream that combines high- and low-level features to compute accurate shape boundaries and is evaluated on four publicly available biomedical datasets.The introduction reports that the model outperforms state-of-the-art segmentation methods on most standard computer vision evaluation metrics and is more generalizable.

II. RELATED WORK · A. Medical image segmentation · B. Residual dense blocks

Related work progresses from convolutional and U-Net-based segmentation toward attention mechanisms and dense residual architectures for improving feature integration and medical image segmentation. These approaches motivate multi-scale feature exchange and residual dense design in MSRF-Net.

  • A. Medical image segmentation: FCN used only convolutional layers for semantic segmentation, while U-Net added an encoder-decoder structure for segmenting HeLa cells and neuronal structures.U-Net combined low- and high-level feature maps through skip connections.
  • A. Medical image segmentation: SE-Net introduced channel-wise attention by modeling channel interdependencies to emphasize relevant features and suppress irrelevant ones.FED-Net incorporated squeeze-and-excitation blocks into a modified U-Net, while FocusNet combined spatial and channel-wise attention.
  • A. Medical image segmentation: MSRF-Net comprises a proposed Dual-Scale Dense Fusion block and Multi-Scale Residual Fusion, with multi-scale feature exchange represented within the MSRF component.The figure identifies DSDF and MSRF as the architecture’s principal components.
  • A. Medical image segmentation: UACA-Net used saliency maps to derive foreground, background, and uncertain-area maps, then modeled relationships between representations for subsequent prediction modules.The passage also directs readers to broader summaries of deep-learning methods in medical image segmentation.
  • B. Residual dense blocks: Dense connections improve information flow and preserve diversified features by giving each layer inputs from all preceding layers.Multiple medical image segmentation methods use these connections to improve segmentation performance.
  • B. Residual dense blocks: Residual Dense Blocks extract local features through densely connected convolutional layers and support global hierarchical feature extraction through cross-block connections and final 1 × 1 convolutions.ResUNet and RD-U-Net incorporate RDBs into standard U-Net architectures to exploit hierarchical features.
  • B. Residual dense blocks: HyperDense-Net introduced a two-stream CNN as another architecture built around dense feature connectivity.The supplied passage begins describing this model but does not provide further architectural details.

C. Multi-scale fusion … B. The DSDF block and MSRF sub-network

MSRF-Net uses multi-scale feature exchange to preserve high-resolution spatial information while combining high- and low-resolution representations. Its DSDF blocks and MSRF sub-network maintain resolution, propagate features across scales, and support global multi-scale fusion within the segmentation architecture.

  • C. Multi-scale fusion: Multi-scale fusion exchanges high- and low-resolution features throughout segmentation, improving high-resolution feature flow and preserving spatial information for accurate maps.Skip-connections, pyramid pooling, ASPP, and atrous convolutions are cited as prior multi-resolution fusion strategies.
  • D. Our approach: The DSDF block accepts two scale features, propagates information within each resolution, and performs cross-resolution fusion to retain both feature types.An encoder feeds feature representations into the MSRF sub-network, which contains multiple DSDF blocks.
  • D. Our approach: The fusion strategy is interchangeable, allowing information exchange from low to high resolution and from high to low resolution.Decoder layers use skip-connections from the MSRF sub-network and process fused feature maps with a triple attention mechanism and shape stream.
  • III. THE MSRF-NET ARCHITECTURE: MSRF-Net comprises an encoder block, MSRF sub-network, gated shape stream, and decoder block with triple attention modules.The encoder uses squeeze-and-excitation modules, while the MSRF sub-network processes low-level feature maps extracted at each encoder resolution.
  • A. Encoder: Squeeze-and-excitation modules increase representational power by computing channel interdependencies through global average pooling and per-channel excitation weights.Each encoder block contains two consecutive convolutions followed by a squeeze-and-excitation module.
  • B. The DSDF block and MSRF sub-network: The DSDF block uses two parallel streams at different resolution scales to exchange information, preserve low-level features, improve information flow, and maintain resolution.A 3 × 3 convolution followed by LeakyReLU is denoted CLR(·); cross-scale inputs are transformed before concatenation.
  • B. The DSDF block and MSRF sub-network: The MSRF sub-network chains several DSDF blocks, passing same-resolution inputs forward to retain multi-scale representations and achieve global multi-scale context.The contiguous memory mechanism preserves representations across successive DSDF blocks.
  • B. The DSDF block and MSRF sub-network: After the fourth MSRF layer, features have been exchanged across all scales, enabling global fusion in which each initial resolution representation can reach all parallel resolutions.The sub-network demarcates resolution-scale pairs and feeds them into corresponding DSDF blocks.

C. Shape stream

MSRF-Net incorporates a gated shape stream that uses DSDF-derived representations to model shape and boundaries. Its output is fused with input-image gradients and the original segmentation stream to increase segmentation-map spatial accuracy.

  • Shape stream: MSRF-Net incorporates a gated shape stream for shape prediction, using DSDF blocks to extract high-level representations containing shape and boundary information.The shape stream uses X, the MSRF-sub-network output, with bilinear interpolation applied so X can match the required spatial dimensions.
  • Shape stream: S_l+1 is computed as RB(S_l×α), where RB is a residual block with two CLR operations followed by a skip-connection.The passage defines S_l as the shape-stream feature maps, with l denoting the number of layers.
  • Shape stream: The shape-stream output is concatenated with input-image gradients and merged with the original segmentation stream before the last CLR operation.This fusion is intended to increase the spatial accuracy of the segmentation map.

D. Decoder

The decoder combines skip connections with channel, spatial, and gated spatial attention to refine and propagate relevant features. Its triple-attention output is fused and followed by two CLR operations.

  • Skip connections: Decoder blocks D2–D4 receive skip connections from the MSRF sub-network and previous decoder output, except D2, which connects to E4’s MSRF output.This design links decoder stages to both multi-scale features and preceding decoder representations.
  • Attention mechanisms: The decoder applies channel and spatial attention through an S&E block, alongside a second attention mechanism based on gating.The S&E block calculates channel-wise scale coefficients, while the gated mechanism uses MSRF-Net features and the previous decoder output.
  • Gated attention: The gated attention combines transformed MSRF and decoder features to produce an activation map for propagating relevant regions.Convolutions transform and combine the inputs, followed by activation and transpose convolution operations.
  • Feature refinement: The gated-attention output is multiplied with features so irrelevant content is pruned while relevant target structures are propagated.The updated gated-attention features are denoted ˜DAG.
  • Decoder output: The triple-attention decoder output fuses channel-spatial and gated-spatial attention features, producing Dα before two CLR operations.The fusion is expressed as Dα = Dsc ⊕˜DAG.

E. Loss computation

MSRF-Net trains with a combined binary cross-entropy and dice loss, while the shape stream uses binary cross-entropy alone. Deep supervision and fixed unit weighting are used to improve gradient flow, regularization, and loss balancing.

  • E. Loss computation: The main loss combines binary cross-entropy and dice loss for gradient minimization between predicted maps and labels.The combined loss is Lcomb = λ1LBCE + λ2LDCS.
  • E. Loss computation: 1: λ1 and λ2 are both set to 1 in the combined loss.This fixes the relative weighting of the binary cross-entropy and dice components.
  • E. Loss computation: The shape stream uses only binary cross-entropy, computed with predicted edge maps and ground-truth maps.The latter loss refers to the edge-map computation for the shape stream.
  • E. Loss computation: Deep supervision is used to improve gradient flow and regularization across the supervised outputs.The final loss includes the two deep-supervision output losses and the shape-stream loss.
  • E. Loss computation: 1: α, β1, β2, and γ are each set to 1 for the experiments.These coefficients weight the deep-supervision and shape-stream loss terms in the final loss.

IV. EXPERIMENTAL SETUP … 1) Comparison on Kvasir-SEG

MSRF-Net was evaluated on four public biomedical image-segmentation datasets using standard segmentation and speed metrics under specified training settings. On Kvasir-SEG, it outperformed the compared state-of-the-art methods across all reported metrics.

  • A. Dataset: The experiments used Kvasir-SEG, CVC-ClinicDB, the 2018 Data Science Bowl dataset, and ISIC-2018 Challenge dataset.Each dataset contains images and corresponding ground-truth masks.
  • B. Evaluation metrics: Evaluation used Dice Coefficient, mean Intersection over Union, recall, precision, and Frames Per Second.Standard deviations were reported for DSC, mIoU, recall, and precision, alongside paired t-tests of DSC against other state-of-the-art methods.
  • C. Implementation details: The architecture was implemented in Keras with TensorFlow as backend and trained on NVIDIA V100 Tensor Core GPUs in an NVIDIA DGX-2 machine.The Adam optimizer used learning rate 0.0001, dropout p = 0.2, and DSDF/MSRF scaling factor w = 0.4.
  • A. SOTA method comparisons: The results section compares MSRF-Net with other state-of-the-art methods.The paper presents these comparisons as part of its experimental results.
  • 1) Comparison on Kvasir-SEG: On Kvasir-SEG, MSRF-Net outperformed all other state-of-the-art methods on every reported metric.Quantitative results are reported in Table II, while qualitative results are shown in Figure 3.
  • 1) Comparison on Kvasir-SEG: 1.39% improvement on DSC was achieved by MSRF-Net on Kvasir-SEG.The passage introduces this improvement while describing the quantitative comparison, but the cited excerpt truncates before identifying the comparison baseline.

2) Comparison on CVC-ClinicDB · 3) Comparison on 2018 Data Science Bowl

On CVC-ClinicDB, MSRF-Net surpasses other state-of-the-art methods across key segmentation metrics, with consistent Dice scores and competitive precision. On the 2018 Data Science Bowl dataset, it outperforms ColonSegNet in most reported metrics.

  • 2) Comparison on CVC-ClinicDB: MSRF-Net shows large improvements for some CVC-ClinicDB samples while maintaining consistent Dice scores, and its prediction masks closely match ground-truth polyp boundaries and shapes.
  • 2) Comparison on CVC-ClinicDB: MSRF-Net achieves a DSC of 0.9420 ± 0.0804 on CVC-ClinicDB, improving 1.76% over the best-performing HRNetV2-W48.
  • 2) Comparison on CVC-ClinicDB: MSRF-Net reports a mIoU of 0.9043 ± 0.1009, a 1.44% improvement over the SOTA combination of ResUNet++ and conditional random field.
  • 2) Comparison on CVC-ClinicDB: MSRF-Net reports a recall of 0.9567 ± 0.0620, a 2.82% improvement over UACANet-S [35].
  • 2) Comparison on CVC-ClinicDB: MSRF-Net achieves a precision of 0.9427, competitive with the best-performing DoubleUNet.
  • 3) Comparison on 2018 Data Science Bowl: MSRF-Net obtains a DSC of 0.9224 ± 0.0538, mIoU of 0.8534 ± 0.0870, recall of 0.9402 ± 0.0734, and precision of 0.9022 ± 0.0601 on the 2018 Data Science Bowl dataset.These results outperform the best-performing ColonSegNet in most metrics.

4) Comparison on ISIC-2018 Skin Lesion Segmentation challenge … C. Ablation study

MSRF-Net achieves strong ISIC-2018 segmentation results and maintains competitive performance across institution-shifted datasets. Ablation results show that adding the MSRF sub-network substantially improves DSC.

  • 4) Comparison on ISIC-2018 Skin Lesion Segmentation challenge: 0.8824 ± 0.1602 DSC and 0.8373 ± 0.1818 mIoU were achieved on the ISIC-2018 challenge.Recall was 0.8893 ± 0.1889 and precision was 0.9348 ± 0.1488.
  • 4) Comparison on ISIC-2018 Skin Lesion Segmentation challenge: The ISIC-2018 results improved over Deeplabv3+ with the Mobilenet backbone by 0.43% in DSC and 1.37% in mIoU.
  • B. Generalization study: Generalizability was evaluated by training on Kvasir-SEG or CVC-ClinicDB and testing on the other dataset from a different institution, cohort population, and imaging protocol.The two directions were Kvasir-SEG to CVC-ClinicDB and CVC-ClinicDB to Kvasir-SEG.
  • 1) Generalizability results on CVC-ClinicDB: 0.7921 ± 0.2564 DSC was obtained when MSRF-Net was trained on Kvasir-SEG and tested on CVC-ClinicDB, outperforming other SOTA methods in DSC.The model also achieved 0.6498 ± 0.2729 mIoU, 0.9001 ± 0.2980 recall, and 0.7000 ± 0.1572 precision; HRNetV2-W48 obtained 0.7901 DSC.
  • 2) Generalizability results on Kvasir-SEG: 0.7575 ± 0.2643 DSC and 0.6337 ± 0.2815 mIoU were achieved when training on CVC-ClinicDB and testing on Kvasir-SEG, outperforming other SOTA methods.PraNet achieved 0.7293 ± 0.3004 DSC and 0.6262 ± 0.3128 mIoU; MSRF-Net exceeded it by 2.82% in DSC and 0.75% in mIoU.
  • C. Ablation study: The ablation study examined the MSRF sub-network, scaling mechanism, number of DSDF blocks, shape prediction, shape-stream removal, and deep-supervision removal.
  • C. Ablation study: 0.8771 DSC was obtained after removing the MSRF sub-network, while adding a subset of it increased DSC to 0.8986.The study also removed DSDF blocks with second- and third-scale inputs from the original MSRF sub-network.

VI. DISCUSSION

MSRF-Net addresses limitations of prior multi-scale fusion methods through repeated, low-complexity feature exchange across high- and low-resolution scales. Its design improves cross-dataset generalization and segmentation quality, while ablations validate the MSRF sub-network and complementary components, with failures remaining on difficult images.

  • Architecture: DSDF blocks enable continuous high- and low-resolution feature exchange for global multi-scale fusion while reducing computational complexity relative to HRNetV2.The residual structure also preserves relevant high- and low-level features.
  • Generalizability: MSRF-Net achieves its highest DSC of 0.7921 when trained on Kvasir-SEG and tested on CVC-ClinicDB, and its highest DSC of 0.7575 in the reciprocal generalization setting.The discussion attributes this generalizability to multi-scale fusion that preserves class-representative features.
  • Ablation study: Removing the MSRF sub-network reduces the combination of high- and low-level multi-scale features needed to recognize target shapes and boundaries.The ablation study links these features to improved segmentation performance.
  • Ablation study: A 0.80% drop in DSC occurs without DSDF output scaling, while empirical experiments identify w = 0.4 as optimal.The scaling factor is defined in Equation 3.
  • Ablation study: Disabling triple attention causes a 1.50% DSC drop, whereas disabling deep supervision causes a 2.29% DSC drop.The experiments also investigate the combination of LBCE and LDCS in Lcomb.
  • Limitations: Qualitative results show poor performance on oblique polyp samples, extremely low-contrast 2018 DSB images, and scattered similar patches in ISIC 2018.These cases identify limitations of the proposed model on difficult visual conditions.

VII. CONCLUSION

MSRF-Net is a medical image segmentation architecture that passes multiscale resolution features through sequential DSDF blocks. Its densely connected residual blocks enable efficient feature extraction across varying receptive fields and help capture complete object boundaries despite variable object sizes.

  • Conclusion: MSRF-Net passes multiscale resolution features through a sequence of DSDF blocks for medical image segmentation.The architecture uses densely connected residual blocks with dual-scale feature exchange.
  • Conclusion: Densely connected residual blocks with dual-scale feature exchange enable efficient feature extraction with varying receptive fields.
  • Conclusion: DSDF-block features better capture an entire target object’s shape boundaries, including for objects with variable sizes.
Loading 2105.07451v2…