PaperlayerView exact revision on arXiv

Source-linked AI summary

FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation

Nikhil Kumar Tomar, Debesh Jha, Michael A. Riegler, Håvard D. Johansen, Dag Johansen, Jens Rittscher, Pål Halvorsen, Sharib Ali

arXiv:2103.17235v3cs.CVeess.IV

TL;DR

Biomedical segmentation methods do not effectively use information across learning epochs and often cannot rectify single-step predictions. FANet propagates previous-epoch masks as hard attention and iteratively refines masks at training and test time, improving segmentation across seven biomedical datasets.

  • Problem

    Existing biomedical segmentation methods do not effectively use information from different learning epochs and commonly rely on single-step predictions that cannot rectify masks.

  • Method

    FANet unifies each previous-epoch mask with current feature maps to provide hard attention and iteratively updates prediction masks during training and testing.

  • Results

    FANet improved performance over existing state-of-the-art methods across seven publicly available biomedical datasets.

  • Takeaways & Limitations

    Feedback attention enables FANet to self-rectify predicted masks using a single end-to-end trainable network without heavy transformations or ensemble strategies.

Abstract

from arXiv · show

The increase of available large clinical and experimental datasets has contributed to a substantial amount of important contributions in the area of biomedical image analysis. Image segmentation, which is crucial for any quantitative analysis, has especially attracted attention. Recent hardware advancement has led to the success of deep learning approaches. However, although deep learning models are being trained on large datasets, existing methods do not use the information from different learning epochs effectively. In this work, we leverage the information of each training epoch to prune the prediction maps of the subsequent epochs. We propose a novel architecture called feedback attention network (FANet) that unifies the previous epoch mask with the feature map of the current training epoch. The previous epoch mask is then used to provide a hard attention to the learned feature maps at different convolutional layers. The network also allows to rectify the predictions in an iterative fashion during the test time. We show that our proposed \textit{feedback attention} model provides a substantial improvement on most segmentation metrics tested on seven publicly available biomedical imaging datasets demonstrating the effectiveness of FANet. The source code is available at \url{https://github.com/nikhilroxtomar/FANet}.

I. INTRODUCTION

Biomedical segmentation faces limited label variability and single-step prediction constraints. FANet addresses these issues by propagating sample-specific masks across epochs and refining predictions iteratively during training and testing.

  • Biomedical segmentation methods often require scarce expert labels that may not capture sufficient imaging variability.
  • Existing semantic segmentation networks generally make single-step predictions and cannot rectify their predicted masks.
  • FANet propagates each sample’s previous-epoch mask to the next epoch and combines it with current feature maps for feedback attention.
  • The feedback mechanism provides hard attention across feature scales, while SE-Residual blocks model channel interdependencies.
  • During inference, FANet updates the input mask with predictions for up to 10 empirically selected iterations.
  • The architecture includes embedded run-length encoding, evaluates seven biomedical datasets, and reaches near-SOTA performance with fewer training epochs.

II. RELATED WORK

Prior biomedical segmentation work builds on encoder-decoder, pyramid-pooling, and attention architectures, while iterative refinement has been explored for mask and latent-space updates. FANet is positioned as a self-rectifying approach that seeks convergence in fewer iterations.

  • Modern biomedical segmentation commonly uses FCN or encoder-decoder architectures such as U-Net, with modifications for multi-scale contextual information.
  • Attention mechanisms have been applied to focus segmentation models on target regions in medical images.
  • Earlier iterative methods refined segmentation masks or latent representations during inference, and related refinement pipelines improved predictions under shape variability or domain shifts.
  • A key bottleneck of prior iterative refinement methods is their need for many iterations to reach convergence.
  • FANet instead prunes predicted masks in less than ten iterations while providing attention to specific regions of interest.

III. METHOD

FANet combines SE-Residual blocks with MixPool blocks to apply feedback-based hard attention across feature scales. Previous-epoch masks are resized, fused with spatial attention, and used to suppress irrelevant features.

  • SE-Residual block: SE-Residual blocks use residual shortcuts and squeeze-and-excitation channel reweighting to emphasize significant features and suppress irrelevant ones.The SE layer globally pools each channel, reduces and expands the resulting feature vector, and uses it to scale the original feature maps.
  • MixPool block: MixPool propagates sample-wise feedback between epochs and applies binary hard attention to learned features in contraction and expansion paths.Its attention values are 0 or 1, retaining selected regions rather than estimating a probability map.
  • MixPool block: The spatial attention map is formed by processing SE-Residual features with convolution, batch normalization, ReLU, and sigmoid activation, then thresholding at 0.5.The resulting binary mask supplies hard attention for the input feature map.
  • MixPool block: The previous-epoch mask is max-pooled and resized to the spatial attention resolution, then unioned with the current attention map before feature modulation.Element-wise multiplication suppresses irrelevant features and enhances important ones; the resulting and original maps are processed by convolution, BN, and ReLU.
  • MixPool block: The MixPool output concatenates the enhanced and original activation outputs, while Figure 2 depicts SE-Residual feature fusion and mask-based attention.The supplied table caption identifies dataset details rather than a method operation.

C. Proposed FANet architecture

The proposed FANet uses an encoder-decoder FCNN with SE-Residual blocks, skip connections, and MixPool feedback. It refines masks across training epochs by combining previous masks with learned features.

  • C. Proposed FANet architecture: FANet combines SE-Residual blocks with MixPool blocks to propagate information from the current learning paradigm and the previous epoch.The architecture implements recurrent learning in both encoder and decoder layers for segmentation.
  • C. Proposed FANet architecture: The model initializes training with an Otsu-thresholded mask and refines it over epochs using MixPool hard attention with learned semantic features.The feedback process prunes input or previous-epoch masks as training progresses.
  • C. Proposed FANet architecture: The FCNN contains four encoder and four decoder blocks, with the encoder progressively downsampling the image into a compact representation.This establishes the architecture’s contraction path before decoding.
  • C. Proposed FANet architecture: The decoder upsamples encoded features, combines them with encoder skip connections, and produces pixel-wise semantic categorization.SE-Residual blocks form both paths, while original-resolution encoder features are concatenated at each scale.
  • C. Proposed FANet architecture: Each encoder uses SE-Residual blocks for feature extraction and skip connections, followed by MixPool blocks that receive previous-epoch segmentation masks.The decoder uses transpose convolutions, corresponding encoder skips, and additional SE-Residual blocks.

A. Setup

The evaluation covers seven biomedical segmentation datasets and standard segmentation metrics, with ablations comparing baseline, MixPool, and feedback configurations. The setup also includes image augmentation and implementation-specific training details.

  • A. Setup: Seven datasets represent different biomedical segmentation tasks, organs, lesions, and imaging protocols.The selected datasets include DRIVE and CHASE-DB1 for retinal vessel segmentation and ISIC-2018 for dermoscopy.
  • A. Setup: Evaluation uses Dice Coefficient, mean Intersection over Union, precision, recall, and, where previously benchmarked, specificity.Dice Coefficient is also identified as F1.
  • A. Setup: Training uses PyTorch on Volta 100 and NVIDIA DGX-2 hardware, while inference uses an NVIDIA GTX 1050 Ti for FANet and comparison methods.The common inference hardware is described as widely available.
  • A. Setup: Images are resized to 512 × 512 except for 2018 Data Science Bowl and CVC-ClinicDB images, which use 256×256 resolution.Augmentation includes cropping, flipping, rotation, elastic and grid distortions, grayscale conversion, and brightness or contrast changes.
  • A. Setup: The ablation study evaluates all seven datasets using baseline, baseline with MixPool, and the full baseline-plus-MixPool-plus-feedback configuration.The comparisons are performed across several segmentation metrics.

3) Ablation study:

FANet was evaluated against state-of-the-art methods across seven biomedical image datasets, with dataset-specific results reported for colon, nuclei, skin, retinal, and EM segmentation.

  • FANet achieved an F1 score of 0.8803 on Kvasir-SEG, exceeding DeepLabv3+ with ResNet101 by 1.6% and HRNet by 3.57%.
  • FANet attained F1 0.9355, mIoU 0.8937, recall 0.9339, and precision 0.9401 on CVC-ClinicDB.
  • On the 2018 Data Science Bowl dataset, FANet produced F1 0.9176, mIoU 0.8569, and recall 0.9222, improving F1 by 2.02% over UNet++ and recall by 28.15% over DoubleU-Net.

3) Results on 2018 Data Science Bowl:

FANet results span skin, retinal vessel, and EM segmentation datasets, with quantitative comparisons and qualitative observations of segmentation-mask quality.

  • FANet achieved F1 0.8731 and recall 0.8650 on ISIC 2018, improving over BCDU-Net by 2.21% and 8.00%, respectively.
  • Qualitative results report that FANet improves under-segmented Otsu input masks and produces masks with smooth boundaries.
  • FANet achieved F1 0.8183, mIoU 0.6927, recall 0.8215, and precision 0.8189 on DRIVE, with recall 4.24% higher than IterNet.
  • On CHASE-DB1, FANet achieved F1 0.8108, mIoU 0.6820, and recall 0.8544, improving recall by 3.67% over DenseBlock-UNet.
  • On the EM dataset, FANet obtained F1 0.9547, mIoU 0.9134, and recall 0.9568, with results described as state of the art.

7) Results on EM dataset:

The paper presents qualitative and ablation-study evidence for FANet across seven datasets, emphasizing iterative pruning and the contribution of feedback-enabled architecture.

  • Figure 3 compares Otsu-thresholded input masks with FANet-predicted segmentation masks across seven biomedical datasets.
  • For colonoscopy and retinal datasets, FANet prunes broad input masks and segments challenging small retinal-vessel bifurcations while resembling the ground-truth mask.
  • The ablation study evaluates four configurations differing in MixPool placement and whether feedback-driven iterative pruning is used during inference.
  • Table IX reports ablation results for the four configurations across all seven datasets.
  • The MixPool block uses previously predicted masks as attention to improve semantically meaningful features and enable higher-level abstractions.

1) Effectiveness of MixPool block:

The ablation study attributes FANet’s performance to the MixPool block and feedback mechanism, evaluating their placement and use during inference.

  • Effectiveness of MixPool block: B4, the full FANet with MixPool throughout encoder and decoder blocks plus feedback, outperformed B1 without MixPool on all seven datasets.
  • Effectiveness of MixPool block: B4 improved F1 over B1 by 2.87% on Kvasir-SEG, 1.89% on CVC-ClinicDB, 0.55% on Data Science Bowl, and 2.92% on CHASE-DB1.
  • Effectiveness of MixPool block: Comparing MixPool placement, B4 improved F1 over B3 by 3.43% on Kvasir-SEG, 1.93% on CVC-ClinicDB, 0.11% on Data Science Bowl, and 0.5% on EM.
  • Feedback mechanism: The feedback mechanism propagates input masks during training and supports iterative pruning during evaluation, with B2-versus-B4 comparisons focused on mIoU and F1 gains.

E. Algorithm efficiency

FANet reduces architectural complexity while improving segmentation through feedback-based iterative pruning, but the added pruning steps increase inference time and introduce an iteration hyperparameter.

  • Efficiency: 7.72 million parameters and 94.75 GMac characterize FANet as the least-parameterized and least-flop model among the compared deeper architectures.Its inference time is nevertheless higher than baseline networks because of the MixPool block.
  • Ablation study: Removing a SE-Residual block reduces F1 by 11.62% and mIoU by 13.43%, while adding blocks also decreases performance.Adding three blocks reduces F1 by 2.33% and mIoU by 2.45%; adding one more reduces them by 6.45% and 7.36%.
  • Interpretation: FANet’s reported improvements are attributed to self-rectification through feedback, which steers masks toward relevant learned features and captures dataset variability.The paper connects these effects to the joint use of masks and feature embeddings.
  • Qualitative analysis: FANet produces stronger feature representations and more distinguishable segmentation regions than compared networks in qualitative analyses.Heatmaps identify red and yellow regions as the most important features and blue regions as less important.
  • Ablation study: The MixPool block unifies feature-map attention with the previous-epoch input mask, enabling feedback and iterative pruning during evaluation.The ablation compares FANet with and without MixPool, while extended experiments examine MixPool components, SE-Residual blocks, and series concatenation.
  • Inference trade-off: Ten evaluation iterations were selected across datasets, although most improvement occurs from iterations 1 to 5 and FANet benefits from only two iterations.Fewer iterations are advised when trading accuracy against inference efficiency.
  • Inference trade-off: Inference timing depends on the pruning configuration and hardware, with the reported GTX 1050 Ti setup limiting the stated timing context.A newer GPU could provide better inference time.

VI. CONCLUSION

FANet introduces feedback-based self-rectification for biomedical image segmentation and improves results across seven public datasets. The architecture uses hard attention with learned feature maps and achieves accurate, consistent segmentation with a simple network design.

  • Conclusion: FANet self-rectifies predicted masks through a feedback mechanism for biomedical image segmentation.The approach improves segmentation across seven publicly available biomedical datasets compared with existing state-of-the-art methods.
  • Conclusion: The feedback mechanism acts as hard attention over existing feature maps, strengthening feature representations.The authors report that FANet achieves accurate and consistent results despite its simple architecture.
  • Conclusion: FANet requires less training time to reach near-state-of-the-art performance according to the ablation study.The paper proposes contrastive learning and additional multimodal biomedical-image testing as future work.
Loading 2103.17235v3…