PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-scale self-guided attention for medical image segmentation

Ashish Sinha, Jose Dolz

arXiv:1906.02849v3cs.CV

TL;DR

Medical segmentation models may redundantly reuse multi-scale information and inadequately model long-range dependencies. The paper proposes multi-scale guided self-attention with position-channel modeling and auxiliary guidance, and reports stronger segmentation across three medical tasks, including improved ablation and state-of-the-art comparisons.

  • Problem

    Standard medical segmentation models can redundantly reuse low-level multi-scale features and inadequately model long-range dependencies, limiting discriminative class representations.

  • Method

    The proposed architecture combines multi-scale features with position and channel self-attention, progressive guided refinement, and additional losses between modules.

  • Results

    86.75%(DSC), 93.85%(VS) and 0.66 voxels (MSD) were the best reported values, improving 4.5%, 4% and 26%, respectively, over baseline.

  • Takeaways & Limitations

    Experiments on abdominal organs, cardiovascular structures and brain tumors support precise and reliable automatic medical-image segmentations from the proposed approach.

Abstract

from arXiv · show

Even though convolutional neural networks (CNNs) are driving progress in medical image segmentation, standard models still have some drawbacks. First, the use of multi-scale approaches, i.e., encoder-decoder architectures, leads to a redundant use of information, where similar low-level features are extracted multiple times at multiple scales. Second, long-range feature dependencies are not efficiently modeled, resulting in non-optimal discriminative feature representations associated with each semantic class. In this paper we attempt to overcome these limitations with the proposed architecture, by capturing richer contextual dependencies based on the use of guided self-attention mechanisms. This approach is able to integrate local features with their corresponding global dependencies, as well as highlight interdependent channel maps in an adaptive manner. Further, the additional loss between different modules guides the attention mechanisms to neglect irrelevant information and focus on more discriminant regions of the image by emphasizing relevant feature associations. We evaluate the proposed model in the context of semantic segmentation on three different datasets: abdominal organs, cardiovascular structures and brain tumors. A series of ablation experiments support the importance of these attention modules in the proposed architecture. In addition, compared to other state-of-the-art segmentation networks our model yields better segmentation performance, increasing the accuracy of the predictions while reducing the standard deviation. This demonstrates the efficiency of our approach to generate precise and reliable automatic segmentations of medical images. Our code is made publicly available at https://github.com/sinAshish/Multi-Scale-Attention

I. INTRODUCTION

Medical image segmentation is clinically important but still relies on time-consuming, variable manual annotation. Existing CNN and multi-scale approaches improve representation yet may redundantly reuse features, model dependencies non-adaptively, and leave noisy attention unresolved.

  • Motivation: Manual medical-image annotation remains common in clinical practice despite segmentation automation, because it is time-consuming and prone to inter- and intra-observer variability.Accurate and reliable automation is sought to improve workflow efficiency and reduce expert workload.
  • Existing approaches: Fully convolutional and encoder-decoder networks are standard segmentation architectures, using contracting paths for high-level features and expanding paths to reconstruct pixel-wise masks.These models may reconstruct masks through one or multiple upsampling steps.
  • Existing approaches: Multi-scale context fusion captures objects at different scales, but its contextual dependencies are homogeneous and non-adaptive across image regions.The supplied passage truncates the final clause describing the resulting limitation.
  • Attention mechanisms: Attention mechanisms integrate local and global features, focus on relevant features without additional supervision, and avoid multiple similar feature maps.Prior work also reported improved segmentation over average- and max-pooling approaches for merging multi-scale predictions.
  • Proposed direction: The proposed network uses multi-scale guided attention with position and channel self-attention to progressively remove noise and emphasize class-relevant regions.The multi-scale design assigns lower-level stacks to local appearance and higher-level stacks to global representations.
  • Research gap: Stacked attention can progressively filter irrelevant noise and emphasize class-relevant regions, but its application to semantic segmentation was described as unexplored.This motivates extending progressive attention beyond the previously discussed visual question-answering and zero-shot-learning settings.

C. Medical image segmentation with deep attention

Medical-image segmentation attention research remains comparatively scarce and often uses simple modules. The paper combines multi-scale features with guided attention so local detail and high-level semantics are jointly represented at each scale.

  • Medical image segmentation with deep attention: Medical-image segmentation has used relatively scarce attention research, with prior approaches characterized as limited to simple attention modules.The paper positions its work as exploring more complex attention mechanisms for medical segmentation.
  • Medical image segmentation with deep attention: Target structures vary in size, shape, and texture, while local receptive fields can produce inconsistent representations for pixels sharing the same label.The paper links this limitation to the need for associations between features.
  • Multi-scale features: The multi-scale strategy uses features at multiple resolutions to encode both global and local context, with scale s denoting the architectural level.The approach follows a previously proposed multi-scale setting.
  • Multi-scale features: Features from all scales are concatenated and convolved into a common map, FMS, combining shallow-layer detail with deeper-layer semantics.FMS is then combined with each scale-specific feature map before guided attention generates As.
  • Multi-scale features: Combining FMS with each scale-specific map jointly encodes complementary low-level information and high-level semantics into the guided-attention representation.The resulting attentive features are denoted As for each scale.

C. Spatial and Channel self-attention modules

The architecture uses separate position and channel self-attention modules to model spatial and inter-channel dependencies. Their outputs are combined to produce position-channel attention features that improve class-specific discrimination.

  • Spatial and Channel self-attention modules: Traditional CNNs have local receptive fields, while channel maps can represent class-specific responses whose dependencies also require modeling.The paper addresses both spatial context and channel relationships through separate attention mechanisms.
  • Position attention module: The position attention module receives F ∈ R^C×W×H, where C, W, and H denote channel, width, and height dimensions.Its spatial attention map models interactions among image positions.
  • Position attention module: The spatial attention map S evaluates the impact of one position on another and is used to selectively aggregate global context into learned features.The position-attention scaling parameter λp is initialized to 0 and learned progressively.
  • Channel attention module: The channel attention module computes a channel attention map Sc ∈ R^C×C from reshaped feature maps, representing dependencies between channel maps.The impact of channel i on channel j is represented in the attention map.
  • Channel attention module: Channel attention aggregates weighted features across channels to highlight class-dependent maps and increase feature discriminability between classes.Its scaling parameter λc is initially zero and gradually learned.
  • Combined attention: The position and channel attention outputs pass through convolution and are summed element-wise to form position-channel attention features.This combines the two dependency types within the attention module.

D. Guiding attention

Guided attention progressively refines features at each scale by combining dual self-attention with encoder-decoder representations. Auxiliary losses constrain latent and reconstructed features to guide attention toward class information and away from noise.

  • D. Guiding attention: The guided attention module applies sequential refinement to progressively weight local regions and mask irrelevant noise.The input feature map F at scale s is processed through multiple refinement steps.
  • D. Guiding attention: At the first refinement step, position and channel modules generate self-attention features while an encoder-decoder compresses F into a latent representation.The latent representation is intended to embed class information for subsequent guided-attention modules.
  • Guided loss: A guided loss forces encoded representations from successive encoder-decoder networks to remain close across refinement iterations.Ei denotes the encoded representation of the i-th encoder-decoder, and M denotes the number of iterations.
  • Feature reconstruction: Reconstructed features from the first encoder-decoder are matrix-multiplied with first-step self-attention features to generate FSA.The reconstruction is designed to correspond to features entering the position-channel attention modules.
  • Losses: The module aggregates guided loss across scales and reconstruction losses across encoder-decoder blocks.LRec1 and LRec2 denote reconstruction losses for the first and second guided-attention blocks.
  • D. Guiding attention: With two refinement steps, the semantic guided attention module provides attentive features As for each scale.The figure depicts the multi-step refinement structure.

E. Deep supervision

The model uses reconstruction losses and segmentation losses across its guided-attention components, combined through a weighted total objective.

  • Deep supervision: Extra supervision at each scale improved the proposed model’s segmentation performance.The authors note this benefit despite attention modules not requiring auxiliary objectives.
  • Deep supervision: The segmentation loss combines predictions from raw features and attention features using multi-class cross-entropy.The first term evaluates raw-feature segmentation, while the second evaluates attention-feature segmentation.
  • Deep supervision: LTotal = αLSegTotal + βLGTotal + γLRecTotal combines segmentation, guided-attention, and reconstruction losses.The coefficients α, β, and γ control the importance of the three terms.

A. Experimental setting

The study evaluates the proposed guided-attention architecture across public medical-image segmentation benchmarks, ablations, and comparisons with competing networks. Results show gains in overlap, volume, surface-distance, and qualitative detail, while excessive refinement can over-focus attention.

  • Experimental setting: Three public benchmarks cover abdominal organs, cardiovascular structures, and brain tumors, with repeated three-fold evaluation reported for CHAOS.CHAOS includes spleen, liver, and kidneys; HVSMR targets myocardium and blood pool.
  • Experimental setting: The ablation compares spatial attention, channel attention, and their parallel combination in the proposed multi-scale architecture.The combined model is termed Proposed (DANet).
  • Results: 2–3% gains in overlap and volume similarity and 12–18% gains in surface-distance metrics followed insertion of spatial or channel attention.The averages are reported relative to the baseline on the CHAOS ablation.
  • Results: 86.75% DSC, 93.85% VS, and 0.66 voxels MSD were achieved by the guided model, improving on baseline by 4.5%, 4%, and 26%, respectively.These are the best values reported for the three metrics in the cited ablation.
  • Experimental setting: Table I reports three-fold average attention-module ablations, while Table II reports three-fold average MS-Dual and MS-Dual-Guided ablations.Both tables mark best and second-best results using red and blue, respectively.
  • Results: Best results often used only two guided-attention modules because additional refinement could over-focus features on discriminative regions.The authors caution that excessive focus may fail to cover the whole object.
  • Results: On CHAOS, the proposed network outperformed UNet, Attention UNet, DAF, and PAN across DSC, VS, and MSD comparisons.Reported mean improvements over PAN were 2.4% DSC, 1.9% VS, and 12% MSD.
  • Results: Qualitatively, the proposed model avoided ambiguous-region distractions and recovered finer organ and tumor details than competing architectures.Examples include anatomically plausible left-kidney segmentation, oedema ramifications, and complex necrotic areas.

V. CONCLUSION

The proposed attention architecture combines multi-scale semantic information, self-attention, and guided refinement for medical image segmentation. Experiments across abdominal organs, cardiovascular structures, and brain tumors support its effectiveness and attention components.

  • V. CONCLUSION: The architecture combines multi-scale semantic information with self-attention modules that progressively aggregate relevant contextual features.A guided refinement module filters noisy regions and focuses on relevant class-specific regions.
  • V. CONCLUSION: The model was evaluated on abdominal-organ, cardiovascular-structure, and brain-tumor segmentation tasks.
  • V. CONCLUSION: The paper presents channel-map visualizations from the spatial and channel attention modules in the guided module.
  • V. CONCLUSION: The proposed model outperformed previous attention-based approaches quantitatively and qualitatively.The authors associate this result with enhanced modeling of rich contextual dependencies over local features.

Supplemental Materials

The supplemental materials define the evaluation metrics used to assess segmentation quality. They cover overlap, volume similarity, and surface-distance behavior.

  • Supplemental Materials: The evaluation metrics include Dice similarity coefficient, volume similarity, and mean surface distance.These metrics assess overlap, volume agreement, and sensitivity to segmentation outlines, respectively.
  • Supplemental Materials: The Dice similarity coefficient compares two volumes based on their overlap.Values near 1 indicate high overlap, while values near 0 indicate no overlap.
  • Supplemental Materials: Volume similarity is used to assess segmentation performance based on volume agreement.
  • Supplemental Materials: Mean surface distance measures sensitivity to segmentation outlines using distances between contours.The distance from a point on one surface to the other surface is defined using the minimum Euclidean distance.

Additional results

Additional experiments show that the proposed architecture generally achieves strong and consistent segmentation results across organs, metrics, and compared networks. Its attention modules improve accuracy distributions and reduce variance.

  • Additional results: The proposed architecture ranks first or second for almost all organs and evaluation metrics across the extended comparisons.The stated exception is liver volume similarity, where all models obtain almost identical results.
  • Additional results: Progressively adding the proposed attention modules improves segmentation accuracy distributions and reduces variance across the 15 evaluation subjects.
  • Additional results: The improvement pattern remains constant across organs and metrics, indicating better and more robust results than current state-of-the-art architectures.State-of-the-art networks are represented by bluish box plots in Figure 7.
  • Additional results: Figure 8 compares convergence using mean DSC over the four structures on one validation fold for each network.The reported convergence behavior differs even when several networks achieve broadly similar results.

DSC (%)

The tables and figures report evaluation results for multi-organ, cardiovascular, and brain-tumor segmentation, alongside ablations, convergence, and model complexity. The provided passages identify metric and dataset layouts but do not include numerical table values.

  • DSC (%): The ablation study on the CHAOS dataset reports average results over three folds for different attention modules.Best and second-best performances are visually distinguished in the table.
  • DSC (%): The CHAOS comparison table reports average results over three folds for the proposed network and other state-of-the-art architectures.
  • DSC (%): The HVSMR 2016 comparison reports average results over five folds for the evaluated architectures.
  • DSC (%): Figure 8 depicts the evolution of mean validation DSC over time.
  • DSC (%): The BRATS 2018 comparison reports average results over three folds and identifies the best and second-best performances.
  • DSC (%): Model complexity is measured by the number of parameters for the evaluated models.
Loading 1906.02849v3…