Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images

Jo Schlemper; Ozan Oktay; Michiel Schaap; Mattias Heinrich; Bernhard Kainz; Ben Glocker; Daniel Rueckert

Source-linked AI summary

Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images

Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, Daniel Rueckert

arXiv:1808.08114v2Aug 22, 2018cs.CV

TL;DR

Medical image models must handle variable target structures without inefficient external localisation stages. The paper introduces trainable attention gates that focus standard CNNs on salient regions, then evaluates them in ultrasound classification and CT segmentation. Across these tasks, attention-gated models improve prediction performance while avoiding multiple CNN models and preserving computational efficiency.

Problem
Variable-shape medical targets often require cascaded CNNs and external localisation models, which repeatedly extract features and consume additional computational resources.
Method
The paper integrates grid-based soft attention gates into standard CNN architectures to learn task-specific salient regions and suppress irrelevant feature activations.
Results
Attention-gated models consistently improve prediction accuracy across fetal ultrasound scan-plane detection and CT segmentation datasets and training sizes without requiring multiple CNN models.
Takeaways & Limitations
Attention gates can provide localisation and task-adaptive feature selection within standard medical-image CNNs, while also producing visualisable attention maps for interpreting predictions.
Takeaways & Limitations
Quantitative comparisons between Attention 3D-U-Net and state-of-the-art 2D cascaded models may be insufficient for a final conclusion.

Abstract

from arXiv · show

We propose a novel attention gate (AG) model for medical image analysis that automatically learns to focus on target structures of varying shapes and sizes. Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules when using convolutional neural networks (CNNs). AGs can be easily integrated into standard CNN models such as VGG or U-Net architectures with minimal computational overhead while increasing the model sensitivity and prediction accuracy. The proposed AG models are evaluated on a variety of tasks, including medical image classification and segmentation. For classification, we demonstrate the use case of AGs in scan plane detection for fetal ultrasound screening. We show that the proposed attention mechanism can provide efficient object localisation while improving the overall prediction performance by reducing false positives. For segmentation, the proposed architecture is evaluated on two large 3D CT abdominal datasets with manual annotations for multiple organs. Experimental results show that AG models consistently improve the prediction performance of the base architectures across different datasets and training sizes while preserving computational efficiency. Moreover, AGs guide the model activations to be focused around salient regions, which provides better insights into how model predictions are made. The source code for the proposed AG models is publicly available.

1. Introduction

Medical image analysis needs accurate automation because manual labelling is tedious, while variable target anatomy often prompts inefficient cascaded CNNs. The paper proposes grid-based attention gates that focus CNNs on salient local regions and evaluates them for ultrasound classification and CT segmentation.

Manual medical-image labelling is tedious and error-prone, motivating fast automated extraction of quantitative measurements.
Cascaded CNNs localise regions of interest before dense prediction but repeatedly extract similar low-level features, increasing computational and parameter costs.
Attention gates learn task-specific local attention coefficients that suppress irrelevant activations and highlight salient structures without additional supervision.
The proposed gates integrate into standard CNNs, including VGG- and U-Net-based architectures, reducing the need for separate localisation and prediction models.
Experiments cover real-time fetal ultrasound scan-plane classification and pancreas or multi-organ CT segmentation across two datasets and different training sizes.
Grid-based gating supports local-region specificity, dense predictions, visualisable fine-scale attention maps, and consistent segmentation improvements with minimal computational overhead.

2. Methodology

The method integrates attention gates into CNNs to select task-relevant regions using local activations and coarser contextual signals. The gates scale feature responses, filter U-Net skip connections, and aggregate attended features for classification.

Motivation: Attention gates are designed to reduce false positives from irrelevant regions in CNN-based medical image analysis.They address the difficulty of small, shape-variable targets while retaining CNN processing.
Attention Gate Module: The gate scales each feature vector by its corresponding coefficient, preserving activations judged relevant to the task.The output feature map is formed by coefficient-wise scaling of the input features.
Attention Gate Module: Each attention gate combines a feature-map vector with a coarser global feature vector to compute local attention coefficients.The gating signal provides contextual information for disambiguating task-irrelevant content.
Attention Gate Module: Additive attention is used to compute coefficients, with sigmoid or softmax serving as normalization functions.The proposed gate uses learned linear transformations, bias terms, and channel-wise 1 × 1 × 1 convolutions.
Attention U-Net: In Attention U-Net, gates filter skip-connection features using contextual information from coarser scales before feature concatenation.The encoder progressively downsamples by a factor of 2 at each scale, while gating selects salient propagated features.
AG-Sononet: In AG-Sononet, attention units are inserted at layers 11 and 14, and attended maps are aggregated into vectors for final prediction.The architecture applies attention before pooling and combines scale-specific features through fully connected layers.

3. Experiments and Results

The study evaluates modular attention-gated fully convolutional models on abdominal CT multi-label segmentation and 2D ultrasound scan-plane classification. Models are compared with standard 3D U-Net and Sononet across prediction, capacity, computational, and memory measures.

Evaluation Tasks: The experiments test attention-gated models on challenging abdominal CT segmentation and 2D ultrasound image-plane classification tasks.The tasks involve pancreas boundary delineation with shape variability and poor tissue contrast, plus variable-quality ultrasound classification.
Model Evaluation: The proposed AG model is modular and independent of application type, supporting both pixel-level and image-level classification tasks.This evaluation compares the models against standard 3D U-Net and Sononet.

3.1. Evaluation Datasets

The evaluation uses two annotated 3D abdominal CT datasets and a 2D fetal ultrasound dataset spanning standard scan planes and background images. The datasets cover pancreas and multi-organ segmentation as well as scan-plane classification.

CT Datasets: The CT evaluation uses CT-150, containing 150 abdominal 3D scans with pancreas, liver, and spleen boundaries annotated and clinician-verified.CT-150 was also used to benchmark U-Net pancreas segmentation.
CT Datasets: CT-82 contains 82 contrast-enhanced 3D CT scans with slice-by-slice manual pancreas annotations from the publicly available NIH-TCIA dataset.It is commonly used to benchmark CT pancreas segmentation frameworks.
Ultrasound Dataset: The ultrasound dataset contains 2694 examinations from volunteers at 18–22 weeks of gestation, covering 13 standard scan-plane types and background.The scan-plane labels follow the UK NHS fetal anomaly screening programme.
Ultrasound Dataset: Ultrasound images include substantial background anatomy, and images were cropped to 208 × 272 to prevent learning surrounding screen annotations.The background was labelled separately from the standard scan planes.

3.2. Model Training and Implementation Details

Training uses class-imbalance handling, normalization, augmentation, and optimization procedures shared across attention and baseline networks. The experiments compare AG-Sononet and U-Net variants across capacities and report implementation details for CT segmentation.

Weighted sampling balances foreground and background sampling for ultrasound, while Sorensen-Dice loss addresses segmentation class imbalance.
Both attention and baseline networks use batch normalization, deep supervision, affine transformations, axial flips, random crops, and intensity normalization.Intensity values are scaled to N(0, 1).
AG-Sononet is initialized from a partially trained Sononet and compared across models with initial feature counts of 8, 16, and 32.For U-Net and Attention U-Net, F1 = 8 and doubles after each max-pooling operation.
Table 1 reports CT-150 multi-class segmentation using DSC and pancreas-only S2S distances across Attention U-Net and U-Net training/testing splits.Forward-pass inference time is measured for 160 × 160 × 96 input tensors.

3.3. 3D-CT Abdominal Image Segmentation Results

Attention U-Net improves pancreas segmentation over U-Net across CT-150 capacity and training-size comparisons and transfers consistently across CT-82 evaluation settings. The method also compares favorably with multi-model and state-of-the-art frameworks while avoiding cascaded localization models.

AGs increase pancreas recall over standard U-Net on CT-150, with significant improvement at full and reduced training sizes.The reported p-values are .005 for the full-size experiment and .01 for the reduced-training experiment.
Reduced training data lowers spleen DSC for both models, but the proposed framework shows a less significant drop; kidney accuracy remains similar.The kidney result is attributed to higher tissue contrast.
Adding 8% capacity to standard U-Net improves DSC by 2-3%, while AGs contribute more than uniformly increasing capacity across layers (p = .007).
Across CT-82 transfer, fine-tuning, and from-scratch settings, AGs improve pancreas accuracy and lower surface distances (p = .03), with increased pancreas recall (p = .09).
The 3D Attention U-Net performs similarly to state-of-the-art methods and significantly improves over single-model frameworks without requiring multiple CNNs for localization and segmentation.Five-fold CT-82 cross-validation achieved 81.48 ± 6.23 DSC for pancreas labels.

3.4. 2D Fetal Ultrasound Image Classification Results

AG-Sononet is evaluated for fetal ultrasound scan-plane classification using subject-level dataset splits and standard classification metrics. It improves over Sononet across model capacities, particularly by increasing precision and reducing false positives.

The ultrasound dataset is split by subject into 122,233 training, 30,553 validation, and 38,243 testing frames.
Evaluation reports macro-averaged precision, recall, F1, overall accuracy, parameter count, and execution speed.
AG-Sononet improves results over Sononet at all capacity levels and achieves higher precision by suppressing background noise and emphasizing class-specific features.
For AG-Sononet-8, precision increases by around 5% for kidney, profile, and spine classes, while 4CH and 3VV improve by an average of 3% (p < 0.05).

3.5. Attention Map Analysis

Attention coefficients evolve from broadly distributed activation to localized organ-focused responses during training. Coarser gates provide rough organ outlines, finer gates refine them, and different gates specialize in organ subsets or discriminative regions.

U-Net attention gates initially pass features across spatial locations, then localize toward target organ boundaries as training progresses.
Across epochs 3, 6, 10, 60, and 150, attention coefficients gradually focus on the pancreas, kidney, and spleen.The coefficients are visualized on sagittal and axial planes from testing CT scans.
Coarser-scale gates produce rough organ outlines that are refined at finer resolutions, while multiple gates learn to focus on particular organ subsets.
AG-Sononet produces localization maps with almost no additional computational cost and consistently focuses on the object of interest across subjects.The maps outline discriminative regions and support bounding-box generation.

4. Discussion

The discussion examines soft-attention design choices, training stability, and the limits of comparing 3D attention models with 2D cascaded models.

Attention design: Soft-max attention maps allow more information to flow but introduce gradient saturation that makes training non-trivial.
Training: Attention-map aggregation across scales affects both attention learning and network performance.Applying losses at each scale encourages the network to attend at each scale.
Related architectures: Residual connections did not significantly improve segmentation performance, although future work may examine whether they improve training behaviour.
Comparison limits: Quantitative comparisons between Attention 3D-Unet and 2D cascaded models are insufficient for a final conclusion because the models use different spatial information.The 3D approach uses contextual information across dimensions, whereas 2D models retain higher-resolution axial information without downsampling.

5. Conclusion

The conclusion presents attention gates as modular components for CNN classification and segmentation that learn salient regions without external localization models. Experiments report improved scan-plane detection and CT organ localization, while attention maps also support fine-grained localization and potential explainability.

Attention gates are modular and can be incorporated into existing segmentation and classification architectures.
AGs implicitly highlight salient image regions, reducing the necessity for external object-localization models.In classification, they also support task-adaptive feature pooling.
Scan-plane detection: AGs improved fetal-ultrasound scan-plane detection, especially precision, while using fewer parameters.The gating signal identifies useful local and global information for classification.
CT segmentation: CT segmentation experiments found AGs beneficial for tissue and organ identification and localization, particularly for variable small organs such as the pancreas.Similar behaviour was observed in image-classification tasks.
Fine-grained attention maps can be exploited for object localization and may support explainable deep learning in medical imaging.

Appendix - Weakly Supervised Object Localisation (WSL)

The WSL appendix derives object locations from attention maps without backpropagation, then evaluates localization with standard and bias-adjusted correctness measures. The attention outlines classification-relevant regions, which may not exactly match the object of interest.

Object locations are generated by blurring attention maps, thresholding low activations, finding connected components, and selecting overlapping components for bounding boxes.The heuristic can be executed efficiently because it does not require backpropagation.
Attention maps outline salient regions used for classification but do not necessarily coincide with the object of interest.Some object regions may also appear in background frames before the ideal plane is reached.
Relative Correctness adjusts localization evaluation using IOU > 0.5×max(IOUclass) to account for classification-related bias.The appendix reports high results under this metric, indicating detection of relevant nearby object features.