Source-linked AI summary
Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks
Abhijit Guha Roy, Nassir Navab, Christian Wachinger
TL;DR
F-CNN segmentation architectures need ways to improve feature representations beyond spatial encoding and connectivity changes. This paper integrates channel, spatial, and joint squeeze & excitation blocks for adaptive feature-map recalibration, reporting consistent gains across segmentation applications with minimal added complexity.
Problem
The paper addresses limited attention to independently encoding spatial and channel-wise patterns in F-CNNs for fine-grained image segmentation.
Method
The paper integrates cSE, sSE, and scSE blocks into F-CNNs to recalibrate intermediate feature maps channel-wise, spatially, or jointly.
Results
The authors report consistent segmentation improvements across three F-CNN architectures and three segmentation applications, with SE blocks adding minimal model complexity.
Takeaways & Limitations
SE-based recalibration is presented as a generic concept for boosting F-CNN segmentation performance.
Abstract
from arXiv · showhide
In a wide range of semantic segmentation tasks, fully convolutional neural networks (F-CNNs) have been successfully leveraged to achieve state-of-the-art performance. Architectural innovations of F-CNNs have mainly been on improving spatial encoding or network connectivity to aid gradient flow. In this article, we aim towards an alternate direction of recalibrating the learned feature maps adaptively; boosting meaningful features while suppressing weak ones. The recalibration is achieved by simple computational blocks that can be easily integrated in F-CNNs architectures. We draw our inspiration from the recently proposed 'squeeze & excitation' (SE) modules for channel recalibration for image classification. Towards this end, we introduce three variants of SE modules for segmentation, (i) squeezing spatially and exciting channel-wise, (ii) squeezing channel-wise and exciting spatially and (iii) joint spatial and channel 'squeeze & excitation'. We effectively incorporate the proposed SE blocks in three state-of-the-art F-CNNs and demonstrate a consistent improvement of segmentation accuracy on three challenging benchmark datasets. Importantly, SE blocks only lead to a minimal increase in model complexity of about 1.5%, while the Dice score increases by 4-9% in the case of U-Net. Hence, we believe that SE blocks can be an integral part of future F-CNN architectures.
I. INTRODUCTION
The paper extends squeeze & excitation recalibration from image classification to F-CNN-based segmentation, introducing spatial and joint variants alongside channel-wise recalibration. These blocks are evaluated across multiple architectures and challenging medical segmentation applications with minimal added complexity.
- Motivation: SE blocks adaptively recalibrate intermediate feature maps by emphasizing useful features and suppressing weaker ones.The original channel SE block uses global average pooling to derive channel descriptors and rescale feature maps.
- Proposed blocks: The proposed sSE block squeezes channels and excites spatial locations, providing attention for fine-grained segmentation regions.The sSE design is presented as complementary to cSE because it adds spatial attention without changing the receptive field.
- Evaluation: SE blocks are integrated into state-of-the-art F-CNNs and tested on whole-brain, whole-body, and retinal-layer segmentation tasks.The experiments cover MRI brain scans, contrast-enhanced CT abdominal-organ labeling, and retinal segmentation applications.
- Proposed blocks: The scSE block combines cSE and sSE outputs to recalibrate feature maps jointly along channel and spatial dimensions.The contribution summary specifies element-wise max-out as the combination strategy.
- Evaluation: The work reports consistent segmentation improvements across three tasks and four F-CNN architectures with only a minimal increase in model complexity.The paper also describes the work as an extension adding details, broader experiments, and training-dynamics analysis.
II. METHODS
The method inserts SE blocks after intermediate F-CNN feature maps and recalibrates their activations. The cSE variant uses global spatial pooling and channel excitation to adaptively rescale channels.
- F-CNN formulation: An F-CNN maps an input image I to a segmentation map S through cascaded encoder and decoder functions.Encoder blocks are separated by max-pooling, while decoder blocks are separated by upsampling.
- SE integration: SE blocks are placed on intermediate output feature maps U to produce recalibrated maps Û.The paper defines U as the output of an encoder or decoder block and introduces three SE variants for recalibration.
- cSE: The cSE block applies global average pooling to U, producing a channel descriptor z that embeds global spatial information.The descriptor is transformed through fully connected layers and a ReLU before channel excitation.
- cSE: The excitation pathway uses a bottleneck parameter r and sigmoid activations to generate channel-wise scaling factors.The paper reports r = 2 as the best-performing bottleneck setting and uses the resulting values to rescale each channel.
- cSE: Each cSE output channel is multiplied by its learned sigmoid weight, adaptively suppressing less important channels and emphasizing important ones.The resulting channel-wise recalibration is illustrated in Fig. 1(a).
B. Channel Squeeze and Spatial Excitation Block (sSE)
The sSE block converts channel information at each spatial location into a spatial attention map. Sigmoid-weighted locations are then used to recalibrate the feature map.
- Spatial excitation: The sSE block squeezes feature map U across channels using a 1 × 1 convolution to produce a spatial projection q.Each q_i,j is a linearly combined representation of all channels at spatial location (i, j).
- Spatial excitation: A sigmoid transforms each spatial projection value into a [0, 1] rescaling weight.These weights represent the relative importance of individual spatial locations.
- Spatial excitation: The recalibrated sSE map multiplies every location’s feature vector by its corresponding spatial weight.This increases emphasis on relevant regions and reduces attention to irrelevant ones.
C. Spatial and Channel Squeeze & Excitation Block (scSE)
The scSE block combines channel and spatial recalibration, using complementary attention signals to make feature maps more informative in both dimensions. Several aggregation strategies combine the two recalibrated responses, with max-out favoring selectivity and concatenation increasing complexity.
- Complementary recalibration: cSE uses global spatial information to recalibrate channels, whereas sSE preserves the receptive field and acts as a spatial attention map.cSE's global average pooling provides whole-image receptive fields; sSE uses a 1 × 1 convolution for channel squeezing.
- Aggregation strategies: Max-out selects the larger cSE or sSE activation at each location and channel, enforcing element-wise competition between the two excitations.The resulting selectivity combines spatial and channel excitation.
- Aggregation strategies: Addition combines the cSE and sSE feature maps element-wise, giving equal importance to both recalibration methods.
- Aggregation strategies: Multiplication combines spatial and channel importance by multiplying corresponding activations, each scaled to [0, 1].
- Aggregation strategies: Concatenation preserves information from both responses but doubles output channels, requiring subsequent convolutional layers to process more channels.
D. Position of SE Block in F-CNNs
The paper evaluates six possible positions for inserting SE blocks in encoder/decoder F-CNNs, including a highlighted recommended configuration. These positions span individual stages and combinations of stages.
- Candidate positions: The six candidate positions place SE blocks after encoder blocks, decoder blocks, the bottleneck, the classifier, encoder and decoder blocks, or all listed stages.
- Architecture illustration: Fig. 2 illustrates the reference encoder/decoder F-CNN and the defined SE-block positions P1 through P6.
E. Model Complexity
SE blocks add relatively few parameters to the evaluated F-CNNs. The paper illustrates this with U-Net and notes that sSE contributes especially little additional complexity.
- Parameter overhead: 1.5% is the approximate model-complexity increase from adding scSE to the U-Net used in the experiments.The U-Net has about 2.1 × 10^6 parameters, while scSE adds 3.3 × 10^4 parameters.
- Parameter overhead: sSE causes only about a 0.01% increase in model complexity, whereas most added complexity comes from cSE.
- Compared architectures: The complexity analysis reports model parameters, additional parameters, and percentage increases for FC-DenseNet, SD-Net, and U-Net.
- Evaluation setting: The experiments use three datasets spanning brain MRI, whole-body contrast-enhanced CT, and retinal-layer segmentation, evaluated with Dice score.The MALC brain dataset has 15 training and 15 testing scans; the Visceral setup trains on 65 silver-corpus scans and tests on 20 gold-corpus scans.
B. Model Architecture and Learning
The experiments use encoder/decoder F-CNNs with standardized training settings and examine SE aggregation strategies through quantitative and qualitative analyses. For SD-Net on MALC, max-out and concatenation provide the best reported aggregation performance, with max-out offering the best complexity trade-off.
- Model architecture: The evaluated architectures use four encoder blocks, a bottleneck, four decoder blocks, and 64 output feature maps per block.U-Net uses padded 3 × 3 convolutions with ReLU, while SD-Net uses a 7 × 7 convolution with batch normalization and ReLU.
- Learning setup: Training uses 2D slice-wise processing, median-frequency-balanced logistic loss, scheduled learning-rate reduction, momentum 0.95, weight decay 10^-4, and batch size 4.
- Evaluation design: The study evaluates quantitative and qualitative performance, aggregation schemes, SE-block positions, and spatial-excitation dynamics during training.
- Aggregation results: For SD-Net on MALC, max-out and concatenation provide the best performance among the tested sSE/cSE aggregation strategies.
- Aggregation results: Max-out offers the best trade-off between performance and model complexity because concatenation increases the number of output channels.The paper attributes max-out's advantage to element-wise selectivity from competing excitations.
B. Position of SE Blocks
The study evaluates where to place scSE blocks, how to configure their channel bottleneck, and whether recalibrated or uncalibrated features should feed skip connections.
- scSE blocks improved segmentation quality at every tested position in SD-Net on MALC.
- The experiments place SE blocks after every encoder and decoder in subsequent experiments.
- The cSE bottleneck hyper-parameter r was evaluated at 2, 4, 8, and 16 on MALC using SD-Net.
- Both tested relationships between skip connections and scSE blocks were equally effective, so either configuration could be used.
E. Whole Brain Segmentation
SE blocks consistently improve whole-brain segmentation across F-CNN architectures, with spatially informed variants especially helping difficult small structures.
- Adding any SE block significantly increased mean Dice scores over the reference architectures on 15 MALC testing images (p ≤0.001, Wilcoxon signed-rank).
- scSE increased global Dice by around 4% for FC-DenseNets and around 8-9% for U-Net and SD-Net.
- SE blocks corrected failures to segment very small structures, including the 3rd ventricle and amygdala, in U-Net and SD-Net.
- sSE and scSE outperformed the FC-DenseNet reference for almost all left-hemisphere structures, while cSE degraded performance on several challenging structures.
- scSE improved an under-segmented left putamen region in qualitative FC-DenseNet results.
G. Retinal Layer and Fluid Segmentation
Adding scSE blocks to ReLayNet improves retinal layer and fluid segmentation, including difficult small-fluid and artifact-containing regions.
- Adding scSE blocks increased Dice scores by 1-2% in most retinal segmentation classes.
- The retinal experiments added scSE blocks after every ReLayNet encoder and decoder block while keeping the original training procedure consistent.
- scSE detected two visually imperceptible fluid pools that normal ReLayNet failed to detect.
- scSE rectified a discontinuous ReLayNet prediction caused by a bright imaging artifact.
I. Dynamics of Spatial Excitation
Training dynamics differ across spatial excitation layers: shallow sE-1 maps stabilize early, while deeper sD-4 maps change across epochs and increasingly focus on anatomical structures and boundaries.
- Shallow spatial excitation: sE-1 activation maps mainly distinguish foreground from background and are not class-specific because the layer encodes lower-order features.In brain MRI, skull is also highlighted as a spatial reference for locating brain tissues.
- Shallow spatial excitation: sE-1 maps change little across epochs, indicating that this shallow layer is learned largely during the first epoch.
- Deeper spatial excitation: sD-4 activation maps show more dynamic behavior, with clear changes in spatial activation during training epochs.
- Deeper spatial excitation: Across later epochs, sD-4 maps increasingly resemble the input scan while focusing on anatomical boundaries.Brain MRI maps shift emphasis between hemispheres, highlight cerebellar structures from the fifth epoch, and become more balanced across hemispheres at other epochs.
- Overall implication: The broader validation finds that SE recalibration consistently improves segmentation performance across three F-CNN architectures and three segmentation applications.This training-dynamics analysis accompanies the reported performance improvements and examines spatial excitation during learning.