GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond

Motivation

• Long-range dependency modeling benefits image and video classification, object detection, and segmentation, but deeply stacking convolutions is computationally inefficient and difficult to optimize.Convolutions primarily model local neighborhoods, making distant-message delivery challenging.Source: §1 · p. 1

Observation

• Non-local attention maps for different query positions are almost identical, indicating that the learned dependency is largely query-independent.The observation is supported by visualizations and statistical analysis showing very small distances between attention maps.Source: §1 · p. 1, Figure 1 · p. 1

Simplification

• The simplified non-local block uses one query-independent attention map for all query positions, substantially reducing computation while maintaining nearly the original accuracy.It adds the same globally aggregated features to every query position.Source: §1 · pp. 1–2

Framework

• The simplified non-local and SE blocks are unified as a three-step global-context framework that separates context modeling, transformation, and feature fusion.Their implementations differ in aggregation strategy, transformation, and strengthening functions.Source: §1 · p. 2

GC block

• The GC block combines global attention pooling and addition from the simplified non-local block with the two-layer bottleneck transform from SE, outperforming both on multiple recognition tasks.This design is lightweight enough to apply across multiple backbone layers.Source: §1 · p. 2

Results

• GCNet outperforms NLNet and SENet on COCO while increasing FLOPs by only 0.07%, and improves over task-specific baselines on COCO, ImageNet, and Kinetics with less than 0.26% extra computation.On COCO, gains over NLNet and SENet are reported for APbox and APmask; broader gains are reported for detection, segmentation, classification, and action recognition.Source: §1 · p. 2

Deep architectures

• Deep architecture research improves basic components through designs such as group convolution, deformable convolution, and channel-wise rescaling.Squeeze-Excitation Networks explicitly model channel dependencies through channel-wise rescaling.Source: §2 · p. 2

Long-range dependency modeling

• Long-range dependency methods are categorized into self-attention approaches for pairwise relations and approaches modeling query-independent global context.The paper places NLNet among self-attention methods and SENet-like methods among global-context approaches.Source: §2 · p. 2

Self-attention methods

• NLNet uses pixel-level pairwise self-attention, but the paper reports that it actually learns query-independent attention maps, making its pairwise computation wasteful.CCNet accelerates NLNet by stacking two criss-cross blocks for semantic segmentation.Source: §2 · p. 2

Global-context methods

• SENet, GENet, and PSANet use global context to rescale channels, while CBAM rescales spatial positions and channels; the paper characterizes rescaling fusion as insufficiently effective for global-context modeling.This limitation motivates a different fusion strategy.Source: §2 · p. 2

GCNet

• GCNet uses addition fusion to retain effective global-context modeling while remaining lightweight, and reports better performance than NLNet and SENet across major recognition benchmarks.The paper frames this as combining NLNet’s addition-based modeling with SENet’s lightweight property.Source: §2 · p. 2

Non-local block

• Non-local blocks strengthen each query position by aggregating features from all positions using query-specific attention weights.The output adds the weighted aggregate to the features at each query position.Source: §3.1 · p. 3
• The Embedded Gaussian instantiation computes normalized pairwise relationships in an embedding space using learned transformations.Its attention weights use transformed query and key features before normalization.Source: §3.1 · p. 3
• The non-local block has quadratic time and space complexity in the number of feature-map positions because it computes attention maps for every query position.The number of positions is Np, such as H·W for an image or H·W·T for a video.Source: §3.1 · p. 3

Visualization

• In COCO visualizations, attention maps for three different query positions in each image are almost the same.The visualizations use red points for query positions and heatmaps for their query-specific attention maps.Source: Figure 2 · p. 3, §3.2 · p. 4

Statistical analysis

• Statistical analysis compares cosine distances and Jensen-Shannon divergences among inputs, outputs before fusion, and attention maps.Attention maps are treated as discrete probability distributions for Jensen-Shannon divergence when their weights sum to one.Source: §3.2 · p. 4, Table 1 · p. 4
• Large input-feature distances but small output and attention-map distances show that non-local global contexts are nearly identical across query positions.This pattern is reported across two standard tasks and all four non-local instantiations in the analysis.Source: §3.2 · p. 4

Simplifying the Non-local Block

• A query-independent attention map is shared across positions, preserving comparable performance to non-local blocks with significantly lower FLOPs.The simplified block aggregates a weighted average of features from all positions and adds the resulting global context to every query position.Source: §4.1 · pp. 4–5

Global Context Modeling Framework

• The simplified non-local block consists of global attention pooling, 1x1-convolution feature transformation, and broadcast addition of context to each position.The pooling uses a 1x1 convolution and softmax to obtain attention weights before feature aggregation.Source: §4.2 · p. 5
• Squeeze-excitation is another framework instantiation, using global average pooling, a bottleneck excitation transform, and element-wise multiplication for channel recalibration.Global average pooling is a special case of global attention pooling.Source: §4.2 · p. 5, §4.3 · p. 6

Global Context Block

• The GC block combines global attention pooling, a bottleneck transform, and broadcast element-wise addition to model long-range dependencies with lightweight computation.The bottleneck reduces transform parameters from C·C to 2·C·C/r, while layer normalization is added before ReLU to ease optimization and act as a regularizer.Source: §4.3 · pp. 5–6
• 0.26% relative increase in computation accompanies GC-ResNet-50 when GC blocks are added to all c3+c4+c5 layers, while adding ∼2.52M parameters.Computation rises from ∼3.86 GFLOPs to ∼3.87 GFLOPs, and parameters increase by ∼9.86% over ResNet-50.Source: §4.3 · p. 6

Global Context Network

• GC blocks are applied to image recognition, object detection/segmentation, and action recognition, with significant improvements observed across all three tasks.Their lower computation cost than non-local blocks allows use in multiple layers.Source: §4.3 · p. 6

• GCNet generally outperforms non-local networks with lower FLOPs and squeeze-excitation networks with comparable FLOPs across object detection, segmentation, classification, and action recognition.Source: §5 · p. 6, §5.2 · p. 8, §5.3 · p. 9

Ablation Study

• Adding GC blocks to all residual stages improves COCO performance by 1.1% on APbbox and 0.9% on APmask with a slight increase in FLOPs and parameters.Source: §5.1.1 · p. 7
• Reducing the bottleneck ratio from 32 to 4 improves performance by 0.8% on APbbox and 0.5% on APmask, while LayerNorm achieves performance similar to the uncompressed transform with fewer parameters.Source: §5.1.1 · p. 7

Image Classification on ImageNet

• GC blocks perform slightly better than NL and SNL on ImageNet with fewer parameters and less computation; all-stage insertion raises top-1 accuracy by 0.82% with a 0.26% relative FLOPs increase.Source: §5.2 · p. 8

Ablation Study

• Attention pooling with addition outperforms vanilla average pooling with scale by 0.44% on ImageNet top-1 accuracy with almost identical parameters and FLOPs.Source: §5.1.1 · p. 7, §5.2 · p. 8

• NLNet intends to model query-specific global context but empirically models query-independent context, motivating its simplification.Source: §6 · p. 9
• The GC block is a lightweight framework instantiation that effectively models long-range dependency and can be applied across multiple backbone layers.Source: §6 · p. 9
• GCNet generally outperforms simplified NLNet and SENet on major benchmarks for various recognition tasks.Source: §6 · p. 9