PaperlayerView exact revision on arXiv

Source-linked AI summary

GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond

Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, Han Hu

arXiv:1904.11492v1cs.CVcs.AIcs.LG

TL;DR

Long-range dependency modeling is computationally costly, and the paper finds that NLNet’s supposedly query-specific contexts are nearly query-independent. It simplifies NLNet, unifies it with SENet in a three-step framework, and develops GCNet, which generally outperforms both while adding little computation.

  • Problem

    Deep convolutional stacking is inefficient and difficult to optimize for modeling long-range dependencies, while NLNet’s query-specific computation may be redundant because its attention maps are nearly identical across query positions.

  • Method

    The paper simplifies NLNet with query-independent attention, unifies it with SENet in a three-step framework, and instantiates the framework as a lightweight GC block applied across backbone layers.

  • Results

    GCNet generally outperforms simplified NLNet and SENet across major recognition benchmarks, with gains across COCO detection and segmentation, ImageNet classification, and Kinetics action recognition.

  • Takeaways & Limitations

    A query-independent global-context design can preserve NLNet-like recognition accuracy with substantially lower computation and support deployment across multiple backbone layers.

Abstract

from arXiv · show

The Non-Local Network (NLNet) presents a pioneering approach for capturing long-range dependencies, via aggregating query-specific global context to each query position. However, through a rigorous empirical analysis, we have found that the global contexts modeled by non-local network are almost the same for different query positions within an image. In this paper, we take advantage of this finding to create a simplified network based on a query-independent formulation, which maintains the accuracy of NLNet but with significantly less computation. We further observe that this simplified design shares similar structure with Squeeze-Excitation Network (SENet). Hence we unify them into a three-step general framework for global context modeling. Within the general framework, we design a better instantiation, called the global context (GC) block, which is lightweight and can effectively model the global context. The lightweight property allows us to apply it for multiple layers in a backbone network to construct a global context network (GCNet), which generally outperforms both simplified NLNet and SENet on major benchmarks for various recognition tasks. The code and configurations are released at https://github.com/xvjiarui/GCNet.

1. Introduction

Long-range dependencies are important for visual recognition, but conventional convolutional stacking is inefficient and difficult to optimize. The paper finds that non-local attention maps are nearly query-independent, then uses this observation to design a cheaper global-context framework and GCNet.

  • Motivation: Long-range dependency modeling benefits image and video classification, object detection, and segmentation, but deeply stacking convolutions is computationally inefficient and difficult to optimize.Convolutions primarily model local neighborhoods, making distant-message delivery challenging.
  • Observation: Non-local attention maps for different query positions are almost identical, indicating that the learned dependency is largely query-independent.The observation is supported by visualizations and statistical analysis showing very small distances between attention maps.
  • Simplification: The simplified non-local block uses one query-independent attention map for all query positions, substantially reducing computation while maintaining nearly the original accuracy.It adds the same globally aggregated features to every query position.
  • Framework: The simplified non-local and SE blocks are unified as a three-step global-context framework that separates context modeling, transformation, and feature fusion.Their implementations differ in aggregation strategy, transformation, and strengthening functions.
  • GC block: The GC block combines global attention pooling and addition from the simplified non-local block with the two-layer bottleneck transform from SE, outperforming both on multiple recognition tasks.This design is lightweight enough to apply across multiple backbone layers.
  • Results: GCNet outperforms NLNet and SENet on COCO while increasing FLOPs by only 0.07%, and improves over task-specific baselines on COCO, ImageNet, and Kinetics with less than 0.26% extra computation.On COCO, gains over NLNet and SENet are reported for APbox and APmask; broader gains are reported for detection, segmentation, classification, and action recognition.

2. Related Work

Related work addresses long-range dependencies through self-attention or query-independent global context, while architectural improvements also include channel-wise recalibration. GCNet combines lightweight computation with addition-based global-context fusion and reports superior performance across vision tasks.

  • Deep architectures: Deep architecture research improves basic components through designs such as group convolution, deformable convolution, and channel-wise rescaling.Squeeze-Excitation Networks explicitly model channel dependencies through channel-wise rescaling.
  • Long-range dependency modeling: Long-range dependency methods are categorized into self-attention approaches for pairwise relations and approaches modeling query-independent global context.The paper places NLNet among self-attention methods and SENet-like methods among global-context approaches.
  • Self-attention methods: NLNet uses pixel-level pairwise self-attention, but the paper reports that it actually learns query-independent attention maps, making its pairwise computation wasteful.CCNet accelerates NLNet by stacking two criss-cross blocks for semantic segmentation.
  • Global-context methods: SENet, GENet, and PSANet use global context to rescale channels, while CBAM rescales spatial positions and channels; the paper characterizes rescaling fusion as insufficiently effective for global-context modeling.This limitation motivates a different fusion strategy.
  • GCNet: GCNet uses addition fusion to retain effective global-context modeling while remaining lightweight, and reports better performance than NLNet and SENet across major recognition benchmarks.The paper frames this as combining NLNet’s addition-based modeling with SENet’s lightweight property.

3. Analysis on Non-local Networks

The analysis revisits non-local blocks, their query-specific attention formulation, and the empirical behavior of attention maps and global context features. It finds that these contexts are nearly identical across query positions despite position-discriminative inputs.

  • Non-local block: Non-local blocks strengthen each query position by aggregating features from all positions using query-specific attention weights.The output adds the weighted aggregate to the features at each query position.
  • Non-local block: The Embedded Gaussian instantiation computes normalized pairwise relationships in an embedding space using learned transformations.Its attention weights use transformed query and key features before normalization.
  • Non-local block: The non-local block has quadratic time and space complexity in the number of feature-map positions because it computes attention maps for every query position.The number of positions is Np, such as H·W for an image or H·W·T for a video.
  • Visualization: In COCO visualizations, attention maps for three different query positions in each image are almost the same.The visualizations use red points for query positions and heatmaps for their query-specific attention maps.
  • Statistical analysis: Statistical analysis compares cosine distances and Jensen-Shannon divergences among inputs, outputs before fusion, and attention maps.Attention maps are treated as discrete probability distributions for Jensen-Shannon divergence when their weights sum to one.
  • Statistical analysis: Large input-feature distances but small output and attention-map distances show that non-local global contexts are nearly identical across query positions.This pattern is reported across two standard tasks and all four non-local instantiations in the analysis.

4. Method

The method simplifies non-local modeling through shared query-independent global context, then unifies it with squeeze-excitation into a three-step framework. The GC block combines global attention pooling, a bottleneck transform, and additive fusion, enabling lightweight deployment across multiple backbone layers.

  • Simplifying the Non-local Block: A query-independent attention map is shared across positions, preserving comparable performance to non-local blocks with significantly lower FLOPs.The simplified block aggregates a weighted average of features from all positions and adds the resulting global context to every query position.
  • Global Context Modeling Framework: The simplified non-local block consists of global attention pooling, 1x1-convolution feature transformation, and broadcast addition of context to each position.The pooling uses a 1x1 convolution and softmax to obtain attention weights before feature aggregation.
  • Global Context Modeling Framework: Squeeze-excitation is another framework instantiation, using global average pooling, a bottleneck excitation transform, and element-wise multiplication for channel recalibration.Global average pooling is a special case of global attention pooling.
  • Global Context Block: The GC block combines global attention pooling, a bottleneck transform, and broadcast element-wise addition to model long-range dependencies with lightweight computation.The bottleneck reduces transform parameters from C·C to 2·C·C/r, while layer normalization is added before ReLU to ease optimization and act as a regularizer.
  • Global Context Block: 0.26% relative increase in computation accompanies GC-ResNet-50 when GC blocks are added to all c3+c4+c5 layers, while adding ∼2.52M parameters.Computation rises from ∼3.86 GFLOPs to ∼3.87 GFLOPs, and parameters increase by ∼9.86% over ResNet-50.
  • Global Context Network: GC blocks are applied to image recognition, object detection/segmentation, and action recognition, with significant improvements observed across all three tasks.Their lower computation cost than non-local blocks allows use in multiple layers.

5. Experiments

Experiments across COCO, ImageNet, and Kinetics show that GCNet generally matches or exceeds NLNet and SENet while using modest computational overhead. Ablations support lightweight multi-layer deployment, attention-plus-addition fusion, and bottleneck designs with reduced parameters.

  • GCNet generally outperforms non-local networks with lower FLOPs and squeeze-excitation networks with comparable FLOPs across object detection, segmentation, classification, and action recognition.
  • Ablation Study: Adding GC blocks to all residual stages improves COCO performance by 1.1% on APbbox and 0.9% on APmask with a slight increase in FLOPs and parameters.
  • Ablation Study: Reducing the bottleneck ratio from 32 to 4 improves performance by 0.8% on APbbox and 0.5% on APmask, while LayerNorm achieves performance similar to the uncompressed transform with fewer parameters.
  • Image Classification on ImageNet: GC blocks perform slightly better than NL and SNL on ImageNet with fewer parameters and less computation; all-stage insertion raises top-1 accuracy by 0.82% with a 0.26% relative FLOPs increase.
  • Ablation Study: Attention pooling with addition outperforms vanilla average pooling with scale by 0.44% on ImageNet top-1 accuracy with almost identical parameters and FLOPs.

6. Conclusion

The paper argues that NLNet effectively learns query-independent rather than query-specific global context, motivating a simplified formulation and a unified global-context framework. Its GC block enables lightweight multi-layer deployment, and GCNet generally outperforms simplified NLNet and SENet across recognition benchmarks.

  • NLNet intends to model query-specific global context but empirically models query-independent context, motivating its simplification.
  • The GC block is a lightweight framework instantiation that effectively models long-range dependency and can be applied across multiple backbone layers.
  • GCNet generally outperforms simplified NLNet and SENet on major benchmarks for various recognition tasks.
Loading 1904.11492v1…