PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Context Networks

Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, Han Hu

arXiv:2012.13375v1cs.CV

TL;DR

NLNet aims to model long-range image dependencies with query-specific global context, but the paper finds that these contexts are nearly the same across query positions. It therefore introduces lightweight query-independent GC blocks and GCNet, which generally outperform NLNet and other baselines across recognition tasks with minimal added computation.

  • Problem

    NLNet models query-specific global context, but empirical analysis finds that its contexts are nearly identical across query positions, challenging the assumed need for query-specific pairwise modeling.

  • Method

    The paper replaces query-specific attention with query-independent global-context aggregation, adds a two-layer bottleneck transform, and applies GC blocks across backbone layers to form GCNet.

  • Results

    GCNet generally outperforms NLNet and other baselines across major recognition benchmarks, including COCO, Cityscapes, ImageNet, and Kinetics, with less than a 0.26% computation increase.

  • Takeaways & Limitations

    The results support lightweight global-context modeling as an effective alternative to query-specific non-local modeling for the evaluated visual recognition tasks.

  • Takeaways & Limitations

    The analysis and experiments mainly target pixel-pixel relation modeling in visual recognition, leaving broader self-attention applications for future study.

Abstract

from arXiv · show

The Non-Local Network (NLNet) presents a pioneering approach for capturing long-range dependencies within an image, via aggregating query-specific global context to each query position. However, through a rigorous empirical analysis, we have found that the global contexts modeled by the non-local network are almost the same for different query positions. In this paper, we take advantage of this finding to create a simplified network based on a query-independent formulation, which maintains the accuracy of NLNet but with significantly less computation. We further replace the one-layer transformation function of the non-local block by a two-layer bottleneck, which further reduces the parameter number considerably. The resulting network element, called the global context (GC) block, effectively models global context in a lightweight manner, allowing it to be applied at multiple layers of a backbone network to form a global context network (GCNet). Experiments show that GCNet generally outperforms NLNet on major benchmarks for various recognition tasks. The code and network configurations are available at https://github.com/xvjiarui/GCNet.

1 INTRODUCTION

The paper finds that NLNet’s attention maps are nearly query-independent, motivating a simpler global-context formulation that reduces computation while preserving accuracy. The resulting GCNet applies lightweight GC blocks broadly and generally improves recognition benchmarks.

  • 1 INTRODUCTION: NLNet attention maps for different query positions are almost identical, indicating that the learned dependency is largely query-independent across COCO detection, ImageNet classification, and Kinetics recognition.The observation is supported by visualizations and statistical analyses of attention-map differences.
  • 1 INTRODUCTION: The proposed simplification uses one query-independent attention map to aggregate features for all query positions, substantially reducing computation with almost no accuracy loss.The output retains the weighted aggregation structure while removing query-specific attention computation.
  • 1 INTRODUCTION: The GC block combines context modeling, channel-wise feature transformation, and fusion, replacing NLNet’s one-layer transform with a two-layer bottleneck.Its lightweight computation allows deployment across multiple residual blocks rather than only one or a few layers.
  • 1 INTRODUCTION: 1.9% higher APbox and 1.5% higher APmask than NLNet are achieved on COCO with only a 0.07% relative FLOPs increase.The comparison is reported for COCO object detection and instance segmentation.

2 RELATED WORK

The related work positions GCNet as a general-purpose architecture for long-range modeling, distinct from locally stacked networks and prior pairwise or context-fusion approaches. It also frames GCNet’s analysis as relevant to broader self-attention applications, while leaving those extensions for future study.

  • 2.1 Deep architectures: GCNet is presented as a general-purpose architecture that improves object detection, instance segmentation, image classification, and action recognition.
  • 2.2 Long-range dependency modeling: Long-range dependency methods are categorized into pairwise approaches, such as NLNet, and context-fusion approaches that strengthen each position with globally aggregated features.These methods complement architectures that otherwise rely mainly on locally operating stacked layers.
  • 2.2 Long-range dependency modeling: The paper reports that NLNet often learns the same global context vector for different pixels, so its effectiveness is mainly attributed to global-context modeling rather than pairwise relation modeling.For semantic segmentation, some pairwise relation is observed, but accuracy improvement is still mostly attributed to global-context modeling.
  • 2.2 Long-range dependency modeling: GCNet uses attention pooling and addition for context fusion, combining NLNet’s context modeling and information fusion with a lightweight design associated with SENet.The paper reports that GCNet generally performs better than SENet.
  • 2.3 Self-attention modeling: The analysis targets pixel-pixel self-attention in vision, while extending the approach to object relations, language, and graph applications remains an open direction.The paper specifically identifies whether pairwise relations are learned and how global context contributes in those applications as future questions.

3 ANALYSIS OF NON-LOCAL NETWORKS

The analysis revisits non-local blocks and finds that their learned global context is often nearly query-independent, motivating a simpler interpretation and design.

  • 3.1 Revisiting the Non-local Block: Non-local blocks model each query position by aggregating features from all positions with weights determined by pairwise relationships.The block strengthens a query feature using a weighted sum over all positions, but its time and space complexity are quadratic in the number of positions.
  • 3.2.1 Visualization: Visualization on COCO shows almost identical attention maps for different query positions, suggesting that object detection may use global context rather than pixel-pixel relations.The visualization uses three query positions across six COCO images with the Embedded Gaussian instantiation in Mask R-CNN with FPN and ResNet50.
  • 3.2.2 Statistical Analysis: On COCO, Kinetics, and ImageNet, output cosine distances are at least one order of magnitude smaller than input distances, indicating nearly identical output global contexts across positions.The analysis compares input, output, and attention-map features across four non-local instantiations and four standard tasks.
  • 3.2.2 Statistical Analysis: Higher network stages produce more query-dependent output features than lower stages, showing that query dependence varies with layer depth.This stage-wise trend is reported for the Embedded Gaussian non-local block across four tasks.
  • 3.2.2 Statistical Analysis: Fine-grained analysis finds that Embedded Gaussian attention becomes more query-independent after inner-product computation, whereas Gaussian attention remains query-dependent until later processing.For Gaussian, query dependence persists because the instantiation lacks query and key transformations; after attention pooling and output transformation, the resulting features become more similar.
  • 3.2.2 Statistical Analysis: Cityscapes semantic segmentation is an exception: non-local networks can learn pairwise relations there, yet simplified variants indicate gains may mainly come from global-context modeling.The paper explicitly cautions that the query-independent behavior does not hold across every task.

4 METHOD

The method reformulates non-local context as query-independent global context, then packages it into a lightweight GC block that can be inserted across backbone layers.

  • 4.1 Simplifying the Non-local Block: The method starts from the finding that non-local blocks tend to learn query-independent attention maps, motivating an explicit shared global attention map.The simplified formulation preserves the non-local aggregation pattern while removing query-specific attention computation.
  • 4.2 Global Context Modeling Framework: The simplified block comprises global attention pooling, feature transformation, and feature aggregation by broadcasting the transformed context to every position.Pooling uses learned attention weights; the transform is implemented with a 1x1 convolution, and fusion uses element-wise addition.
  • 4.2 Global Context Modeling Framework: The framework unifies the simplified non-local block and the SE block as alternative instantiations of global context modeling.SE uses global average pooling, a bottleneck transform, and channel-wise excitation within the same three-part framework.
  • 4.3 Global Context Block: The proposed GC block combines global attention pooling, a two-layer bottleneck transform, and broadcast addition to model long-range dependencies with low overhead.The bottleneck reduces transform parameters from C·C to 2·C·C/r, while layer normalization is added before ReLU to ease optimization and support generalization.
  • 4.3 Global Context Block: GCNet applies GC blocks across multiple backbone layers, enabling broader global-context modeling than the heavier non-local block.For ImageNet ResNet-50, GC-ResNet-50 adds GC blocks to c3+c4+c5 with a bottleneck ratio of 16 and a 0.26% relative computation increase.

5 EXPERIMENTS

Experiments show that GC blocks retain comparable accuracy with less computation, while their lightweight design supports broader placement and generally improves recognition across tasks.

  • 5.1.1 Ablation Study: GC blocks match non-local alternatives with fewer parameters and less computation, while inserting them throughout residual stages further improves COCO detection and segmentation.Single-block SNL and GC performance is comparable to NL; all-residual-block placement improves APbbox by 1.1% and APmask by 0.9%.
  • 5.1.1 Ablation Study: GC placement across c3, c4, and c5 improves APbbox and APmask by 0.7%-1.7%, with c4 or c5 outperforming c3 and all-stage placement performing best.The results associate stronger gains with more semantic features and only a slight FLOPs increase.
  • 5.1.1 Ablation Study: LayerNorm preserves the one-layer transform's performance with far fewer parameters, addressing the optimization difficulty of narrower two-layer bottlenecks.Batch normalization and group normalization perform poorly in the 1 × 1 intermediate feature map because their statistics are insufficient.
  • 5.1.1 Ablation Study: Attention pooling with addition is more effective than scaling, while pooling choice matters less than how global context is fused into query features.The att+add configuration significantly outperforms avg+scale.
  • 5 EXPERIMENTS: Attention maps learned by NLNet are usually similar across query points and resemble GCNet's maps, with both often focusing on small or thin objects.The authors suggest this focus may facilitate detection of objects such as frisbees, skateboards, and snowboards.

6 CONCLUSION

The paper finds that non-local networks empirically model query-independent rather than query-specific context on several visual tasks. It uses this finding to build lightweight GC blocks and a multi-layer GCNet that generally outperforms simplified NLNet.

  • 6 CONCLUSION: Non-local networks empirically model query-independent context on several visual recognition tasks, despite being designed for query-specific global context.This conclusion motivates replacing query-specific attention with an explicitly query-independent formulation.
  • 6 CONCLUSION: GC blocks provide a lightweight framework for long-range dependency modeling, and GCNet applies them across multiple backbone layers to generally outperform simplified NLNet.The paper verifies benefits across multiple visual recognition tasks and identifies extensions to generative, graph-learning, and self-supervised models as future work.
Loading 2012.13375v1…