Source-linked AI summary
Graph-Based Global Reasoning Networks
Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Shuicheng Yan, Jiashi Feng, Yannis Kalantidis
TL;DR
CNNs efficiently model local relations but are inefficient at capturing global relations between distant regions. The paper introduces GloRe, which reasons over globally aggregated features in an interaction-space graph and projects relation-aware features back to coordinate space. Across multiple backbones and image/video tasks, GloRe consistently boosts performance.
Problem
CNNs are inefficient at capturing relations between distant regions because convolution models information locally and requires multiple layers for global coverage.
Method
GloRe projects globally aggregated coordinate-space features into a small interaction-space graph, applies graph convolution, and broadcasts relation-aware features back.
Results
GloRe consistently boosts performance across ResNet, ResNeXt, SE-Net, and DPN backbones for 2D and 3D CNN tasks including classification, segmentation, and action recognition.
Takeaways & Limitations
The lightweight, end-to-end trainable GloRe unit can be plugged into existing CNNs to provide complementary globally reasoned features.
Abstract
from arXiv · showhide
Globally modeling and reasoning over relations between regions can be beneficial for many computer vision tasks on both images and videos. Convolutional Neural Networks (CNNs) excel at modeling local relations by convolution operations, but they are typically inefficient at capturing global relations between distant regions and require stacking multiple convolution layers. In this work, we propose a new approach for reasoning globally in which a set of features are globally aggregated over the coordinate space and then projected to an interaction space where relational reasoning can be efficiently computed. After reasoning, relation-aware features are distributed back to the original coordinate space for down-stream tasks. We further present a highly efficient instantiation of the proposed approach and introduce the Global Reasoning unit (GloRe unit) that implements the coordinate-interaction space mapping by weighted global pooling and weighted broadcasting, and the relation reasoning via graph convolution on a small graph in interaction space. The proposed GloRe unit is lightweight, end-to-end trainable and can be easily plugged into existing CNNs for a wide range of tasks. Extensive experiments show our GloRe unit can consistently boost the performance of state-of-the-art backbone architectures, including ResNet, ResNeXt, SE-Net and DPN, for both 2D and 3D CNNs, on image classification, semantic segmentation and video action recognition task.
1. Introduction
The paper addresses CNNs’ inefficient modeling of long-range relations by projecting globally aggregated features into an interaction space for direct relational reasoning, then returning relation-aware features to coordinate space. It instantiates this approach as the lightweight, trainable GloRe unit and reports consistent gains across backbones and vision tasks.
- Motivation: CNN layers model local relations, so capturing arbitrary distant regions requires stacking layers until the receptive field covers them.In ResNet50, the receptive field reaches the full 224×224 image only at the 11th residual unit.
- Positioning: Unlike methods focused on information delivery or image-level context, GloRe directly reasons over relations among multiple regions.The paper contrasts it with Non-local and Double Attention networks, and with SE-Net’s single-node interaction graph.
- Approach: The proposed approach globally aggregates coordinate-space features, projects them into an interaction space, reasons over relations there, and broadcasts the results back.The interaction space represents semantically similar regions with shared features rather than scattered coordinate-specific features.
- Approach: The GloRe unit implements coordinate-interaction mapping with weighted global pooling and broadcasting, while graph convolution performs relation reasoning.This design is differentiable, lightweight, and end-to-end trainable.
- Evidence: Experiments show consistent performance boosts when GloRe is inserted into ResNet, ResNeXt, SE-Net, and DPN backbones across 2D and 3D CNN tasks.Evaluated tasks include image classification, semantic segmentation, and video action recognition.
2. Related Work
Related work improves global context modeling through enlarged receptive fields, learned sampling, or global pooling, while graph-based methods provide efficient relational reasoning. GloRe builds on these directions by targeting global region relations within CNN feature processing.
- Deep Architecture Design: Deep architecture research improves optimization or efficiency through designs including residual, dense, dual-path, grouped, and depth-wise convolutional networks.Examples include ResNet, DenseNet, DPN, Xception, MobileNet, and ResNeXt.
- Global Context Modeling: Global context methods enlarge receptive fields or incorporate broader information using multi-scale features, deformable sampling, or global average pooling.PSP-Net and DenseASPP use multi-scale features, Deformable CNNs learn sampling offsets, and SE-Nets use global average pooling.
- Graph-based Reasoning: Graph-based approaches use graph models for relational reasoning in image segmentation, semi-supervised classification, and object relations in video recognition.Examples include CRFs, random walk networks, and graph convolution networks.
3. Graph-based Global Reasoning
The GloRe unit addresses inefficient global reasoning in CNNs by projecting coordinate-space features into a latent interaction space, reasoning over a smaller graph, and projecting relation-aware features back.
- Overview: The GloRe unit projects coordinate-space features into an interaction space where disjoint regions can be represented by shared node features for global reasoning.The interaction space uses a smaller fully connected graph whose nodes store the projected features.
- Overview: The unit performs a three-step process: projection into interaction space, graph convolution over node relations, and reverse projection to coordinate space.The returned features provide complementary information for subsequent task-specific convolutional layers.
- Coordinate-to-interaction projection: Weighted global pooling forms interaction-space nodes as linear combinations of coordinate-space features using learnable projection weights.Convolution layers generate the projection weights directly, avoiding extra object annotations and a time-consuming detector.
- Graph convolution: Graph convolution diffuses information across interaction-space nodes and updates their states using a learned adjacency matrix and state transformation.The identity shortcut helps alleviate optimization difficulties, while Laplacian smoothing propagates node features over the graph.
- Interaction-to-coordinate projection: Reverse projection performs weighted broadcasting from graph nodes back to the coordinate space, reusing the projection weights as D = B⊤ to reduce computation.The broadcast output is followed by dimension expansion so the unit can form a residual path compatible with CNN backbones.
4. Experiments
Experiments evaluate GloRe across image classification, semantic segmentation, and video action recognition, using multiple CNN backbones and datasets. The unit consistently improves performance, including gains on shallow and deep networks and under constrained segmentation settings.
- Experimental Setup: Experiments cover ImageNet classification, Cityscapes segmentation, and Kinetics action recognition with 2D and 3D CNN backbones.The evaluated backbones include ResNet, ResNeXt, DPN, and SE-Net for classification, FCN-based models for segmentation, and 3D ResNet models for video.
- ImageNet: Figure 4 compares Top-1 accuracy against FLOPs, distinguishing ResNet, Nonlocal, GloRe, and SE variants by insertion location and model size.The ablation includes reproduced baselines and models with one or multiple added blocks.
- ImageNet: 1.5%: adding one GloRe unit improves ResNet-50 accuracy, while GloRe exceeds Nonlocal accuracy at the same computation budget and model size.The gain is larger when the unit is inserted on Res4 rather than Res3 in ResNet-50.
- ImageNet: 78.4% Top-1 accuracy: three GloRe units enhance ResNet-50 to slightly outperform ResNet-200 using about 30% of its GFLOPs and 50% of its parameters.The authors also report consistent accuracy increases as more blocks are added and similar improvement on SE-ResNet-50.
- ImageNet: GloRe consistently improves deeper ResNet-200, ResNeXt-101, DPN-98, and DPN-131 models, with Res3-plus-Res4 insertion helping deeper networks.The authors attribute the differing insertion behavior to deeper networks containing more informative Res3 features.
- Cityscapes: 1.8%: one GloRe unit improves Cityscapes validation mIoU over the FCN-plus-multi-grid baseline, while two sequential units add no further gain.On the test set, the method uses no extra coarse training data and achieves better accuracy than some popular baselines with ResNet-50.
5. Visualizing the GloRe Unit
The paper visualizes GloRe’s learned representations and projection weights to analyze how the unit supports global reasoning. The visualizations focus on feature improvements and the weighted pooling that forms interaction-space descriptors.
- Analysis: Experiments report that GloRe boosts accuracy across various CNN backbones and datasets for both 2D and 3D tasks.The visualization analysis follows these performance experiments to examine the learned feature representations.
- Segmentation Visualization: Figure 5 compares Cityscapes FCN predictions with and without GloRe, highlighting differences in yellow boxes.The authors state that GloRe alleviates ambiguity, captures more details, and enhances backbone generalization in these visualizations.
- Projection Weights: Figure 7 visualizes four projection maps for two ImageNet images, with red indicating positive weights, green negative weights, and brightness indicating magnitude.The weights are coefficients for weighted-average pooling over the image to produce one interaction-space feature descriptor.
6. Conclusion
The GloRe unit provides an efficient global-reasoning module that projects features into an interaction-space graph, reasons over globally aware features, and projects them back. It is lightweight, easy to optimize, and consistently improves diverse CNNs across 2D and 3D tasks.
- 6. Conclusion: GloRe projects coordinate-space information into interaction-space graph nodes for direct reasoning over globally aware discriminative features.Projection and reverse projection use weighted pooling and weighted broadcasting, while graph interactions use graph convolution.
- 6. Conclusion: The GloRe unit is lightweight, easy to implement, and easy to optimize.
- 6. Conclusion: Extensive experiments show GloRe learns features complementary to popular CNNs and consistently boosts performance on 2D and 3D tasks across multiple datasets.