Source-linked AI summary
Vision GNN: An Image is Worth Graph of Nodes
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, Enhua Wu
TL;DR
Irregular image objects expose limits in grid and sequence representations used by common vision architectures. The paper constructs patch graphs and introduces ViG with graph information exchange and node-feature transformation, reporting strong results across image recognition and object detection. The study positions graph neural networks as a viable architecture for general visual tasks.
Problem
Grid and sequence representations are inflexible for modeling irregular and complex objects, while GCNs have mainly been applied to tasks with naturally constructed graphs.
Method
ViG treats image patches as graph nodes, connects nearest neighbors, and combines Grapher graph convolution with FFN node-feature transformation in isotropic and pyramid architectures.
Results
ViG demonstrates superiority on image recognition and object detection, including 82.1% top-1 accuracy for Pyramid ViG-S on ImageNet at about 4.5G FLOPs.
Takeaways & Limitations
Graph neural networks can serve as general visual backbones, with graph representations offering flexibility for irregular and complex image objects.
Abstract
from arXiv · showhide
Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes. ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research. The PyTorch code is available at https://github.com/huawei-noah/Efficient-AI-Backbones and the MindSpore code is available at https://gitee.com/mindspore/models.
1 Introduction
Existing vision models represent images as grids or patch sequences, which are inflexible for irregular objects. ViG instead represents patches as graph nodes and exchanges information through graph-based modules.
- 1 Introduction: CNNs represent images as regular pixel grids, while ViT and MLP models represent them as patch sequences.ViT divides a 224 × 224 image into 16 × 16 patches, producing a sequence of length 196.
- 1 Introduction: Grid and sequence representations are redundant and inflexible because objects in images are usually irregular rather than quadrate.Object parts, such as a human’s head, body, arms, and legs, can instead be connected as a graph.
- 1 Introduction: Graph representations link nodes by content rather than local position and generalize both grid and sequence structures.This flexibility supports modeling objects as compositions of connected parts.
- 1 Introduction: ViG divides images into patches as nodes, constructs their graph, and uses Grapher and FFN modules to exchange information and transform node features.The FFN also encourages node diversity to alleviate over-smoothing in graph convolution.
- 1 Introduction: 82.1% top-1 accuracy is achieved by Pyramid ViG-S on ImageNet with about 4.5G FLOPs, outperforming comparable CNN, MLP, and transformer models.The comparison uses ResNet, CycleMLP, and Swin-T as representative networks.
2 Related Work
Computer vision has progressed from CNNs toward transformers and MLP-based models, while graph neural networks have mainly served tasks with naturally constructed graphs. The related work motivates graph-based backbones for general image processing.
- 2 Related Work: CNNs became the mainstream architecture for visual tasks, with representative developments including ResNet, MobileNet, and NAS.CNNs have been applied to image classification, object detection, and semantic segmentation.
- 2 Related Work: Vision transformers were introduced for visual tasks from 2020 and achieved competitive performance alongside CNNs.The passage places transformers within the recent development of computer vision architectures.
- 2 Related Work: Graph neural network research includes spatial-based and spectral-based graph convolutional networks and their extensions.The early work spans recursive formulations, spatial graph convolutions, and spectral graph theory.
- 2 Related Work: Prior computer-vision applications of GCNs mainly involve point-cloud classification, scene-graph generation, and action recognition.These tasks use point clouds, object-relation graphs, or naturally linked human joints.
- 2 Related Work: Because existing GCNs primarily address naturally constructed graphs, general computer vision requires a GCN-based backbone that directly processes image data.This scope motivates constructing graph structure from image patches.
3 Approach
ViG represents images as graphs of patch nodes, then learns visual features through graph convolution and node-wise feed-forward transformations. The approach includes isotropic and pyramid architectures with positional encoding for different visual-network designs.
- Graph Structure of Image: Images are divided into patches represented as nodes, with each node connected to its K nearest neighbors to form a graph.Patch features form unordered nodes, and directed edges connect neighboring patches.
- Graph-level Processing: Graph convolution exchanges information by aggregating neighbor features and updating each node representation.The model uses max-relative graph convolution and can process the graph as X′ = GraphConv(X).
- ViG Block: Multi-head graph updates split aggregated features into multiple representation subspaces, update them with different weights, and concatenate the results.The heads can be updated in parallel, supporting feature diversity.
- ViG Block: The ViG block combines a Grapher module with feature transformations and nonlinear activation, followed by an FFN applied independently to each node.These transformations increase feature diversity and help alleviate over-smoothing and layer collapse.
- Network Architecture: ViG provides isotropic and pyramid architectures to support equal-sized features or progressively smaller multi-scale features.Isotropic models use N = 196 nodes with K increasing from 9 to 18, while pyramid models use spatial reduction in their first two stages.
- Network Architecture: Positional encoding is added to node features, with pyramid ViG additionally using relative positional distance during graph construction.Absolute encoding is used in both architectures; relative encoding contributes to feature-distance calculations in pyramid ViG.
4 Experiments
Experiments evaluate ViG on ImageNet recognition, COCO detection, ablations, and graph visualizations. ViG outperforms or compares favorably with CNN, transformer, and MLP backbones across the reported tasks.
- Main Results on ImageNet: ViG-Ti achieves 73.9% top-1 accuracy on ImageNet, 1.7% above DeiT-Ti at similar computational cost.
- Main Results on ImageNet: Pyramid ViG matches or outperforms representative CNN, MLP, and transformer pyramid networks on ImageNet.The reported comparison includes ResNet, Swin Transformer, and CycleMLP.
- Ablation Study: Different graph convolutions outperform DeiT-Ti in top-1 accuracy, with Max-Relative GraphConv providing the best FLOPs–accuracy trade-off.Max-Relative GraphConv is used by default in subsequent experiments.
- Ablation Study: Neighbor counts from 9 to 15 perform well on ImageNet, while fewer neighbors reduce information exchange and more can cause over-smoothing.
- Object Detection: Pyramid ViG-S outperforms representative ResNet, CycleMLP, and Swin Transformer backbones on both RetinaNet and Mask R-CNN.The evaluation uses ImageNet-pretrained backbones and the COCO 2017 validation set.
- Visualization: Visualization shows shallow layers select neighbors by local color and texture, whereas deep layers select more semantic, same-category nodes.The graph is visualized across the first and twelfth ViG blocks.
5 Conclusion
The paper presents ViG as a graph-based vision architecture that represents image patches as nodes and supports visual recognition through improved graph blocks. Experiments on image recognition and object detection demonstrate the reported superiority of ViG.
- ViG represents image patches as graph nodes and combines graph processing with additional node feature transformation.
- The proposed isotropic and pyramid ViG architectures demonstrate superiority on image recognition and object detection experiments.
- The conclusion identifies ViG as a potential basic architecture for general visual tasks.
Checklist
The checklist records the paper’s reported experimental materials, training details, compute information, and asset citations, while noting that error bars were not reported.
- The checklist states that the paper describes its limitations and that its main claims accurately reflect the paper’s contributions and scope.
- The paper reports experimental results and training details in the referenced experimental section.
- Error bars across multiple random-seed runs were not reported, with common ImageNet and COCO settings cited instead.
- The paper reports the total compute and resource information in the referenced experimental section.
- Existing assets are cited and their licenses are discussed; no new assets are included.
- Human-subjects and participant-risk items are marked not applicable, and no potential negative societal impacts are reported.
A.1 Theoretical Analysis
The analysis argues that the FFN module can increase node-feature diversity, with the effect governed by its Lipschitz constant.
- Feature diversity: Feature diversity measures how far graph-convolution outputs are from being constant across nodes.The measure uses the ℓ1,∞ norm of the deviation from the closest row-constant matrix.
- Theoretical result: Theorem 1 bounds the diversity of FFN(X) using the input diversity and the FFN Lipschitz constant.The theorem is derived for a FFN applied to graph-convolution features.
- Proof: FFN operations preserve the property that constant-across-row inputs remain constant across rows.The proof covers weight multiplication, bias addition, and elementwise nonlinear activation.
- Interpretation: The FFN Lipschitz constant is usually much larger than 1 because it relates to the norms of its weight matrices.This makes the theorem predict increased feature diversity when FFN is introduced into the ViG block.
A.2 Pseudocode
The pseudocode implements ViG as stacked Grapher and FFN modules with residual connections, normalization, nonlinearities, and configurable graph-convolution parameters.
- ViG block: The ViG block combines a Grapher module with an FFN module as its two core components.The implementation constructs both modules with shared channel dimensions and configurable drop-path regularization.
- Implementation: The implementation is based on commonly used layers and does not introduce complex operations.The core ViG block is presented in PyTorch-like pseudocode.
- Grapher module: The Grapher module applies pointwise projection, dynamic graph convolution, normalization, GELU activation, and pointwise projection.Its graph-convolution stage uses k-neighbor and dilation parameters.
- Grapher module: The Grapher forward path reshapes spatial features into nodes, applies projection and graph convolution, then restores the spatial layout.A residual shortcut is added after drop-path before reshaping the output back to B × C × H × W.
- FFN module: The FFN module uses two 1×1 convolutional layers with batch normalization and GELU between them.Its second layer returns to the input channel dimension, and drop-path output is added to the shortcut.