Source-linked AI summary
A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective
Chaoqi Chen, Yushuang Wu, Qiyuan Dai, Hong-Yu Zhou, Mutian Xu, Sibei Yang, Xiaoguang Han, Yizhou Yu
TL;DR
Computer vision lacks systematic coverage of how GNNs and graph Transformers address relational structure across diverse visual modalities and tasks. The paper surveys these methods using a task-oriented taxonomy, covering five input modalities and representative approaches, then discusses their insights, limitations, and future directions. It highlights strong relational modeling across vision tasks while identifying unresolved challenges in long-range video dependencies and unified, explainable vision-language architectures.
Problem
Computer vision needs methods for relational and irregular visual structure, but existing work lacks a comprehensive task-oriented survey of GNNs and graph Transformers across vision modalities.
Method
The paper organizes GNN and graph Transformer applications into five input-modality categories and further divides them by vision task, reviewing representative methods and challenges.
Results
The survey systematically covers methods showing GNN flexibility for relational vision tasks, including 3D understanding, vision-language reasoning, and cross-domain knowledge transfer.
Takeaways & Limitations
Task-oriented organization clarifies how graph-based methods model relations across visual data types and identifies directions for improving graph construction, reasoning, and global dependency modeling.
Takeaways & Limitations
Video methods often capture only partial frame dependencies or local region contexts, while vision-language research still lacks unified architectures and stronger reasoning beyond message passing.
Abstract
from arXiv · showhide
Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (\emph{e.g.,} social network analysis and recommender systems), computer vision (\emph{e.g.,} object detection and point cloud learning), and natural language processing (\emph{e.g.,} relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, \emph{i.e.,} 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.
1 INTRODUCTION
Computer vision needs methods that represent irregular topology and relations, while graph construction from regular visual data remains underdeveloped. This survey organizes GNN and graph Transformer research by input modality and vision task to review approaches, challenges, insights, limitations, and future directions.
- Irregular visual topology and relations are important for representation learning but remain insufficiently studied by CNN-centered vision systems.
- Graph construction from regular grid data lacks principled methods and often depends on domain knowledge, while point clouds and meshes contain sophisticated relational structure.
- GNN vision research combines GNNs with CNN backbones or uses pure GNN architectures for visual representation learning.
- The survey provides a task-oriented review covering task definitions and challenges, representative approaches, insights, limitations, and future directions.
- Applications are divided into five input-modality categories and then organized by the computer vision tasks they perform.
2 BACKGROUND AND CATEGORIZATION
The survey reviews GNNs, graph convolutional paradigms, deeper GNN techniques, and graph Transformers for vision. It emphasizes neighborhood aggregation, attention, graph structure, and the contrast between local relational biases and broader self-attention.
- Vision approaches often combine CNNs and GNNs, while this background focuses on the GNN component and also reviews graph Transformers.
- Convolutional GNNs: Graph convolution is commonly categorized into spectral approaches based on the Laplacian spectrum and spatial approaches based on graph neighborhoods.
- Spectral approaches: Spectral graph convolution uses graph Fourier transforms and Laplacian-based filtering, with ChebNet approximating filters through Chebyshev polynomials.
- Spatial approaches: Spatial GNNs aggregate neighbor information through sampling, concatenation, and learned attention weights, as illustrated by GraphSAGE and GAT.
- New GNN Techniques: Deep GNN techniques use residual, dense, and dilated connections, edge dropout, and reversible designs to improve depth, efficiency, and trainability.
- Graph Transformers for 3D Data: Graph Transformers for 3D data use local or global self-attention, with designs such as Point Transformer, Point Cloud Transformer, and Mesh Graphormer modeling relational interactions.
3 2D NATURAL IMAGES
GNN-based methods in 2D natural-image tasks construct and propagate relational information among objects, labels, classes, and domains. Their applications emphasize multi-object understanding and data- or label-efficient learning, while graph construction remains a key limitation.
- GNNs provide relational inductive biases for modeling structural dependencies among visual categories and improving context-aware understanding.
- Image Classification: For multi-label recognition, methods construct per-image label graphs so category representations and label relationships can be learned jointly or independently.
- Few-Shot Learning: Few-shot learning methods construct embedding-space graphs and propagate labels from support samples to query samples.
- Zero-Shot Learning: Zero-shot learning uses predefined or learnable knowledge graphs to represent semantic relations and support recognition of unseen classes.
- Transfer Learning: Domain adaptation and generalization use class-relation knowledge graphs, pseudo-label propagation, curriculum learning, and prototype alignment for structural transfer.
- Discussion: Current image-classification work relies on task-specific heuristic knowledge graphs, motivating general automatic graph construction and finer-grained node and edge design.
3.2 Object Detection
GNN-based object detection methods address the limitations of treating objects independently by modeling semantic, spatial, and cross-domain relationships. Approaches range from knowledge-guided global reasoning to adaptively learned sparse graphs and bipartite domain graphs.
- Object detection must model semantic dependencies, co-occurrence, and relative locations alongside individual object appearance.
- SGRN adaptively discovers semantic and spatial relationships by learning a sparse adjacency matrix among image regions.
- Domain-adaptive detectors use bipartite and cross-image graphs to model homogeneous, heterogeneous, and class-conditional dependencies across domains.
3.3 Image Segmentation
GNNs provide a framework for combining object appearance with image context and reasoning over long-range relationships in semantic segmentation. The reviewed methods include coordinate- and feature-space reasoning, graph attention, and cross-image or cross-scale relational modeling.
- GNNs model object appearances and image contexts jointly to support pixel-wise semantic segmentation.The formulation constructs a graph from the image, encodes appearance and context, then applies a pixel-wise predictor.
- Dual GCNs separately model spatial relations among pixels and dependencies among feature channels before mapping features back to coordinate space.
- Pyramid graph attention and scale-aware GNNs perform relational reasoning between query and support images for one-shot and few-shot segmentation.
- GNN-based methods are reported to improve relation modeling and training efficiency over prior approaches with limited representation ability.
3.4 Scene Graph Generation (SGG)
Scene graph generation detects objects and their relationships to produce a visually grounded scene graph. Its graph structure aligns naturally with GNNs, which propagate contextual information over object and relation nodes.
- Scene graph generation detects object pairs and their relations, representing objects as nodes and relations as edges.
- Graph R-CNN prunes a dense proposal graph, then uses an attentional GCN to update node features and edge relations.
- GNN-based scene graph methods apply shared latent spaces, spatially weighted message passing, GAT aggregation, bipartite propagation, and energy-based inference.
4 VIDEO UNDERSTANDING
GNNs are applied across video understanding tasks to model spatial, semantic, and temporal relations among visual constituents. The survey highlights both graph constructions for specific tasks and the remaining challenge of capturing long-range global dependencies without redundancy.
- Video understanding: GNNs support video action recognition, temporal action localization, and video object segmentation by reasoning over spatio-temporal or semantic relations.They are also used for group activity recognition, action performance assessment, and human motion prediction.
- Video Action Recognition: Video action recognition constructs graphs from objects, humans, frames, or skeletons, then applies GNNs to model relationships and predict actions.Skeleton-based methods connect joints within frames and across consecutive frames to learn spatial and temporal action patterns.
- Long-range modeling: Long-range video reasoning connects visual constituents across frames, including human-object and object-object relations, while multi-scale dilated windows capture longer temporal dependencies.These designs extend beyond local consecutive-frame modeling through graph-based reasoning over space-time regions and multiple temporal scales.
- Temporal Action Localization: Temporal action localization uses proposal or snippet graphs to capture relations for classifying actions and refining temporal boundaries.Edges can encode temporal overlap, feature correlations, consecutive-time relations, or semantic connections, with dynamic graphs supporting temporal and semantic context modeling.
- Discussion: Existing methods often capture only partial frame-level dependencies or local region-level contexts, leaving efficient long-range global modeling as an open direction.The survey specifically calls for capturing crucial contexts without redundant information.
5 VISION + LANGUAGE
GNNs are used in vision-and-language tasks to represent visual, linguistic, and cross-modal relations for alignment and reasoning. The survey covers task-specific graph pipelines while identifying open questions about unified architectures, explainability, and knowledge-aware reasoning.
- Vision + language: Vision-and-language applications include visual question answering, visual grounding, image captioning, image-text matching, and vision-language navigation.GNNs model relations among visual, linguistic, or multimodal components and perform cross-modal alignment.
- Visual Question Answering: In visual question answering, graphs represent image objects, question dependencies, and, for fact-based questions, external knowledge before GNN-based answer prediction.The surveyed methods use relational modeling, adaptive attention, and higher-order reasoning over visual and textual content.
- Visual Grounding: Visual grounding uses GNNs to encode dependencies between visual and linguistic components and replace answer prediction with bounding-box matching or refinement.Language-guided visual relation graphs support multi-order semantic contexts and stepwise graph reasoning.
- Image Captioning: Image captioning methods construct semantic, spatial, or sub-graphs to capture object context, select important structures, and control described objects, attributes, and relations.These graph representations are used to generate more context-aware captions and improve fine-grained control or diversity.
- Discussion: Open problems include designing unified GNN architectures across vision-and-language tasks, extending reasoning beyond neighborhood aggregation, improving explainability, and incorporating knowledge graphs.The survey presents these as research questions rather than settled conclusions.
6.1 3D Representation Learning
The survey reviews graph-based representation learning for point clouds and meshes, emphasizing relational modeling for irregular 3D structures. It covers graph convolutions, attention, spectral methods, and graph Transformers across 3D data.
- 3D representations: Point clouds and meshes are irregular 3D representations whose geometric and topological relationships motivate GNN-based representation learning.GNNs can handle varying 3D data sizes and resolutions while modeling interactions among points or mesh components.
- Point Cloud Representation: Point-based and geometry-based methods process points independently or model shape structure without directly representing point relationships, limiting robustness on noisy, complex, or incomplete clouds.Graph-based methods instead represent point clouds as spatial graphs to model local or global point interactions.
- Point Cloud Representation: Point-cloud GNN approaches include adaptive edge-conditioned convolutions, spectral filtering on dynamic local graphs, and attention based on point positions and learnable features.These methods use edge information, graph pooling, or adaptive attention to improve relational feature aggregation.
- Graph Transformers: Graph Transformers for point clouds use self-attention to represent global relations, with PT applying attention on local k-NN graphs and capturing global relations in deeper layers.This combines local graph structure with the global-relation capacity of Transformer self-attention.
- Mesh Representation Learning: Mesh representation learning applies convolution and pooling to vertices, edges, faces, or combinations of mesh components, while newer operators expand receptive fields and improve efficiency or invariance.Examples include local vertex aggregation, intrinsic multi-scale convolutions, and dilated mesh convolutions.
6.2 3D Understanding
GNNs support 3D understanding by modeling local, global, and multi-level relations across point clouds, regions, objects, and scenes. The survey covers graph construction, feature aggregation, and emerging combinations with self-attention.
- Overview: GNN-based 3D methods extract low-level features from input graphs and can aggregate them through region- or instance-level graphs for downstream tasks.This hierarchy incorporates high-level relations into features used for tasks such as segmentation and detection.
- 3D Segmentation: 3D segmentation requires both global scene semantics and fine-grained pointwise signatures, motivating graph construction and improved local aggregation or global interaction.GNNs model inter-point, inter-region, and inter-object relationships in irregular geometric structures.
- 3D Segmentation: On S3DIS-Area5, point-transformer methods achieve state-of-the-art segmentation accuracy among the compared GNN-based and non-GNN-based approaches.The comparison is reported in Table 6.
- 3D Object Detection: 3D object detection uses GNNs to capture cross-point and inter-object relations, improving point-level feature extraction and proposal understanding in multi-object scenes.Examples include object-level relation graphs and dynamic graph convolutions for object queries.
- Discussion: GCN methods remain popular for 3D understanding, while Transformer self-attention offers stronger feature extraction at an efficiency cost.The survey identifies combining graph convolution and self-attention across point-, object-, and scene-level understanding as an open direction.
6.3 3D Generation
3D generation methods use GNNs for input encoding, geometric manipulation, and reconstruction of point clouds or meshes. The survey highlights their structural benefits while noting limited exploration of graph Transformers and efficiency challenges.
- Overview: 3D generation commonly uses encoder-decoder architectures, with GNNs supporting feature extraction, input manipulation, and reconstruction from reference inputs.Examples include point-position adjustment for denoising and generation of 3D shapes from reference data.
- Point Cloud Completion and Upsampling: Point cloud completion and upsampling add points to sparse or incomplete scans, while GNNs capture geometric structure and local and global topological information.The target is a dense or complete point cloud I′ = {I, △I}.
- Point Cloud Completion: Completion methods use graph convolutions and hierarchical local-global features to recover complete shapes from incomplete information.Representative designs use skeletons, EdgeConv, cross-cascade modules, or GCN-guided optimization.
- Point Cloud Upsampling: Upsampling methods aggregate features on k-NN graphs and use dense, residual, multiresolution, or adversarial graph mechanisms to generate higher-resolution point clouds.The surveyed approaches include PU-GCN, AR-GCN, and PUGeo-Net.
- 3D Reconstruction: GNNs support point cloud and mesh reconstruction by modeling sparse dependencies and geometric structure within encoder-decoder or graph-based generative frameworks.Examples include FoldingNet, StructureNet, graph-convolutional decoders, and Pixel2Mesh.
- Discussion: Graph Transformers remain insufficiently explored for general 3D object and scene reconstruction, while large data sizes make efficiency an important direction.The discussion calls for more efficient graph convolutional and graph Transformer methods.
7 MEDICAL IMAGE ANALYSIS
Medical image applications of GNNs span brain activity analysis, disease diagnosis, and anatomical segmentation. The survey organizes brain activity methods by subject- or region-based graph granularity and reviews graph-based approaches across neurological, chest, and anatomical tasks.
- Overview: GNNs in medical image analysis address brain activity investigation, disease diagnosis, and anatomy segmentation.The survey covers these applications across brain and chest medical data.
- Brain Activity Investigation: Brain activity analysis uses subject-based graphs, where nodes represent subjects, or region-based graphs, where nodes represent brain areas.Node and edge features may vary with the task, and graph construction is performed over subjects or brain regions.
- Disease Diagnosis: GNN-based diagnosis covers autism spectrum disorder, Alzheimer’s disease, Parkinson’s disease, major depressive disorder, bipolar disorder, and other brain conditions.The reviewed methods use subject-level, region-level, multimodal, spatial-temporal, and attention-based graph representations.
- Brain Activity Investigation: Subject-based methods construct population graphs from imaging and non-imaging similarity, whereas region-based methods learn from functional graphs of individual subjects.Reported applications include spectral, hybrid, siamese, and triplet GCNs.
- Chest Disease Analysis: Chest disease methods build graphs over anatomies, textual attributes, or similar scans, including anatomy-aware GCNs for mammogram detection.These approaches combine image features with relation-aware or structural information.
- Anatomical Segmentation: GNN-based anatomical segmentation includes brain surface, vessel, and other structure segmentation, often extending region-based graph schemas.GCNs and GATs are applied to meshes, surface representations, and vessel graphs.
8 CONCLUSIONS
The survey frames relational reasoning as a major challenge for computer vision and organizes GNN and graph Transformer research by input modality and task. It synthesizes innovations, limitations, and future directions toward more human-like visual understanding.
- Conclusions: The survey presents a task-oriented review of GNNs and graph Transformers for relational computer vision tasks.It groups algorithms by input modalities including images, video, and point clouds.
- Conclusions: The review organizes methods by modality and task while discussing key innovations, limitations, and potential research directions.Its stated goal is to provide insights for advancing visual understanding.