PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective

Chaoqi Chen, Yushuang Wu, Qiyuan Dai, Hong-Yu Zhou, Mutian Xu, Sibei Yang, Xiaoguang Han, Yizhou Yu

arXiv:2209.13232v4cs.CVcs.AIcs.LG

TL;DR

Earlier surveys covered selected vision tasks, while this paper reviews GNNs and graph Transformers across computer vision using a modality- and task-oriented taxonomy. It concludes that these methods show flexibility and superiority for relational tasks.

  • Problem

    Earlier surveys addressed selected vision tasks, leaving a need for a comprehensive, task-oriented examination of GNNs and graph Transformers across computer vision.

    Source: §1 · p. 2

  • Method

    The survey organizes GNN applications into five input-modality categories and reviews approaches by vision task, including graph Transformers.

    Source: §1 · p. 2

  • Results

    The survey concludes that GNNs and graph Transformers show significant flexibility and superiority in handling relational tasks.

    Source: §8 · p. 19

  • Takeaways & Limitations

    The authors intend this task-wise synthesis to shed light on future progress in computer vision.

    Source: §8 · p. 19

    Video methods often capture only partial frame-level dependencies or local contexts, leaving long-range global dependencies as a future research direction.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (\emph{e.g.,} social network analysis and recommender systems), computer vision (\emph{e.g.,} object detection and point cloud learning), and natural language processing (\emph{e.g.,} relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, \emph{i.e.,} 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.

1 INTRODUCTION

The introduction argues that vision needs stronger modeling of irregular topology and relations, and presents a task-oriented survey of GNNs and graph Transformers across visual modalities and tasks.

Introduction

  • CNNs excel on regular grids, but irregular topology and object interactions remain underexplored; human relational reasoning motivates studying structures in vision [4].The introduction notes that object-recognition systems tend to perceive objects separately, while dependencies and interactions among them receive scant attention.Source: §1 · p. 1
  • Both grid-based and naturally irregular visual data could benefit from modeling topological structures and relations, particularly in complex scenes, limited-data learning, and cross-domain transfer.Building graphs from regular-grid data requires principled methods and domain knowledge, whereas point clouds and meshes already contain sophisticated relational information.Source: §1 · p. 1
  • Vision GNN research combines GNNs with CNN backbones to improve long-range modeling or uses pure GNNs as feature extractors, including for point clouds,.For point-cloud shape classification, the introduction distinguishes pointwise MLP, convolution-based, and graph-based approaches.Source: §1 · pp. 1–2
  • The survey organizes GNN applications into five input-modality categories and further by vision task, while also reviewing graph Transformers and discussing methods, limitations, and future directions [7],.It aims to provide task definitions and challenges, in-depth coverage of representative approaches, and a more comprehensive taxonomy than earlier surveys of selected vision tasks.Source: §1 · p. 2

2 BACKGROUND AND CATEGORIZATION

This section reviews GNNs and graph Transformers used in computer vision, while focusing on the GNN components of approaches that may combine CNNs and GNNs [11], [12], [13].

2.1 Recurrent GNNs

  • Recurrent GNNs update node representations iteratively with shared weights; Scarselli et al. extended earlier DAG-focused work to cyclic and undirected graphs.Updates can use node features, edge features, and neighboring nodes’ hidden states.Source: §2.1 · p. 2

2.2 Convolutional GNNs

  • Among approaches generalizing convolution to the graph domain, two series garner the most attention in recent years: spectral approaches, [16],,,, and spatial approaches,,,,,.Figure 2 shows two types of graph convolutional operations.Source: §2.2 · p. 2, Figure 2 · p. 2

2.2.1 Spectral Approaches

  • Spectral approaches define graph filters using the Laplacian spectrum; ChebNet approximates filtering with Chebyshev polynomials, and GCNs use a first-order approximation to aggregate neighbor information.The normalized Laplacian is decomposed into eigenvectors and eigenvalues, which provide the graph Fourier basis for convolution.Source: §2.2.1 · pp. 2–3

2.2.2 Spatial Approaches

  • Spatial approaches include GraphSAGE, which samples and aggregates local neighbors, and GAT, which learns attention weights between connected nodes.GraphSAGE supports mean, max, LSTM, or pooling aggregators; GAT uses multi-head attention to increase capacity and stabilize self-attention.Source: §2.2.2 · p. 3

2.2.3 New GNN Techniques

  • DeepGCNs adapt CNN techniques to deepen graph convolution, while DropEdge and other methods address deep-network training challenges.DeepGCNs use residual and dense connections and dilated convolutions, including a 56-layer GCN for point cloud semantic segmentation. DropEdge randomly removes input edges during training to mitigate over-fitting and over-smoothing; Li et al. find reversible connections combined with deep architectures can enable networks of up to 1001 layers with 80 channels or 448 layers with 224 channels. Graph pooling methods include predefined subgraphs, hierarchical differentiable pooling, feature-informed assignments, and U-shaped pooling and up-sampling. ViG builds image graphs from patches and K-nearest-neighbor edges, then uses graph convolution and positional encoding; it outperforms DeiT by 1.7% (top-1 accuracy) on ImageNet classification and Swin-T by 0.3% (mAP) on MSCOCO object detection.Source: §2.2.3 · p. 3

2.3 Graph Transformers for 3D Data

  • Graph Transformers for 3D data combine local and global attention strategies, while GNNs allow flexible connectivity and multiple relation types unlike the fully connected or local graphs used by ViTs [41].Point Transformer uses local vector self-attention, whereas Point Cloud Transformer adopts global attention; Mesh Graphormer combines graph convolution for local interactions with self-attention for non-local relations. Fast Point Transformer, Stratified Transformer, and Point Transformer V2 target efficiency or long-range dependencies.Source: §2.3 · pp. 3–4

3 2D NATURAL IMAGES · 3.1 Image Classification

For 2D natural-image classification, GNN approaches model relationships among objects and classes to support prediction, especially in multi-object and data- or label-scarce settings. The surveyed methods use graphs for relational reasoning, label propagation, adaptation, and knowledge transfer, while the discussion calls for more general graph construction and stronger integration with task-specific classifiers.

3 2D NATURAL IMAGES

  • GNNs use arbitrary connectivity to model dependencies among image categories, supporting visual and semantic understanding, prediction transparency, and overall performance.The survey notes that current approaches often construct task-specific knowledge graphs from data.Source: §3.1 · p. 4

3.1 Image Classification

  • Image-classification pipelines can build adjacency from feature distances and combine language-based label dependencies, image structural priors, and learnable matrices.Updated features may be projected into the visual domain, combined with other feature extractors, or used directly for task-specific prediction.Source: §3.1 · p. 4

3.1.1 Multi-Label Classification

  • For multi-label classification, ML-GCN maps a directed label-embedding graph into interdependent binary classifiers, while other methods propagate region semantics or model label dependencies with varied graph architectures.The surveyed alternatives include statistical co-occurrence, hypergraph neural networks, cross-modality attention, graph Transformers, and Transformers.Source: §3.1.1 · p. 4

3.1.2 Few-Shot Learning (FSL)

  • Few-shot methods use graphs to exploit novel-class structure, propagate labels from support to query samples, model class hierarchies or similarities, and adapt features to new tasks.Other approaches address data scarcity through meta-learning and class-based information transfer,,.Source: §3.1.2 · p. 5

3.1.3 Zero-Shot Learning (ZSL)

  • Zero-shot methods use predefined or learnable knowledge graphs and prior relations to transfer structural or semantic information from seen to unseen classes.Approaches include hierarchical propagation, region-based visual relations, generated attributes, prototype propagation, local-region appearance relations, and compositional relations.Source: §3.1.3 · p. 5

3.1.4 Transfer Learning

  • For domain adaptation and generalization, graph methods encode instance structure or category relations, with other approaches aggregating cross-domain information through curriculum learning, pseudo-labeling, and prototype alignment.Examples include adversarial GCNs for adaptation and global prototypical relation graphs with graph self-attention for generalization.Source: §3.1.4 · p. 5
  • The review characterizes classification GNNs as especially explored for multi-object and data- or label-efficient settings, and identifies heuristic task-specific graphs as a limitation.Future directions include general automatic graph construction, stronger interaction between graph structures and task-specific classifiers, and finer-grained node and edge building blocks.Source: §3.1.4 · p. 5

3.2 Object Detection

GNN-based object detection addresses the limits of treating objects independently by reasoning over object and part relationships. Methods model relations within images and across domains, while adding modules introduces optimization and feature-structure challenges.

3.2 Object Detection

  • Object detection must reason about semantic dependencies, co-occurrence, and relative locations while embedding object-level relations in the detection pipeline.GNN-based methods aim to perform local and global reasoning over objects or their parts.Source: §3.2 · p. 5
  • Reasoning-RCNN, SGRN, RelationNet, RelationNet++ [73], Li et al., and GraphFPN model commonsense, learned, long-range, heterogeneous, or hierarchical relations.SGRN learns sparse semantic and spatial relations; GraphFPN enables within-scale and cross-scale interactions through a superpixel hierarchy.Source: §3.2 · p. 5
  • Domain adaptive object detection has drawn attention in real-world applications beyond improving in-distribution performance.Source: §3.2 · p. 5
  • Chen et al., use intra- and inter-domain graphs, SIGMA frames DAOD as graph matching, and SRR-FSD propagates relations between base and novel classes.These methods model cross-domain dependencies in pixel and semantic spaces, class-conditional distributions across domains, or semantic relations for novel object detection, respectively.Source: §3.2 · p. 6
  • GNN detection methods model between-object, cross-scale, cross-domain, and base-to-novel-class relations, but added modules complicate optimization and Euclidean/non-Euclidean compatibility.Future directions include better region-to-node mapping, stronger initial node features through Transformer or pure GNN encoders, and reasoning directly in the original feature space.Source: §3.2 · p. 6

3.3 Image Segmentation

For image segmentation, GNNs model object appearance and image context as graphs to support pixel-wise prediction. The surveyed methods use graph reasoning for global context, specialized supervision settings, and panoptic segmentation.

3.3 Image Segmentation

  • Semantic segmentation assigns pixel-wise labels, and the described GNN formulation encodes object appearance and image context before pixel-wise prediction.It represents the image as a graph, applies a GNN, and uses a pixel-wise predictor to produce the segmentation.Source: §3.3 · p. 6
  • Backbone graph methods capture global contextual and semantic relations through spatial, feature, dynamic, multiscale, or class-wise reasoning.Zhang et al. use coordinate- and feature-space GCNs; Chen et al. reason over a fully connected graph; DGMN and Yu et al. dynamically sample nodes. Li et al. reason in the original feature space, while Hu et al. aggregate pixels within each class.Source: §3.3 · p. 6
  • Graph methods also address one-shot, few-shot, weakly supervised semantic, and panoptic segmentation.Zhang et al. connect query and support maps; Xie et al. perform cross-scale reasoning with self-node collaboration; Zhang et al. propagate semantics through affinity attention; Wu et al. use bidirectional reasoning between things and stuff branches.Source: §3.3 · p. 6
  • Non-local self-attention has quadratic time complexity when comparing every pixel with every other pixel, while CRFs and MRFs have limited representation ability.The discussion identifies these as prior approaches to contextual modeling.Source: §3.3 · p. 6
  • GNN-based methods show clear superiority in relation modeling and training efficiency over prior efforts.The cited discussion contrasts them with contextual modeling approaches including non-local networks and probabilistic graphical models.Source: §3.3 · pp. 6–7

3.4 Scene Graph Generation (SGG)

Scene graph generation detects objects and their relationships to produce a visually grounded graph, making it a natural fit for GNNs. Existing approaches use staged detection, graph refinement, and relation prediction, while direct single-stage generation remains an open direction.

3.4 Scene Graph Generation (SGG)

  • SGG detects object pairs and their relations to build a visually grounded scene graph, and GNN methods typically use object detection, relation-graph construction and refinement, then relation prediction.Scene graphs represent image objects as nodes and relations as edges; their structure aligns with GNNs’ ability to construct task-specific graphs from inputs.Source: §3.4 · p. 7
  • Graph R-CNN, Li et al., Qi et al., Li et al., and Suhail et al. use graph pruning, message passing, attention, confidence propagation, or energy-based inference for SGG.Graph R-CNN prunes a dense candidate graph with a relation proposal network, then uses an attentional GCN to update object features and edge relations.Source: §3.4 · p. 7
  • Although many graph-representation paradigms suit SGG, current approaches are often stage-wise; direct single-stage graph generation and graph-based diffusion models remain open directions.The proposed future direction is to better exploit relational inductive bias when generating scene graphs.Source: §3.4 · p. 7

4 VIDEO UNDERSTANDING

The section surveys GNNs for video understanding, where they model spatial, semantic, and temporal relations among objects, frames, and joints. It covers action recognition, temporal action localization, other video tasks, and the challenge of capturing long-range global dependencies without redundant information.

4 VIDEO UNDERSTANDING

  • GNNs model video relations across objects and frames, at multiple temporal scales, and among joints within or across frames.The survey organizes video applications around these forms of spatio-temporal reasoning.Source: §4 · p. 7

4.1 Video Action Recognition

  • Action-recognition methods use graphs of humans, objects, frames, or skeletons to model spatio-temporal context and predict actions.Approaches include long-range human-object and object-object reasoning, actor-centric object and relation graphs, and multi-head temporal graph reasoning; related work addresses zero-shot, long-term, group activity, and performance assessment.Source: §4.1 · p. 7
  • ST-GCN [99] connects naturally linked joints within frames and corresponding joints across consecutive frames to learn spatial and temporal action patterns.Follow-on methods add learnable or input-dependent edges [110], connect physically distant joints [111], [112], expand within-frame links [113], model edge features [115], or aggregate multi-scale graphs over dilated temporal windows.Source: Figure 6 · p. 8, §4.1 · p. 8

4.2 Temporal Action Localization

  • Temporal action localization uses proposal graphs or snippet graphs to capture relations for predicting action boundaries and categories.A video self-stitching graph network exploits multi-level correlations among cross-scale snippets; GNNs also support temporal proposal generation and video action segmentation.Source: §4.2 · p. 8

4.3 Others

  • GNNs support video object segmentation through higher-order frame relations and graph memory, and multi-object tracking through message passing over detection graphs,.For tracking, detections are nodes, links represent candidate associations across consecutive frames, and predicted edge activity determines object trajectories.Source: §4.3 · pp. 8–9
  • Human motion prediction methods model body-part relations at multiple scales or encode separable spatial, temporal, and spatio-temporal interactions.GNNs are also used to estimate 3D human poses from consecutive 2D poses.Source: §4.3 · p. 9
  • Trajectory-prediction methods use pedestrian locations and temporal histories to model interactions, including social relations and multi-scale scene context.Approaches include spatio-temporal GNNs, graph Transformers, and sparse directed spatial and temporal graphs.Source: §4.3 · p. 9

4.4 Discussion

  • Many video GNNs capture only frame-level dependencies or local region context, motivating methods that capture long-range global dependencies without redundant information.The discussion identifies temporal connectivity and spatio-temporal context as central to video understanding.Source: §4.4 · p. 9

5 VISION + LANGUAGE

The survey organizes vision-and-language applications of GNNs around modeling relations within visual and linguistic components and aligning them across modalities. It also identifies open questions about unified architectures, reasoning, explainability, and knowledge-aware methods.

5 VISION + LANGUAGE

  • Vision-and-language GNNs model visual relations, linguistic dependencies, or both jointly to support cross-modal semantic alignment and understanding.Applications include visual question answering, visual grounding, and image captioning.Source: §5 · p. 9

5.1 Visual Question Answering

  • GNN-based VQA uses image and question graphs to capture object relations, focus on relevant image parts, and support higher-order reasoning.Approaches align static image and question graphs, learn question-conditioned or dynamically updated visual graphs,, and use symbolic graphs for interpretable reasoning,. Fact-based methods retrieve supporting facts or combine graph-based knowledge reasoning with Transformer reasoning.Source: §5.1 · pp. 9–10

5.2 Visual Grounding

  • Visual grounding uses GNNs to capture visual and linguistic dependencies; uses an expression-conditioned visual relation graph and gated GCN, whereas performs dynamic graph attention and convolution for stepwise reasoning.Phrase grounding uses graphs of noun phrases and their dependencies,. Video grounding methods represent spatial relations across objects in frames and temporal relations between objects or intervals [178],.Source: §5.2 · pp. 10–11

5.3 Image and Video Captioning

  • Image and video captioning methods use graph-based object relations and temporal interactions to improve contextual representations for text generation.Image-captioning approaches encode semantic and spatial relations, hierarchical image-to-object relations, or language priors and selected scene subgraphs,; video methods model object relations across frames, or adapt temporal graphs with the decoder state.Source: §5.3 · p. 11

5.4 Others

  • Other applications include image-text matching, navigation, and video question answering, using graphs for cross-modal matching, action prediction, or temporal and object-level reasoning.Image-text methods reason over visual regions and sometimes textual or commonsense graphs,,,; navigation methods build evolving or topological environment graphs,,, while video QA methods model question, appearance, motion, or object graphs,.Source: §5.4 · pp. 11–12

5.5 Discussion

  • The discussion highlights open problems in unifying GNN architectures across tasks, extending reasoning and explainability beyond message passing, and incorporating knowledge graphs.These questions arise because GNNs are used to capture relations and align visual, linguistic, or multimodal graph representations across vision-and-language tasks.Source: §5.5 · p. 12

6 3D DATA ANALYSIS · 6.1 3D Representation Learning

GNNs support 3D representation learning by modeling geometric and topological relationships in point clouds and meshes. The section surveys methods that capture local and global interactions across these representations.

6.1 3D Representation Learning

  • GNNs capture geometric and topological relationships in irregular 3D data, making them useful for analysis across varying data sizes and resolutions.The survey focuses on point clouds and meshes as representations of complex 3D shapes.Source: §6.1 · p. 12

6.1.1 Point Cloud Representation

  • Point-cloud GNNs model points as spatial graphs: ECC uses edge-conditioned kernels, while DGCNN dynamically rebuilds feature-space graphs to capture local and global interactions.Fixed local-neighborhood graphs can limit flexibility and non-local modeling; CurveNet instead uses random walks to form non-local point-segment descriptors.Source: §6.1.1 · p. 12
  • Spectral and specialized point-cloud methods include Local-SpecGCN for local spectral filtering and pooling, HGNN for high-order correlations, and DiffConv for adaptive views.Wiersma et al. combine geometric operators on scalar and vector fields to encode surface-point direction.Source: §6.1.1 · p. 13
  • Point-cloud graph attention,,, weights neighbors using positions and learned features; graph-RNN models spatial and temporal relations in point-cloud sequences.Graph Transformers include PT, which attends over local k-NN graphs, and PCT, which uses global attention for local and non-local interactions.Source: §6.1.1 · p. 13
  • On ModelNet40, advanced GNN-based methods,,, achieve state-of-the-art point-cloud classification performance.3DCTN combines a Transformer with graph convolutions, while other methods,,,, simplify or accelerate point-cloud sampling, grouping, and pooling.Source: §6.1.1 · p. 13

6.1.2 Mesh Representation

  • Meshes are graphs of vertices and edges, and mesh-learning methods operate on vertices, edges, faces, or both edges and vertices.FeaStNet dynamically selects convolution weights; MeshCNN uses fixed-size edge neighborhoods; DCM-Net combines geodesic and Euclidean convolutions.Source: §6.1.2 · p. 13
  • Mesh methods extend feature extraction with multi-scale intrinsic convolution in SpiralNet++ [252], scale-invariant operations in PolyNet, and gauge-equivariant, rotation-invariant attention in GET.Other approaches use hierarchical subdivision, spectral mappings [259], or random walks along mesh edges, enhanced by AttWalk.Source: §6.1.2 · p. 13

6.2 3D Understanding

3D understanding methods use GNNs to capture inter-instance and contextual information, often building an input graph for low-level features and then a region- or instance-level graph for higher-level relations. The section covers 3D point-cloud segmentation and object detection, along with limitations and future directions.

6.2.1 3D Point Cloud Segmentation

  • For segmentation, Qi et al. fuse RGB semantics and point-cloud geometry on point-level graphs, while SPG uses smaller superpoint graphs for efficient, high-quality segmentation.Landrieu et al. use graph-structured contrastive loss and cross-partition weighting to form compact superpoints with high contrast at object boundaries.Source: §6.2.1 · p. 14
  • Segmentation aggregation includes coordinate-and-feature attention, dense local pair interactions, hierarchical point-edge graphs, and scale-aware or specialized methods [269],,,,,,.These methods model local geometry, accelerate graph construction, capture cross-scale correlations, or distinguish nodes within and across instances.Source: §6.2.1 · p. 14
  • On S3DIS-Area5, point-transformer-based methods, achieve SOTA segmentation accuracy; Tab. 6 compares GNN-based and non-GNN-based methods,,.The supplied passage gives no table values, so no numerical comparison is reported.Source: §6.2.1 · p. 14

6.2.2 3D Object Detection

  • Detection methods fuse camera imagery with LiDAR,, use radius graphs and auto-registration in Point-GNN, or hierarchical semantics in HGNet; object-level graphs model relations,, [282].PointRGCN [282] combines point- and object-level GNNs to capture inter-point and inter-object relations and context.Source: §6.2.2 · pp. 14–15
  • GCNs are the most popular 3D-understanding solutions for local neighborhoods and global context, but combining graph convolution with self-attention remains underexplored; future frameworks could balance MLP, convolution, and attention.The proposed scope spans point-, object-, and scene-level understanding, with efficiency and performance as balancing considerations.Source: §6.2.2 · p. 15
  • Automated panoptic symbol spotting, models symbolwise dependencies and is gaining attention for creating 3D prototypes in architecture, engineering, and construction.Source: §6.2.2 · p. 15

6.3 3D Generation

The section surveys GNN use in 3D generation, where models extract input features, manipulate 3D data, or generate shapes from references [s36.p1#9eb8d7].

6.3 3D Generation

  • For 3D generation, GNNs extract input features and learn manipulations such as point-position adjustment or generate shapes from reference inputs [s36.p1#9eb8d7].The section frames these approaches around encoder-based input representation and decoder transformation into 3D outputs.Source: §6.3 · p. 15

6.3.1 Point Cloud Completion and Upsampling

  • Completion methods address missing geometry with skeleton-based ECG, local/global fusion, and GCN-guided points; upsampling includes k-NN aggregation, PU-GCN, AR-GCN [290], and PUGeo-Net.GNNs capture local and global point-cloud topology to help avoid inconsistent generated content.Source: §6.3.1 · p. 15

6.3.2 3D Data Denoising

  • Denoising GNNs learn local neighborhood structure and integrate point features; representative approaches use dynamic aggregation, manifold learning, hierarchical features, rotation-invariant patches, and mesh dual graphs,,,,,.The task estimates noise N from input I to produce I′ = I − N.Source: §6.3.2 · p. 15

6.3.3 3D Reconstruction

  • Reconstruction uses graph-based encoders and decoders for point clouds, meshes, scenes, and humans; general 3D graph Transformers remain underexplored, and efficiency is a practical direction.Examples include FoldingNet and StructureNet for point clouds, Pixel2Mesh and Mesh R-CNN for meshes, and Mesh Graphormer for human meshes.Source: §6.3.3 · pp. 15–16

7 MEDICAL IMAGE ANALYSIS

This section surveys GNN applications in medical imaging, including brain activity analysis, disease diagnosis, and anatomical segmentation. It groups methods by task and describes graph-based approaches and future directions.

7.1 Brain Activity Investigation

  • Subject-based brain graphs represent people as nodes, while region-based graphs represent brain areas; both schemas support brain activity analysis.Subject-based graphs are constructed over subjects, whereas region-based graphs are constructed over brain areas and use area-specific representations.Source: §7.1 · p. 16, Figure 11 · p. 16, Figure 12 · p. 16

7.1.1 Brain Signal Disorder Recognition

  • Parisot et al. reported spectral GCNs outperforming ridge classification by at least 5% across datasets for ASD identification.Other approaches include ensemble GCNs to address sparse population graphs, linear or hybrid GCNs,, federated training across institutions, and region-based siamese and triplet GCNs,. Later methods add pooling regularization, ROI-aware convolution, and multi-kernel inception GCNs,,.Source: §7.1.1 · pp. 16–17

7.1.2 Brain State Decoding

  • Brain-state decoding uses graph methods to interpret rs-fMRI cognitive tasks, predict age and gender, and learn dynamic connectome representations.Kim et al. use spatial-temporal attention and a Transformer encoder to process short sequences of brain connectome graphs.Source: §7.1.2 · p. 17

7.2.1 Brain Disease Identification

  • For brain disease identification, population GCNs for Alzheimer’s disease surpassed prior state of the art by about 5% at least, while Parkinson’s methods use regional multimodal graphs.Alzheimer’s approaches also address MCI and EMCI, adaptive graph uncertainty, attention-guided dynamic-graph features, and cortical geometry –; Parkinson’s methods fuse modalities, rank subject-specific features, or encode cross-modality relations –.Source: §7.2.1 · p. 17

7.2.2 Chest Disease Analysis

  • Chest-disease methods model relations among similar CT scans, image and textual features, abnormalities, labels, diseases, and anatomy.Anatomy-aware GCNs for mammogram classification are reported to achieve state-of-the-art results and provide interpretable diagnoses,. Other approaches include graph embeddings for classification and report generation, label co-occurrence embeddings, and attribute-based probabilistic and neural reasoning.Source: §7.2.2 · pp. 17–18

7.3.1 Brain Surface Segmentation

  • Brain-surface segmentation methods use graph convolutions on cortical meshes and manifolds, with a spectral graph Transformer aligning surfaces and outperforming traditional GCN models.GCNs and GATs also outperformed traditional mesh segmentation approaches, sometimes by over 5%; other methods learn spectral embeddings or handle surfaces with impaired spherical topology.Source: §7.3.1 · p. 18

7.3 Anatomy Segmentation

  • Vessel segmentation uses GNNs and CNN-GNN hybrids to encode vascular connectivity, structure, and shape, while other anatomical applications include airway, organ, prostate, optic, lymph-node, and brain-tissue segmentation.Vessel approaches cover coronary, pulmonary, retinal, and intracranial vessels, including tubular priors and multi-scale fusion –; other methods use graph refinement or hybrid U-shaped models, among other strategies –.Source: §7.3.2 · p. 18, §7.3.3 · p. 18

7.4 Discussion

  • The discussion identifies self-supervised graph representation learning, multi-disease knowledge graphs, and graph-based reasoning as promising directions for medical-image analysis and segmentation.Proposed examples include contrasting neighboring and non-neighboring nodes, integrating knowledge graphs built from domain knowledge or clinical descriptions, and using annotated classes to support rare diseases or unseen organs.Source: §7.4 · pp. 18–19

8 CONCLUSIONS

The survey presents a task-oriented review of GNNs and graph Transformers in computer vision, organizing methods by input modality and discussing their innovations, limitations, and research directions.

8 CONCLUSIONS

  • The survey groups classical and recent GNN and graph Transformer methods into five categories based on input modality.Examples include images, videos, and point clouds.Source: §8 · p. 19
  • GNNs and graph Transformers show flexibility and strength on relational tasks, while visual reasoning remains a major challenge.Source: §8 · p. 19
  • The survey discusses key innovations, limitations, and potential research directions to support progress toward human-like visual understanding.Source: §8 · p. 19

Related papers

Loading 2209.13232v4…