PaperlayerView exact revision on arXiv

Source-linked AI summary

Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification

Yuankai Luo, Lei Shi, Xiao-Ming Wu

arXiv:2406.08993v2cs.LG

TL;DR

The paper addresses whether Graph Transformers genuinely outperform classic message-passing GNNs for node classification. It reevaluates GCN, GAT, and GraphSAGE across diverse datasets with tuned configurations and finds that classic GNNs achieve the top rank on 17 of 18 datasets. The results support more rigorous, better-controlled empirical comparisons of graph models.

  • Problem

    The paper examines whether reported Graph Transformer advantages over classic GNNs reflect model superiority or incomplete GNN hyperparameter evaluation.

  • Method

    The study benchmarks GCN, GAT, and GraphSAGE against Graph Transformers across 18 datasets and analyzes normalization, dropout, residual connections, and network depth.

  • Results

    Classic GNNs achieve the top rank on 17 of 18 evaluated datasets, matching or surpassing advanced Graph Transformers.

  • Takeaways & Limitations

    The findings challenge perceived Graph Transformer superiority and support more rigorous empirical evaluation of graph-model capabilities.

  • Takeaways & Limitations

    The study focuses only on node classification and does not evaluate graph classification or link prediction.

Abstract

from arXiv · show

Graph Transformers (GTs) have recently emerged as popular alternatives to traditional message-passing Graph Neural Networks (GNNs), due to their theoretically superior expressiveness and impressive performance reported on standard node classification benchmarks, often significantly outperforming GNNs. In this paper, we conduct a thorough empirical analysis to reevaluate the performance of three classic GNN models (GCN, GAT, and GraphSAGE) against GTs. Our findings suggest that the previously reported superiority of GTs may have been overstated due to suboptimal hyperparameter configurations in GNNs. Remarkably, with slight hyperparameter tuning, these classic GNN models achieve state-of-the-art performance, matching or even exceeding that of recent GTs across 17 out of the 18 diverse datasets examined. Additionally, we conduct detailed ablation studies to investigate the influence of various GNN configurations, such as normalization, dropout, residual connections, and network depth, on node classification performance. Our study aims to promote a higher standard of empirical rigor in the field of graph machine learning, encouraging more accurate comparisons and evaluations of model capabilities.

1 Introduction

The paper reevaluates whether Graph Transformers truly outperform classic message-passing GNNs for node classification. Across 18 diverse datasets, tuned GCN, GAT, and GraphSAGE models achieve the top rank on 17 datasets, while ablations identify influential configurations.

  • Study scope: The study compares GCN, GAT, and GraphSAGE against recent Graph Transformers across 18 real-world benchmark datasets.The benchmark spans homophilous, heterophilous, and large-scale graphs.
  • Main findings: With proper hyperparameter tuning, classic GNNs achieve the top rank on 17 of 18 node-classification datasets, challenging claims of GT superiority.The evaluated graphs include homophilous, heterophilous, and large-scale settings with up to millions of nodes.
  • Motivation: The authors argue that previously reported GT advantages may reflect suboptimal hyperparameter configurations in GNN evaluations.This conclusion motivates a more comprehensive reassessment of classic models.
  • Ablations: The analysis examines normalization, dropout, residual connections, and network depth as factors affecting GNN node-classification performance.These configurations are investigated through ablation studies.

2 Classic GNNs for Node Classification

The paper formulates node classification with message-passing GNNs and compares three classic architectures: GCN, GraphSAGE, and GAT. These models differ in how they aggregate or weight neighboring-node information before prediction.

  • Message Passing: Message-passing GNNs iteratively aggregate neighboring-node information and update each node representation, beginning from its node features.The final-layer representation is passed to a prediction head for label prediction.
  • Classic GNNs: GCN, GraphSAGE, and GAT are the three classic GNNs studied, differing in how they learn node representations.GAT uses masked self-attention to assign different weights to neighboring nodes.
  • GAT: GAT computes neighbor-dependent attention using trainable weights and concatenation for edges between nodes.The propagation rule applies to each edge (v, u) in the graph.
  • Node Classification: Node classification predicts unlabeled-node labels by minimizing training loss between predicted and ground-truth labels.The prediction head receives the representation produced by the final GNN layer.
  • Graph Types: The paper distinguishes homophilous graphs, whose edges often join same-class nodes, from heterophilous graphs, whose connected nodes may have different classes.Although GNNs implicitly assume homophily, cited empirical work reports that standard GCNs can also work well on heterophilous graphs.

3 Key Hyperparameters for Training GNNs

The paper studies normalization, dropout, residual connections, and network depth as key GNN training hyperparameters. These choices address training stability, overfitting, gradient problems, and depth-related representation collapse.

  • Normalization: Normalization stabilizes GNN training by reducing covariate shift, maintaining embedding distributions, and enabling higher learning rates.Layer Normalization or Batch Normalization can be applied before activation in every layer.
  • Dropout: Dropout reduces hidden-neuron co-adaptation and addresses overfitting effects that accumulate through message passing.It is typically applied to feature embeddings after the activation function.
  • Residual Connections: Residual connections directly link layer inputs to outputs, mitigating vanishing gradients and improving GNN expressiveness.The linear component helps preserve distinguishable node representations while addressing over-smoothing and over-squashing.
  • Network Depth: Deeper GNNs can extract more complex features but risk over-smoothing, so typical practice uses only 2 to 5 layers.The section frames network depth as a trade-off between representational capacity and indistinguishable node representations.

4 Experimental Setup for Node Classification

The study evaluates classic GNNs and graph-transformer baselines across diverse graph datasets using standardized splits, metrics, and tuned training configurations.

  • Evaluation Protocol: The dataset suite uses established splits and metrics, including semi-supervised citation-network evaluation, 60%/20%/20% train/validation/test splits for several homophilous datasets, and official or benchmark-provided settings elsewhere.Detailed experimental setup and hyperparameters are provided in Appendix A.
  • Datasets: The evaluation covers homophilous, heterophilous, and large-scale graphs, including OGB datasets with 0.16M–2.4M nodes and pokec with 1.6M nodes.The study maintains OGB standard evaluation settings and follows established evaluation settings for pokec.
  • Baselines: The benchmark compares GCN, GraphSAGE, and GAT against scalable and powerful graph transformers, including SGFormer, Polynormer, GOAT, NodeFormer, NAGphormer, GraphGPS, and Exphormer.
  • Hyperparameter Configurations: Hyperparameter tuning varies learning rate, hidden dimension, normalization, residual connections, dropout, and depth for classic GNNs, while graph-transformer baselines are retrained under the same search space and environments.The learning-rate search is {0.001, 0.005, 0.01}, hidden dimensions are {64, 256, 512}, dropout rates are {0.2, 0.3, 0.5, 0.7}, and depth ranges from 1 to 10 layers.
  • Evaluation Protocol: Results are reported as means and standard deviations over five independent runs with different initializations.

5 Empirical Findings

Across 18 datasets, lightly tuned classic GNNs often match or outperform graph transformers, ranking first on 17 datasets. Ablations show that normalization, dropout, residual connections, and depth affect performance differently across graph types and scales.

  • Overall performance: Classic GNNs achieve the top rank on 17 of 18 datasets, often outperforming or matching state-of-the-art graph transformers across homophilous, heterophilous, and large-scale graphs.The overall comparison is reported across Tables 2–4, with Table 3 covering heterophilous graphs and Table 4 covering large-scale graphs.
  • Homophilous graphs: On homophilous graphs, slight hyperparameter adjustments make classic GNNs highly competitive; GAT∗ reaches first place on WikiCS after a 4.16% accuracy increase, while GCN∗ improves 1.54%–3.50% across four datasets.GraphSAGE∗ also becomes the top model on Photo and CS, surpassing Polynormer and SGFormer.
  • Heterophilous graphs: On heterophilous graphs, classic GNNs rank among the top models on five of six datasets, with GCN∗ gains of 4.98% on Chameleon, 6.34% on Squirrel, and 17.58% on Roman-Empire.The reported improvements primarily stem from residual connections, and classic GNNs surpass specialized GNNs and leading graph transformers.
  • Large-scale graphs: On large-scale graphs, classic GNNs rank first across all four datasets; GAT∗ improves 12.99% on ogbn-proteins and exceeds SGFormer by 5.09%.GCN∗ leads on ogbn-arxiv and pokec, while GraphSAGE∗ leads on ogbn-products; all three models improve by over 10% on pokec.
  • Normalization: Normalization matters most on large-scale graphs, where removing it reduces GraphSAGE∗ and GAT∗ accuracy by 4.79% and 4.69% on ogbn-proteins, respectively.Its ablation causes less substantial changes on small graphs.
  • Dropout, residual connections, and depth: Dropout is consistently important, residual connections are especially beneficial on heterophilous graphs, and deeper networks help heterophilous graphs more than homophilous graphs.Removing dropout lowers accuracy by up to 6.57% on Roman-Empire, while removing residual connections lowers GCN∗ accuracy by 16.43% there; 2–6 layers are optimal on homophilous graphs and ogbn-arxiv.

6 Conclusion

The study reevaluates foundational GNN models for node classification and finds that they can match or surpass Graph Transformers across diverse graph datasets.

  • The reevaluation shows classic GNNs can reach or surpass Graph Transformer performance on various graph datasets.The study also examines how GNN configuration choices affect performance and advocates more rigorous empirical evaluation.

A Datasets and Experimental Details

The experiments use PyG and DGL and run on a workstation equipped with eight RTX 3090 GPUs.

  • The implementation uses PyG and DGL for the experiments.
  • Experiments are conducted on a single workstation with 8 RTX 3090 GPUs.

A.2 Hyperparameters and Reproducibility

The reproducibility setup combines expanded depth searches for heterophilous graphs, dataset-specific settings, validation-based model selection, and repeated runs.

  • Hyperparameter selection: For heterophilous graphs, the layer search includes 12, 15, and 20 layers because deeper networks showed empirical performance improvements.
  • Training strategy: Large graphs use neighbor sampling for ogbn-proteins and random partitioning for pokec and ogbn-products, while other datasets use full-batch training.
  • Evaluation protocol: The model with the highest validation result supplies the test evaluation, with means and standard deviations reported over 5 independent runs.
  • Reproducibility: The implementation is released under the MIT License.
  • Hyperparameter selection: The appendix provides dataset-specific GCN∗ hyperparameter settings in Table 8.

B.1 GAT∗with Edge Features on ogbn-proteins

The edge-feature analysis on ogbn-proteins compares GAT∗ with deeper GNN baselines and reports that moderate depth can be sufficient under the study’s setup.

  • Comparison scope: DeeperGCN is excluded from the main comparison because its 85.50% result uses edge features, unlike the standard ogbn-proteins baseline setup.
  • GAT∗ with Edge Features on ogbn-proteins: 87.47% accuracy from a 6-layer GAT exceeds DeeperGCN’s 85.50% on ogbn-proteins with edge features.The comparison uses the edge-feature-enhanced GAT∗ and results reported in Table 11.
  • GAT∗ with Edge Features on ogbn-proteins: The study concludes that GNNs need not be as deep as DeeperGCN, with 2 to 10 layers typically sufficient.
  • Additional results: The appendix tables report node-classification results for homophilous, heterophilous, and large-scale graphs.
  • Depth ablation on heterophilous graphs: The study examines layer counts beyond 10 for GCN∗ and GraphSAGE∗ on Roman-Empire and Minesweeper, reporting the results in Table 12.

B.3 Jumping Knowledge Mode and Early Results

The study examines Jumping Knowledge (JK) as a GNN configuration and reports that detailed tuning often makes it unnecessary. Early visualization results also show greater inter-class separation for GCN* embeddings than Polynormer* embeddings.

  • B.3 Jumping Knowledge Mode and Early Results: Summation JK forms a node representation by adding representations from successive GNN layers.The displayed formulation combines layer outputs through hL.
  • B.3 Jumping Knowledge Mode and Early Results: Detailed tuning found that omitting JK usually produced comparable, and sometimes better, results than using it, so JK was removed from the search space.Earlier experiments had selected only subsets of configurations based on experience rather than exhaustively searching the space.
  • C Visualization: GCN* embeddings display greater inter-class distances than Polynormer* embeddings in the t-SNE visualization.

D Limitations & Broader Impacts

The paper reports no societal consequences requiring specific emphasis and limits its study to node classification. It identifies graph classification and link prediction as important directions for extending the benchmark.

  • Broader Impacts: The authors state that no societal consequences of the work require specific highlighting.
  • Limitations: The study focuses solely on node classification and does not evaluate graph classification or link prediction.
Loading 2406.08993v2…