PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Graph Neural Networks

Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, Xavier Bresson

arXiv:2003.00982v5cs.LGstat.ML

TL;DR

GNN research lacks standardized evaluations that make progress and architectural differences difficult to assess. This paper presents an open-source, modular benchmark with diverse datasets, fair fixed-parameter comparisons, and infrastructure for testing new designs; its positional-encoding study improved message-passing GCNs and helped motivate subsequent PE research.

  • Problem

    Inconsistent experiments on traditionally used datasets make it difficult to track progress and distinguish architectural effects from other factors.

  • Method

    The paper presents an open-source, modular framework with diverse medium-scale datasets, standardized protocols, fixed parameter budgets, and facilities for testing new GNN ideas.

  • Results

    The Laplacian positional-encoding study improved message-passing GCNs across synthetic and real-world datasets, including AQSOL, and subsequent PE ideas followed.

  • Takeaways & Limitations

    The benchmark provides a practical setting for fair GNN comparison and has helped steer research on positional encodings and other GNN designs.

Abstract

from arXiv · show

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

1. Introduction

GNNs are increasingly applied across diverse domains, but inconsistent evaluations make progress and architectural contributions difficult to assess. The framework addresses this gap with an open-source benchmark using diverse datasets and fixed parameter budgets.

  • GNNs are being developed for applications spanning chemistry, physics, social sciences, transportation, knowledge graphs, recommendation, and neuroscience.
  • Inconsistent comparisons on traditionally used datasets make it difficult to distinguish complex, simple, and graph-agnostic architectures.
  • The framework provides diverse medium-scale datasets for fair GNN comparison under fixed parameter budgets.
  • The open-source infrastructure has been widely used, reaching 2,000+ GitHub stars, 380+ forks, and 470+ citations by the reported counts.

2. Overview of GNN Benchmarking Framework

The benchmark assembles medium-scale datasets designed to expose GNN performance across graph properties, tasks, and application domains. Its standardized infrastructure and fixed parameter budgets support fair architectural comparisons while remaining practical for academic research.

  • Datasets: Dataset construction faces unresolved questions about representativeness, graph preparation choices, and the computational cost of very large datasets.
  • Datasets: The benchmark includes 12 datasets from real-world sources and mathematical models, covering graph-, node-, and edge-level tasks across diverse domains.
  • Datasets: These datasets are intended to separate GNN performance on specific graph properties and support identifying first principles.
  • Coding Infrastructure: Standard protocols and fixed model-parameter budgets enable fair comparisons by separating architectural gains from additional learning capacity.

3. How can the benchmark be used to explore new insights?

The modular benchmark supports controlled experiments on data preprocessing, GNN layers, normalization, and new architectures. Its demonstrated use with Laplacian positional encodings produced improvements and helped stimulate subsequent positional-encoding research.

  • Researchers can modify modular components and run multiple experiments fairly and easily across the benchmark’s diverse datasets.
  • The framework was used to study graph positional encodings based on Laplacian eigenvectors as an example of exploring new GNN insights.
  • Laplacian positional encoding improved message-passing GCNs on three synthetic datasets and additional real-world datasets, including AQSOL.
  • Subsequent literature proposed further positional-encoding ideas after the benchmark introduced Laplacian positional encoding.

4. Conclusion

The paper presents a modular, open-source GNN benchmarking framework designed for fair, reproducible comparison across diverse graph tasks and architectures. It also uses the framework to study graph positional encodings and support further GNN research.

  • Conclusion: The framework combines diverse datasets, rigorous experimental protocols, and modular code to benchmark GNN architectures fairly and reproducibly.Its pipelines cover message-passing GCNs and Weisfeiler–Lehman GNNs, with task-specific prediction layers.
  • Conclusion: The framework enabled the study of graph positional encodings, including Laplacian eigenvectors, as a way to improve message-passing GCN performance.The paper describes positional features as graph coordinates intended to distinguish distant nodes while keeping nearby nodes similar.
  • Conclusion: Benchmarking aims to identify which GNN architectures, principles, and mechanisms are universal, generalizable, and scalable on larger, more challenging datasets.The paper motivates benchmarking as a way to quantify progress and distinguish architectural design from additional learning capacity.
  • Conclusion: The framework supports standard GCN pipelines that embed node and edge features, perform message passing, and produce task-specific predictions.Message-passing updates depend on local neighborhoods and have O(E) complexity, reducing to O(n) for sparse graphs.
  • Conclusion: WL-GNN pipelines represent graph information as dense tensors and apply successive higher-order GNN computations before task-specific prediction.The benchmark presents WL-GNNs as a class intended to improve the theoretical limitations of message-passing GNNs.

B.2.3 Task-based network layers

Task-based network layers convert final or intermediate GNN representations into predictions for graph, node, and edge tasks. The benchmark standardizes these downstream components while preserving model-specific readout designs where required.

  • Graph classifier layer: RingGNN classifiers use features from all intermediate layers, while 3WL-GNNs apply diagonal and off-diagonal max pooling at every layer.These readout choices follow the respective model implementations and differ from the generic final-layer averaging scheme.
  • Task-based network layers: The benchmark describes task-specific graph regression, node classification, and edge classification layers for higher-order GNNs.For link prediction, edge features are obtained by concatenating node features.
  • Experimental setup: Experiments use standardized training procedures and parameter budgets to compare models rather than optimize each architecture independently.The benchmark uses 100k parameters for all tasks and 500k when scaling models to larger parameter counts and deeper layers.
  • Task-based network layers: Final node or dense-tensor representations are passed to downstream network suffixes, usually three-layer MLPs, for task-specific prediction.GIN, RingGNN, and 3WL-GNN may instead use outputs from every intermediate layer.

C.2 Graph Regression with AQSOL dataset

AQSOL adds a medium-sized molecular regression dataset with experimentally measured aqueous-solubility targets. It uses filtered molecular graphs, scaffold splitting, and mean absolute error evaluation within the benchmark framework.

  • Dataset: AQSOL contains 9,823 molecular graphs derived from AqSolDB after filtering graphs without edges and graphs with missing node features.Node features encode heavy-atom types, while edge features encode bond types.
  • Dataset: Its regression target is experimentally measured aqueous solubility standardized to LogS units.These standardized values are used as the property to regress in AQSOL.
  • Role in benchmark: The benchmark positions AQSOL alongside ZINC as a molecular graph regression dataset with real-world chemical targets.AQSOL complements benchmark datasets spanning multiple graph-learning tasks and application domains.
  • Splitting: The dataset uses scaffold splitting with 7,831 training, 996 validation, and 996 test graphs.The split ratio is 8:1:1.
  • Performance measure: AQSOL performance is evaluated with mean absolute error, matching the regression evaluation used for ZINC.The benchmark table reports results averaged over four runs with four different seeds and lower values are better.

C.3 Link Prediction with OGBL-COLLAB dataset

The supplied passages describe OGBL-COLLAB as a large collaboration-network link-prediction dataset with scientist features and collaboration edge information. They also include benchmark context for evaluating graph models and unrelated dataset descriptions.

  • Dataset: OGBL-COLLAB represents collaborations among approximately 235K scientists indexed by Microsoft Academic Graph.Nodes are scientists and edges denote collaborations.
  • Dataset: Node features are 128-dimensional vectors formed by averaging word embeddings from each scientist’s papers.
  • Evaluation: Matrix factorization provides a feature-agnostic comparison for testing whether GNNs use node features in addition to graph structure.

C.6 Node Classification with SBM (PATTERN/CLUSTER) datasets

The SBM datasets benchmark node-level pattern recognition and semi-supervised clustering by controlling community structure and noise. PATTERN searches for embedded graph patterns, while CLUSTER infers community labels from sparse node supervision.

  • Dataset construction: SBM datasets use stochastic block models to control task difficulty through intra- and inter-community connection probabilities.The inter-community probability q acts as the noise level.
  • PATTERN: PATTERN finds a fixed 20-node graph pattern embedded in larger graphs with five randomly sized communities.The generated graphs use p = 0.5, q = 0.35, and node features from a three-value vocabulary.
  • CLUSTER: CLUSTER infers six community classes from graphs containing mostly unknown node features and one labeled node per community.Graphs contain 40–190 nodes, with p = 0.55 and q = 0.25.

C.8 Graph Classification and Isomorphism Testing with CSL dataset

CSL is a small symmetric graph dataset designed to test GNN expressivity through classification across skip-link isomorphism classes. The supplied passages also position CSL alongside synthetic datasets used to validate graph positional encodings.

  • CSL dataset: CSL contains 4-regular cycle graphs whose skip-link lengths define the isomorphism classes.The benchmark uses G41,C with ten specified values of C and 150 class-balanced graphs.
  • Evaluation: CSL results use 5-fold cross-validation repeated across 20 runs with different seeds.The paper reports maximum, minimum, average, and standard deviation scores.
  • Positional encodings: The paper uses CSL primarily to validate the impact of graph positional encodings.CSL is considered together with CYCLES and GraphTheoryProp for this validation.

D. Analysis and Discussion of Benchmarking Results

The benchmark experiments compare GNN families across diverse tasks and show strong performance from scalable GCN-based models, particularly anisotropic variants. They also identify training challenges for WL-GNNs and demonstrate the framework’s use for developing Laplacian positional encodings.

  • Benchmark findings: MLP baselines consistently score poorly, indicating that the proposed tasks require graph structure.The comparison spans all datasets in Tables 3–10.
  • Benchmark findings: GCNs outperform WL-GNNs on the proposed medium-scale datasets while scaling to 16 layers with linear complexity on sparse graphs.3WL-GNNs and RingGNNs encounter loss divergence or out-of-memory errors when made deeper.
  • Benchmark findings: Anisotropic models, particularly GAT and GatedGCN, generally outperform isotropic GCN variants across the reported graph-level and node-level datasets.Exceptions include vanilla GCN on WikiCS, GraphSage on MNIST, and MoNet on CIFAR10.
  • Training challenges: WL-GNN training shows relatively high standard deviation because dense variable-sized tensors hinder batching and batch normalization.The authors report that layer normalization did not solve the issue and that WL-GNNs could not be trained in CPU memory for the single COLLAB graph.
  • Studies using the framework: The framework supports experiments on graph positional encodings, including adding Laplacian eigenvectors to MP-GCNs to address their representation limitations.The positional features are added before graph processing, with smaller eigenvectors providing smooth coordinates for neighboring nodes.
  • Studies using the framework: Laplacian eigenvectors are defined from graph-Laplacian spectral information and used as node positional features, with random sign flipping during training.The experiments use the k smallest non-trivial eigenvectors as positional-encoding dimensions.

E.1.3 Experiments and Analysis

The experiments show that positional encodings improve message-passing GNNs across synthetic and real-world tasks, while Laplacian encodings outperform index-based alternatives under the tested settings.

  • Synthetic datasets: 99% mean accuracy was achieved by MP-GCNs with Laplacian eigenvectors on CSL, compared with 97% for 3WL-GNN and 25% for RingGNN.GINs and GatedGCNs augmented with Laplacian positional encodings also improved consistently on CYCLES and GraphTheoryProp.
  • Real-world datasets: Laplacian positional encodings improved performance on ZINC, AQSOL, PATTERN and CLUSTER, while COLLAB remained statistically unchanged.MNIST, CIFAR10 and TSP were excluded because their nodes already contain positional features in R2.
  • Challenges with Laplacian eigenvectors: Laplacian eigenvectors reduce positional ambiguity from n! node orderings to 2^k sign choices when k ≪ n.Taking absolute values removes sign ambiguity but seriously degrades positional-feature expressivity.
  • Positional-encoding variants: Laplacian positional encodings with random sign flipping consistently produced the best results among the tested positional encodings.Randomly permuting node order improved index encodings, but Laplacian encodings still clearly outperformed them.
  • Anisotropic GNNs: The experiments compare isotropic and anisotropic GNN variants using edge features or explicit edge representations, including GAT and GatedGCN designs.The variants differ in whether edge information is computed from adjacent node features or maintained across layers.

Numerical Experiments and Analysis

Experiments on TSP and COLLAB show that edge-aware anisotropic models can outperform isotropic variants under equal parameter budgets, but explicit edge representations are task-dependent.

  • Experimental constraints: COLLAB experiments used a reduced 27K-parameter budget and a maximum 24-hour training time because explicit edge representations incurred time and memory costs.GatedGCN-E and GAT-E initialize edge representations from dataset input edge features.
  • Edge features: 0.75 vs. 0.64 F1 score on TSP and 50.6% vs. 35.9% Hits@50 on COLLAB were obtained by GatedGCN with edge features versus its isotropic variant.These comparisons use the same model parameters.
  • Explicit edge representations: 0.78 vs. 0.67 F1 score on TSP was achieved by GAT-E versus standard GAT when explicit edge representations were maintained and initialized with euclidean distances.On COLLAB, adding explicit edge representations and input features degraded performance.
  • Aggregation functions: Max and mean aggregation provided a powerful inductive bias for COLLAB, while sum-aggregation models failed to beat the simple matrix-factorization baseline.GraphSage with max aggregation performed close to anisotropic GAT and GatedGCN models.
  • GraphSage variants: Adding edge features did not significantly improve anisotropic GraphSage with max aggregation, while maintaining explicit edge representations hurt performance.The authors associate the latter result with very small hidden dimensions under the memory constraints of representing 235K nodes and 2.3M edges.

F. Experiments on TU datasets

The TU-dataset experiments indicate that small conventional graph-classification datasets provide unstable and weak evidence for distinguishing GNN architectures.

  • Experimental protocol: The experiments used 10-fold cross validation with stratified 8:1:1 train-validation-test splits and reported mean accuracy with standard deviation.The saved indices and shared protocol were used across experiments for fair comparisons.
  • Results: All neural networks had similar statistical test performance on ENZYMES, DD and PROTEINS because the standard deviations were large.Repeating the same protocol with a different seed changed the model ranking.
  • Results: Graph-agnostic MLP baselines performed as well as GNNs on DD and PROTEINS.This limits the evidential value of these datasets for identifying graph-specific architectural advantages.
  • Interpretation: The benchmark authors identify small TU datasets as difficult settings for determining which GNNs are powerful and robust.The reported ranking changes are attributed to small dataset sizes and gradient non-determinism.
  • Graph size normalization: GraphNorm normalizes node features with respect to graph size before BatchNorm, but its design lacks a concrete theoretical basis.The authors motivate it from initially promising empirical results on ZINC and CLUSTER and call for more principled normalization methods.

H. Elaboration on Benchmarking Design Choices

The benchmark is designed to identify robust architectural principles through fixed parameter budgets and reproducible experiments, while recognizing practical hardware and scaling constraints.

  • Framework design: The framework supports fair experimentation by combining standardized pipelines with controlled parameter budgets and modular components.Researchers can modify data, layers or other components and run multiple experiments across the benchmark datasets.
  • Parameter budgets: Fixed parameter budgets are intended to distinguish architectural gains from gains caused by additional trainable capacity.The framework is aimed at identifying first principles and performance trends rather than directly optimizing state-of-the-art scores.
  • Timing: The framework reports average wall-clock time per epoch and total training time despite implementation and hardware differences.Experiments were implemented in DGL/PyTorch and run on single GPUs with a maximum duration of 12 hours.
  • Memory usage: GPU memory varies across batches for variable-size graphs, so memory usage was measured during training on ZINC and SBM-CLUSTER.The reported forward-pass batches contain 128 ZINC graphs and 64 SBM-CLUSTER graphs.
  • Memory usage: GatedGCN was relatively more memory-intensive than GCN and GAT in the reported forward-pass measurements.All displayed GNNs used 500k learnable parameters.
Loading 2003.00982v5…