PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Graph Neural Networks

Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, Xavier Bresson

arXiv:2003.00982v5cs.LGstat.ML

TL;DR

Without a community-standard benchmark, inconsistent evaluations make GNN progress difficult to track and architectures difficult to compare. The paper presents an open-source benchmarking framework and reports that Laplacian positional encoding improved message-passing GCNs across synthetic and real-world datasets.

  • Problem

    Without a community-standard benchmark, inconsistent evaluations make it difficult to track progress and differentiate GNN architectures.

    Source: §1 · p. 2

  • Method

    The paper introduces an open-source framework with diverse medium-scale datasets, fixed parameter budgets, and modular infrastructure for fair GNN comparisons.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Laplacian positional encoding improved message-passing GCNs on three synthetic datasets and other real-world datasets, including AQSOL.

    Source: §3 · p. 4

  • Takeaways & Limitations

    The framework supports quick, robust testing of new GNN ideas and exploration of insights that can direct further research.

    Source: §4 · p. 4

Abstract

from arXiv · show

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

1. Introduction

The paper presents an open-source GNN benchmarking framework designed for fair comparison across diverse datasets, addressing the difficulty of tracking progress without community-standard benchmarks. Its update adds mathematical datasets and AQSOL, a molecular dataset with experimentally measured targets, and uses the framework to explore graph positional encoding.

1. Introduction

  • Without a community-standard benchmark, inconsistent comparisons on traditionally used datasets make it difficult to distinguish complex, simple, and graph-agnostic models.The passage cites Hoang and Maehara (2019), Chen et al. (2019a), and Errica et al. (2019).Source: §1 · p. 2
  • The framework combines diverse, discriminative medium-scale datasets, fixed parameter budgets, and modular infrastructure to support fair GNN comparisons.The updated collection includes 12 datasets.Source: §1 · p. 2
  • The updated benchmark adds mathematical datasets for testing theoretical graph properties and AQSOL, whose real-world experimental solubility targets contrast with ZINC’s computed targets.Together, the collection comprises 12 datasets.Source: §1 · p. 2
  • Since its March 2020 release, the framework has gained 2,000+ GitHub stars, 380+ forks, and 470+ citations, and has supported studies across several GNN design areas.These areas include aggregation, expressive power, pooling, normalization and regularization, robustness, and efficiency.Source: §1 · p. 2

2. Overview of GNN Benchmarking Framework

The framework combines diverse medium-scale datasets and modular, reproducible infrastructure with fixed parameter budgets to support fair and informative GNN comparisons.

Datasets

  • Dataset design faces unclear standards for representativeness, arbitrary feature-preparation choices, and substantial GPU demands for very large graphs (Chiang et al., 2019; Rossi et al., 2020; Hu et al., 2021).Source: §2 · p. 2
  • The 12 medium-scale datasets span real-world and mathematical sources, graph-, node-, and edge-level tasks, and diverse application domains to distinguish GNN performance on graph properties.They are intended to support the academic mission of identifying first principles.Source: §2 · p. 3

Coding Infrastructure

  • The PyTorch- and DGL-based infrastructure unifies data pipelines, GNN models, training and evaluation, configurations, and reproducibility scripts in a modular framework.Its objectives include ease of use, experimental rigour and fairness, and support for future graph ML tasks and new GNNs.Source: §2 · p. 3

Parameter Budgets for Fair Comparison

  • The benchmark fixes 100k parameters per GNN across datasets and uses 500k when investigating scalability to larger models and deeper layers.The fixed budgets help compare architectures without confusing architectural gains with gains from extra learning capacity; the goal is not to optimize each model’s hyperparameters.Source: §2 · p. 3

Discussion on Design Choices

  • Medium-scale datasets enable swift yet reliable prototyping, with statistical performance differences achievable within 12 hours of single-experiment runs.The paper argues they are more informative than small datasets and more feasible than large datasets for academic-scale research.Source: §2 · p. 3

3. How can the benchmark be used to explore new insights?

The benchmark supports fair, modular experimentation across diverse datasets and was used to show that Laplacian positional encodings improve message-passing GNNs. That work also helped spur later research on positional encodings.

3. How can the benchmark be used to explore new insights?

  • Researchers can modify modular components and fairly test preprocessing, GNN layers, normalization schemes, or novel models across diverse datasets under the same parameter budgets.The framework enables multiple experiments to be conducted conveniently and with ease.Source: §3 · p. 4
  • Laplacian PE improved MP-GCNs on the three synthetic datasets and real-world datasets, including the newly added AQSOL.The paper directs readers to Appendix E.1 for the detailed positional-encoding experiments.Source: §3 · p. 4
  • After Laplacian PE was introduced through the benchmark, later work explored improved positional encodings (Beaini et al., 2021; Wang et al., 2022; Lim et al., 2022; Kreuzer et al., 2021; Mialon et al., 2021).The authors present this as an example of how identifying first principles with the benchmark can steer GNN research.Source: §3 · p. 4

4. Conclusion … B.2.2 WL-GNN layers

The section presents an open-source, modular GNN benchmark for testing architectures and ideas, including its experimental pipelines and graph positional encoding as an avenue of exploration. It also outlines representative message-passing and Weisfeiler–Lehman layers, with implementation details for WL-GNNs and Ring-GNNs.

4. Conclusion

  • The benchmark is modular and easy to use for testing GNN ideas; it prompted graph positional encoding research and includes further studies of GNN categories and link-prediction edge representations.The additional studies are described in the appendix.Source: §4 · p. 4

A. Related Work

  • Prior GNNs span convolutional, anisotropic, and theoretically more expressive classes, while limited datasets and inconsistent experimental settings hinder fair assessment and progress.The paper argues that rigorous comparisons require reproducible frameworks and datasets that can statistically separate model performance.Source: A. Related Work · p. 6

B. Graph Neural Network Pipeline

  • The benchmark covers message-passing GCNs and WL-GNNs, using task-specific prediction layers; its WL pipeline represents available node and edge information in a dense tensor.The described instances include vanilla GCN, GraphSage, MoNet, GAT, GatedGCN, GIN, Ring-GNNs, and 3WL-GNNs.Source: B. Graph Neural Network Pipeline · pp. 6–7, Figure 4 · p. 8

B.1 Message-Passing GCNs

  • Message-passing GCNs update nodes from local neighborhoods with complexity O(E), or O(n) on sparse graphs, enabling GPU parallelization and sparse-matrix implementations.The framework’s input layer linearly projects optional node and edge features into d-dimensional hidden representations.Source: §B.1 · p. 7, §B.1.1 · p. 7

B.1.2 GCN layers

  • GCN variants differ in aggregation: vanilla GCN averages neighbors, GraphSage adds the node’s own features, and GAT weights neighbors with multi-head attention.The experiments use GraphSage’s max-pooling aggregator.Source: §B.1.2 · pp. 9–10
  • MoNet uses learnable Gaussian-mixture parameters, GatedGCN updates edge features with gates, and GIN is based on the Weisfeiler–Lehman isomorphism test.GatedGCN also uses residual connections and batch normalization; the framework augments message-passing layers with batch normalization and residual connections.Source: §B.1.2 · pp. 10–12

B.1.3 Task-based Layer

  • Task-specific MLP prediction heads use graph-level pooled features for graph tasks and node or concatenated endpoint features for node and edge classification.Graph regression minimizes L1 loss; classification uses cross-entropy, with node classification weighting it inversely by class size.Source: §B.1.3 · p. 13

B.2.2 WL-GNN layers

  • The 3-WL GNN uses rank-2 tensors while remaining 3-WL provable, improving Morris et al. (2019)’s space/time complexities from O(n^3)/O(n^4) to O(n^2)/O(n^3), respectively.Its layer uses 2-layer MLPs applied along the feature dimension.Source: §B.2 · p. 14, §B.2.2 · p. 15
  • For h ∈ R^(n×n×d), the MLP is implemented as a standard 2D convolution with a 1 × 1 kernel.The matrix multiplication is carried out along the first and second dimensions.Source: §B.2.2 · p. 15
  • Ring-GNNs improve order-2 equivariant GNNs by multiplying two equivariant linear layers.The passage attributes the proposal to Chen et al. (2019b) and describes the order-2 models as those of Maron et al. (2019b).Source: §B.2.2 · p. 15
  • The equivariant linear-function basis comprises 15 operations, while the bias basis uses L_i for i=16 and 17; matrix multiplication has O(n^3) time complexity.The passage points to Appendix A of Maron et al. (2019b) for the complete list of 15 operations.Source: §B.2.2 · p. 15

B.2.3 Task-based network layers · C. Datasets and Benchmarking Experiments · C.1 Graph Regression with ZINC dataset

The merged sections describe task-specific network suffixes and a benchmark protocol for comparing GNNs under fixed parameter budgets. They also define the ZINC graph-regression task and its evaluation, with numerical results deferred to another section.

B.2.3 Task-based network layers

  • RingGNN graph classifiers pass features from all intermediate layers to an MLP, while 3WL-GNNs use diagonal and off-diagonal max pooling at every layer.The classifier designs follow Maron et al. (2019a,b) and Chen et al. (2019b); the RingGNN readout is similar to Xu et al. (2018, 2019).Source: §B.2.3 · p. 15
  • The graph-regression suffix uses the classifier approach, with P ∈ Rd×1 for RingGNNs and Pℓ ∈ R2d×1 for 3WL-GNNs.The reported loss functions for each task are the same as those used for GCNs.Source: §B.2.3 · pp. 15–16
  • For node classification, RingGNN and 3WL-GNN have task-specific prediction scores, while link prediction concatenates node features to form edge features.The supplied passages introduce the node-prediction definitions but do not include their equations.Source: §B.2.3 · p. 16

C. Datasets and Benchmarking Experiments

  • The benchmarking experiments include a graph-agnostic baseline that independently applies an MLP to each node’s feature vector, and dataset statistics are summarized in Table 2.This baseline does not use information from other nodes.Source: C. Datasets and Benchmarking Experiments · p. 16, Table 2 · p. 17
  • Final-layer representations feed a usually three-layer downstream MLP, while GIN, RingGNN, and 3WL-GNN suffixes use outputs from each network layer.The all-layer approach follows the original suffix instructions and is similar to Jumping Knowledge Networks.Source: C. Datasets and Benchmarking Experiments · p. 17
  • The benchmark compares models within 100k-parameter budgets, with a 500k budget also used for GNNs tested at larger scale and depth.Hidden-layer counts and dimensions are selected to match the chosen budgets; the goal is benchmarking models and building blocks, not per-model hyperparameter optimization.Source: C. Datasets and Benchmarking Experiments · p. 17

C.1 Graph Regression with ZINC dataset

  • ZINC regression uses 12K graphs from a 250K-molecule dataset to predict constrained solubility, with 10,000/1,000/1,000 train/validation/test graphs and MAE evaluation.The target is logP − SA − cycle; node features encode heavy-atom types and edge features encode bond types. Table 3 presents four-seed results, but the supplied passages give no cell values.Source: §C.1 · pp. 17–18, Table 3 · p. 18

C.2 Graph Regression with AQSOL dataset

AQSOL is a graph-regression dataset of 9,823 molecules with experimentally measured aqueous solubility targets standardized to LogS units. The section describes its molecular features, scaffold split, training settings, and MAE evaluation.

C.2 Graph Regression with AQSOL dataset

  • The dataset derives from AqSolDB, a standardized database of 9,982 molecular graphs and solubility values collected from nine data sources.The section uses the database’s final LogS values as regression targets.Source: §C.2 · p. 18
  • AQSOL contains 9,823 molecular graphs after filtering, with heavy-atom types as node features and bond types as edge features.Its targets are aqueous solubility values collected from experimental measurements and standardized to LogS units in AqSolDB.Source: §C.2 · pp. 18–19
  • The scaffold split uses an 8:1:1 ratio, yielding 7,831 training, 996 validation, and 996 test graphs.Across GNNs, the initial learning rate is 1 × 10^-3, the reduction factor is 0.5, and the stopping learning rate is 1 × 10^-5; patience is 5 for 3WLGNN and RingGNN and 10 for other GNNs.Source: §C.2 · p. 19
  • Performance is measured by mean absolute error (MAE) between predicted and actual aqueous solubility values.Numerical results are presented in Table 4 and analyzed in Section D.Source: §C.2 · p. 19
  • Table 4 reports AQSOL graph-regression results averaged over four runs with four seeds, with lower values better; -E denotes edge features and -PE denotes four-dimensional Laplacian Eigenvector node positional encodings.The supplied caption does not provide the table’s cell values.Source: Table 4 · p. 19

C.3 Link Prediction with OGBL-COLLAB dataset · C.4 Node Classification with WikiCS dataset · C.5 Graph Classification with Super-pixel (MNIST/CIFAR10) datasets

The three subsections define datasets and evaluation protocols for link prediction, node classification, and graph classification. They describe task-specific splits, training procedures, and performance measures; numerical results are deferred to Section D.

C.3 Link Prediction with OGBL-COLLAB dataset

  • OGBL-COLLAB evaluates prediction of future scientist collaborations using OGB’s temporal edge splits and Hits@50 against 100,000 sampled negative collaborations.Training uses collaborations through 2017, validation uses 2018, and testing uses 2019; all GNNs use the same learning-rate schedule.Source: §C.3 · p. 20
  • A 256-dimensional matrix-factorization baseline provides a feature-agnostic comparison for assessing whether GNNs use node features as well as graph structure.Numerical results are presented in Table 5 and discussed in Section D.Source: §C.3 · pp. 20–21

C.4 Node Classification with WikiCS dataset

  • WikiCS contains 11,701 Wikipedia articles and 216,123 hyperlinks across 10 classes, with 300-dimensional GloVe features; its dense, cross-class neighborhoods motivate testing GNN neighborhood functions.The passage also notes that its average shortest path length is smaller than Cora’s and Citeseer’s.Source: §C.4 · p. 21
  • WikiCS uses 20 class-stratified splits, combining the original validation and early-stopping sets into validation, and reports accuracy averaged over 80 runs.Each split has 5% training, 45% validation, and 50% test nodes per class; the 20 splits are each run with four seeds.Source: §C.4 · p. 21

C.5 Graph Classification with Super-pixel (MNIST/CIFAR10) datasets

  • The MNIST and CIFAR10 super-pixel graph datasets serve as sanity checks, with near-100% MNIST and strong CIFAR10 performance expected.They also illustrate how image datasets can be used in graph-learning research.Source: §C.5 · p. 22
  • Images are converted to SLIC super-pixel graphs (Knyazev et al., 2019), using k=8 and yielding 40–75 MNIST or 85–150 CIFAR10 nodes.The maximum node counts are 75 for MNIST and 150 for CIFAR10.Source: §C.5 · p. 22
  • Standard MNIST and CIFAR10 splits are used, with shared randomly sampled validation graphs; 3WLGNN and RingGNN receive adjusted learning-rate schedules after training difficulties.MNIST uses 55,000/5,000/10,000 train/validation/test graphs and CIFAR10 uses 45,000/5,000/10,000; RingGNN’s initial and stopping learning rates are changed to 1 × 10−4 and 1 × 10−6.Source: §C.5 · p. 22

C.6 Node Classification with SBM (PATTERN/CLUSTER) datasets · C.7 Edge Classification/Link Prediction with TSP dataset

The sections describe node-classification benchmarks built from stochastic block models and a TSP edge-classification benchmark that uses GNNs to score candidate tour edges. They specify how the datasets and evaluation protocols are constructed.

C.6 Node Classification with SBM (PATTERN/CLUSTER) datasets

  • The SBM benchmarks cover graph-pattern recognition in PATTERN and semi-supervised graph clustering in CLUSTER, with community connectivity controlled by within- and between-community probabilities p and q.For SBM graphs, p sets connection probability within a community, while q sets it between communities and acts as the noise level.Source: §C.6 · pp. 22–23
  • PATTERN graphs contain five randomly sized communities, embed one of 100 randomly generated 20-node patterns, and label nodes by whether they belong to the pattern.Graph sizes are 44–188 nodes; community probabilities are p = 0.5 and q = 0.35, and node features take values {0, 1, 2}.Source: §C.6 · p. 23
  • CLUSTER graphs contain six communities and one labeled node per community; unlabeled nodes have feature value 0 and are assigned their community class.Graphs have 40–190 nodes, with p = 0.55 and q = 0.25; PATTERN and CLUSTER use fixed train, validation, and test splits shared across models.Source: §C.6 · pp. 23–24

C.7 Edge Classification/Link Prediction with TSP dataset

  • For TSP, a GNN assigns tour-membership probabilities to edges, which graph search converts into discrete decisions, following Joshi et al. (2022).The task requires reasoning over local neighborhoods and global graph structure because TSP is multi-scale.Source: §C.7 · p. 25
  • TSP instances sample 50–500 locations uniformly in the unit square, and the benchmark uses sparse 25-nearest-neighbor graphs to scale to large instances.Ground-truth tour edges are provided by Concorde to isolate backbone GNN performance from the search component.Source: §C.7 · p. 25
  • TSP evaluation uses positive-class F1 for its imbalanced edge labels and compares GNNs with a k-nearest-neighbor heuristic using k = 2.The dataset has 10,000 training, 1,000 validation, and 1,000 test graphs.Source: §C.7 · p. 25

C.8 Graph Classification and Isomorphism Testing with CSL dataset · C.9 Cycle Detection with CYCLES dataset · C.10 Multi-task graph properties with GraphTheoryProp dataset

These sections describe three synthetic graph benchmarks used to validate Graph Positional Encodings: CSL for graph classification, CYCLES for cycle detection, and GraphTheoryProp for multi-task graph properties. They specify each dataset’s tasks, data splits, and evaluation setup.

C.8 Graph Classification and Isomorphism Testing with CSL dataset

  • CSL tests GNN expressivity on class-balanced, 4-regular graphs with cycle edges and skip-links, using 150 graphs across 10 isomorphism classes.The benchmark uses graphs with 41 nodes and 15 examples for each skip-link length: 2, 3, 4, 5, 6, 9, 11, 12, 13, and 16.Source: §C.8 · p. 26
  • CSL uses stratified 5-fold cross-validation, 20 initialization seeds, and classification accuracy to evaluate Graph Positional Encodings.The folds use a 3:1:1 train/validation/test ratio; the reported statistics summarize 100 test scores. CSL is used to validate the positional encodings proposed in this framework.Source: §C.8 · pp. 26–27

C.9 Cycle Detection with CYCLES dataset

  • CYCLES is a balanced binary classification benchmark for detecting fixed-length cycles, using the 56-node, cycle-length-6 variant selected for its reported difficulty.Graphs contain 37–65 nodes, and the task is to predict whether a graph has a cycle; this dataset variant follows Loukas (2020).Source: §C.9 · p. 27
  • CYCLES uses 9,000 training, 1,000 validation, and 10,000 test graphs; training varies sample size from 200 to 5,000 and reports accuracy over four seeds.The benchmark follows Loukas (2020)’s splits, samples class-balanced validation graphs, and trains only GIN and GatedGCN MP-GCNs to validate Graph Positional Encodings.Source: §C.9 · pp. 27–28

C.10 Multi-task graph properties with GraphTheoryProp dataset

  • GraphTheoryProp benchmarks six graph-theoretic properties: shortest paths, eccentricity, Laplacian features, connectivity, diameter, and spectral radius.It includes three node-level and three graph-level tasks on randomly generated undirected, unweighted graphs with 15–24 nodes and random identifier input features.Source: §C.10 · p. 28
  • GraphTheoryProp uses 5,120 training, 640 validation, and 1,280 test graphs, evaluating each task and their combined average with Log10MSE.The reported results average four runs with different seeds; the dataset is used to validate Graph Positional Encodings.Source: §C.10 · pp. 28–29

D. Analysis and Discussion of Benchmarking Results … E.1.2 Laplacian eigenvectors as Positional Encodings

The benchmark experiments favor graph-aware GCNs, especially anisotropic variants, while revealing practical and training challenges for WL-GNNs. The framework also supports research explorations, including Laplacian eigenvector positional encodings for MP-GCNs.

D. Analysis and Discussion of Benchmarking Results

  • The graph-agnostic MLP baseline scored consistently low, supporting the need for graph structure, and the proposed datasets statistically separate GNN performance.Source: D. Analysis and Discussion of… · p. 29
  • GCNs outperformed 3WL-GNNs and RingGNNs on the proposed medium-scale datasets and scaled to 16 layers, while the WL models faced complexity and training limits.Within the WL-GNN class, 3WL-GNN outperformed RingGNN and came close to the best AQSOL performance; GIN scaled better and provided good overall performance.Source: D. Analysis and Discussion of… · pp. 29–30
  • Anisotropic GNNs generally outperformed isotropic models across the reported datasets, with exceptions for vanilla GCN on WikiCS, GraphSage on MNIST, and MoNet on CIFAR10.The compared anisotropic models were GCN with symmetric normalization, GAT, MoNet, and GatedGCN; the discussion also points to attention-based aggregation as a research direction.Source: D. Analysis and Discussion of… · p. 29
  • WL-GNNs showed relatively high performance variability and practical training barriers tied to dense variable-sized tensors, including limited batching and normalization options.Layer normalization did not resolve the issue, and the authors could not train WL-GNNs on CPU memory for the single COLLAB graph; they identify batching, normalization, and residual connections as areas needing redesign.Source: D. Analysis and Discussion of… · pp. 29–30

E. Studies using the Benchmarking Framework

E.1 Laplacian Positional Encodings

  • Laplacian positional encodings aim to address MP-GCNs’ theoretical shortcomings while retaining message-passing computational efficiency and improving performance.This motivation follows the benchmark’s finding that MP-GCNs outperform WL-GNNs despite their theoretical limitations, while WL-GNNs were computationally infeasible on medium- and large-scale datasets.Source: §E.1 · p. 30

E.1.1 Related Work

  • Earlier approaches address anonymous-node or positional limitations, but the authors identify scalability, generalization, or task-applicability drawbacks and propose Laplacian eigenvectors instead.Graph Relational Pooling requires accounting for n! node permutations (Murphy et al., 2019); random anchor embeddings have limitations on inductive node tasks (You et al., 2019), while random identifiers may not generalize to unseen graphs (You et al., 2019; Loukas, 2020). Distance encodings based on random-walk matrix powers can fail on distance-regular graphs and incur computation costs (Li et al., 2020).Source: §E.1.1 · p. 31

E.1.2 Laplacian eigenvectors as Positional Encodings

  • The method adds node positional features before MP-GCN processing, using Laplacian eigenvectors as graph coordinates that capture local relationships while preserving global structure.Experiments use the k smallest non-trivial eigenvectors; eigenvector signs are randomly flipped during training because they are defined up to ±1. Computing them has complexity O(E^3/2), which can be improved with methods such as Nyström.Source: §E.1.2 · p. 31

E.1.3 Experiments and Analysis … E.2.1 With GatedGCN and GAT

Laplacian positional encodings improve message-passing GNN performance across mathematical and real-world graph tasks, though eigenvector sign ambiguity remains a challenge. The sections also examine isotropic and anisotropic GatedGCN and GAT variants for edge-classification tasks.

E.1.3 Experiments and Analysis

  • 99% mean accuracy for MP-GCNs on CSL exceeded 3WL-GNN’s 97% and RingGNN’s 25%; Laplacian PE also improved GIN and GatedGCN on CYCLES and GraphTheoryProp.These results show positional features can help detect cycles and predict theoretical and geometric graph properties.Source: §E.1.3 · p. 32
  • Laplacian PE boosted performance on ZINC, AQSOL, CLUSTER, and PATTERN, while COLLAB results were statistically unchanged.The authors also report gains on real-world graph learning tasks and say PE can overcome MP-GCN limitations in detecting simple graph symmetries.Source: §E.1.3 · p. 32

E.1.4 Challenges with using Laplacian eigenvectors

  • Positional features cannot be canonical for structurally symmetric nodes, so networks must learn to handle arbitrary assignments among n! possible node orderings.Training uniformly samples these orderings to encourage independence from arbitrary choices.Source: §E.1.4 · p. 32
  • Laplacian eigenvectors are invariant to node re-parametrization but retain 2^k sign choices; random sampling reduces ambiguity, while absolute values degrade expressivity.With k ≪ n, the 2^k sign possibilities are much fewer than n! node orderings.Source: §E.1.4 · pp. 32–33
  • Random sign flipping produced the best Laplacian PE results, while randomly permuted index PEs beat fixed orderings but remained inferior to Laplacian PEs.Table 13 compares positional encodings with GatedGCN on CSL, ZINC, PATTERN, CLUSTER, and COLLAB.Source: Table 13 · p. 33, §E.1.4 · p. 33

E.2.1 With GatedGCN and GAT

  • On TSP and COLLAB edge classification, anisotropic GCNs using adjacent-node representations as edge features consistently outperformed isotropic vanilla GCN and GraphSage.The section studies anisotropy through GAT and GatedGCN variants.Source: §E.2.1 · p. 34
  • The GatedGCN variants range from isotropic node updates to anisotropic edge features and anisotropic updates that maintain explicit edge representations across layers.Input edge features can initialize representations, including TSP distances and COLLAB year and frequency features.Source: §E.2.1 · pp. 34–35
  • The GAT variants likewise compare isotropic aggregation, anisotropic joint representations of adjacent nodes, and anisotropic models with explicit edge representations updated across layers.Dataset edge features can optionally initialize those representations.Source: §E.2.1 · p. 35

Numerical Experiments and Analysis · E.2.2 With GraphSage

The experiments show that aggregation choice and edge modeling have task-dependent effects: COLLAB favors degree-invariant max or mean aggregation, while edge features and explicit edge states help TSP but can hurt COLLAB.

Numerical Experiments and Analysis

  • For TSP and COLLAB, models use a reduced 27K-parameter budget and train for at most 24 hours to fit GPU memory and allow convergence.Maintaining edge representations is costly in time and memory on the large COLLAB graph.Source: Numerical Experiments and Analysis · p. 35
  • 0.75 vs. 0.64 F1 score on TSP and 50.6% vs. 35.9% Hits@50 on COLLAB show gains from edge features for GatedGCN over its isotropic variant.The comparison uses the same model parameters.Source: Table 14 · p. 36, Numerical Experiments and Analysis · p. 36
  • 0.78 vs. 0.67 F1 score for GAT-E vs. standard GAT on TSP shows further gains from explicit edge representations, especially when initialized with node-to-node Euclidean distances.On COLLAB, GatedGCN-E scores 47.2 vs. 51.5 Hits@50 for GatedGCN, indicating that adding explicit edge representations and inputs can degrade performance.Source: Numerical Experiments and Analysis · p. 36
  • The authors suggest modeling COLLAB as a multi-graph with temporal edges to motivate task-specific anisotropic edge representations beyond generic attention and gating.They note that COLLAB’s collaboration-frequency and year features may not be useful for link prediction.Source: Numerical Experiments and Analysis · p. 36

E.2.2 With GraphSage

  • On COLLAB, GraphSage with max aggregation performs close to GAT and GatedGCN, while sum-aggregation GIN and MoNet fail to beat matrix factorization.The authors interpret max and mean aggregation, which are invariant to node degree, as a powerful inductive bias for COLLAB.Source: §E.2.2 · p. 36
  • With GraphSage on COLLAB, edge features do not significantly improve max aggregators, while maintaining explicit edge representations across layers hurts performance.The authors attribute the latter result presumably to very small hidden dimensions, required because representations for 235K nodes and 2.3M edges consume substantial GPU memory.Source: §E.2.2 · p. 36, Table 15 · p. 37
  • The experiments compare isotropic GraphSage, anisotropic variants with intermediate edge features, and anisotropic variants that update explicit edge representations across layers.Input edge features can optionally initialize those representations.Source: §E.2.2 · p. 37

F. Experiments on TU datasets · G. A Note on Graph Size Normalization

Experiments on ENZYMES, DD, and PROTEINS showed similar statistical test performance across neural networks, with rankings varying between runs. The note describes GraphNorm as an empirically motivated normalization technique without a concrete theoretical basis.

F. Experiments on TU datasets

  • The TU experiments used ENZYMES, DD, and PROTEINS to highlight challenges of benchmarking GNNs on conventional datasets.The paper characterizes these TU datasets as small, making it difficult to determine which GNNs are powerful and robust.Source: F. Experiments on TU datasets · pp. 37, 39
  • Across ENZYMES, DD, and PROTEINS, all NNs had similar statistical test performance with large standard deviations; rankings changed across runs, and MLPs matched GNNs on DD and PROTEINS.The second run used the same 10-fold splits and hyperparameters but different initialization; the authors attribute ranking changes to small dataset size and optimizer non-determinism.Source: Table 16 · p. 38, F. Experiments on TU datasets · pp. 38–39
  • Each TU dataset used stratified 10-fold cross-validation with an 8:1:1 train-validation-test ratio and shared saved indices across experiments.Per fold, ENZYMES had 480/60/60 train/validation/test graphs, DD had 941/118/119, and PROTEINS had 889/112/112.Source: F. Experiments on TU datasets · p. 38
  • Models used Adam, per-model grid-searched learning rates, and test accuracy averaged with standard deviation across the 10 folds.The learning-rate search ranged from 1×10−3 to 7 × 10−5; the reduce factor was 0.5, patience was 25, and the stopping rate was 1 × 10−6.Source: F. Experiments on TU datasets · p. 38

G. A Note on Graph Size Normalization

  • GraphNorm normalizes node features with respect to graph size before BatchNorm; variable graph sizes in batches can make BatchNorm statistics difficult to learn.The note links this difficulty to node representations at different scales across irregular batch sizes and variable-sized graphs.Source: G. A Note on Graph Size Normalization · p. 39
  • GraphNorm has no concrete theoretical basis and was proposed from initially promising empirical results on ZINC and CLUSTER; more principled graph normalization remains future work.Source: G. A Note on Graph Size Normalization · p. 39

H. Elaboration on Benchmarking Design Choices · I. Hardware · J. Memory Usage

The framework’s design choices support robust, fair GNN comparisons while allowing extensions; experiments use specified hardware and report that GatedGCN consumes more GPU memory than GCN and GAT on ZINC and SBM-CLUSTER.

H. Elaboration on Benchmarking Design Choices

  • The benchmark uses medium-scale, diverse datasets for rapid prototyping and cross-domain testing, while allowing researchers to add datasets and adapt the framework to other libraries.The paper reports that robust analysis can be generated in experiments taking at most 12 hours and points to convenient dataset extensions and adoption in PyTorch Geometric.Source: H. Elaboration on Benchmarking Design… · pp. 39–40
  • Fixed parameter budgets, such as 100k or 500k, help attribute performance gains to architecture rather than parameter count, supporting identification of robust principles rather than direct SOTA tuning.After identifying such principles, models can be scaled without constraints for targeted SOTA benchmarks.Source: H. Elaboration on Benchmarking Design… · p. 40
  • The open-source standard codebase addresses inconsistent GNN comparisons through shared training and evaluation settings, while permitting fair extensions to training methods.The paper notes that researchers can augment the codebase with methods or modules applied consistently across models.Source: H. Elaboration on Benchmarking Design… · p. 40

I. Hardware

  • Experiments ran on two specified CPU-and-GPU servers, with each experiment using one GPU, four concurrent experiments per server, and a 12-hour maximum.MNIST, CIFAR10, ZINC, AQSOL, TSP, COLLAB, WikiCS, CSL, CYCLES, GraphTheoryProp, and TUs used an Intel Xeon E5-2690 v4 with four Nvidia 1080Ti GPUs; PATTERN and CLUSTER used an Intel Xeon Gold 6132 with four Nvidia 2080Ti GPUs.Source: §I · p. 40
  • The benchmark reports average wall-clock time per epoch and total training time, acknowledging that implementation and hardware acceleration complicate timing comparisons.Experiments were implemented in DGL/PyTorch.Source: §I · p. 40

J. Memory Usage

  • GatedGCN is relatively more memory-intensive than GCN and GAT on ZINC and SBM-CLUSTER.Figure 16 measures GPU memory allocated during a training forward pass; all models have 500k learnable parameters, with batch sizes of 128 for ZINC and 64 for SBM-CLUSTER.Source: J. Memory Usage · pp. 40–41, Figure 16 · p. 41

Related papers

Loading 2003.00982v5…