PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphFM: A Comprehensive Benchmark for Graph Foundation Model

Yuhao Xu, Xinqi Liu, Keyu Duan, Yi Fang, Yu-Neng Chuang, Daochen Zha, Qiaoyu Tan

arXiv:2406.08310v2cs.LG

TL;DR

GraphFM addresses limited evidence about Graph Foundation Models’ cross-task generalization, scalability, efficiency, and training-stop criteria. It benchmarks eight GSSL methods under standardized settings across tasks, datasets, and training strategies, finding that existing methods often struggle to generalize across key graph reasoning tasks simultaneously. The study also identifies early stopping as an important factor in model generalization and highlights scope boundaries involving homogeneity and textual attributed graphs.

  • Problem

    Existing GSSL methods lack clear evidence of cross-task generalization, scalability, efficiency, and suitable early stopping for Graph Foundation Models.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    GraphFM reimplements and compares GSSL methods using standardized data processing, evaluation, and hyperparameter-search settings across downstream tasks and training strategies.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Existing self-supervised GNN pre-training efforts often struggle to generalize well across node classification, link prediction, and node clustering simultaneously, with early stopping substantially affecting generalization capability.

    Source: §6 · p. 9

  • Takeaways & Limitations

    GraphFM provides a benchmark for analyzing Graph FM generalization, training strategies, efficiency, and early stopping while motivating future research.

    Source: §6 · p. 9

    Current GraphFM experiments primarily use conventional graphs with numerical node features, leaving extension to textual attributed graphs open.

    Source: §5 · p. 8

Abstract

from arXiv · show

Foundation Models (FMs) serve as a general class for the development of artificial intelligence systems, offering broad potential for generalization across a spectrum of downstream tasks. Despite extensive research into self-supervised learning as the cornerstone of FMs, several outstanding issues persist in Graph Foundation Models that rely on graph self-supervised learning, namely: 1) Homogenization. The extent of generalization capability on downstream tasks remains unclear. 2) Scalability. It is unknown how effectively these models can scale to large datasets. 3) Efficiency. The training time and memory usage of these models require evaluation. 4) Training Stop Criteria. Determining the optimal stopping strategy for pre-training across multiple tasks to maximize performance on downstream tasks. To address these questions, we have constructed a rigorous benchmark that thoroughly analyzes and studies the generalization and scalability of self-supervised Graph Neural Network (GNN) models. Regarding generalization, we have implemented and compared the performance of various self-supervised GNN models, trained to generate node representations, across tasks such as node classification, link prediction, and node clustering. For scalability, we have compared the performance of various models after training using full-batch and mini-batch strategies. Additionally, we have assessed the training efficiency of these models by conducting experiments to test their GPU memory usage and throughput. Through these experiments, we aim to provide insights to motivate future research. The code for this benchmark is publicly available at https://github.com/NYUSHCS/GraphFM.

1 Introduction

GraphFM addresses unclear generalization, comparability, scalability, and efficiency in Graph Foundation Models built with graph self-supervised learning. It introduces a unified benchmark spanning datasets, training strategies, GSSL methods, downstream tasks, and early stopping.

  • Existing GSSL methods lack comprehensive evaluation of generalization across node classification, link prediction, and clustering tasks.Prior studies often focus on one task, limiting understanding of performance across tasks.Source: §1 · p. 2
  • Existing results are difficult to compare because studies use different classifiers, hyperparameter ranges, data processing, and evaluation settings.These methodological differences can substantially affect reported performance.Source: §1 · p. 2
  • GraphFM provides a unified benchmark of eight representative GSSL methods across six datasets with varying scales.The benchmark standardizes data processing, splitting, and hyperparameter-search budgets.Source: §1 · p. 2
  • The benchmark evaluates node representations on node classification, link prediction, and node clustering using both full-batch and mini-batch training.It also analyzes performance and efficiency under different settings.Source: §1 · p. 2
  • GraphFM investigates how downstream-task performance and alternative metrics affect early stopping during Graph FM pre-training.The study treats stopping criteria as part of multi-task evaluation.Source: §1 · p. 3
  • The authors release code and pre-trained models and use benchmark findings to outline future research directions.Source: §1 · p. 3

2 Preliminaries

GraphFM defines Graph Foundation Models through the quality and transferability of learned node representations. Its preliminaries frame homogenization, scalable message passing, and early stopping as central evaluation concerns.

  • A Graph FM maps graph structure and node features to node embeddings that can support multiple downstream tasks.The objective is to learn high-quality representations using graph self-supervised learning.Source: §2 · p. 3
  • Homogenization denotes a Graph FM’s generalization capability across node classification, link prediction, and node clustering.Source: §2 · p. 3
  • Full-batch GNN training retains the entire graph for message passing, whereas mini-batch training samples subgraphs to reduce memory consumption.The benchmark considers node sampling and subgraph sampling as mini-batch strategies.Source: §2 · p. 3
  • In full-batch training, each layer’s adjacency covers all nodes; in mini-batch training, it covers only a sampled subset.The resulting adjacency is a sub-matrix of the full graph adjacency.Source: §2 · p. 3
  • Early stopping is straightforward for one task and metric but remains underexplored when Graph FMs must perform across multiple tasks and metrics.Relevant metrics include accuracy and AUC.Source: §2 · p. 3

3 Benchmark Design

GraphFM builds a standardized benchmark to compare GSSL methods across datasets, tasks, training strategies, and efficiency measures. Its design explicitly tests generalization, scalability, and early-stopping choices.

3.1 Dataset and Implementations

  • The benchmark uses six graph datasets spanning citation, social-network, and large-scale citation settings to assess methods across data scales.The datasets include Cora, Citeseer, Pubmed, Flickr, Reddit, and Arxiv.Source: §3.1 · pp. 3–4
  • GraphFM reproduces eight state-of-the-art methods—five contrastive and three generative—under equal hyperparameter-search budgets.All methods are implemented from their papers or source code within the benchmark framework.Source: §3.1 · p. 4

3.2 Research Questions

  • Node classification is evaluated on Cora, Citeseer, and Pubmed with full-batch training and accuracy-based early stopping.Source: §3.2 · p. 4
  • Generalization is tested by evaluating learned representations on node classification, link prediction, and node clustering with standard task metrics.Link prediction uses AUC and AP; clustering uses NMI and ARI.Source: §3.2 · p. 4
  • Scalability experiments compare full-batch, node-sampling, and subgraph-sampling strategies on Flickr, Reddit, and Arxiv while reporting throughput and GPU memory.Source: §3.2 · p. 4
  • The benchmark separately tests whether validation performance from different downstream tasks or metrics changes the effectiveness of saved pre-trained models.The experiments compare stopping based on link prediction or clustering as well as other downstream settings.Source: §3.2 · p. 5

4 Experiments Results and Analyses

Under standardized full-batch settings, GraphFM generally reproduces or improves prior node-classification results. The largest reported improvement is on PubMed, while S2GAE is the exception.

4.1 Performance Comparison in Node Classification (RQ1)

  • PubMed node-classification results exceed previous benchmarks by up to 4 percentage points under standardized full-batch evaluation.The authors attribute the improvement to more suitable hyperparameters identified by Optuna after standardization.Source: §4.1 · p. 5
  • Reproduced full-batch results are generally comparable to or higher than results reported in the original papers.The comparison uses standardized settings and hyperparameter search.Source: §4.1 · p. 5
  • S2GAE performs worse than its original result, possibly because the original study used an SVM classifier whereas GraphFM uses an MLP head for comparability.Source: §4.1 · p. 5

2 The performance gap between leading contrastive and generative paradigms on node clas-

GraphFM finds a marginal node-classification gap between leading contrastive and generative methods, while their strengths diverge across link prediction, clustering, and training strategies.

2 The performance gap between leading contrastive and generative paradigms on node clas-

  • Advanced generative models achieve highly competitive node-classification results, with generative methods averaging above contrastive models on Cora and Citeseer.This narrows the presumed advantage of contrastive learning for node classification.Source: §2 · p. 5

4.2 Performance Comparison in Link Prediction and Node Clustering (RQ2)

  • GraphMAE and GraphMAE2 underperform on link prediction despite competitive classification results, while they outperform other baselines on node clustering.The paper relates the link-prediction weakness to reconstructing node features rather than network structure.Source: §4.2 · p. 6
  • Both paradigms show strong homogenization on small datasets, with models selected by node-classification performance remaining effective on link prediction and clustering.The evaluation uses pre-trained models saved according to node-classification performance.Source: §4.2 · pp. 5–6

4.3 Performance and Efficiency Comparison in Large-Scale Dataset (RQ3)

  • Mini-batch training generally lowers performance versus full-batch training across downstream tasks, with generative models experiencing the larger drop.GBT improves in node classification, and GraphMAE improves in link prediction, but these are exceptions to the overall pattern.Source: §4.3 · p. 6

7 On large datasets, the performance of existing GSSL methods varies significantly across

On large datasets, GSSL performance varies across downstream tasks and training choices, while early stopping on the target task improves that task’s performance, especially for contrastive models.

7 On large datasets, the performance of existing GSSL methods varies significantly across

  • On large datasets, contrastive and generative methods show similar node-classification performance, whereas generative models consistently outperform contrastive methods in node clustering.Negative clustering scores are valid because the Adjusted Rand Index ranges from -1 to 1.Source: §7 · p. 7
  • Generative models avoid out-of-memory issues, while subgraph sampling uses the least memory and trains fastest among the two mini-batch methods.The efficiency comparison is reported for large-scale datasets.Source: §7 · p. 7

4.4 Performance Using Alternative Early Stopping Criterion (RQ4)

  • Using the same downstream task as the early-stopping criterion improves performance on that task, with especially substantial gains for contrastive models in link prediction.Generative methods show no substantial performance enhancement under this criterion.Source: §4.4 · p. 7

5 Future Direrctions

GraphFM identifies future directions centered on improving cross-task homogeneity, early stopping, and applicability to textual attributed graphs.

  • GraphFM finds that contrastive and generative models struggle to generalize consistently across datasets and node-level or edge-level tasks.Source: §5 · p. 8
  • No single early stopping criterion currently improves performance across various downstream tasks, motivating more effective GNN pre-training stopping strategies.Source: §5 · p. 8
  • Extending GraphFM beyond numerical node features to textual attributed graphs remains an open research question.Source: §5 · p. 8

6 Conclusion

GraphFM benchmarks eight leading graph self-supervised learning methods to assess their generalization, scalability, and training behavior. The findings indicate limited simultaneous generalization across key graph tasks and show that training setup and early stopping materially affect outcomes.

  • GraphFM reimplements and fairly compares 8 leading GSSL methods across diverse datasets.Source: §6 · p. 8
  • Existing self-supervised GNNs often struggle to generalize simultaneously across node classification, link prediction, and node clustering tasks.Source: §6 · p. 9
  • Full-batch and mini-batch training produce variations in model performance, while GPU-memory results include out-of-memory cases in Flickr.Source: §6 · p. 9, Table 6 · p. 9
  • Early stopping criteria in GNN pre-training significantly affect model generalization capability and were previously overlooked in this research area.Source: §6 · p. 9

B Additional Experimental Details

The benchmark standardizes evaluation across graph self-supervised models, downstream tasks, sampling strategies, and early-stopping metrics. Additional results document metric definitions, sampling procedures, full-batch clustering, and experimental settings.

B.1 RQ1

  • Standardized optimizers, evaluation methods, edge splitting, and data processing support fair comparisons across node classification, link prediction, and node clustering.Source: §B.1 · p. 14
  • Optuna-based hyperparameter tuning uses comprehensive search spaces to provide an impartial evaluation of the GSSL models.Source: §B.1 · p. 14

B.2 RQ2

  • AUC and AP evaluate link prediction, while NMI and ARI evaluate node clustering.Source: §B.2 · p. 14

B.2.1 AUC

  • AUC measures discrimination between positive and negative edges, whereas AP summarizes precision across recall thresholds.Source: §B.2.1 · p. 14, §B.2.2 · p. 14

B.2.3 NMI

  • NMI measures agreement between predicted node clusters and ground-truth labels, while ARI measures clustering similarity after correcting for chance.Source: §B.2.3 · p. 14, §B.2.4 · p. 15

B.3 RQ3

  • Mini-batch training uses node or subgraph sampling, with sampled layer neighborhoods determining the message-passing computation.Source: §B.3 · p. 15

B.3.1 Node Sampling

  • Node sampling selects a fixed number of one-hop neighbors per layer, whereas subgraph sampling reuses nodes from a sampled induced subgraph across layers.Source: §B.3.1 · p. 15, §B.3.2 · p. 15

B.4 RQ4

  • Early-stopping experiments replace node-classification accuracy with link-prediction AUC or clustering NMI to test how task-specific validation criteria affect model selection.Source: §B.4 · p. 15

C.2 Result of Mini-Batch

Mini-batch results show that scalability and performance depend on the downstream task, while subgraph sampling improves efficiency and S2GAE is consistently efficient. Early-stopping comparisons focus on task-specific validation criteria across Flickr and ogbn-arxiv.

C.2 Result of Mini-Batch

  • Mini-batch results cover node classification with node and subgraph sampling, with out-of-memory cases explicitly marked in the reported tables.Source: §C.2 · p. 16, Table 8 · p. 16, Table 9 · p. 16
  • S2GAE performs worse than other models for node classification on large-scale datasets but typically performs better in link prediction.Source: §C.2 · p. 16
  • S2GAE’s mini-batch performance in node clustering is not significantly different from other models, indicating task-dependent scalability.Source: §C.2 · p. 16

C.3 Results of Efficiency

C.4 Results of Early Stop Criteria

Related papers

Loading 2406.08310v2…