Open Graph Benchmark: Datasets for Machine Learning on Graphs

Weihua Hu; Matthias Fey; Marinka Zitnik; Yuxiao Dong; Hongyu Ren; Bowen Liu; Michele Catasta; Jure Leskovec

Source-linked AI summary

Open Graph Benchmark: Datasets for Machine Learning on Graphs

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec

arXiv:2005.00687v7May 2, 2020cs.LGcs.SIstat.ML

TL;DR

Graph ML lacks comprehensive, realistic benchmarks spanning diverse datasets, scales, domains, and task types with consistent evaluation. OGB addresses this by providing such datasets, application-specific splits, evaluation protocols, and benchmark experiments, which reveal challenges in large-scale learning and out-of-distribution generalization.

Problem
Graph ML benchmarks lack comprehensive, large-scale datasets and consistent evaluation across diverse domains and tasks.
Method
OGB constructs diverse realistic graph datasets across scales, domains, and task categories, using application-specific data splits and extensive benchmark experiments.
Results
OGB experiments reveal significant challenges in scaling models to large graphs and achieving accurate predictions under realistic data splits.
Takeaways & Limitations
OGB provides a unified benchmark foundation and identifies research opportunities in large-scale graph learning and out-of-distribution generalization.
Takeaways & Limitations
Some benchmark experiments are omitted for ogbn-proteins because its graph fits common GPUs, while efficiently handling its many edge features remains challenging.

Abstract

from arXiv · show

We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at https://ogb.stanford.edu .

1 Introduction

OGB addresses limitations in graph ML benchmarks by providing large, diverse, realistic datasets spanning multiple tasks and domains, with standardized evaluation and an automated pipeline.

Existing graph benchmarks are often too small, inconsistent, and based on unrealistic random splits, limiting reliable comparison and real-world relevance.
OGB targets scalable, robust, and reproducible graph ML research through challenging datasets designed around scale, domain diversity, and realistic tasks.
OGB datasets range from GPU-manageable small graphs to medium and large graphs supporting mini-batching and distributed training.
The benchmark covers node, link, and graph property prediction across diverse application domains, with domain-specific splits more realistic than random splits.
Extensive experiments identify scalability to large graphs and out-of-distribution generalization under realistic splits as central research challenges.
OGB standardizes data loading, experimental setup, model evaluation, and leaderboard-based comparison through an automated open-source pipeline.

2 Shortcomings of Current Benchmarks

Existing graph benchmarks suffer from limited scale, inconsistent protocols, unrealistic splits, and narrow domain coverage, motivating OGB’s broader and more standardized design.

Node property prediction: Common node-level datasets are small, and performance differences among graph neural networks can become statistically indistinguishable.
Node property prediction: Larger node datasets still use training-heavy splits that create artificially small distribution shifts between training, validation, and test sets.
Link property prediction: Link-level benchmarks are frequently either very small or missing input node features, while random splits fail to reflect temporal application settings.
Link property prediction: Existing link datasets emphasize sparse domains, potentially limiting generalization to dense biological and medical graphs.
Graph property prediction: Graph-level benchmarks often contain fewer than 1,000 graphs and combine random splits, inconsistent evaluation, missing molecular features, and isomorphism bias.
Graph property prediction: Molecular benchmarks offer larger realistic datasets but lack consensus on splitting and molecular features, complicating fair model comparison.

3 OGB: Overview

OGB combines diverse datasets, realistic splits, benchmark analyses, and software tools into a reproducible graph ML evaluation framework.

OGB spans real-world applications and provides common graph representations, established metrics, and framework-compatible tools for loading and evaluation.
The datasets are documented through graph properties, prediction tasks, splitting schemes, and statistics spanning diverse graph characteristics.
Benchmark experiments use representative embedding models, GNNs, and mini-batch GNNs, repeated across 10 random seeds with reported means and unbiased standard deviations.
The Python package provides automatic data loaders and evaluators, with public code, documentation, and reproducible baseline implementations.
OGB’s dataset identifiers encode node, link, or graph prediction categories, and every dataset has a specified realistic split scheme.

4 OGB Node Property Prediction

OGB’s node-property benchmarks span diverse domains, graph structures, and scales, pairing realistic splits with standardized evaluation. Results expose challenges in scalable training and out-of-distribution generalization across products, proteins, citation networks, and heterogeneous academic graphs.

Datasets: OGB’s five node-property datasets cover Amazon products, proteins, and academic citation networks, including heterogeneous graphs and a network exceeding 100 million nodes.The datasets differ substantially in density, diameter, clustering, node types, and relations.
ogbn-products: GNNs achieve the highest ogbn-products test performance, but realistic distribution shifts create a large generalization gap; mini-batch GNNs slightly outperform full-batch GraphSAGE that exceeds ordinary GPU memory.The conventional random split is contrasted with the more challenging popularity-based split.
ogbn-proteins: On ogbn-proteins, simple MLPs outperform Node2Vec and GCN, while GraphSAGE performs best, indicating that central-node information is important for prediction.The dataset also raises scalability questions for handling numerous edge features and multiple association types.
ogbn-arxiv: On ogbn-arxiv, graph-based models outperform the feature-only MLP, and the realistic time split is more challenging than a same-ratio random split.The results motivate using graph structure together with edge direction and node temporal information.
ogbn-papers100M and ogbn-mag: On ogbn-papers100M, SGC substantially outperforms MLP but underfits, while ogbn-mag results favor models exploiting heterogeneous graph information and mini-batch GNNs.The benchmarks highlight the need for more expressive models that scale to gigantic graphs and methods that use heterogeneous relations.

5 OGB Link Property Prediction

OGB’s link-property benchmarks span biological, academic, and knowledge-graph domains, using diverse datasets and standardized prediction protocols. Results expose challenges in realistic generalization, graph scale, embedding capacity, and scalable training.

Six link-prediction datasets cover protein associations, author collaborations, drug interactions, paper citations, and knowledge graphs.
ogbl-ppa: On ogbl-ppa, matrix factorization outperforms GNN and NODE2VEC baselines, while poor generalization motivates positional information and improved negative sampling.
ogbl-collab: On ogbl-collab, NODE2VEC performs best in the conventional setting, reflecting the predictive value of past collaborations for future links.
ogbl-ddi: On ogbl-ddi, GNNs transfer training performance to test data better than matrix factorization, while the protein-target split is harder than a random split.GRAPHSAGE reaches 80.88±2.42% test Hits@20 under the random split.
ogbl-citation2: On ogbl-citation2, full-batch GNNs perform best but require more than 40GB of GPU memory, and mini-batch methods perform worse.
Knowledge graphs: Knowledge-graph results show that larger embeddings improve MRR, yet ogbl-wikikg2 remains highly non-trivial; COMPLEX achieves the best test MRR on the second knowledge graph.The ogbl-wikikg2 test MRR uses 500 negative entities per triplet, fewer than common existing KG datasets.

6 OGB Graph Property Prediction

OGB’s graph-property benchmarks span diverse graph domains, structures, and prediction tasks, while testing GNN scalability and generalization under realistic splits.

Datasets: OGB provides molecular, protein-association, and source-code AST datasets spanning three application domains.The graph-property suite includes ogbg-molhiv, ogbg-molpcba, ogbg-ppa, and ogbg-code2.
Datasets: The datasets range from tree-like molecular and AST graphs to denser, larger protein-association neighborhoods.ogbg-ppa has larger graphs, higher average degree, clustering, and diameter than the molecular and AST datasets.
Baselines: Baseline GNNs obtain graph embeddings through node-level message passing, graph pooling, and a final linear prediction model.The baselines include GCN, GCN with virtual nodes, GIN, and GIN with virtual nodes.
Molecular graphs: GIN with additional features and virtual nodes performs best on the two molecular datasets.OGB therefore includes additional node and edge features in its molecular graphs.
Generalization: Random splits substantially outperform scaffold or species splits, revealing challenging distribution shifts in molecular and protein-association prediction.For ogbg-molhiv, random-split ROC-AUC is 82.73±2.02%, 5.66 points above scaffold; ogbg-ppa random-split accuracy is 92.91±0.27%, over 20 points above species split.
Source code ASTs: ogbg-code2 contains approximately 450 thousand Python-method ASTs and predicts method-name subtokens from AST structure and node features.The dataset covers 13,587 repositories and evaluates predictions with subtoken F1.
Generalization: The project split is harder than random splitting, with GCN plus virtual nodes reaching 21.64±0.26% test F1 on the random split.The random result is approximately 6 percentage points higher than the project split, and the reported generalization gap is around 15 points.

7 OGB Package

The OGB package standardizes graph-ML data loading, dataset splits, and evaluation across compatible machine-learning frameworks, allowing researchers to focus on model development.

Pipeline: OGB automates dataset loading, experimental setup, and model evaluation through data loaders and evaluators.The package also provides a public leaderboard for reproducible graph-ML research.
Compatibility: OGB is compatible with PyTorch, PyTorch Geometric, Deep Graph Library, TensorFlow, MXNet, and other Python deep-learning frameworks.It additionally exposes library-agnostic dataset objects.
Data loading: Researchers can obtain a dataset object and standardized train, validation, and test indices through a concise loader interface.The loader downloads, processes, stores, and returns the requested dataset object.
Evaluation: Dataset-specific Evaluator objects standardize the expected prediction inputs and return model-performance results.For ogbg-molpcba, the evaluator accepts ground-truth and predicted binary-label matrices.

8 Conclusions

OGB introduces realistic graph benchmarks across scales, domains, and task categories, together with reusable tooling for reproducible evaluation and research.

Conclusions: OGB combines diverse realistic datasets, application-specific splits, data loaders, evaluation scripts, baseline code, and public leaderboards.The initiative is open-source and invites community contributions of state-of-the-art graph-ML models.
Conclusions: Benchmark experiments show challenges in scaling to large graphs and predicting accurately under realistic data-splitting scenarios.The paper presents these challenges as opportunities for future graph-ML research.

A More Benchmark Results on ogbg-mol* Datasets

Additional MoleculeNet experiments show that feature augmentation and virtual nodes usually improve generalization, while the best GNN architecture varies by dataset.

Experimental scope: The supplementary benchmark covers 10 additional MoleculeNet datasets using standardized protocols and task-appropriate metrics.It uses ROC-AUC, Average Precision for heavily imbalanced classification, and RMSE for regression.
Findings: Additional features give comparable or improved performance on 9 of 10 datasets when added to GIN with virtual nodes.The only exception reported is ogbg-molbace.
Findings: Adding virtual nodes to GIN gives comparable or improved performance on 9 of 10 datasets.The reported exception is ogbg-clintox.
Findings: The optimal architecture varies across datasets, with GCN or GIN preferred depending on the molecule benchmark.The results motivate research into architectures that perform well across molecule datasets.
Implications: The benchmark results provide baselines for future molecule-specific graph-ML models.The authors frame the experiments as useful for further research on molecular methods.