Source-linked AI summary
OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, Jure Leskovec
TL;DR
Large-scale graph ML has lacked realistic public benchmarks, despite the importance of handling massive graphs. OGB-LSC introduces three datasets spanning core graph tasks and evaluates scalable baseline and expressive models. Expressive models significantly outperform simpler baselines, while the KDD Cup demonstrates further advances from innovative techniques.
Problem
Large-scale graph ML has been limited by the lack of suitable public benchmarks, although massive graphs matter for industrial and scientific applications.
Method
OGB-LSC provides three realistic large-scale datasets covering node-, link-, and graph-level prediction, with dedicated baseline analyses and models scaled to massive graphs.
Results
Advanced expressive models significantly outperform simpler graph ML baselines, and KDD Cup winners achieved significant accuracy improvements over strong baselines.
Takeaways & Limitations
OGB-LSC and its baseline code support research on large-scale graph ML, while the KDD Cup results highlight techniques and best practices for further advances.
Takeaways & Limitations
The datasets do not fully capture dynamic real-world graphs, and PCQM4M inference is constrained to less than 0.1 second per molecule on a single GPU and CPU.
Abstract
from arXiv · showhide
Enabling effective and efficient machine learning (ML) over large-scale graph data (e.g., graphs with billions of edges) can have a great impact on both industrial and scientific applications. However, existing efforts to advance large-scale graph ML have been largely limited by the lack of a suitable public benchmark. Here we present OGB Large-Scale Challenge (OGB-LSC), a collection of three real-world datasets for facilitating the advancements in large-scale graph ML. The OGB-LSC datasets are orders of magnitude larger than existing ones, covering three core graph learning tasks -- link prediction, graph regression, and node classification. Furthermore, we provide dedicated baseline experiments, scaling up expressive graph ML models to the massive datasets. We show that expressive models significantly outperform simple scalable baselines, indicating an opportunity for dedicated efforts to further improve graph ML at scale. Moreover, OGB-LSC datasets were deployed at ACM KDD Cup 2021 and attracted more than 500 team registrations globally, during which significant performance improvements were made by a variety of innovative techniques. We summarize the common techniques used by the winning solutions and highlight the current best practices in large-scale graph ML. Finally, we describe how we have updated the datasets after the KDD Cup to further facilitate research advances. The OGB-LSC datasets, baseline code, and all the information about the KDD Cup are available at https://ogb.stanford.edu/docs/lsc/ .
1 Introduction
Large-scale graph ML lacks benchmarks matching realistic industrial and scientific graphs, while expressive models remain difficult to scale. OGB-LSC addresses this gap with three large datasets, baseline studies, and a KDD Cup deployment.
- Existing graph ML benchmarks remain small relative to industrial and scientific graphs, limiting large-scale progress.
- Scaling expressive GNNs is challenging because node predictions depend on information from many other nodes and require specialized training algorithms.
- OGB-LSC introduces MAG240M, WikiKG90M, and PCQM4M, covering node-, link-, and graph-level prediction tasks at scales 10 to 100 times larger than corresponding original OGB datasets.
- Advanced expressive models benefit from large data and significantly outperform simple baseline models, despite requiring more effort to scale.
- The ACM KDD Cup 2021 built around OGB-LSC attracted more than 500 team registrations and 150 leaderboard submissions, generating innovative methods and performance gains.
2 OGB-LSC Datasets, Baselines, and KDD Cup Summary
OGB-LSC provides large-scale datasets and baselines spanning node-, link-, and graph-level prediction, with experiments showing benefits from expressive models. KDD Cup results further demonstrate substantial gains and identify promising techniques for large-scale graph ML.
- Datasets and baselines: OGB-LSC introduces datasets for node-, link-, and graph-level prediction, with practical task designs and publicly available code.The datasets are MAG240M, WikiKG90M, and PCQM4M.
- MAG240M: Node-Level Prediction: MAG240M evaluates subject-area classification on arXiv papers, where selected heterogeneous meta-paths produce substantially different homophily levels.Paper-Author-Paper paths are more homophilous than Paper-Author-Institution-Author-Paper paths, motivating analysis of heterogeneous connectivity.
- Datasets and baselines: Expressive models outperform simpler scalable baselines across the OGB-LSC experiments.The paper reports this pattern for the benchmark overall, while dataset-specific analyses show advantages for expressive GNNs and multimodal encoders.
- MAG240M: Node-Level Prediction: 73.71% validation accuracy from a single BD-PGL model improves the best R-GAT baseline by 3.7%, while KDD Cup winners achieved 5–6% test-accuracy gains.The winning solutions used expressive architectures, label propagation, self-supervised learning, temporal information, and model ensembling.
- WikiKG90M: Link-Level Prediction: 0.92 validation MRR from a single BD-PGL model improves the best COMPLEX-CONCAT baseline by 0.07, and winning systems reached 0.97 test MRR.The strongest encoders combine textual and structural information; candidate-tail frequency also provided a strong statistical signal.
3 Updates after the KDD Cup
After the KDD Cup 2021, OGB-LSC updates make WikiKG90M harder, add training 3D structures to PCQM4M, and introduce public leaderboard splits.
- The datasets and leaderboards were updated based on lessons learned from the KDD Cup 2021.These changes are intended to facilitate further research advances.
- Updates on WikiKG90M: WikiKG90Mv2 removes validation and test candidate entities because the original candidates were mostly easy negatives.The best COMPLEX-CONCAT model achieves 0.1833 MRR on WikiKG90Mv2 versus 0.8637 MRR on WikiKG90M.
- Updates on PCQM4M: PCQM4Mv2 provides DFT-calculated 3D structures for training molecules while withholding 3D structures for validation and test molecules.Models must still predict validation and test molecules from their 2D molecular graphs.
- Updates on leaderboards: The test set is split into public test-dev and future-competition test-challenge subsets for post-KDD Cup evaluation.Test-dev supports public leaderboard submissions during the year, while test-challenge is reserved for future competitions.
4 Conclusions
OGB-LSC addresses the shortage of suitable large-scale graph ML benchmarks with three realistic datasets spanning node, link, and graph prediction. Its baseline analysis finds that advanced expressive models can outperform simpler scalable models, while the KDD Cup generated community engagement and performance improvements.
- OGB-LSC provides three large-scale realistic benchmark datasets covering node classification, link prediction, and graph regression.The challenge targets graph data with billions of edges or millions of graphs.
- Advanced expressive models can significantly outperform simpler baseline models on large-scale graph learning tasks.The paper presents this result as an opportunity for further dedicated effort to improve performance.
- The ACM KDD Cup 2021 attracted substantial community engagement and produced significant performance improvement through innovative methods.The paper summarizes winning solutions and current best practices in large-scale graph ML.
Checklist
The checklist reports that the paper documents its experimental materials, training details, compute resources, asset licensing, and relevant ethical considerations, while error bars were not reported.
- The paper reports that supplementary experimental results, training details, and reproducibility materials are provided.It points to Appendix A, Section 2, and the relevant code, data, and instructions.
- Error bars were not reported because the datasets are very large and use hidden test sets, following conventions from comparable benchmarks.The paper states that model performance is often stable on large datasets.
- The paper reports compute resources, cites existing assets, discusses their licenses, and provides URLs for new assets.It also states that public datasets are used and license rules are followed.
- The paper reports that its datasets contain no private or offensive information and that human-subject procedures were not applicable.Potential societal impacts and ethics-review compliance are addressed in Appendix A.
A Key Information about OGB-LSC
OGB-LSC provides datasets, code, documentation, hosting, maintenance, licensing, and intended-use guidance for large-scale graph ML research. The benchmark’s resources are accompanied by stated limitations and potential social-impact concerns.
- OGB-LSC is intended for machine learning and data scientists developing models for large-scale graph ML.
- The official website documents the datasets, Python-package usage, KDD Cup leaderboards, winner reports, and code.
- The GitHub repository hosts the OGB package, dataset evaluation, and released baseline code.
- The datasets total around 300GB, are hosted on AWS with DGL support, and are downloaded and processed through the Python package.
- The package uses the MIT license, while MAG240M, WikiKG90M, and PCQM4M use ODC-BY, CC-0, and CC BY 4.0, respectively.
- The current datasets omit representative large-scale recommender, social, and financial graphs because privacy and cooperation concerns make them difficult to obtain.
B Basic Graph Statistics of the Datasets
Table 8 provides the basic graph statistics of the OGB-LSC datasets, including statistics computed on standardized graph representations.
- Table 8 reports basic graph statistics for the OGB-LSC datasets and standardized graph variants.Standardized graphs are converted to undirected, unlabeled homogeneous graphs with duplicate edges removed before statistics are computed.
C Details about Dataset Updates after the KDD Cup 2021
After KDD Cup 2021, OGB-LSC updated MAG240M, WikiKG90M, and PCQM4M to improve evaluation settings and dataset content. The updates include unrestricted tail prediction, newer Wikidata snapshots, molecular 3D structures, and corrected molecular labels.
- MAG240M updates: MAG240M itself was unchanged; only its test set was updated, with test-dev accuracy reported for all models.
- WikiKG90Mv2 updates: WikiKG90Mv2 removes provided validation and test candidate tails, requiring prediction of target entities from all Wikidata entities.
- WikiKG90Mv2 updates: WikiKG90Mv2 uses more recent Wikidata dumps collected on May 17, June 7, and June 28, 2021 for training, validation, and testing.
- WikiKG90Mv2 updates: Textual features provide important signals for validation triples whose head entities have very sparse training connections.
- WikiKG90Mv2 updates: For WikiKG90Mv2 inference, relation-specific candidate sets contain 20K most frequent training tails per relation because scoring nearly 90M entities does not scale well.
- PCQM4Mv2 updates: PCQM4Mv2 adds DFT-calculated 3D molecular structures and updates some SMILES strings after identifying mismatches with HOMO-LUMO gaps.
- PCQM4Mv2 updates: All models achieved lower MAE on PCQM4Mv2 than PCQM4M, probably because the identified mismatch bug was fixed.