Source-linked AI summary
TUDataset: A collection of benchmark datasets for learning with graphs
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, Marion Neumann
TL;DR
Existing graph-learning studies often rely on small, non-diverse benchmarks and non-standardized protocols, hindering comparisons across publications. TUDataset addresses this with over 120 standardized datasets, baseline implementations, and evaluation procedures; experiments find classical graph kernels remain competitive, while specialized molecular architectures provide gains over generic baselines.
Problem
Graph-learning research commonly uses small, non-diverse benchmarks with non-standardized experimental protocols and baselines, hindering comparison across publications.
Method
TUDataset provides over 120 graph classification and regression datasets, standard data formats, kernel and GNN baseline implementations, and evaluation procedures including cross-validation.
Results
Classical graph kernels with SVMs remain highly competitive in graph classification, while specialized MPNN architectures significantly outperform the generic GINE-ε baseline on large-scale molecular learning tasks.
Takeaways & Limitations
The collection and unified evaluation procedures are intended to improve result comparability and support further progress in graph representation learning.
Takeaways & Limitations
The study uses one-hot degree features for neural baselines on datasets without node labels, unlike the kernel baselines.
Abstract
from arXiv · showhide
Recently, there has been an increasing interest in (supervised) learning with graph data, especially using graph neural networks. However, the development of meaningful benchmark datasets and standardized evaluation procedures is lagging, consequently hindering advancements in this area. To address this, we introduce the TUDataset for graph classification and regression. The collection consists of over 120 datasets of varying sizes from a wide range of applications. We provide Python-based data loaders, kernel and graph neural network baseline implementations, and evaluation tools. Here, we give an overview of the datasets, standardized evaluation procedures, and provide baseline experiments. All datasets are available at www.graphlearning.io. The experiments are fully reproducible from the code available at www.github.com/chrsmrrs/tudataset.
1. Introduction
Graph learning research has advanced through graph kernels and graph neural networks, but progress is constrained by small, non-diverse benchmarks and inconsistent evaluation practices. TUDataset addresses this gap with a broad dataset collection, standardized tools, and comparative experiments.
- Graph learning methods exploit graph structure and node or edge features across domains including cheminformatics, bioinformatics, imaging, and social-network analysis.
- Most recent studies use fixed small-scale, non-diverse benchmarks with non-standardized protocols and baselines, hindering comparison across publications.
- TUDataset provides over 120 datasets spanning graph classification and regression across a wide range of domains.
- The collection offers standard data formats, access through popular graph-learning frameworks, baseline implementations, and evaluation procedures.
- The study compares graph kernels and graph neural networks on a subset of the collection to support standardized benchmarking.
2. The TUDataset collection
TUDataset combines diverse graph datasets with accessible baselines and evaluation procedures for supervised graph learning. Its collection spans application domains, graph models, temporal and synthetic settings, and molecular, biological, visual, and social-network data.
- TUDataset contains over 120 datasets with Python access to datasets, baseline methods, and evaluation tools.
- Datasets differ in graph models and annotations, including discrete or continuous node and edge attributes, even within the same domain.
- Small molecules: The collection covers molecular datasets in which nodes represent atoms, edges represent chemical bonds, and labels encode properties such as toxicity or biological activity.
- Bioinformatics: Bioinformatics datasets represent macromolecules using graph models for proteins, with tasks including six-class enzyme classification and enzyme prediction.
- Computer vision: Computer-vision datasets include handwritten signs, fingerprints, 3D point clouds for robot grasping, and semantic image processing.
- Social networks: Social-network datasets model discussion threads, researcher collaborations, and actor collaborations for tasks such as subreddit, research-field, and genre prediction.
- The collection includes temporal datasets derived from online, collaboration, social, and contact networks to support temporal graph classification.
- Baseline support includes common graph kernels, GNN architectures, Python accessibility, and procedures using SVMs, ADAM, cross-validation, validation splits, and repeated evaluation.
3. Experimental evaluation
The evaluation compares graph kernels and GNN baselines across classification and molecular regression datasets using standardized experimental procedures. Results show strong kernel competitiveness, scalability constraints, and gains from specialized molecular architectures.
- The study compares graph kernels and GNN baselines on graph classification and large-scale multi-target molecular regression datasets.
- Evaluation uses kernel and neural baselines, with classification accuracies summarized for small- and mid-scale datasets and MAE-based results for molecular regression.
- WL-OA performs best overall on small-scale datasets but does not scale to large datasets because it relies on Gram matrix computation.
- 1-WL performs well across the large datasets except GITHUB_STARGAZERS, where neural baselines perform best overall.
- Classical graph kernels combined with SVMs remain highly competitive in graph classification despite extensive recent work on GNNs.
- Specialized MPNN architectures produce significant gains over the generic GINE-ε baseline on large-scale molecular learning tasks.
- For neural baselines, datasets without node labels use one-hot degree features, unlike the kernel baselines.
4. Conclusion
The paper presents TUDataset and a comparative study of graph kernels and GNNs, aiming to support more comparable and reproducible graph-learning research. It identifies expanded dataset coverage and broader molecular regression comparisons as future directions.
- TUDataset provides a collection and experimental comparison intended to improve the comparability of graph-learning results through unified evaluation procedures.
- Future work includes adding datasets and conducting a more extensive comparison of kernel and neural approaches on molecular regression with continuous features.
A. Evaluation examples
Further documentation for the evaluation examples is available at graphlearning.io.
- Further documentation is available at www.graphlearning.io.
- The evaluation examples direct readers to graphlearning.io for additional documentation.
- Readers can consult www.graphlearning.io to learn more about the examples.
Kernelized SVM for graph kernels based on Gram matrices
The example computes 1-WL graph-kernel Gram matrices for ENZYMES across one to five iterations, normalizes them, and evaluates them with repeated LIBSVM cross-validation.
- The example downloads the dataset and imports utilities for datasets, kernel baselines, and kernel SVM evaluation.
- It computes the 1-WL kernel for iterations 1 through 5 on ENZYMES using node labels but no edge labels.
- Cosine normalization is applied to each computed Gram matrix before the matrices are collected.
- The resulting matrices are evaluated with LIBSVM using 10 repetitions of 10-fold cross-validation.
Linear SVM for graph kernels based on sparse feature maps
The sparse-feature example computes 1-WL kernels with node and edge labels, applies ℓ_2 normalization, and evaluates the resulting feature maps with repeated LIBLINEAR cross-validation.
- The example downloads the dataset and imports utilities for datasets, kernel baselines, and linear SVM evaluation.
- It computes the 1-WL kernel for iterations 1 through 5 using both node labels and edge labels.
- LIBLINEAR evaluation uses 10 repetitions of 10-fold cross-validation.
GNN evaluation
The GNN example evaluates a GIN after downloading the dataset, tuning layer and hidden-feature counts, and training under fixed epoch, batch-size, learning-rate, and repetition settings.
- The example imports GNN evaluation utilities and the GIN architecture, then downloads the selected dataset.
- It optimizes the number of layers over {1,2,3,4,5} and the number of hidden features over {32,64,128}.
- Training uses a maximum of 200 epochs and a batch size of 64.
- The evaluation configuration sets the starting learning rate to 0.01 and uses 10 repetitions for 10-fold cross-validation.
Loading graphs in NetworkX format
The loading workflow downloads a selected dataset and converts it into a list of graphs represented in NetworkX format.
- The interface imports auxiliary dataset methods and a NetworkX conversion reader.
- The selected dataset is downloaded through dp.get_dataset(dataset).
- The resulting dataset is output as a list of graphs.
B. Experimental protocol and hyperparameters for ZINC, ALCHEMY, QM9
The molecular regression experiments use specified architectures, dataset splits, target normalization, and repeated evaluation, with a stated distinction between benchmark baselines and state-of-the-art physical models.
- Hyperparameters: ZINC uses four GINE-ε layers with hidden dimension 256, batch normalization, mean pooling, and a 4-layer MLP for joint regression of twelve targets.
- Hyperparameters: ALCHEMY and QM9 use six layers with 64 hidden node features, set2seq graph pooling, and a 2-layer MLP for joint regression of twelve targets.
- Data splits and metrics: 80% of ALCHEMY and QM9 graphs are used for training, with 10% each for validation and testing, while ZINC uses its given splits.
- Data splits and metrics: Targets for ALCHEMY and QM9 are normalized using training-split mean and variance, and results are reported with mean standardized MAE and mean standardized logMAE over five repetitions.
- Baselines and scope: QM9 additionally uses MPNN as a baseline, while the experiments explicitly do not aim to beat physical knowledge-incorporating architectures.
- Dataset caveat: The full ALCHEMY dataset differs from the contest dataset and does not provide normalized targets.
C. Dataset statistics
Table 4 presents dataset statistics and properties, including a notation for continuous vertex labels and an encoding detail for the final three components.
- Table 4 reports dataset statistics and properties.
- The dagger symbol marks continuous vertex labels following Gilmer et al. (2017).
- The caption specifies that the last three components encode an additional property, though the supplied passage does not identify it.