Source-linked AI summary
Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks
Maya Bechler-Speicher, Ben Finkelshtein, Fabrizio Frasca, Luis Müller, Jan Tönshoff, Antoine Siraudin, Viktor Zaverkin, Michael M. Bronstein, Mathias Niepert, Bryan Perozzi, Mikhail Galkin, Christopher Morris
TL;DR
Graph-learning research lacks benchmarks that consistently reflect meaningful applications, faithful graph constructions, and reliable comparisons. This position paper diagnoses these shortcomings and advocates revised datasets, rigorous evaluation, tuned baselines, and domain collaboration; its reassessment shows that reported progress can change substantially with better tuning.
Problem
Graph-learning benchmarks often use narrow or poorly justified datasets and inconsistent evaluations, limiting reliable assessment of model progress and real-world relevance.
Method
The paper presents a position-based critique, proposes benchmark and evaluation remedies, and empirically reassesses baselines across molecular and other graph-learning settings.
Results
Re-tuning GINE reduced error by over 20% to 0.0913 without RWSE features and to 0.0898 with structural features, showing reported evaluations can be brittle.
Takeaways & Limitations
The paper calls for meaningful real-world benchmarks, standardized evaluations, scalable datasets, and stronger collaboration with domain experts.
Takeaways & Limitations
Benchmark utility cannot be conclusively determined without well-defined evaluation protocols, even for promising datasets.
Abstract
from arXiv · showhide
While machine learning on graphs has demonstrated promise in drug design and molecular property prediction, significant benchmarking challenges hinder its further progress and relevance. Current benchmarking practices often lack focus on transformative, real-world applications, favoring narrow domains like two-dimensional molecular graphs over broader, impactful areas such as combinatorial optimization, relational databases, or chip design. Additionally, many benchmark datasets poorly represent the underlying data, leading to inadequate abstractions and misaligned use cases. Fragmented evaluations and an excessive focus on accuracy further exacerbate these issues, incentivizing overfitting rather than fostering generalizable insights. These limitations have prevented the development of truly useful graph foundation models. This position paper calls for a paradigm shift toward more meaningful benchmarks, rigorous evaluation protocols, and stronger collaboration with domain experts to drive impactful and reliable advances in graph learning research, unlocking the potential of graph learning.
1. Introduction
Graph learning benchmarks often emphasize narrow or saturated tasks rather than transformative real-world applications, while inconsistent practices weaken comparisons. The paper proposes revising datasets, modalities, and evaluation to restore the field’s impact and relevance.
- Current shortcomings: Many benchmarks focus on narrow domains such as 2D molecular graphs or synthetic datasets with questionable practical relevance.These choices can neglect critical 3D structures and lack sufficient real-world justification for graph-based modeling.
- Suggested remedies: The paper calls for shifting toward naturally graph-structured problems with real-world impact, including combinatorial optimization and relational databases.The proposed direction targets applications where graphs encode meaningful problem structure.
- Current shortcomings: Fragmented evaluation protocols and inconsistent benchmarking practices make performance comparisons unreliable and contribute to the absence of true graph foundation models.The paper identifies evaluation fragmentation alongside limited datasets and modalities as a central obstacle.
- Suggested remedies: The paper proposes actionable recommendations, including revised benchmarks, tuned baselines, and broader evaluation across molecular prediction, heterophilic datasets, and transfer settings.Its empirical work tunes baselines and reference models across these task categories.
- Related work: Earlier benchmark suites improved standardization but often used saturated synthetic tasks, transformed datasets, restrictive model-size ceilings, or incompletely covered graph regimes.The discussion highlights limitations in suites from Dwivedi et al., OGB, and later long-range benchmarks.
2. Missing transformative real-world applications and supporting benchmarks
The paper argues that graph benchmarks should target transformative problems naturally represented as graphs rather than narrow 2D molecular tasks. It highlights combinatorial optimization and other domains as promising sources of realistic, scalable benchmarks.
- Problem: Small 2D molecular graphs cover niche chemistry and drug-discovery subfields, while many real applications depend on 3D molecular geometry.The paper notes that 3D structure is important for tasks such as molecular docking and fragment linking.
- Suggested remedies: The community should prioritize graph-native problems such as vehicle routing and integer-linear programming.Combinatorial optimization offers clear applications, easy large-dataset generation, and testbeds for size generalization.
- Suggested remedies: Other promising areas include satisfiability solving, recommender systems, social networks, power-flow networks, relational databases, computer systems, and automated chip design.Examples include RelBench, 4DBInfer, TpuGraphs, and AlphaChip.
- Limitations: Sensitive industrial data, especially social-network data, limits accessibility and makes anonymized graph generation an open challenge.The paper identifies better anonymization as a possible way to address this constraint.
- Suggested remedies: Graph generation can support design-related tasks, including sampling solutions to combinatorial problems, but efficiency and GNN capabilities remain open issues.The paper also hypothesizes value in conditional generation when targets are easy to evaluate or predictive models are robust.
3. Graphs are not necessarily constructed in a meaningful way
A graph representation is useful only when its relational structure meaningfully and predictively reflects the underlying task. The paper recommends justifying graph construction and comparing graph models against non-graph baselines.
- Problem: Benchmarks often fail to establish whether their constructed graphs meaningfully, relevantly, and completely represent the underlying data.Some graphs use unsuitable formalisms or omit important information.
- Examples: Image benchmarks may connect super-pixels with rag-boundary edges without theoretical or empirical justification, leaving the benefit of graph modeling unclear.The paper uses PASCALVOC-SP and COCO-SP as examples.
- Examples: Sensor-network benchmarks can impose graph connectivity by thresholding a Gaussian kernel over pairwise sensor distances.This converts an otherwise fully connected weighted graph into a thresholded structure.
- Examples: Molecular benchmarks such as ZINC encode atoms and bonds but omit structural information including 3D coordinates and SMILES-derived features.The paper treats this as incomplete representation of molecular information.
- Examples: MPNNs can overfit to spurious structural correlations, allowing set-based models that use only node features to generalize better.This shows that natural relational information is not necessarily informative for the prediction task.
- Suggested remedies: Benchmark authors should explain graph construction, feature choices, edge determination, and why relational processing should help the target prediction task.Evaluations should also include tuned baselines using unstructured node-feature sets or similarity-based connectivity.
4. Bad benchmarking culture
Graph-learning evaluations are weakened by inconsistent protocols, outdated or unfair baselines, saturated tasks, and incomplete reporting. The paper calls for practical, expert-validated benchmarks with stronger standardization and broader data.
- Problem: Inconsistent protocols and dataset splits produce highly variable results, while small datasets can yield large standard deviations and unreliable comparisons.The paper cites MUTAG’s 188 graphs and validation-only reporting as examples.
- Problem: New architectures are often compared with outdated or poorly tuned baselines, inflating apparent performance gains.The paper notes that baseline results may be reused for years without reevaluation.
- Problem: Incremental improvements on saturated ZINC tasks can receive attention even though standard chemoinformatics tools readily solve them.Limited molecular and materials expertise can also obscure the importance of 3D structure–property relationships.
- Problem: Molecule-generation benchmarks remain dominated by near-perfect QM9 and ZINC250K results, while more robust alternatives are underused because of computational demands.The paper also identifies inconsistent splits, FCD reporting, and insufficiently studied conditional generation.
- Problem: Graph generative-model benchmarking beyond molecules is less standardized, with limited datasets, omitted metrics, absent error bounds, and additional difficulties for large graphs.Practical datasets and standardization are especially scarce for large-graph evaluation.
- Suggested remedies: The paper recommends practical tasks, robust evaluation frameworks, hidden-test competitions, larger domain-relevant datasets, and multidisciplinary curation.Expert-validated benchmarks are presented as a route to improving evaluation and adoption.
5. Implication: No true foundation model exists for graph learning
The paper argues that truly impactful graph foundation models have not yet emerged because current training data and evaluations do not support broad, reliable generalization. It recommends unified models, diverse large-scale datasets, and task-specific adaptation around shared backbones.
- Graph foundation models have not yet emerged with impact comparable to foundation models in other deep-learning domains.Existing domain-specific graph models often show only marginal gains over standard supervised GNNs.
- Small datasets and academic tasks without rigorous evaluations hamper progress toward useful graph foundation models.Additional challenges include differing task symmetries, varying graph scales and feature spaces, and the absence of a universal graph vocabulary.
- The paper recommends training one model across datasets rather than one model per dataset to assess performance across diverse graphs.For mixed classification and regression settings, it suggests a shared encoder-processor-decoder design with task-specific encoders and decoders.
- Large-scale, high-quality datasets should cover diverse graph structures, use synthetic data to address gaps, and exclude known test sets from pre-training corpora.
6. Alternative views
The paper contrasts graph learning's benchmarking challenges with adjacent geometric deep learning, which has achieved notable advances by focusing on three-dimensional objects and their symmetries. These adjacent successes motivate attention to richer graph representations.
- Geometric deep learning has advanced structural biology and materials science.
- Geometric deep learning focuses on three-dimensional objects and addresses symmetries and invariances inherent to the input data.
7. Empirical evidence
The empirical evidence shows that graph-learning conclusions can change substantially when graph structure, baseline tuning, and architectural modifications are evaluated more carefully. The results support stricter, standardized protocols across benchmark settings.
- 7.1. Graphs not necessarily constructed in a meaningful way: Across 5 of 9 experiments, a noninformative regular Cayley graph matched or outperformed the original graph.For MOLBBBP, GraphConv achieved the highest ROC-AUC when the graph structure was completely dropped.
- 7.2. Reassessing simple baselines on PCQM4MV2: Old, untuned reference results can make newer architectures' performance gains appear larger than they are.
- 7.2. Reassessing simple baselines on PCQM4MV2: Over 20% lower GINE validation error on PCQM4Mv2 was obtained through configuration tuning alone, reaching 0.0913 without additional RWSE features.With additional structural features, the error reached 0.0898 and was competitive with several graph-transformer baselines.
- 7.3. The meaningfulness of architectural changes: The heterophilous benchmark comparisons were less meaningful because baseline and specialized models used unequal architectural and evaluation conditions.The paper states that the benchmark datasets require an accompanying evaluation protocol to assess their quality reliably.
- 7.3. The meaningfulness of architectural changes: +43.56%, +14.63%, and +19.49% average baseline performance gains arose from architectural modifications on roman-empire, amazon-ratings, and minesweeper, respectively.The analysis motivates fixed model-size limits and clear rules for architectural modifications.
- 7.3. The meaningfulness of architectural changes: The study evaluates upstream datasets with single-task and multi-task training and tests transfer to downstream datasets with varying similarity.It reports that MPNNs and graph transformers can learn representations that often transfer to in-domain and cross-domain tasks.
8. Conclusion
The conclusion calls for graph-learning benchmarks to reflect real-world complexity, use standardized evaluations, and include scalable datasets. It presents these changes as necessary for maintaining the field's impact and relevance.
- Current graph benchmarks often overlook real-world problems, narrow their data modalities, and lack consistent protocols or large-scale datasets.
- The paper proposes designing realistic benchmarks, standardizing evaluations, and creating scalable datasets.
- These changes are presented as a way for graph learning to align with machine-learning advances and maintain impact and relevance.
A. Extended related work
Graph-learning benchmarks remain limited by narrow or unrealistic graph distributions, while evaluation practices and model implementations can make reported comparisons fragile. The paper reviews broader benchmark directions and details architectures and training choices used to reassess these issues.
- Benchmark directions: GRAPHWORLD showed that existing datasets cover a narrow distribution of graphs, and that tuning common MPNNs may not predict performance in less common domains.It proposed stochastic block model graphs with more diverse connectivity patterns.
- Benchmark directions: CLRS evaluates graph-based algorithmic reasoning through simulations of 30 classical algorithms under difficult size-generalization settings.The benchmark includes node- and edge-level prediction tasks.
- Benchmark limitations: Real-world social-network benchmarks often lack large-scale, realistic structure, including high-degree hubs and dense communities.Random-walk sampling can further reduce average node degree and weaken representativeness of the original graph.
- Benchmark limitations: Temporal graph benchmarks expose another gap: simple heuristics and overlooked baselines can outperform sophisticated temporal GNNs.This finding questions the relevance of some benchmarks and the significance of reported progress.
- Evaluation practice: The reviewed experiments tune architecture and optimization choices across layers, widths, batch sizes, learning rates, dropout, weight decay, and training schedules.The GINE reassessment uses 20 layers, 512-dimensional embeddings, and approximately 20 million trainable parameters.
- Model architectures: The reviewed encoder-processor-decoder setup uses domain-specific encoders, task-specific decoder MLPs, and either GINE-based MPNNs or Graphormer-based graph transformers.The graph transformer uses soft attention bias and RWSE structural encodings; downstream fine-tuning freezes processor weights.
- Model architectures: Graph-transformer attention incorporates edge features as a learned soft bias, while graphs exceeding the context limit are truncated at 8192 nodes.The bias is added before softmax, and [cls] tokens use separate learnable incoming and outgoing vectors.
B.3.2. MULTI-TASK PRE-TRAINING
The multi-task pre-training setup combines heterogeneous graph datasets spanning domains, tasks, and structural properties. It selects checkpoints using aggregate validation loss after training graph transformers and GINE models on the mixed tasks.
- Pretraining mix: The pretraining mix contains datasets differing in application domain, prediction task, graph size, sparsity, and structural properties.The authors use this diversity to investigate whether processors learn representations useful across potentially unrelated tasks.
- Pretraining datasets: COCO-SP represents images as sparse super-pixel graphs for node-level semantic-segmentation prediction.Nodes contain pixel-value statistics and center-of-mass coordinates.
- Pretraining datasets: PCQM4MV2 contains many small molecular graphs with HOMO-LUMO energy-gap prediction as the task.The passage notes that two-dimensional molecular information is practically relevant because obtaining 3D structures requires expensive DFT-based geometry optimization.
- Pretraining datasets: MALNET-TINY contains larger, unattributed function-call graphs labeled for association with malicious code execution.It contributes a graph domain with relatively few larger graphs and no node or edge attributes.
- Training and selection: Pretraining sweeps learning rates separately for graph transformers and GINE, trains for 100K gradient steps, and selects checkpoints by the summed validation loss across three tasks.The selection criterion aggregates all task losses rather than optimizing a single task.
B.3.3. SINGLE-TASK FINE-TUNING
The single-task fine-tuning study evaluates pretrained and from-scratch models across downstream datasets that vary in domain, task, structure, and similarity to pretraining data. The experiments vary training duration and tune learning rates while comparing matched-parameter baselines.
- Fine-tuning setup: Pretrained processors and reusable feature encoders remain frozen during fine-tuning, while prediction heads and otherwise unavailable encoders are trained from scratch.This setup distinguishes transferred components from newly learned task-specific components.
- Downstream datasets: Downstream datasets span widely varying domains, tasks, structural features, and similarity levels to the pretraining mix.Examples include closely aligned PASCALVOC-SP, structurally distinct PEPTIDES-STRUCT, and fully out-of-domain STARGAZERS datasets.
- Training setup: Models are trained for 1K, 3K, and 10K steps, with batch sizes corresponding roughly to 2, 10, and 30 epochs.Single-task baselines are trained from scratch on the same amount of data.
- Hyperparameter tuning: The learning rate is the only tuned hyperparameter, swept from 1 · 10^-5 to 1 · 10^-2, while baselines match the parameter counts of pretrained counterparts.This design controls model size while varying the learning-rate setting across fine-tuning regimes.
- Additional results: Additional results compare a GNN with random-walk structural encoding against a graph transformer without additional structural encodings.These comparisons extend the main fine-tuning results shown in Figure 2.
B.4. Variance of results reported on ENZYMES
Reported ENZYMES test accuracy varies sharply across publications and years, even for similar base architectures. The paper attributes this instability partly to inconsistent evaluation protocols and warns that cross-study comparisons can confound methodological progress with experimental artifacts.
- Reported performance: Figure 4 plots reported ENZYMES test accuracy against publication year across graph kernels, GNN benchmarking, graph pooling, and graph rewiring studies.The figure aggregates results from multiple lines of graph-learning research.
- Reported performance: ENZYMES results range from over 60% for early simple GCN-based evaluations to sometimes below 30% in publications from 2023 and 2024.For the same base architecture, reported results can differ by a factor of two across publications.
- Evaluation protocols: Older publications typically used stratified 10-fold cross-validation, whereas newer studies often used repeated random 80/10/10 splits.The newer split protocol is more noise-prone and explains some, but not all, of the observed variance.
- Evaluation protocols: Internal consistency within each publication is insufficient when measurements vary drastically over time, because missing standardization can conflate methodological improvements with experimental-design artifacts.The paper argues for shared protocols and rigorous benchmarking on more suitable datasets.