Source-linked AI summary
Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark
Jan Tönshoff, Martin Ritzert, Eran Rosenbluth, Martin Grohe
TL;DR
LRGB was designed around graph tasks with strong long-range dependencies, where Graph Transformers were reported to outperform message-passing GNNs. The paper reevaluates those comparisons through baseline tuning, feature normalization, and scrutiny of the link-prediction metric, finding that the gap closes on multiple datasets and that transformers show clear benefits only on the superpixel tasks.
Problem
LRGB tasks depend strongly on long-range vertex interactions, and prior reports found Graph Transformers outperforming message-passing GNNs on the benchmark.
Source: §1 · p. 1Method
The paper reevaluates MPGNN baselines and GPS with hyperparameter tuning, applies channel-wise feature normalization to vision datasets, and examines PCQM-Contact's filtered-MRR evaluation.
Source: §1 · p. 2, §2 · pp. 2–3Results
Takeaways & Limitations
Extensive baseline tuning is important for assessing methods, and whether other tasks can be solved without long-range interactions remains an open research question.
Source: §4 · p. 5
Abstract
from arXiv · showhide
The recent Long-Range Graph Benchmark (LRGB, Dwivedi et al. 2022) introduced a set of graph learning tasks strongly dependent on long-range interaction between vertices. Empirical evidence suggests that on these tasks Graph Transformers significantly outperform Message Passing GNNs (MPGNNs). In this paper, we carefully reevaluate multiple MPGNN baselines as well as the Graph Transformer GPS (Ramp\'a\v{s}ek et al. 2022) on LRGB. Through a rigorous empirical analysis, we demonstrate that the reported performance gap is overestimated due to suboptimal hyperparameter choices. It is noteworthy that across multiple datasets the performance gap completely vanishes after basic hyperparameter optimization. In addition, we discuss the impact of lacking feature normalization for LRGB's vision datasets and highlight a spurious implementation of LRGB's link prediction metric. The principal aim of our paper is to establish a higher standard of empirical rigor within the graph machine learning community.
1 Introduction
LRGB was introduced to test graph tasks with strong long-range dependence, where global-attention Graph Transformers were expected to help. The paper reassesses that comparison and reports that tuning can reduce or close the gap on multiple datasets.
1 Introduction
- LRGB comprises five datasets with long-range interactions: peptide property prediction, superpixel segmentation, and prediction of graph-distant molecular vertices in 3D contact.The tasks include graph-level classification and regression, node classification, and link prediction.Source: §1 · p. 1
- Earlier LRGB reports found Graph Transformers outperforming GCN, GINE, and GatedGCN, and GPS later reported similar conclusions.The paper notes that the two works are closely related and share a code base.Source: §1 · p. 1
- The authors report that tuning reduces or closes MPGNN–transformer gaps on multiple datasets, feature normalization benefits vision tasks, and PCQM-Contact results depend strongly on filtering implementation.They also report that GCN achieves state-of-the-art results on Peptides-Struct, with much of its improvement tied to a multi-layer prediction head.Source: §1 · p. 2
1.1 Related Work
- The related-work discussion places the study among newer graph transformers, non-transformer approaches, and benchmarking projects advocating rigorous architecture testing.It names Exphormer, GRIT, Graph ViT / GraphMLPMixer, CRaWl, and DRew as examples.Source: §1.1 · p. 2
2 Concerns
The paper questions whether prior LRGB results used representative configurations, whether vision inputs were appropriately normalized, and whether PCQM-Contact was evaluated with its specified filtered metric.
Hyperparameters
- A basic hyperparameter sweep within the official 500k parameter budget improves MPGNN baselines and GPS, while tuning positional or structural encodings for each method.The sweep covers main hyperparameters such as depth and dropout; GPS also improves significantly on three datasets.Source: §2 · pp. 2–3
Feature Normalization
- Channel-wise linear normalization of vision-dataset node and edge features benefits all evaluated models, including GPS.The input channels span different orders of magnitude, and omitting normalization can cause poorly conditioned activations.Source: §2 · p. 3, Figure 2 · p. 3
Link Prediction Metrics
- The PCQM-Contact code computes raw MRR despite the benchmark specifying filtered MRR, which removes other true tails from each head's candidate list.The paper also evaluates an extended filtering setting that removes self-loops, which the scoring function is biased toward.Source: §2 · p. 3
3 Experiments
Tuned MPGNNs close or erase reported gaps on peptide and PCQM-Contact tasks, while vision-dataset results improve with normalization and tuning. Prediction-head depth and the PCQM-Contact filtering setting materially affect comparisons.
Peptides-Func and Peptides-Struct
- 30% relative MAE reduction on Peptides-Struct closes the gap to recent graph transformers; all three tuned MPGNNs outperform GPS on both peptide tasks.GCN reaches 68.60% on Peptides-Func, competitive with newer graph transformers.Source: §3 · p. 3
- A 2-layer prediction head accounts for almost the complete Peptides-Struct gap between MPGNNs and graph transformers, while GPS is largely unchanged by head depth.The head-depth ablation changes the linear head to a 2-layer perceptron while keeping other hyperparameters fixed; the benefit on Peptides-Func is also considerable and highly significant.Source: §3 · pp. 3–4
PascalVOC-SP and COCO-SP
- GPS reaches 44.40% F1 on PascalVOC-SP and 38.84% F1 on COCO-SP; GatedGCN reaches 38.80% F1 on PascalVOC-SP, exceeding GPS’s originally reported score.On PascalVOC-SP, GPS is one percentage point below CRAWL, the only reported result with normalized features.Source: §3 · p. 4
PCQM-Contact
- Tuning improves raw MRR by around 3% on PCQM-Contact, and GPS’s previously reported slight edge is no longer observable.Figure 3 presents MRR scores under varying filtering settings.Source: Figure 3 · p. 4, §3 · p. 4
- On PCQM-Contact, filtered MRR is roughly 3% above raw MRR, and additionally removing self-loops raises it by about 10 percentage points; GPS’s relative edge returns in the latter setting.The reported comparison indicates that negative-sample filtering can change method rankings.Source: §3 · p. 4
4 Conclusion
The experiments find considerable gains for all three MPGNN baselines. Graph transformers show clear performance benefits over MPGNNs only on the two superpixel datasets, leaving open how much the other tasks require long-range interactions.
4 Conclusion
- All three MPGNN baselines show considerable performance gains in the experiments.The conclusion identifies extensive baseline tuning as important for properly assessing methods, especially on relatively recent datasets.Source: §4 · p. 5
- Graph transformers show clear performance benefits over MPGNNs only on the PascalVOC-SP and COCO-SP superpixel datasets.The conclusion leaves open whether other tasks can be solved without long-range interactions or whether graph transformers are not inherently better at exploiting them.Source: §4 · p. 5
- The authors identify evaluation of long-range dependency use and discussion of the best-suited PCQM-Contact link-prediction metric as future directions.Source: §4 · p. 5
A.1 Hyperparameters
The hyperparameter search tunes selected choices individually from an empirical default, then tests a combined configuration and selects by validation performance. The sweep is explicitly limited and includes several additional setup changes.
A.1 Hyperparameters
- The search tunes each hyperparameter individually from an empirical default, then evaluates a configuration combining the best choices and selects the best validation performer.Each search configuration uses one run; final evaluations average results across four random seeds.Source: §A.1 · p. 7
- The main tuned choices are dropout, model depth, prediction-head depth, learning rate, and positional or structural encoding; GPS also varies its internal MPGNN and normalization type.The tested ranges include dropout [0, 0.1, 0.2], depth [6] [8] [10], head depth [1] [2] [3], and encoding [none, LapPE, RWSE].Source: §A.1 · p. 7
- The sweep is not exhaustive and uses limited ranges, especially for network depth; PCQM-Contact and COCO receive smaller tuning budgets for efficiency.Learning rate is not tuned for those larger datasets, and dropout 0 is omitted.Source: §A.1 · p. 7
- Additional setup changes include AdamW with cosine annealing and warmup, zero weight decay, larger batches, GeLU, and GCN skip connections around the nonlinearity.Graph-level prediction-head hidden layers are kept at the GNN hidden dimension; these setup tweaks are not used in the head-depth and normalization ablations.Source: §A.1 · p. 7
A.2 Feature Normalization
On PascalVOC-SP and COCO-SP, node and edge features are normalized channel-wise using each dataset’s training-split statistics before being passed to the model.
A.2 Feature Normalization
- Channel-wise normalization is applied to node and edge features on PascalVOC-SP and COCO-SP.Source: §A.2 · p. 7
- For each dataset, channel-wise means and standard deviations are computed on the training split.Source: §A.2 · p. 7
- Each feature vector is normalized linearly before it is passed to the model.Source: §A.2 · p. 7