PaperlayerView exact revision on arXiv

Source-linked AI summary

Can You Hear Me Now? A Benchmark for Long-Range Graph Propagation

Luca Miglior, Matteo Tolloso, Alessio Gravina, Davide Bacciu

arXiv:2512.17762v2cs.LG

TL;DR

Long-range graph propagation remains a challenge for GNNs, and existing benchmarks do not fully test dependencies required in real-world applications. ECHO combines synthetic graph tasks and DFT-based molecular prediction tasks; its experiments reveal performance gaps between global-attention and traditional message-passing models.

  • Problem

    Effectively propagating information over long graph distances remains challenging, while existing benchmarks do not fully capture long-range dependencies in real-world applications.

    Source: §1 · p. 1

  • Method

    ECHO benchmarks long-range propagation with three synthetic graph-property tasks and two DFT-based molecular tasks for atomic charges and total energies.

    Source: §1 · p. 2

  • Results

    Global-attention models significantly outperform traditional message-passing frameworks on synthetic tasks; GRIT achieves an sssp MAE of 0.121, while GCN has consistent limitations across the benchmarks.

    Source: §4 · pp. 8–9

  • Takeaways & Limitations

    ECHO provides a benchmark for long-range graph propagation across synthetic and chemically grounded real-world settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Effectively capturing long-range interactions remains a fundamental yet unresolved challenge in graph neural network (GNN) research, critical for applications across diverse fields of science. To systematically address this, we introduce ECHO (Evaluating Communication over long HOps), a novel benchmark specifically designed to rigorously assess the capabilities of GNNs in handling very long-range graph propagation. ECHO includes three synthetic graph tasks, namely single-source shortest paths, node eccentricity, and graph diameter, each constructed over diverse and structurally challenging topologies intentionally designed to introduce significant information bottlenecks. ECHO also includes two real-world datasets, ECHO-Charge and ECHO-Energy, which define chemically grounded benchmarks for predicting atomic partial charges and molecular total energies, respectively, with reference computations obtained at the density functional theory (DFT) level. Both tasks inherently depend on capturing complex long-range molecular interactions. Our extensive benchmarking of popular GNN architectures reveals clear performance gaps, emphasizing the difficulty of true long-range propagation and highlighting design choices capable of overcoming inherent limitations. ECHO thereby sets a new standard for evaluating long-range information propagation, also providing a compelling example for its need in AI for science.

1 INTRODUCTION

Long-range propagation remains difficult for GNNs, while existing benchmarks often emphasize shorter-range tasks. ECHO addresses the need for controlled and application-driven evaluation of long-range capabilities.

1 INTRODUCTION

  • Existing benchmarks mostly target short- to medium-range tasks, while LRGB and GPP do not fully test true long-range dependencies.The paper attributes this gap to limited graph sizes, undefined propagation ranges, and emphasis on specific issues such as over-smoothing and over-squashing.Source: §1 · p. 2
  • ECHO combines three synthetic graph-property tasks with two chemically grounded molecular tasks to evaluate long-range communication.The synthetic tasks are single-source shortest paths, node eccentricity, and graph diameter; the chemistry tasks predict atomic charges and molecular energies using DFT computations.Source: §1 · p. 2
  • ECHO's synthetic topologies introduce information bottlenecks, requiring models to use global information and traverse entire graphs.The benchmark also includes real-world molecular tasks involving long-range charge redistribution and total energy, relevant to chemical reactivity, stability, and intermolecular interactions.Source: §1 · p. 2
  • The benchmark comprises 10,080 synthetic graphs and approximately 170,000–196,000 chemistry graphs, with required propagation spanning 17–40 hops.The chemistry tasks use DFT references; generating the benchmark required approximately two months of parallel DFT computation on the reported hardware.Source: §1 · p. 2
  • The authors analyze whether ECHO tasks capture long-range dependencies and establish strong baselines across the tasks.They present ECHO-Chem as a benchmark for long-range atomic interactions in chemistry.Source: §1 · p. 2

2 ON THE NEED FOR A NEW BENCHMARK

The paper argues that existing benchmarks incompletely test long-range propagation, particularly on larger graphs and application-driven tasks. ECHO specifies propagation needs across diverse topologies and adds atomic-scale molecular benchmarks.

2 ON THE NEED FOR A NEW BENCHMARK

  • Existing benchmarks often use small graphs with limited diameters, narrow task designs, or tasks that may be local rather than global.The paper also notes performance saturation on LRGB and sensitivity to hyperparameter tuning.Source: §2 · p. 3
  • The proposed benchmark should test traversal of extensive graphs, global-information aggregation, and adaptation to diverse topological constraints.The authors argue that evaluation is timely across established model families, including graph transformers and multi-hop GNNs.Source: §2 · p. 3
  • ECHO tasks specify propagation ranges of 17–40 hops, making models that fail to capture dependencies within that span underreaching by the benchmark's definition.Tasks require shortest paths, long-range charge redistribution, or molecular total energies, depending on graph structure.Source: §2 · p. 3
  • ECHO-Charge and ECHO-Energy target long-range atomic interactions, unlike prior molecular benchmarks focused mainly on short-range properties reducible to small local substructures.The tasks require modeling charge distributions, energy stabilization, and complex atomic interactions.Source: §2 · p. 3
  • The authors present ECHO-Chem as a standard for evaluating long-range propagation and an application of AI for science and chemistry.They note potential relevance to drug and materials design and understanding biological functions.Source: §2 · p. 3
  • ECHO aims to balance long-range complexity and usability, complementing a contemporaneous single-task synthetic benchmark on graphs up to 569k nodes.The authors describe ECHO as a five-task benchmark intended to avoid digital-divide concerns while representing scientific challenges.Source: §2 · pp. 3–4

3 THE ECHO BENCHMARK

ECHO pairs algorithmic graph-property tasks with chemically grounded molecular benchmarks to evaluate long-range propagation. Its synthetic tasks use diverse bottlenecked topologies, while its chemistry tasks target atomic charges and molecular energies shaped by non-local interactions.

3 THE ECHO BENCHMARK

  • The benchmark combines controlled synthetic graph tasks with chemically grounded molecular tasks focused on long-range information propagation.The synthetic tasks cover shortest paths, eccentricity, and diameter; the chemistry tasks cover atomic partial charges and molecular total energies.Source: §3 · p. 4

3.1 THE ECHO-SYNTH DATASET

  • ECHO-Synth tests single-source shortest paths and node eccentricity per node, and graph diameter per graph, across tasks requiring global graph information.The tasks are designed to challenge localized message passing; random node features and a source indicator for shortest paths distinguish input roles.Source: §3.1 · p. 4
  • Six synthetic topology families introduce structural bottlenecks, with chains, trees, caterpillars, lobsters, and edge-deleted grids constraining communication in distinct ways.The families are line, ladder, grid, tree, caterpillar, and lobster; bottlenecks can force information through narrow paths or intermediary nodes.Source: §3.1 · pp. 5–6
  • Synthetic graphs span target diameters 17–40, with 10,080 graphs split evenly by topology and diameter across training, validation, and test sets.For each topology and diameter, 40 graphs are assigned to training and 15 each to validation and test.Source: §3.1 · p. 6

3.2 THE ECHO-CHEM DATASETS

  • ECHO-Chem frames partial-charge prediction as atom-level regression and total-energy prediction as graph-level regression, with energy depending on multiple long-range quantum interactions.The benchmark's energies are not computed as sums of per-atom energies; the stated contributions include electron–nuclear, electron–electron, and nuclear–nuclear interactions.Source: §3.2 · pp. 6–7
  • ECHO-Chem uses ChEMBL molecules with graph diameters of 17–40 and DFT-derived structures, charges, and energies; its charge and energy datasets use different split ratios.ECHO-Charge has approximately 170,000 molecular graphs and ECHO-Energy approximately 196,000; their train/validation/test splits are 80/10/10 and 90/5/5, respectively.Source: §3.2 · p. 7

4 EXPERIMENTS

Experiments compare diverse GNN propagation mechanisms on synthetic graph problems and chemically grounded tasks. Results favor architectures that capture long-range information, while also showing an accuracy–efficiency trade-off.

Experimental setup

  • Evaluated models span classical, multi-hop, transformer, and differential-equation-inspired GNNs, with a shared embedding, GNN stack, and task-specific readout.Node-level readouts use a two-layer MLP; graph-level readouts concatenate mean, max, and sum pooling before a two-layer MLP.Source: §4 · pp. 7–8
  • Training uses log10(MSE), Adam, validation-loss early stopping, 100 Bayesian-optimization trials per model–dataset pair, and four random-seed runs.Early stopping has a patience of 50 epochs, with a maximum of 1,000 training epochs.Source: §4 · p. 8

Synthetic benchmark results

  • GRIT records the lowest MAE on sssp (0.121) and diam (1.014), while DRew leads on ecc (4.651); GCN has the highest diam MAE (3.832).GraphCON does not consistently outperform traditional methods, and performs more weakly than A-DGN and SWAN within its architectural family.Source: Table 1 · p. 8, §4 · pp. 8–9

Chemical benchmark results

  • GPS performs best on ECHO-Energy and competitively on ECHO-Charge, while SWAN is best on ECHO-Charge; GCN and GIN lag on both chemical tasks.A-DGN and SWAN are also consistently among the top performers; DRew surpasses classical GNNs, especially on ECHO-Energy, but trails DE-GNN and transformer-based models.Source: Table 2 · p. 9, §4 · p. 9
  • GPS shows lower ECHO-Charge prediction errors than GCN, especially at peripheral atoms, while GCN accumulates errors at distant or chemically sensitive sites.Figure 3 colors log absolute error from green for lower error to orange for higher error.Source: §4 · p. 9, Figure 3 · p. 10

Additional analyses

  • Deeper networks outperform shallower ones, model rankings remain consistent across synthetic graph topologies, and readout depth appears not to affect final performance.Runtime results show GPS is computationally demanding, while A-DGN offers a more balanced accuracy–efficiency alternative; small errors of 10^-4 e to 10^-6 e may still affect molecular modeling and reproducibility.Source: §4 · pp. 9–10

5 CONCLUSION

ECHO benchmarks long-range information propagation in GNNs through synthetic and chemically grounded tasks. The authors report that current architectures have limitations on these tasks and provide baselines for future comparisons.

Conclusion

  • ECHO combines synthetic graph-property tasks with chemical tasks predicting atomic charge distributions and molecular total energies that depend on long-range quantum interactions.The paper reports analyzing the tasks’ long-range dependencies and establishing baselines for each task.Source: §5 · p. 10
  • The reported results highlight limitations of current GNN architectures on long-range propagation challenges.The authors propose ECHO as a reference point for future research on these challenges.Source: §5 · p. 10
  • The authors present ECHO as a challenge for developing GNNs with more effective information propagation across graph-learning tasks.They describe future architectures as aiming for greater robustness, scalability, and generalizability.Source: §5 · p. 10

ETHICS STATEMENT

The study reports no human subjects, sensitive personal data, or anticipated ethical concerns.

Ethics statement

  • The study does not involve human subjects.The authors report no ethical concerns anticipated from data usage, methodology, or findings.Source: ETHICS STATEMENT · p. 10
  • The study does not use sensitive personal data.The authors report no ethical concerns anticipated from data usage, methodology, or findings.Source: ETHICS STATEMENT · p. 10
  • The authors anticipate no harmful consequences or ethical concerns from the study’s data, methods, or findings.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The section motivates focused benchmarks for long-range propagation and supports reproducibility through experiment details and openly released data and code.

REPRODUCIBILITY STATEMENT

  • Benchmark details appear in Section 3 and Appendix C, experiment setups in Section 4 and Appendix F, and data and code are released for replication and new evaluations.The release is at https://github.com/Graph-ECHO-Benchmark/ECHO.Source: REPRODUCIBILITY STATEMENT · pp. 10–11

B DISCUSSION ON LRGB

  • On Peptides-func, 30 models improved 6.5% in year one, 1.3% in year two, and showed no significant gain in year three.The passage says a similar trend exists for the other LRGB tasks.Source: B DISCUSSION ON LRGB · p. 18
  • Analysis found that only a subset of LRGB tasks genuinely require longer interactions, while the peptides tasks are effectively local.The passage also raises concerns about LRGB’s sensitivity to hyperparameter tuning.Source: B DISCUSSION ON LRGB · p. 18
  • The authors call for more focused benchmarks that explicitly and systematically test GNN long-range propagation capabilities.This need follows from concerns about LRGB’s task requirements and hyperparameter sensitivity.Source: B DISCUSSION ON LRGB · p. 18

C CHEMICAL SIMULATION TECHNICAL INFORMATION

The chemical simulation pipeline generates molecular structures, computes quantum-chemical targets, and extracts partial charges. The appendix also describes the hardware used for these computations and model training.

C CHEMICAL SIMULATION TECHNICAL INFORMATION

  • Molecular structures were generated from SMILES with GAFF and optimized in 100 coarse-minimization and 500 local-refinement steps before export as SDF.The average structure-generation time was 562 ± 124 ms per molecule meeting the dataset diameter criteria.Source: C CHEMICAL SIMULATION TECHNICAL… · p. 18
  • ORCA 6.0.1 with B3LYP DFT computed charges and energies, using TightSCF settings to reduce numerical errors.Löwdin population analysis supplied the atomic partial-charge supervision signals.Source: C CHEMICAL SIMULATION TECHNICAL… · pp. 18–19
  • A single quantum-chemical calculation averaged 634.5 ± 21.3 seconds per molecule, requiring approximately two months on the reported hardware.The simulations used ORCA’s full thread parallelism.Source: C CHEMICAL SIMULATION TECHNICAL… · p. 19

D HARDWARE RESOURCES

  • DFT simulations ran on a 288-core, approximately 1 TiB RAM machine, while model training and inference used a separate node with 8 NVIDIA H100 GPUs.The CPU cores supported parallelization over molecular batches and basis-function evaluations.Source: D HARDWARE RESOURCES · p. 19

E ADDITIONAL DATASET INFORMATION

The appendix reports dataset statistics and structural-property distributions for ECHO-Synth, ECHO-Charge, and ECHO-Energy. It also describes the relationship between graph size and diameter across these datasets.

E ADDITIONAL DATASET INFORMATION

F HYPERPARAMETER SELECTION

The appendix documents task-specific hyperparameter searches and reports that tuning consistently selected deeper GNN configurations. It also outlines additional analyses of model performance.

F HYPERPARAMETER SELECTION

  • Tables 6–16 report each architecture’s task-specific hyperparameter search space and validation-selected values.The tasks include diam, ecc, sssp, ECHO-Charge, and ECHO-Energy.Source: F HYPERPARAMETER SELECTION · p. 23, Table 6 · p. 23, Table 7 · p. 23
  • Bayesian optimization consistently selected configurations with many GNN layers, suggesting deeper models were needed to minimize validation error on these tasks.The authors interpret this tuning pattern as further evidence that the benchmark tasks demand long-range propagation.Source: F HYPERPARAMETER SELECTION · p. 23

G ADDITIONAL EXPERIMENTAL ANALYSIS

G.1 LAYER-WISE PERFORMANCE ANALYSIS

Across tasks, deeper propagation and architectures designed for long-range message passing tend to perform better, though patterns vary by task and graph regime. Diameter, readout-depth, and topology analyses further qualify these results.

G.1 LAYER-WISE PERFORMANCE ANALYSIS

  • Across most tasks, deeper networks tend to perform better, and architectures designed for many message-passing steps consistently outperform others.Figure 8 compares test MAE across shallow (<10 layers), medium (10–17), and deep (>17) regimes.Source: §G.1 · p. 26, Figure 8 · p. 27
  • On synthetic tasks, greater depth improves eccentricity and SSSP results across many models, while diameter results vary by model.GraphCON is the exception to the reported SSSP depth-related improvements; DRew, SWAN, and GPS show large gains there.Source: §G.1 · p. 26, Figure 8 · p. 27
  • For molecular tasks, explicit long-range models retain high ECHO-Charge accuracy at depth, while GPS performs best on ECHO-Energy.ECHO-Energy results vary more across models, although PH-DGN and A-DGN generally benefit from depth.Source: §G.1 · p. 26, Figure 8 · p. 27

G.2 PERFORMANCE ACROSS GRAPH DIAMETERS

  • Across synthetic tasks, diameter-wise results vary: eccentricity error is U-shaped, SSSP MAE rises with diameter, and several models degrade on large-diameter graphs.For graph diameter, non-dissipative architectures, DRew, GPS, and GRIT remain accurate across graph diameters; the eccentricity pattern tracks the node-eccentricity distribution.Source: §G.2 · pp. 27–29, Figure 9 · p. 28
  • On molecular tasks, ECHO-Charge MAE is consistent across graph diameters, whereas ECHO-Energy MAE rises, with SWAN and DRew less sensitive to diameter.All models show a general ECHO-Energy performance drop above diameter 35, which the authors attribute to fewer such graphs in the source dataset.Source: Figure 9 · p. 28, §G.2 · p. 29

G.3 IMPACT OF READOUT LAYER DEPTH

  • Increasing GCN readout depth to three layers did not improve ECHO-Synth performance, so the authors retained a two-layer readout for efficiency.The comparison is reported in Table 17.Source: Table 17 · p. 29, §G.3 · p. 29

G.4 PERFORMANCE ACROSS DIFFERENT GRAPH TOPOLOGIES

  • Model rankings remain consistent across synthetic graph topologies, while line graphs are consistently the most challenging because sequential message paths create bottlenecks.Absolute performance varies slightly by topology; GPS, for example, performs better on lobster than tree-like topologies in SSSP.Source: §G.4 · p. 29, Figure 10 · p. 30, Figure 11 · p. 30

H EXTENDED RESULTS

The extended-results tables report test performance across ECHO-Synth, ECHO-Charge, and ECHO-Energy using MAE, MSE, and loss, with lower values better. The loss is defined as log10(MSE).

H EXTENDED RESULTS

  • Table 18 reports mean ± std test performance for models across ECHO-Synth tasks, with lower values better.The reported metrics are MAE, MSE, and loss, defined as log10(MSE).Source: Table 18 · p. 31, H EXTENDED RESULTS · p. 31
  • Table 19 reports mean ± std test performance for models on ECHO-Charge, with lower values better.Test loss is computed on the normalized dataset, while test MSE and MAE are reported on the original, non-scaled data.Source: H EXTENDED RESULTS · p. 31, Table 19 · p. 32
  • Table 20 reports mean ± std test performance for models on ECHO-Energy, with lower values better.Test loss is computed on the normalized dataset, while test MSE and MAE are reported on the original, non-scaled data.Source: H EXTENDED RESULTS · p. 31, Table 20 · p. 32

I RUNTIMES

GPS performs strongly on long-range tasks, especially ECHO-Charge and ECHO-Energy, but with higher computational overhead than SWAN and A-DGN. Its attention patterns show distant-node communication routed through a few central hubs.

I RUNTIMES

  • DRew’s reported runtime excludes Floyd–Warshall preprocessing, which has cubic time complexity in the number of nodes.The runtime protocols differ in their stated batch size: 512 in the prose and 256 in the Table 21 caption.Source: I RUNTIMES · p. 33, Table 21 · p. 34
  • GPS achieves strong performance on long-range tasks, particularly ECHO-Charge and ECHO-Energy, but has significantly higher computational overhead than SWAN and A-DGN.SWAN and A-DGN strike a more favorable balance between efficiency and accuracy.Source: I RUNTIMES · p. 33

J VISUALIZATION OF GPS ATTENTION PATTERNS

  • GPS’s first-layer attention often links distant, nonadjacent nodes through a few central hubs, reducing maximal traversal distance to a few hops.The visualization highlights the top 40 attention-weighted node pairs alongside the original graph topology.Source: J VISUALIZATION OF GPS ATTENTION… · p. 34, Figure 13 · p. 35
Loading 2512.17762v2…