PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Graph Benchmark for Machine Learning on Temporal Graphs

Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusseau, Reihaneh Rabbany

arXiv:2307.01026v2cs.LGcs.AI

TL;DR

Temporal-graph research lacks sufficiently large, diverse datasets and realistic evaluation protocols, especially for node-level tasks. TGB addresses this gap with benchmark datasets, task-specific evaluation, and an automated research pipeline. Across datasets, model performance varies substantially, and simple heuristics often outperform temporal-graph models on dynamic node property prediction.

  • Problem

    Temporal-graph benchmarks lack large, diverse datasets and realistic evaluation, while large-scale node-level datasets are scarce because dynamic node labels are difficult to acquire.

  • Method

    TGB provides large, diverse datasets for dynamic link and node property prediction, realistic evaluation protocols, and an automated pipeline for processing, evaluating, and comparing models.

  • Results

    Model rankings vary significantly across datasets, and simple heuristics can outperform state-of-the-art temporal-graph methods on node affinity prediction.

  • Takeaways & Limitations

    Reliable temporal-graph evaluation requires diverse datasets and protocols aligned with real-world applications, while node affinity prediction remains open to improved methods.

  • Takeaways & Limitations

    TGB currently focuses on the streaming setting and includes datasets from only five domains, excluding areas such as biological networks.

Abstract

from arXiv · show

We present the Temporal Graph Benchmark (TGB), a collection of challenging and diverse benchmark datasets for realistic, reproducible, and robust evaluation of machine learning models on temporal graphs. TGB datasets are of large scale, spanning years in duration, incorporate both node and edge-level prediction tasks and cover a diverse set of domains including social, trade, transaction, and transportation networks. For both tasks, we design evaluation protocols based on realistic use-cases. We extensively benchmark each dataset and find that the performance of common models can vary drastically across datasets. In addition, on dynamic node property prediction tasks, we show that simple methods often achieve superior performance compared to existing temporal graph models. We believe that these findings open up opportunities for future research on temporal graphs. Finally, TGB provides an automated machine learning pipeline for reproducible and accessible temporal graph research, including data loading, experiment setup and performance evaluation. TGB will be maintained and updated on a regular basis and welcomes community feedback. TGB datasets, data loaders, example codes, evaluation setup, and leaderboards are publicly available at https://tgb.complexdatalab.com/.

1 Introduction

TGB addresses limitations in temporal-graph research with larger, more diverse datasets and realistic evaluation protocols spanning edge- and node-level tasks. It also provides an automated, reproducible pipeline and reports substantial variation in model performance, including strong results from simple heuristics on node affinity prediction.

  • Pipeline: TGB automates dataset downloading, processing, evaluation, and comparison through compatible data formats and public leaderboards.The pipeline supports numpy, PyTorch, and PyG TemporalData formats and is publicly available as a Python library.
  • Motivation: Existing temporal-graph benchmarks are smaller, less diverse, and often disconnected from realistic evaluation settings.Prior datasets focus substantially on social and interaction networks, while real-world temporal networks span additional domains and evolve over time.
  • Contributions: TGB contributes seven novel datasets spanning diverse domains, with orders-of-magnitude larger numbers of nodes, edges, and timestamps than existing benchmarks.The tgbn-token dataset has around 73 million edges, while tgbl-comment has more than 30 million timestamps.
  • Evaluation: TGB covers dynamic link property prediction and dynamic node property prediction with evaluation designed around real-world applications.Dynamic link prediction uses multiple historical and random negative samples with MRR, while node affinity prediction uses NDCG.
  • Empirical findings: Model rankings can vary significantly across datasets, while simple heuristics can outperform state-of-the-art temporal-graph methods on node affinity prediction.For example, the best model on tgbl-wiki experiences a 40% test MRR drop on tgbl-review.

2 Related Work

Related work has expanded temporal-graph datasets and reproducible model libraries, but existing evaluations can still produce over-optimistic and similar performance. TGB differentiates models through broader datasets and evaluation protocols.

  • Temporal Graph Datasets and Libraries: Prior work collected novel continuous-time link-prediction datasets and introduced more difficult negative samples for evaluation.TGB extends this direction with seven datasets spanning both edge- and node-level tasks.
  • Temporal Graph Datasets and Libraries: DyGLib provides reproducible training and evaluation of temporal-graph models on common benchmark datasets.The related work motivates comparison across datasets as an important evaluation concern.
  • Temporal Graph Methods: Existing temporal-graph evaluations often yield over-optimistic and similar dynamic-link-prediction performance, limiting distinctions among methods.TGB’s datasets and evaluation show clearer distinctions between state-of-the-art model performance.

3 Task Evaluation on Temporal Graphs

TGB evaluates temporal-graph link and node prediction under chronological, application-oriented settings. Link prediction uses realistic negative sampling and ranking metrics, while node affinity prediction forecasts changing user preferences over future windows.

  • Temporal graph setting: Continuous-time temporal graphs are represented as timestamped source-destination edge streams with chronological training, validation, and test splits.
  • Temporal graph setting: In the streaming setting, models can access previously observed test edges during inference but cannot update weights through back-propagation.
  • Dynamic link property prediction: TGB treats dynamic link prediction as forecasting future link properties after chronologically splitting edges into training, validation, and test sets.
  • Dynamic link property prediction: Multiple historical and random negative edges are sampled per positive edge, with fixed negatives provided for reproducible model comparisons.
  • Dynamic link property prediction: Filtered MRR evaluates dynamic link prediction because it captures the ranking of positive edges among negatives more directly than AUROC or AP.
  • Dynamic node property prediction: Node affinity prediction forecasts a user's interaction-frequency vector over candidate items during a fixed future window, using NDCG@10 to evaluate top-ranked targets.

4 Datasets

TGB provides nine chronological temporal-graph datasets spanning diverse domains, scales, network types, and time durations. The collection includes both edge-level prediction datasets and node-affinity datasets for forecasting evolving interactions or preferences.

  • Dataset overview: TGB offers nine datasets split chronologically into training, validation, and test sets containing 70%, 15%, and 15% of all edges.
  • Dataset overview: The datasets span social, interaction, rating, traffic, and trade domains, with durations ranging from months to years and scales from thousands to millions of nodes and edges.
  • Dataset characteristics: The surprise index measures the ratio of test edges unseen during training, and its variation across datasets contributes to benchmark diversity.
  • Edge-level datasets: The collection includes Wikipedia co-editing, Amazon reviews, stablecoin transactions, Reddit replies, and international flight networks for dynamic link prediction.
  • Node-level datasets: Node-affinity datasets cover international trade, music-genre listening, Reddit posting, and cryptocurrency-token transactions, each forecasting future interaction proportions or frequencies.

5 Experiments

TGB benchmarks temporal-graph methods across link and node prediction datasets using its standardized evaluation protocols. Performance varies across datasets, simple heuristics are strong on node affinity prediction, and neural methods can require substantially more computation than EdgeBank.

  • Experimental setup: The benchmark compares eight temporal-graph methods and two EdgeBank heuristics for link prediction, plus temporal models and persistence or moving-average heuristics for node prediction.
  • Dynamic link property prediction: With TGB evaluation, model performance is clearly distinguished on tgbl-wiki, where NAT achieves the best result after comparing all possible destinations.
  • Dynamic link property prediction: Higher surprise on tgbl-review is associated with substantial performance reductions for some methods because more test edges are unseen during training.
  • Computational cost: EdgeBank inference, training, and validation are generally at least one order of magnitude faster than neural-network methods, while temporal methods also differ in inference time.
  • Dynamic node property prediction: Persistence forecast is state of the art on tgbn-trade, while moving average performs best on the other node-affinity datasets.
  • Dynamic node property prediction: Simple heuristics are strong contenders against DyRep and TGN, highlighting the need for temporal models with flexible representations of evolving user preferences.

6 Conclusion

TGB provides large, diverse temporal-graph datasets and realistic evaluation for dynamic link and node property prediction. Results show dataset-dependent model rankings and that simple node-prediction heuristics can outperform state-of-the-art temporal models, while the benchmark’s scope remains limited.

  • TGB offers orders-of-magnitude-larger, diverse datasets for dynamic link and node property prediction, with an automated evaluation pipeline and public leaderboards.The benchmark targets realistic, reproducible, and robust temporal-graph evaluation.
  • Model rankings for dynamic link property prediction vary significantly across datasets, supporting evaluation across TGB’s diverse dataset range.
  • Simple persistence forecast and moving average heuristics outperform state-of-the-art methods such as TGN on dynamic node property prediction.This result motivates further development of temporal-graph methods for node-level tasks.
  • Public, well-curated datasets such as TGB are expected to facilitate methodological progress in temporal graph learning.
  • TGB may focus future benchmarking on its own datasets and tasks, potentially limiting use of other temporal-graph benchmarks.The authors plan regular updates with community feedback and additional datasets and tasks.
  • TGB currently covers five domains and uses the streaming evaluation setting, although other applications may require different settings or domains.The authors plan to increase dataset diversity over time.

Checklist

The checklist records reproducibility, licensing, documentation, and ethics disclosures for TGB. It points readers to code, data, training details, compute information, and public documentation.

  • The authors report that code, data, and reproduction instructions are publicly available.
  • Chronological data splits and training details are documented, including dataset splits and hyperparameter information.
  • Error bars across multiple trials and compute-resource details are reported in the paper or appendices.
  • Dataset creators are cited, licenses are described, and processed-dataset download links are provided.
  • The datasets are stated to come from publicly available sources where consent and licensing were provided.
  • The checklist states that the provided datasets contain no personally identifiable information or offensive content.
  • TGB documentation, Python package, tutorials, and API references are available through the project website and PyPI.
  • The benchmark is maintained through community feedback, repository updates, and institutional dataset maintenance.

B Dataset Licenses and Download Links

TGB documents licenses and access routes for its datasets, combining MIT, CC BY-NC, Amazon, and restricted non-commercial research terms. Dataset maintenance is assigned to the Digital Research Alliance of Canada.

  • TGB datasets are maintained by the Digital Research Alliance of Canada, and the authors accept responsibility for rights violations.
  • tgbl-wiki, tgbn-trade, and tgbn-genre use MIT licenses.
  • The tgbl-review dataset is governed by Amazon terms permitting academic research while restricting resale, republication, and commercial use.
  • tgbl-flight is available under a limited, non-exclusive, non-transferable license restricted to specified non-commercial research, education, testing, evaluation, or government purposes.
  • tgbn-reddit, tgbn-token, tgbl-coin, and tgbl-comment use CC BY-NC licenses.

C Evaluation Settings in TG

TGB focuses on continuous-time temporal graphs and evaluates models under a streaming setting, while distinguishing alternative test-information regimes. The benchmark includes neural models and simple memory- or history-based baselines for link and node prediction.

  • Evaluation Settings: In the streaming setting, test-set information updates a model’s memory but cannot trigger back-propagation or model updates.
  • Evaluation Settings: The deployed setting forbids using test information to modify any part of the model after deployment.
  • Evaluation Settings: The live-update setting permits retraining, fine-tuning, or updating model weights using all historical information, including test data.
  • Temporal Graph Learning Models: Continuous-time temporal graphs represent timestamped edge streams, whereas discrete-time temporal graphs use ordered static snapshots.
  • Temporal Graph Learning Models: TGB primarily focuses on continuous-time graphs because their precise temporal information can support discrete-time inference, while the reverse adaptation is non-trivial.
  • Models: Dynamic link prediction comparisons include temporal neural models such as DyRep, TGN, GraphMixer, TCL, CAWN, TGAT, and NAT.
  • Models: EdgeBank baselines predict observed node pairs as true, with EdgeBanktw adding a fixed recent-history window and recency bias.
  • Models: Node property prediction uses TGN and DyRep embeddings alongside persistence forecast and a seven-step moving average.

E Computing Resources

The experiments used substantial GPU and CPU resources across dynamic link and node property prediction tasks. GPU usage varied across temporal graph methods, with GraphMixer requiring significantly more than most others on tgbl-wiki.

  • Computing Resources: Dynamic node property experiments used four standard CPU configurations with RTX8000, V100, A100, or A6000 GPUs.The longest experiments took around two days on tgbl-comment and tgbn-token.
  • Computing Resources: GraphMixer used significantly more GPU memory than other methods on tgbl-wiki, while most methods had similar GPU usage.The comparison reports average GPU usage across five trials; EdgeBank used no GPU because it is CPU-only.

G Additional Dataset Statistics

TGB characterizes datasets using recurrence, temporal edge density, unseen-node ratios, and domain-scale diversity. These properties expose both memorization opportunities and challenges for predicting edges involving new nodes.

  • Additional Dataset Statistics: The reoccurrence index measures the ratio of training edges that reappear during testing and can correlate with EdgeBank performance when appearances are consistent.EdgeBank is a memorization-based approach.
  • Additional Dataset Statistics: New nodes appear during validation and testing, making edge-property prediction harder because models lack historical information for them.The evaluation mainly focuses on transductive dynamic link property prediction despite these unseen nodes.
  • Additional Dataset Statistics: TGB datasets vary in average edges per timestamp and across the additional properties reported in Table 5.These statistics describe how datasets evolve and how much unseen-node exposure occurs during inference.

H Number of Negative Samples

Additional experiments examine how negative-sample count, transductive versus inductive evaluation, and test-time cost affect temporal graph model comparisons. More negatives generally reduce reported MRR, while runtime rankings depend on the dataset.

  • Number of Negative Samples: More negative samples generally lower MRR, while computationally feasible evaluations should use more negatives; TGB uses more for small datasets.With fewer negatives, MRR scores were significantly higher for most methods, especially on tgbl-wiki.
  • Number of Negative Samples: CAWN’s performance drops significantly from tgbl-wiki to tgbl-review, and EdgeBank drops when the surprise index is high on tgbl-review.These findings reaffirm the main-section results under a 20-negative-sample setting.
  • Transductive and Inductive Comparison: Strong performance on tgbl-wiki, such as NAT’s, requires success in both transductive and inductive reasoning.Transductive MRR uses edges with transductive source nodes, whereas inductive MRR uses edges with inductive source nodes.
  • Additional Computational Time Comparison: EdgeBank is one order of magnitude faster than TGN and DyRep on tgbl-coin and tgbl-comment, whereas DyRep is fastest on tgbl-flight.The flight result is attributed to that dataset’s large number of edges.
Loading 2307.01026v2…