Source-linked AI summary
Temporal Graph Networks for Deep Learning on Dynamic Graphs
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein
TL;DR
Deep-learning methods often assume static graphs, while many interaction systems evolve continuously and require models that preserve temporal structure. The paper introduces Temporal Graph Networks, combining memory and graph-based operators for event-sequence dynamics, and reports state-of-the-art results across tasks while improving efficiency.
Problem
Most graph-learning methods assume static graphs, while dynamic-graph methods have often focused on snapshots despite continuous-time interaction systems.
Method
TGNs are an inductive encoder framework for continuous-time dynamic graphs represented as time-stamped events, using memory and graph-based embedding operations.
Results
TGN outperforms baselines on future edge prediction and dynamic node classification, while running up to 30× faster than TGAT per epoch.
Takeaways & Limitations
Ablations identify memory for long-term node information and graph-based embeddings for up-to-date node representations as important TGN components.
Takeaways & Limitations
The raw message store contains no message for a node that has never previously participated in an event.
Abstract
from arXiv · showhide
Graph Neural Networks (GNNs) have recently become increasingly popular due to their ability to learn complex systems of relations or interactions arising in a broad spectrum of problems ranging from biology and particle physics to social networks and recommendation systems. Despite the plethora of different models for deep learning on graphs, few approaches have been proposed thus far for dealing with graphs that present some sort of dynamic nature (e.g. evolving features or connectivity over time). In this paper, we present Temporal Graph Networks (TGNs), a generic, efficient framework for deep learning on dynamic graphs represented as sequences of timed events. Thanks to a novel combination of memory modules and graph-based operators, TGNs are able to significantly outperform previous approaches being at the same time more computationally efficient. We furthermore show that several previous models for learning on dynamic graphs can be cast as specific instances of our framework. We perform a detailed ablation study of different components of our framework and devise the best configuration that achieves state-of-the-art performance on several transductive and inductive prediction tasks for dynamic graphs.
1 INTRODUCTION
Graph learning has largely focused on static graphs, despite many real-world interaction systems evolving continuously. TGNs address this gap with an inductive continuous-time framework, efficient training, ablations, and state-of-the-art results.
- Most graph deep-learning methods assume static graphs, although many interaction systems are dynamic and their temporal structure can contain crucial insights.
- Discrete-time approaches represent dynamic graphs as snapshots and are unsuitable for continuous settings where edges can appear at any time.
- TGNs provide a generic inductive framework for continuous-time dynamic graphs represented as sequences of events.
- The paper introduces efficient training, analyzes speed–accuracy tradeoffs through ablations, and reports state-of-the-art performance across transductive and inductive tasks.
2 BACKGROUND
Dynamic graphs can be represented either as discrete-time snapshots or as continuous-time event sequences. The paper models continuous-time temporal multigraphs through time-stamped node and interaction events whose accumulated history defines the graph state.
- Discrete-time dynamic graphs are sequences of static snapshots, whereas continuous-time dynamic graphs are timed event lists supporting structural and feature changes.
- A temporal multigraph is an ordered sequence of time-stamped events representing node changes or interactions between node pairs.
- Node-wise events create unseen nodes or update features, while interaction events represent directed temporal edges and may produce multiple edges between a pair.
- At time t, the graph snapshot contains all vertices and edges accumulated from events occurring up to t.
3 TEMPORAL GRAPH NETWORKS
Temporal Graph Networks encode continuous-time dynamic graphs with node memories, message processing, and temporal graph-based embeddings. Their training strategy updates memory from prior interactions before prediction, while delaying current-batch updates to preserve gradient flow and avoid information leakage.
- Framework: TGNs use a temporal encoder to produce node embeddings from a dynamic graph and each node’s memory.The encoder operates on continuous-time graphs represented as sequences of time-stamped events.
- Memory: Node memory stores each observed node’s history in compressed form and is updated after node-wise or interaction events.Interaction events update both participating nodes, whereas node-wise events update only the related node.
- Memory: Messages are computed for events, aggregated when multiple same-node events occur in a batch, and passed to a learnable memory updater.Experiments use identity message functions and efficient most-recent-message or mean-message aggregation.
- Embedding: The embedding module combines memory with temporal neighborhood information to reduce memory staleness for inactive nodes.Temporal graph attention can select neighbors using feature and timing information, while temporal graph sum provides a simpler aggregation.
- Embedding: TGNs support identity, time-projection, attention, and sum embedding modules, with attention aggregating information across a node’s temporal neighborhood.Time projection uses the elapsed time since the node’s last interaction, while attention uses multi-head attention over temporal neighbors.
- Training: Training updates memory from stored raw messages of previous batches before computing embeddings and loss, then stores current-batch raw messages for future updates.This ordering lets memory-related modules influence the loss without using the interaction being predicted, although a batch shares one memory state and batch size 200 balances speed with update granularity.
4 RELATED WORK
Related work spans discrete- and continuous-time dynamic graph models, while TGN provides a framework that subsumes many recent continuous-time approaches and variants.
- Discrete-time methods aggregate snapshots, factorize snapshot tensors, or encode snapshots into time-varying embeddings.
- Continuous-time approaches use random walks, RNN-based updates, and dynamic knowledge-graph methods to model timed interactions.
- Many recent continuous-time models are specific cases of TGN, including Jodie, TGAT, DyRep, and static Graph Networks.The framework differs by selecting modules such as time projection, memory, graph attention, and message computation.
- Table 1 organizes prior continuous-time models and TGN variants by graph-convolution depth, neighbor sampling, batching, and message aggregation.
5 EXPERIMENTS
Experiments evaluate TGN on future edge prediction and dynamic node classification across Wikipedia, Reddit, and Twitter, including performance, speed, and module ablations.
- 5 EXPERIMENTS: The experiments use Wikipedia, Reddit, and Twitter for future edge prediction and dynamic node classification.Future edge prediction estimates whether an edge occurs between two nodes at a given time.
- 5.1 PERFORMANCE: TGN outperforms baselines on future edge prediction in both transductive and inductive settings across all datasets.On Twitter, it exceeds DyRep by over 4% transductively and 10% inductively.
- 5.1 PERFORMANCE: Up to 30× faster than TGAT per epoch, TGN uses efficient parallel processing and only one graph attention layer while requiring similar convergence epochs.
- 5.2 CHOICE OF MODULES: Memory raises precision by nearly 4% over TGN-no-mem but makes TGN-attn about 3× slower; memory models consistently outperform memory-free variants.Combining memory with most-recent-neighbor sampling also reduces the neighbors needed for best performance.
- 5.2 CHOICE OF MODULES: Graph-based embedding modules outperform graph-less TGN-id, with TGN-attn performing best while only slightly slower than TGN-sum.The comparison indicates that recent neighbor information and neighbor selection are important for performance.
- 5.2 CHOICE OF MODULES: TGN-mean performs slightly better than the last-message aggregator but is more than 3× slower.
- 5.2 CHOICE OF MODULES: TGN achieves very high performance with one graph-attention layer because memory access to one-hop neighbors indirectly reaches information from farther hops.Using one layer also speeds up the model dramatically.
6 CONCLUSION
TGN is a generic framework for learning on continuous-time dynamic graphs, achieving state-of-the-art results across several tasks and datasets while being faster than previous methods. Ablations highlight the importance of memory modules and graph-based embedding for long-term information and current node representations.
- TGN provides a generic framework for learning on continuous-time dynamic graphs.
- TGN achieves state-of-the-art results on several tasks and datasets while being faster than previous methods.
- Ablations show that memory modules store long-term information, while graph-based embedding generates up-to-date node embeddings.
A.1 DELETION EVENTS
TGN supports edge and node deletion events in continuous-time dynamic graphs. Deletions update the graph structure and can optionally propagate information through messages to neighboring nodes.
- TGN supports edge and node deletion events.
- For an edge deletion, two messages can be computed for the source and target nodes involved in the interaction.
- A node deletion removes the node and its incoming and outgoing edges from the temporal graph.The deleted node is then excluded when computing other nodes’ embeddings with temporal graph attention.
A.2 TGN TRAINING
TGN training processes interactions chronologically in batches while preserving temporal dependencies between nodes. A message store retains recent raw messages so memory-related modules can be updated before subsequent predictions.
- Graph dependencies prevent processing nodes independently because a node update depends on other nodes’ current memories.
- TGN processes interactions in chronological batches and stores each node’s last message for processing before its next prediction.
- The training procedure computes node embeddings before predicting positive and negative interactions.
- Raw messages for source and destination nodes are computed and stored for later processing.
- Updated source and destination memories are stored during training.
- The same message function is used for sources and destinations, and emb̂s denotes an embedding layer operating on updated memory.
A.3 DATASETS
The experiments use Wikipedia, Reddit, and Twitter dynamic interaction datasets with different graph structures and text-based interaction features. The datasets lack node features and contain only edge-creation events, limiting event diversity in evaluation.
- Wikipedia and Reddit are bipartite interaction graphs, whereas Twitter is a non-bipartite graph of users connected by retweets.
- Reddit interactions are user posts to sub-reddits, while Wikipedia interactions are user edits to pages.
- Reddit and Wikipedia interactions use text features, and their labels indicate whether a user is banned.
- Twitter interaction features are BERT-based vector representations of retweet text.
- Node features are absent in all three datasets, so every node receives the same zero feature vector.
- All three datasets contain only edge-creation interaction events, leaving broader event-type evaluation for future work.
A.4 ADDITIONAL EXPERIMENTAL SETTINGS AND RESULTS
The experimental settings specify common optimization and evaluation choices, while additional results compare implementations and neighborhood-sampling strategies. Most-recent neighbor sampling outperforms uniform sampling in the reported TGN-attn comparison.
- Baseline results: Baseline comparisons include static graph, temporal graph, and dynamic graph models, with several results taken from TGAT and others reimplemented in PyTorch.The reimplemented models include Jodie and DyRep.
- Baseline results: The DyRep implementation augments messages with destination-neighborhood temporal attention, while both reimplemented models use vanilla RNN memory updates.The temporal attention is applied to the destination’s neighborhood.
- Neighbor sampling: When sampling 10 neighbors, TGN-attn models using the most recent edges clearly outperform models using uniform sampling.Figure 5 reports means and standard deviations computed over 10 runs.
- Neighbor sampling: Figure 5 visualizes the two sampling strategies with ellipses representing means and standard deviations across 10 runs.The comparison concerns two TGN-attn models evaluated with 10 sampled neighbors.