Source-linked AI summary
Temporal Knowledge Graph Forecasting with Neural ODE
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, Volker Tresp
TL;DR
Future-link forecasting on temporal knowledge graphs is difficult because existing methods commonly represent continuously changing graphs in discrete time. The paper proposes TANGO, which combines neural ODEs, multi-relational graph convolutions, and a graph transition layer, and reports state-of-the-art performance on five benchmark datasets.
Problem
Existing temporal KG methods often use discrete-time snapshots, while continuous-time methods may omit graph structure or fail on unseen nodes.
Method
TANGO couples neural ODEs with multi-relational graph convolutions and a graph transition layer to model continuous dynamics, structure, edge formation, and edge dissolution.
Results
TANGO achieves state-of-the-art performance on five benchmark temporal KG datasets for future-link forecasting.
Takeaways & Limitations
The paper extends neural ODEs to dynamic multi-relational graphs and evaluates forecasting on unseen entities and farther-future links through two new tasks.
Takeaways & Limitations
The model assumes that relation representations do not evolve and remain time-invariant.
Abstract
from arXiv · showhide
There has been an increasing interest in inferring future links on temporal knowledge graphs (KG). While links on temporal KGs vary continuously over time, the existing approaches model the temporal KGs in discrete state spaces. To this end, we propose a novel continuum model by extending the idea of neural ordinary differential equations (ODEs) to multi-relational graph convolutional networks. The proposed model preserves the continuous nature of dynamic multi-relational graph data and encodes both temporal and structural information into continuous-time dynamic embeddings. In addition, a novel graph transition layer is applied to capture the transitions on the dynamic graph, i.e., edge formation and dissolution. We perform extensive experiments on five benchmark datasets for temporal KG reasoning, showing our model's superior performance on the future link forecasting task.
1 Introduction
Temporal knowledge graphs extend static knowledge graphs with time-varying facts, making future-link forecasting more challenging than completion. The paper addresses limitations of discrete-time and structurally limited approaches with a neural-ODE model, a graph transition layer, new evaluation tasks, and experiments on five datasets.
- Motivation: Temporal knowledge graphs represent facts with timestamps because entity relations can change over time.A fact is represented as a quadruple (s, r, o, t), extending static triples with validity time.
- Motivation: Future-link forecasting predicts links at later timestamps and is more challenging than predicting missing links at observed timestamps.The paper focuses on forecasting rather than temporal KG completion.
- Research gap: Existing discrete-time methods use regularly spaced snapshots, while Know-Evolve omits graph structure and cannot handle unseen nodes.These limitations motivate continuous-time, structurally informed forecasting models.
- Approach: TANGO combines neural ODEs with multi-relational graph convolutions to learn continuous-time representations of entities and relations.The ODE models temporal evolution while graph convolutions capture multi-relational structure.
- Approach: A graph transition layer models edge formation and dissolution to focus explicitly on temporal graph changes.The layer is reported to improve the model’s performance.
- Evaluation: The paper introduces inductive link prediction and long horizontal link forecasting to test unseen-entity and farther-future prediction.It evaluates TANGO on five benchmark temporal knowledge graph datasets and reports state-of-the-art performance.
2 Preliminaries and Related Work
The preliminaries define the graph-convolutional and neural-ODE components used for temporal KG reasoning. They also formalize temporal KG forecasting as ranking candidate entities for a future query based on earlier observed events.
- Graph convolution: R-GCN extends graph convolution to multi-relational graphs by using relation-specific weight matrices for message transformation.Common GCNs otherwise target homogeneous graphs.
- Neural ODEs: A neural ODE represents hidden-state derivatives with a neural network and uses an ODE solver to compute states at requested times.The formulation supports continuous-time dynamic systems.
- Neural ODEs: The adjoint sensitivity method computes parameter gradients by solving an augmented ODE backward in time without backpropagating through solver operations.This approach is introduced to reduce memory cost during backpropagation.
- Temporal KGs: A temporal knowledge graph is a multi-relational graph whose edges evolve over time, with snapshots that need not be regularly spaced.Timestamped edges are represented as quadruples (s, r, o, t).
- Forecasting task: Temporal KG forecasting ranks candidate entities to predict the missing object of a future query from observed events before its target time.The candidate set consists of all entities in V.
- Related work: Prior temporal-KG work includes completion and forecasting methods, with some forecasting approaches using temporal point processes to model event sequences.Forecasting methods can generalize to unseen timestamps.
3 Our Model
TANGO models temporal knowledge graphs with a neural ODE that combines multi-relational graph convolutions for structure with a transition layer for edge formation and dissolution. It produces continuous-time entity and relation representations, then forecasts links using KG score functions.
- Neural ODE-based model: TANGO uses a neural ODE-based encoder to learn continuous-time representations of entities and relations in evolving multi-relational graphs.The ODE solver integrates hidden representations over time, while the decoder uses KG score functions.
- Neural ODE-based model: The model combines stacked multi-relational graph convolutional layers with a graph transition layer that captures edge formation and dissolution.The graph convolutional component captures structural information, while the transition component uses changes between observations.
- Graph transition modeling: The transition tensor compares consecutive graph snapshots, assigning -1 to disappearing triples, 1 to formed triples, and 0 to unchanged edges.Its entries are derived from the difference between adjacency tensors at times t and t − ∆t.
- Link forecasting: For a query (s, r, ?, t), TANGO computes hidden representations, scores candidate objects, and predicts the object with the highest score.The decoder compares DistMult and TuckER score functions for evaluating candidate triples.
- Learning and inference: Representation inference uses an ODE solver, with fixed-grid solvers and IRDM employed to keep training time tractable.IRDM approximates reverse-mode hidden-state solutions using interpolation to reduce backpropagation cost while maintaining learning accuracy.
4 Experiments
TANGO is evaluated on five temporal knowledge graph datasets for extrapolated, inductive, and long-horizon future link prediction, with ablations of its graph transition layer and comparisons of training time. It outperforms the reported baselines across the main evaluations, while the transition layer improves performance and TANGO maintains a favorable time–performance trade-off.
- Experimental setup: TANGO is evaluated on ICEWS14, ICEWS18, ICEWS05-15, YAGO, and WIKI using time-aware filtered MRR and Hits@1/3/10.The experiments compare TANGO with static and temporal KG baselines and report averaged results from five runs.
- Time-aware filtered results: TANGO outperforms all static baselines on every dataset, indicating the value of temporal information for temporal KG representation learning.Comparisons with TANGO-Distmult and TANGO-TuckER also support the reported superiority of the NODE-based encoder over their static counterparts.
- Time-aware filtered results: TANGO outperforms all temporal KG baselines on every dataset and consistently exceeds RE-Net by explicitly encoding time into hidden representations.RE-Net is identified as the strongest baseline, but it models temporal order without explicitly encoding time information in the same way.
- Ablation study: The graph transition layer improves time-aware filtered MRR on ICEWS05-15 and WIKI by incorporating edge formation and dissolution information.The ablation uses datasets representing short, repeated events and longer, non-periodic events; Figure 2 focuses on graph snapshots with larger transition norms.
- Time cost analysis: TANGO maintains state-of-the-art future-link performance with shorter training time than RE-Net, which requires almost ten times TANGO’s total training time on ICEWS05-15.TTransE trains quickly but has low performance, while TA-Distmult takes longer than TANGO and is beaten by it in performance.
- Additional forecasting tasks: Long horizontal forecasting evaluates links at t+∆T with ∆T much larger than the usual short forecasting interval, while inductive prediction evaluates future links involving unseen entities.TANGO outperforms RE-Net in long horizontal forecasting, and TANGO-TuckER achieves the best inductive results across all reported metrics on ICEWS05-15.
5 Conclusions
TANGO forecasts future links on temporal knowledge graphs by combining multi-relational graph convolution with neural ODEs and a graph transition layer. It achieves state-of-the-art performance across five benchmark datasets and performs well on inductive and long-horizon forecasting tasks.
- TANGO combines multi-relational graph convolution for structural dependencies with neural ODEs for continuous dynamic representations.The model is presented as the first neural ODE extension to dynamic multi-relational graphs.
- The graph transition layer explicitly captures edge formation and deletion in temporal knowledge graphs.
- TANGO achieves state-of-the-art performance on five benchmark temporal knowledge graph datasets.
- TANGO performs well on inductive link prediction and long horizontal link forecasting.These tasks evaluate prediction for unseen entities and links occurring farther in the future.
A Representation Inference
TANGO infers future links by integrating hidden graph representations continuously between observed timestamps. Its inference procedure uses graph snapshots and transition tensors within a Neural ODE equipped with a graph neural network module.
- TANGO initializes hidden representations from entity and relation embeddings at the start of the history window.The initial hidden state approximates H(t − tk) before ODE integration begins.
- An ODE solver integrates the hidden state from the history start to the query time using the fTANGO function and graph information.
- The inference architecture feeds graph snapshots G and transition tensors T into the Neural ODE's GNN-based module.Dashed arrows denote hidden-state input and output paths, while red arrows represent continuous hidden-state flows.
B Evaluation Metrics
The evaluation reports raw, time-unaware filtered, and time-aware filtered metrics for future link prediction. Time-aware filtering is used to avoid removing facts that are not valid at the query time.
- Evaluation uses raw, time-unaware filtered, and time-aware filtered settings for future link prediction.
- Time-unaware filtering removes repeated triplets from training, validation, or test data regardless of their validity at the query time.
- The result tables mark TANGO as the model label and identify best results in bold.
- Time-aware filtering removes only triplets appearing at the query time, providing fairer temporal reasoning results.A fact valid at another timestamp is retained rather than filtered out.
C Implementation Details
The implementation tunes TANGO across a predefined hyperparameter search, trains the selected configuration repeatedly, and uses fixed-grid numerical integration. Experiments also specify dataset-dependent transition weights, solver settings, and baseline implementations.
- Training and tuning: The hyperparameter search performs 432 trials of 20 epochs and selects configurations by filtered validation MRR.The selected configuration is trained to convergence and run five times for averaged results.
- Numerical integration: TANGO uses a fourth-order Runge-Kutta fixed-grid ODE solver and an interpolated reverse dynamic method with 3 Chebyshev nodes.These choices are described as keeping training tractable while maintaining high precision.
- Dataset-specific settings: The transition coefficient w is tuned separately for each dataset, and scale controls the time-range rescaling parameter.
- Implementations: The experiments implement TANGO and baselines in PyTorch and run them on GeForce RTX 2080 Ti hardware.
- Evaluation setup: Validation uses time-aware filtered MRR, and ICEWS14 has no validation set.
D Datasets
The datasets use chronological splits and distinguish short, recurring ICEWS events from longer-lasting, non-periodic WIKI and YAGO events.
- Except for ICEWS14, datasets are split chronologically into 80% training, 10% validation, and 10% test portions.ICEWS14 uses a 50% training and 50% testing split because it is not pro-...
- ICEWS datasets contain short-lived events that may recur multiple times, whereas WIKI and YAGO contain longer-lasting, non-periodic events.The paper illustrates these contrasts with repeated visits in ICEWS and a multi-year football-career relation in WIKI and YAGO.
- Dataset statistics report training, validation, and test quadruple counts alongside the number of temporal observations.The observation count reflects snapshots taken at each observation time.
E Impact of Past History Length
Longer histories improve MRR but increase forwarding cost, so TANGO selects a history length of four as a performance–computation trade-off.
- TANGO uses the preceding history from t − tk to t, with tk controlling the history length.
- Longer past histories produce higher MRR but require more forwarding inferences.
- When history length reaches four or more, the MRR gain relative to training time is not significant.
- The experiments choose a history length of four because it balances forecasting performance against computational cost.
F Analysis on Temporal KGs with Irregular Time Intervals
TANGO is evaluated on temporal knowledge graphs with irregular observation intervals using a continuous-time dataset and time-aware forecasting metrics.
- Existing temporal KG reasoning models are described as unable to properly handle irregular time intervals, whereas TANGO is designed to model them through Neural ODEs.
- Figure 7 compares time-aware filtered MRR and training time across different history lengths on ICEWS05-15.
- ICEWS05-15_continuous samples ICEWS05-15 timestamps while retaining intervals between consecutive timestamps from 1 to 4.
- The continuous dataset keeps snapshots only at sampled times to represent non-periodic observations in continuous time.
- Models are evaluated on ICEWS05-15_continuous using time-aware filtered MRR and Hits@1/3/10.
- The paper reports that TANGO performs well on temporal KGs with irregular time intervals, while also reporting average training time until convergence.