PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Knowledge Graph Forecasting with Neural ODE

Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, Volker Tresp

arXiv:2101.05151v3cs.LGcs.AI

TL;DR

Temporal knowledge graph forecasting remains challenging because links evolve continuously while existing models often use discrete-time representations. TANGO combines neural ODEs with multi-relational graph convolutions and a graph transition layer; it achieves state-of-the-art performance on five benchmark datasets.

  • Problem

    Temporal KG forecasting is more challenging than completion, while existing approaches often use discrete time and may omit structural information or fail to handle unseen nodes.

    Source: §1 · p. 1

  • Method

    TANGO combines neural ODEs with multi-relational graph convolutions to learn continuous-time representations and uses a graph transition layer to capture edge formation and dissolution.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    TANGO achieves state-of-the-art performance on five benchmark temporal knowledge graph datasets.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Modeling edge formation and dissolution with a graph transition layer effectively improves the model’s performance.

    Source: §1 · p. 2

    The model assumes that relation representations are time-invariant.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

There has been an increasing interest in inferring future links on temporal knowledge graphs (KG). While links on temporal KGs vary continuously over time, the existing approaches model the temporal KGs in discrete state spaces. To this end, we propose a novel continuum model by extending the idea of neural ordinary differential equations (ODEs) to multi-relational graph convolutional networks. The proposed model preserves the continuous nature of dynamic multi-relational graph data and encodes both temporal and structural information into continuous-time dynamic embeddings. In addition, a novel graph transition layer is applied to capture the transitions on the dynamic graph, i.e., edge formation and dissolution. We perform extensive experiments on five benchmark datasets for temporal KG reasoning, showing our model's superior performance on the future link forecasting task.

1 Introduction

TANGO forecasts future links in temporal knowledge graphs by modeling continuous-time graph dynamics and explicitly representing edge transitions. The introduction also presents new evaluation tasks for unseen entities and farther-future links.

1 Introduction

  • Discrete snapshot methods (Jin et al., 2019; Zhu et al., 2020) miss irregular intervals, while KnowEvolve (Trivedi et al., 2017) omits graph structure and cannot handle unseen nodes.TANGO proposes a continuous-time graph neural ODE to model temporal KG dynamics.Source: §1 · pp. 1–2
  • A graph transition layer processes edge formation and dissolution explicitly, emphasizing graph changes that may carry valuable temporal information.It uses graph convolutions on a transition tensor containing formation and dissolution information.Source: §1 · p. 2
  • TANGO extends neural ODEs to multi-relational graphs, learning continuous-time entity and relation representations from structural and temporal information.MGCN layers capture graph structure, while an ODE solver integrates hidden representations over time.Source: §1 · pp. 1–2
  • The paper proposes inductive link prediction and long horizontal link forecasting to test performance on unseen entities and links in the farther future.Temporal KG forecasting targets future links, unlike completion, which predicts missing links at observed timestamps.Source: §1 · pp. 1–2
  • TANGO shows state-of-the-art performance on five benchmark temporal knowledge graph datasets.The introduction reports this overall result without giving benchmark scores.Source: §1 · p. 2

2 Preliminaries and Related Work

The section reviews graph convolution, neural ODEs, and temporal knowledge graph forecasting, establishing the background and task definition for continuous-time multi-relational graph modeling.

2.1 Graph Convolutional Networks

  • Common GCNs handle homogeneous graphs, while R-GCN distinguishes relations but can overfit as its parameters grow; prior temporal GCNs target homogeneous rather than multi-relational graphs.Multi-relational GCNs compatible with KGs have also used entity-relation composition operations.Source: §2.1 · p. 2

2.2 Neural Ordinary Differential Equations

  • A neural ODE represents hidden-state derivatives with a neural network and uses an ODE solver with an initial value to produce hidden states at arbitrary times.This enables modeling continuous-time dynamic systems.Source: §2.2 · p. 3
  • The adjoint sensitivity method reduces backpropagation memory cost by solving an augmented ODE backward in time to compute parameter gradients without differentiating through solver operations.The adjoint is a(t) = ∂L/∂z(t).Source: §2.2 · p. 3

2.3 Temporal Knowledge Graph Reasoning

  • A temporal KG is a multi-relational graph whose edges evolve over time, and neighboring snapshots need not be regularly spaced.A timestamped quadruple (s, r, o, t) represents a labeled edge at time t.Source: §2.3 · p. 3
  • Forecasting ranks candidate objects for a query using events before its target time; reciprocal relations make object prediction general, while prior approaches include temporal point-process event-sequence models.Forecasting methods can generalize to unseen timestamps; Trivedi et al. (2017) and Han et al. (2020b) use temporal point processes to model event sequences and evolving entity representations.Source: §2.3 · p. 3

3 Our Model

TANGO learns continuous-time representations of temporal knowledge graphs by combining structural graph information with edge-formation and dissolution signals in a neural ODE, then decoding with knowledge-graph score functions.

3 Our Model

  • TANGO learns continuous-time entity representations with a neural ODE encoder, combining structural graph convolution and edge-transition inputs before integration, then decoding with KG score functions.The two input modules capture snapshot structure and transitions since the previous observation.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Neural ODE for Temporal KG

  • The ODE parameterizes hidden-state derivatives using structural graph convolution plus a weighted transition-layer term, starting from learnable entity and relation embeddings to evolve representations over time.The transition contribution is controlled by hyperparameter w.Source: §3.1 · p. 4

3.2 Multi-Relational Graph Convolutional Layer

  • Its multi-relational convolution updates entity representations using entity-relation composition; relation representations are assumed time-invariant, and ReLU is the activation.Source: §3.2 · p. 4

3.3 Graph Transition Layer

  • For adjacent snapshots, T(t) marks formed triples +1, dissolved triples -1, and unchanged edges 0; graph convolutions extract these transitions and combine them with MGCN in the derivative network.Source: §3.3 · pp. 4–5

3.4 Learning and Inference

  • At query time, TANGO scores candidate objects with Distmult or TuckER and predicts the highest scorer; fixed-grid ODE solving with IRDM reduces backpropagation time while maintaining learning accuracy.Source: §3.4 · p. 5

4 Experiments

Across five benchmark datasets, TANGO outperforms static and temporal baselines in time-aware filtered future-link prediction. The experiments also assess the graph transition layer, training time, long-horizon forecasting, and prediction involving unseen entities.

4.1 Experimental Setup

  • The experiments use five temporal-KG datasets, evaluate extrapolated prediction with MRR and Hits@1/3/10 under two filtering protocols, and compare against three static and six temporal baselines.The datasets are ICEWS14, ICEWS18, ICEWS05-15, YAGO, and WIKI; the filtering protocols are time-aware and time-unaware.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, §4.1.3 · p. 6

4.2.1 Time-aware filtered Results

  • In time-aware filtered forecasting, TANGO outperforms every static and temporal baseline on all five datasets.The comparison identifies RE-Net as the strongest baseline; TANGO also improves over paired static models, supporting the benefit of its NODE-based encoder and explicit time encoding. Table 2 reports the benchmark results, but the supplied passages do not include its cell values.Source: §4.2.1 · p. 6, Table 2 · p. 7

4.2.2 Ablation Study

  • The graph transition layer improves time-aware filtered MRR on ICEWS05-15 and WIKI, with the ablation focused on snapshots showing larger graph changes.The layer is designed to incorporate edge formation and dissolution; the two datasets represent short, recurring events and longer, non-periodic events, respectively.Source: Figure 2 · p. 6, §4.2.2 · p. 7

4.2.3 Time Cost Analysis

  • On ICEWS05-15, TANGO trains in almost one-tenth the time required by RE-Net while maintaining state-of-the-art future-link prediction performance.The reported comparison says TANGO is also faster than TA-Distmult, which is beaten by TANGO in performance; static methods generally require less training time than temporal methods.Source: §4.2.3 · p. 7, Figure 3 · p. 7

4.3.1 Long Horizontal Link Forecasting

  • Long horizontal forecasting predicts links at t+ΔT from snapshots observed through t, and TANGO outperforms RE-Net, though their performance gap narrows as ΔT grows.TANGO adjusts the final ODE integration interval to the forecast horizon; its performance degrades as the assumption that dynamics at t persist into the farther future becomes less valid.Source: §4.3.1 · pp. 7–8, Figure 4 · p. 7

4.3.2 Inductive Link Prediction

  • For inductive future link prediction on ICEWS05-15, TANGO-TuckER achieves the best results across all metrics, while both TANGO-TuckER and TANGO-Distmult outperform RE-Net.The TANGO models achieve much better results than their variants without graph transition layers, supporting the layer’s essential role in inductive link prediction.Source: §4.3.2 · p. 8, Table 3 · p. 8

5 Conclusions · Appendix

TANGO uses continuous-time graph neural ODEs and a graph transition layer to forecast future links on temporal knowledge graphs. The paper reports state-of-the-art performance on five benchmark datasets and good performance on two proposed forecasting tasks.

5 Conclusions

  • TANGO is a proposed representation method for forecasting future links on temporal knowledge graphs.Source: §5 · pp. 8–9
  • TANGO achieves state-of-the-art performance on five benchmark datasets for temporal knowledge graphs.Source: §5 · p. 9
  • A multi-relational graph convolutional layer captures structural dependencies on temporal knowledge graphs.Source: §5 · p. 9
  • Graph neural ODEs learn continuous dynamic representations, and TANGO demonstrates their extension to dynamic multi-relational graphs.Source: §5 · p. 9
  • A graph transition layer explicitly captures information from edge formation and deletion.Source: §5 · p. 9
  • TANGO performs well on inductive link prediction and long horizontal link forecasting, two tasks proposed to evaluate link forecasting models.Source: §5 · p. 9

A Representation Inference

TANGO infers hidden representations by initializing them from graph embeddings at the start of a history window and integrating a Neural ODE over that window. At each observation time, it supplies graph snapshots and transition tensors to the dynamics module, allowing the hidden state to evolve continuously while learning from graph observations.

A Representation Inference

  • TANGO initializes H(t − tk) from Emb(V, R) and integrates the Neural ODE from t − tk to t to infer H(t).The history length tk determines the graph-history window used for forecasting a link at t.Source: A Representation Inference · p. 10
  • At each observation time, set_graph and set_transition feed graph snapshots and transition tensors into the dynamics module fTANGO.Figure 6 depicts the graph snapshots and transition tensors as inputs to the Neural ODE's GNN-based module.Source: A Representation Inference · p. 10, Figure 6 · p. 10
  • The hidden state evolves continuously over time and learns from graph observations at different timestamps.TANGO uses an ODE solver to integrate the state across the history window.Source: A Representation Inference · p. 10, Figure 6 · p. 10

B Evaluation Metrics

The section reports raw, time-unaware filtered, and time-aware filtered evaluation, and explains why filtering only at the query time is fairer for temporal KG reasoning.

B Evaluation Metrics

  • Future link prediction results on benchmark datasets report raw MRR (%) and Hits@1/3/10 (%), alongside time-unaware filtered MRR (%) and Hits@1/3/10 (%).Source: Table 4 · p. 11, Table 5 · p. 11
  • Time-unaware filtering removes corrupted triplets appearing in training, validation, or test data, following Bordes et al. (2013) and Jin et al. (2019).The triplet of interest is excluded from removal.Source: B Evaluation Metrics · pp. 10–11
  • Time-aware filtering removes only triplets appearing at query time, following Han et al. (2021), and is presented as fairer for temporal KG reasoning.A triplet valid at another time can be invalid at the query time, so filtering it across all times is unreasonable.Source: B Evaluation Metrics · p. 11

C Implementation Details

The implementation details describe TANGO’s hyperparameter selection, ODE integration choices, dataset-specific time settings, and baseline configurations.

C Implementation Details

  • TANGO is tuned in 432 trials of 20 epochs, selected by validation filtered MRR, trained to convergence, and averaged over five runs.The search space is listed in Table 7, and the best settings by dataset are reported in Table 8.Source: C Implementation Details · p. 11, Table 7 · p. 12, Table 8 · p. 12
  • TANGO uses a fixed-grid fourth-order Runge–Kutta solver and a three-Chebyshev-node interpolated reverse dynamic method to limit training time while maintaining precision.The time range is rescaled to 0–0.01 or 0.1 to shorten integration and prevent solver error.Source: C Implementation Details · p. 11, Table 7 · p. 12
  • Input-history windows are four days for ICEWS datasets and four years for WIKI and YAGO; the transition coefficient varies by dataset.TANGO is implemented in PyTorch and experiments run on a GeForce RTX 2080 Ti.Source: C Implementation Details · p. 11
  • Baselines use their stated implementations and Adam with batch size 512; RE-Net uses ground-truth history at multi-step inference and a history length of 10.DistMult uses binary cross-entropy; TA-DistMult uses year, month, and day temporal tokens across datasets.Source: C Implementation Details · pp. 11–12

D Datasets

This section describes the dataset split strategy and distinguishes short, recurring ICEWS events from longer, non-periodic WIKI and YAGO events.

D Datasets

  • Datasets except ICEWS14 are split chronologically into training, validation, and test sets, typically 80%/10%/10%; ICEWS14 uses a 50%/50% training-test split because it lacks validation data.The split strategy follows Jin et al. (2019).Source: D Datasets · p. 12
  • ICEWS datasets contain short-lived events that can recur, whereas WIKI and YAGO events last longer and do not occur periodically.The passage contrasts repeated visits with an event lasting from 2003 to 2010 as examples.Source: D Datasets · p. 12
  • Table 9 reports training, validation, and test quadruple counts, plus the number of observations, defined by snapshots of the temporal knowledge graph.Source: Table 9 · p. 12

E Impact of Past History Length

TANGO’s history length trades higher MRR against increased forwarding cost; because gains relative to training time are not significant at lengths of four or more, the experiments use length four.

E Impact of Past History Length

F Analysis on Temporal KGs with Irregular Time Intervals · G Average runtime for each approach

TANGO is evaluated on an irregular-time version of ICEWS05-15, where the paper reports that it performs well. A separate runtime section reports average training time until convergence across models and datasets.

F Analysis on Temporal KGs with Irregular Time Intervals

G Average runtime for each approach

Related papers

Loading 2101.05151v3…