Source-linked AI summary
Learning Sequence Encoders for Temporal Knowledge Graph Completion
Alberto García-Durán, Sebastijan Dumančić, Mathias Niepert
TL;DR
Temporal knowledge graph completion must predict missing relations when facts hold at specific times or intervals, despite sparse and heterogeneous temporal expressions. The paper uses a digit-level LSTM to encode temporal-token sequences into time-aware relation representations for existing scoring functions, and reports effectiveness across four temporal knowledge graphs.
Problem
Temporal KG completion requires predicting missing relations conditioned on time, while temporal expressions are sparse and heterogeneous.
Method
A digit-level LSTM encodes predicate and temporal-token sequences into time-aware relation representations used with existing link-prediction scoring functions.
Results
Experiments in four temporal knowledge graphs show the effectiveness of the approach, with time-aware models systematically improving TRANSE and DISTMULT in MRR, hits@10, and hits@1 in almost all cases.
Takeaways & Limitations
Learned relation embeddings carry temporal information, and incorporating that information improves training loss and commonly improves link-prediction metrics.
Abstract
from arXiv · showhide
Research on link prediction in knowledge graphs has mainly focused on static multi-relational data. In this work we consider temporal knowledge graphs where relations between entities may only hold for a time interval or a specific point in time. In line with previous work on static knowledge graphs, we propose to address this problem by learning latent entity and relation type representations. To incorporate temporal information, we utilize recurrent neural networks to learn time-aware representations of relation types which can be used in conjunction with existing latent factorization methods. The proposed approach is shown to be robust to common challenges in real-world KGs: the sparsity and heterogeneity of temporal expressions. Experiments show the benefits of our approach on four temporal KGs. The data sets are available under a permissive BSD-3 license 1.
1 Introduction
Temporal knowledge graphs extend link prediction by attaching time information to facts, including point timestamps and open intervals. The paper proposes learning time-aware relation representations to handle sparse, heterogeneous temporal expressions.
- Link prediction in temporal KGs seeks the most probable completion of a fact given its time information.The task extends conventional completion of (s, p, ?) or (?, p, o) triples.
- Temporal KGs augment entity–relation facts with timestamps or temporal modifiers describing when relations hold.Examples include a point timestamp such as 1961 and an open interval marked by “occursSince.”
- Learning temporal representations is challenging because real-world time expressions are sparse and irregular.The paper treats temporal expressions as token sequences to make them suitable for representation learning.
- The proposed method learns time-aware relation representations that can be combined with standard embedding approaches for link prediction.It is motivated by scoring functions that operate on latent entity and relation embeddings.
2 Related Work
Prior work introduced temporal reasoning, event-based prediction, point-process models, and time-aware scoring functions for temporal knowledge graph completion. This paper instead encodes temporal-token sequences with an RNN.
- Temporal reasoning has been studied through temporal logics and extensions of statistical relational learning frameworks.
- Existing temporal KG completion methods use ordering constraints, incoming-event graphs, point processes, or time-aware TransE-style scoring.
- Unlike previous approaches, the paper encodes sequences of temporal tokens with an RNN.The sequences include relation tokens and temporal tokens such as “since,” “until,” and timestamp digits.
3 Time-Aware Representations
The approach converts heterogeneous temporal expressions into token sequences and encodes them with an LSTM to create time-aware predicate representations. These representations are combined with entity embeddings in standard knowledge-graph scoring functions.
- Embedding-based KG completion scores triples using subject, object, and predicate embeddings, but the standard functions omit temporal information.
- Temporal expressions are decomposed into token sequences, including relation and temporal-modifier tokens when available.
- Each token is mapped to a d-dimensional embedding, processed by an LSTM, and represented by its final hidden state epseq = hN.
- The resulting predicate-sequence representation is inserted into temporal-aware variants of TRANSE and DISTMULT, called TA-TRANSE and TA-DISTMULT.
- Character-level time encoding supports a small vocabulary, representations for unseen timestamps, and training on both temporal and atemporal triples.
4 Experiments
Experiments evaluate temporal knowledge graph completion on four data sets using filtered completion queries and standard ranking metrics. Time-aware models generally improve over their non-temporal counterparts, while timestamp-specific baselines are competitive when timestamps are few.
- Datasets: Experiments use four temporal knowledge graph data sets, including ICEWS 2014, ICEWS 2005-15, YAGO15K, and WIKIDATA.ICEWS variants contain political events, while YAGO15K and WIKIDATA include temporal facts with different temporal annotations.
- Evaluation: Evaluation ranks the correct subject or object in completion queries and reports Mean Rank, hits@n, and Mean Reciprocal Rank.Mean Rank is lower-is-better, whereas hits@n and Mean Reciprocal Rank are higher-is-better; the filtered setting is also reported.
- Results: TA-TRANSE and TA-DISTMULT systematically improve TRANSE and DISTMULT in MRR, hits@10, and hits@1 in almost all cases.Mean-rank improvements are less consistent because Mean Rank is susceptible to outliers.
- Results: TTRANSE is competitive in YAGO15K and similar to TA-TRANSE in WIKIDATA, where few distinct timestamps provide enough examples for robust timestamp embeddings.At test time, timestamps unseen during training receive null vectors in TTRANSE.
- Results: In YAGO15K, TA-TRANSE achieves lower training loss than TRANSE under the same setup.The comparison is shown in Figure 3.
- Results: A t-SNE visualization for pseq = [playsFor, occursSince, date] illustrates that learned relation-type embeddings carry temporal information.Here, date corresponds to the date token sequence.
5 Conclusions
The paper proposes a digit-level LSTM for representing time-augmented knowledge-graph facts and demonstrates its effectiveness on four temporal knowledge graphs.
- A digit-level LSTM learns representations for time-augmented knowledge-graph facts.
- The learned representations can be combined with existing scoring functions for link prediction.
- Experiments on four temporal knowledge graphs show the approach is effective.