Source-linked AI summary
Diachronic Embedding for Temporal Knowledge Graph Completion
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, Pascal Poupart
TL;DR
Temporal KG completion must infer time-stamped facts even though many KG embedding methods use static entity representations. The paper introduces model-agnostic diachronic entity embeddings, proves DE-SimplE fully expressive, and reports superior benchmark performance against existing work.
Problem
KG embedding approaches have mostly been developed for static KGs, limiting how entity characteristics are represented for temporal KG completion.
Method
The paper equips static KG models with a diachronic embedding function that maps each entity and timestamp to a hidden representation.
Results
DE-SimplE is fully expressive for temporal KG completion and the proposed models show superior performance compared to existing work on several benchmarks.
Takeaways & Limitations
Diachronic embeddings provide a generic way to model entity features at any point in time while extending static KG embedding models to temporal completion.
Takeaways & Limitations
For the domain-knowledge result, DE-SimplE additionally constrains the activation function in Equation (1) to have a non-negative range.
Abstract
from arXiv · showhide
Knowledge graphs (KGs) typically contain temporal facts indicating relationships among entities at different times. Due to their incompleteness, several approaches have been proposed to infer new facts for a KG based on the existing ones-a problem known as KG completion. KG embedding approaches have proved effective for KG completion, however, they have been developed mostly for static KGs. Developing temporal KG embedding models is an increasingly important problem. In this paper, we build novel models for temporal KG completion through equipping static models with a diachronic entity embedding function which provides the characteristics of entities at any point in time. This is in contrast to the existing temporal KG embedding approaches where only static entity features are provided. The proposed embedding function is model-agnostic and can be potentially combined with any static model. We prove that combining it with SimplE, a recent model for static KG embedding, results in a fully expressive model for temporal KG completion. Our experiments indicate the superiority of our proposal compared to existing baselines.
1 Introduction
Temporal KG completion extends KG completion to facts associated with timestamps, but most KG embedding methods were designed for static graphs. The paper proposes diachronic entity embeddings that represent entity features at each time and can extend static models, with DE-SimplE proved fully expressive.
- Motivation: KG completion infers new facts from existing graph facts, and KG embeddings score tuples using learned entity and relation representations.These methods differ in how they construct hidden representations and score tuples.
- Motivation: Temporal KG edges carry timestamps or intervals, yet most embedding approaches ignore this temporal aspect.Recent extensions add timestamp representations to score functions, improving performance over static approaches.
- Motivation: Static entity representations may be sub-optimal because an entity’s features and relationships can differ across time.The paper motivates modeling features at the timestamp of each fact rather than only current or aggregated features.
- Contribution: Diachronic embedding maps an entity and timestamp to a hidden representation and can potentially extend any static KG embedding model.The function is model-agnostic and is intended to provide entity features at arbitrary times.
- Contribution: Combining diachronic embedding with SimplE yields a fully expressive temporal KG completion model, evaluated on ICEWS and GDELT subsets.The paper presents this as the first temporal KG embedding model with a proof of full expressiveness.
2 Background and Notation
The background formalizes temporal knowledge graphs, KG completion, relation properties, and embedding models. A KG embedding specifies entity and relation representation functions together with a tuple-scoring function learned from data.
- Notation: Vector notation includes element indexing, norms, transposes, concatenation, outer-product flattening, and summed element-wise products.These conventions support the embedding and scoring formulations used throughout the paper.
- Temporal KG completion: A temporal KG consists of entities, relation types, timestamps, true temporal tuples, and a graph subset containing known facts.Temporal KG completion infers the full set of true tuples from the observed graph subset.
- Relation properties: Relations may be symmetric, anti-symmetric, inverse, or entail another relation according to tuple-level logical conditions.These properties describe how facts involving relation direction and implication are related.
- KG embedding: An entity embedding maps each entity to a hidden representation, while a relation embedding is defined analogously.The representation space may contain non-empty tuples of vectors and/or matrices.
- KG embedding: A KG embedding model defines entity and relation embedding functions plus a score function that evaluates tuples using those representations.The parameters of the hidden representations are learned from data.
3 Existing Approaches
Existing approaches extend static KG embedding models by incorporating timestamp representations or temporal transformations. The section reviews static scoring functions and representative temporal variants based on TransE, Tucker, DistMult, and sequence modeling.
- Static approaches: TransE scores tuples by the negative norm of head plus relation minus tail, while DistMult scores the element-wise product of their embeddings.Both use vector embeddings for entities and relations.
- Static approaches: Tucker scores tuples with a shared weight vector applied to the outer product of head, relation, and tail embeddings.RESCAL instead represents each relation with a matrix and scores using a bilinear form.
- Static approaches: CP uses separate head and tail entity vectors, with DistMult as the special case where those vectors are equal.SimplE addresses CP’s information-flow issue using inverse-relation embeddings and averages two CP scores.
- Temporal approaches: TTransE extends TransE by adding a timestamp embedding to the tuple score.Its score uses head, relation, timestamp, and tail vectors in a translated norm.
- Temporal approaches: HyTE projects entity and relation embeddings into a timestamp-specific space before applying a TransE-style score.Its temporal representation is induced through projection rather than a separate time-dependent entity function.
- Temporal approaches: ConT extends Tucker by replacing its shared weight vector with timestamp embeddings, while TA-DistMult encodes timestamp characters through an LSTM.TA-DistMult then applies the DistMult score to the resulting temporal relation embedding.
4 Diachronic Embedding
Diachronic embedding extends static KG embedding models by making entity representations depend on time, while preserving model agnosticism. The DE-SimplE combination is fully expressive for temporal KG completion, and the framework supports temporal learning and transferred domain-knowledge constraints.
- Diachronic Embedding: Diachronic entity embedding maps each entity–timestamp pair to a hidden representation, providing entity features at any given time.This replaces representations that are static or aggregate features across time.
- Diachronic Embedding: Any static KG score function can be made temporal by replacing its entity embeddings with diachronic entity embeddings.The DEEMB choice can vary across temporal KGs, and the function can generalize beyond tuples of vectors.
- Diachronic Embedding: The proposed embedding separates temporal and static features, with γ controlling the percentage of temporal features.Temporal features are modeled with entity-specific learnable parameters and an activation function, while explicit static features can reduce parameters and overfitting to temporal signals.
- Learning: The framework trains on temporal facts using mini-batches, sampled candidate entities, and cross-entropy loss for subject and object queries.Each fact generates queries with one known entity and randomly selected negative candidates.
- Expressivity and Domain Knowledge: DE-SimplE is fully expressive for temporal knowledge graph completion, and static-model domain-knowledge mechanisms can be transferred to its temporal version.Symmetry, anti-symmetry, inversion, and entailment are incorporated through parameter tying and additional non-negativity constraints where specified.
5 Experiments & Results
Experiments on ICEWS and GDELT compare static and temporal baselines with DE-TransE, DE-DistMult, and DE-SimplE. The results support diachronic embeddings, show strong model and activation-function variants, and demonstrate generalization to unseen timestamps.
- Comparative Study: Temporal versions of the models outperform static counterparts in most cases, while ConT performs poorly on ICEWS because its timestamp-specific parameterization is large.On GDELT, ConT performs somewhat better because the dataset has many training facts per timestamp, but its large parameter count also makes training extremely slow.
- Comparative Study: DE-TransE beats TTransE and HyTE on ICEWS14 and GDELT, DE-DistMult beats TA-DistMult and all TransE-based baselines, and DE-SimplE beats the other proposed variants.DE-TransE is on par with HyTE on ICEWS05-15; the authors attribute DE-SimplE’s advantage to SimplE’s higher expressivity.
- Model Variants & Ablation Study: Other activation functions also perform well, with squared exponential almost on par with sine across DE embedding functions.The authors suggest that sine and squared exponential features can represent multiple temporal switches, such as relations that start and later end.
- Model Variants & Ablation Study: Diachronic relation embeddings perform on par with entity-only diachronic embeddings on ICEWS14, ICEWS05-15, and GDELT.These results suggest that modeling relation evolution may not help on the evaluated benchmarks, though the authors leave other relation types and longer horizons for future work.
- Model Variants & Ablation Study: DE-DistMult gains almost 10% MRR over DistMult when evaluated on timestamps absent from training, indicating generalization to unseen timestamps.The held-out timestamps are the 5th, 15th, and 25th day of each month, with excluded facts split randomly into validation and test sets.
- Model Variants & Ablation Study: All three temporal components in Equation 1 are important, while removing b_v may reduce parameters with only a slight accuracy reduction.The ablation varies a_v, w_v, and b_v separately in DE-DistMult on ICEWS14.
- Model Variants & Ablation Study: DE-SimplE performance rises substantially when some features become temporal, then slightly drops as γ increases further, consistent with overfitting to temporal cues.Explicit static features reduce learnable parameters and may help avoid overfitting, at the cost of an additional hyper-parameter.
6 Related Work
Related work spans rule-based, graph-walk, static embedding, temporal embedding, extrapolation, and diachronic word-embedding approaches. Existing temporal KG methods often add timestamp representations, whereas the paper targets broader model extension and timestamp generalization.
- StaRAI: StaRAI approaches primarily use soft rules, sometimes combined with embeddings for property prediction, and include extensions to temporal KGs.These methods assign world probabilities according to satisfied or violated rules and rule confidence.
- Graph Walk: Graph-walk methods define weighted template walks and answer queries through template matching, with links to soft-rule models.They have been shown to be similar to, and sometimes subsumed by, soft-rule approaches.
- Static KG Embedding: Static KG embedding includes translational, bilinear, and deep-learning approaches, which can potentially be extended to temporal completion using diachronic embedding.The bilinear class varies matrix sparsity constraints, while deep models use feed-forward or convolutional layers.
- Temporal KG Embedding: Temporal KG embedding methods extend static models by adding timestamp embeddings, projecting embeddings into timestamp-specific spaces, or incorporating timestamp representations into score functions.Some approaches may struggle with many timestamps and cannot generalize to timestamps observed only outside training.
- KG Embedding for Extrapolation: KG extrapolation predicts future interactions using only facts before the queried time, whereas TKGC is framed as interpolation over missing facts within a time frame.The two problems share some approaches but differ in the information available for scoring a temporal interaction.
- Diachronic Word Embeddings: Diachronic word embeddings learn word representations as functions of time to study how meanings evolve across temporal corpus slices.The paper’s embedding idea is presented as similar to this time-dependent representation strategy.
7 Conclusion
The paper concludes that diachronic entity embeddings provide time-specific representations for temporal KG completion and can be combined with generic score functions. Their combination with SimplE is fully expressive and performs better than existing work on several benchmarks.
- Conclusion: The diachronic embedding function provides hidden representations for temporal-KG entities at any point in time and can combine with any score function.The authors identify alternative functions, function-selection studies, and diachronic word embeddings as future work.
- Conclusion: Combining diachronic embedding with SimplE yields a fully expressive temporal KG embedding model, the first such model with a proof of full expressiveness.The model was evaluated on several ICEWS and GDELT benchmarks.
- Conclusion: The proposed model shows superior performance compared with existing work on several benchmarks.
A Proof of Theorems and Propositions
The paper proves that DE-SimplE is fully expressive for temporal knowledge graph completion and preserves key relational properties through parameter tying and non-negativity constraints.
- Theorem 1: DE-SimplE is fully expressive for temporal knowledge graph completion.
- Theorem 1: The proof constructs relation-, entity-, and timestamp-specific embedding blocks that assign positive scores to facts and negative scores to their complements.Relation blocks isolate each relation, entity sub-blocks isolate each subject–relation pair, and timestamp sub-sub-blocks isolate each temporal fact.
- Proposition 1: Symmetry and anti-symmetry are incorporated by tying relation embeddings or tying them to their negations, respectively.These ties make scores identical for symmetric relations and opposite for anti-symmetric relations.
- Proposition 1: Inverse relations are incorporated by cross-tying the two relation embeddings, making the inverse fact score identical to the original fact score.
- Proposition 2: Entailment is incorporated by constraining entity embeddings and activation outputs to be non-negative, then tying the entailed relation to a non-negative offset from the entailing relation.The resulting score for the entailed relation is greater than or equal to the score for the entailing relation.
- Proposition 2: The entailment construction ensures that the entailed relation has probability greater than or equal to the entailing relation.