Source-linked AI summary
Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, Yan Zhan
TL;DR
Incomplete temporal knowledge graphs contain recurring facts that can inform future-fact prediction. CyGNet combines time-aware copying from historical entities with generation over the full vocabulary, and experiments on five benchmarks demonstrate effective recurrent and de novo prediction.
Problem
Temporal knowledge graphs are incomplete, while many facts recur over time; existing methods generally do not explicitly model this recurrence.
Method
CyGNet uses a time-aware copy-generation mechanism combining historical-vocabulary copying with whole-entity-vocabulary generation.
Results
Across five benchmark datasets, CyGNet achieves more precise future-fact predictions than state-of-the-art methods lacking special consideration of recurrence patterns.
Takeaways & Limitations
CyGNet supports both recurring-fact prediction from history and de novo fact prediction from the whole entity vocabulary.
Takeaways & Limitations
On WIKI and YAGO, especially WIKI, CyGNet does not always perform best; WIKI’s imbalanced subject–object recurrence rates hinder combined learning.
Abstract
from arXiv · showhide
Large knowledge graphs often grow to store temporal facts that model the dynamic relations or interactions of entities along the timeline. Since such temporal knowledge graphs often suffer from incompleteness, it is important to develop time-aware representation learning models that help to infer the missing temporal facts. While the temporal facts are typically evolving, it is observed that many facts often show a repeated pattern along the timeline, such as economic crises and diplomatic activities. This observation indicates that a model could potentially learn much from the known facts appeared in history. To this end, we propose a new representation learning model for temporal knowledge graphs, namely CyGNet, based on a novel timeaware copy-generation mechanism. CyGNet is not only able to predict future facts from the whole entity vocabulary, but also capable of identifying facts with repetition and accordingly predicting such future facts with reference to the known facts in the past. We evaluate the proposed method on the knowledge graph completion task using five benchmark datasets. Extensive experiments demonstrate the effectiveness of CyGNet for predicting future facts with repetition as well as de novo fact prediction.
1 Introduction
Temporal knowledge graphs capture evolving entity interactions but remain incomplete. CyGNet addresses recurring temporal facts by combining historical copying with whole-vocabulary generation for future-fact prediction.
- Motivation: Temporal knowledge graphs extend static knowledge graphs by assigning temporal properties to evolving entity interactions.Representative event-based resources include GDELT and ICEWS.
- Motivation: Existing temporal representation methods can fall short because recurrence and trending patterns complicate temporal-fact evolution.Many methods incorporate past snapshots but do not explicitly model these evolution patterns.
- Motivation: More than 80% of ICEWS events from 1995–2019 had already appeared in a previous time period.This recurrence motivates using known historical facts when predicting future facts.
- CyGNet: CyGNet combines Copy and Generation modes to predict from either a historical entity vocabulary or the whole entity vocabulary.The Copy mode targets recurring facts, while Generation supports newly appearing facts.
- Evaluation: Experiments on five benchmark TKG datasets show that CyGNet more precisely predicts future facts than state-of-the-art methods lacking special treatment of recurrence patterns.The reported evaluation covers future fact link prediction.
2 Related Work
Prior work includes static and temporal knowledge-graph embeddings, graph-sequence models, and copy mechanisms from natural-language generation. CyGNet adapts hybrid copying and generation to recurring temporal knowledge-graph facts.
- Static KG Embeddings: Static KG embedding methods model facts without temporal dynamics and therefore do not capture temporally dynamic facts.They include translational and semantic-matching approaches.
- Temporal KG Embeddings: Temporal KG embedding models encode evolving facts with time-specific representations, temporal scoring, point processes, diachronic embeddings, or timestamp-specific tensors.Examples include TTransE, HyTE, Know-Evolve, DE-SimplE, and ConT.
- Temporal KG Embeddings: Earlier temporal embedding models do not provide a mechanism to capture long-term dependency across consecutive time snapshots.Graph-sequence methods use recurrent or graph-convolutional architectures to address long-term temporal structure.
- Copy Mechanism: Pointer-style copy mechanisms select outputs from input sequences, while hybrid COPYNET combines copying with generation for external lexemes.SeqCopyNet further copies subsequences rather than only single lexemes.
- Copy Mechanism: CyGNet is presented as the first work to incorporate a copy mechanism into temporal knowledge-graph modeling.Its design is motivated by recurrence patterns along temporal timelines.
3 Method
CyGNet models temporal knowledge graph completion with sequentially maintained historical vocabularies and two complementary inference modes. Copy predicts recurrent facts from history, while Generation predicts de novo facts from the whole entity vocabulary, and their probabilities are combined for final prediction.
- Training Process: CyGNet sequentially trains on temporal knowledge graph snapshots while extending each subject–predicate pair’s historical vocabulary from previous snapshots.The historical vocabulary contains entities previously observed as objects for the same subject and predicate; evaluation uses the maximum vocabulary from the training set.
- Copy Mode: The Copy mode identifies recurrent facts by predicting object entities from the historical vocabulary associated with a query’s subject and predicate.It delimits the candidate space using a modified multi-hot historical indicator, reducing probability for uninterested entities before softmax prediction.
- Generation Mode: The Generation mode predicts potentially new facts by selecting an object entity from the whole entity vocabulary without reference to history.Its softmax probabilities cover the complete candidate space, complementing Copy mode with de novo fact prediction.
- Learning Objective: Training treats missing-object prediction as a multi-class classification task optimized with cross-entropy over temporal graph facts.The same inference formulation can be extended to predicting missing subjects or predicates.
- Inference: CyGNet combines Copy and Generation probabilities for every entity, with α controlling their interpolation, and selects the entity with the highest combined probability.Both modes produce predictions within their candidate spaces before the final whole-vocabulary decision.
4 Experiments
CyGNet is evaluated for temporal knowledge graph link prediction across five benchmark datasets using time-aware copy and generation modes. It outperforms baselines on several datasets, while performance varies with recurrence balance and the availability of historical information.
- Experimental Setup: CyGNet is evaluated on link prediction using five benchmark datasets: ICEWS18, ICEWS14, GDELT, WIKI, and YAGO.The experiments compare CyGNet with static knowledge graph embedding and temporal knowledge graph embedding methods.
- Results: CyGNet significantly outperforms other baselines on ICEWS18, ICEWS14, and GDELT.Static methods generally struggle to capture temporal dynamics, while some temporal baselines fail to capture long-term dependency.
- Results: 10.80% MRR improvement over state-of-the-arts is reported for CyGNet on GDELT, alongside gains of 12.10% Hits@1, 11.29% Hits@3, and 7.19% Hits@10.GDELT’s denser training facts provide more historical information that CyGNet can use for prediction.
- Results: CyGNet does not always perform best on WIKI and YAGO, especially WIKI, where recurring objects are much more common than recurring subjects.The reported recurrence rates are 62.3% for object groups and around 23.4% for subject groups on WIKI.
- Ablation Study: Removing Copy mode decreases MRR by 12.11%, whereas removing Generation mode decreases MRR by 4.09% on ICEWS18.The ablation results associate Copy mode with historical-fact prediction and Generation mode with de novo fact prediction.
- Ablation Study: CyGNet outperforms CyGNet-Generation-new because its original Generation mode can also strengthen predictions for repeated future facts.The modified variant restricts Generation mode to new facts outside the historical vocabulary.
5 Conclusion
The paper presents CyGNet as a copy-based approach for predicting future temporal knowledge graph facts from historical recurrence and the open entity vocabulary. Results show promising future-fact prediction, with further work aimed at salient entities, dated-document grounding, and dynamic event processes.
- 5 Conclusion: CyGNet combines historical copying with open-world prediction to identify and predict recurring and new future facts in temporal knowledge graphs.The model is motivated by the hypothesis that future facts can be predicted from facts in history.
- 5 Conclusion: The presented results demonstrate CyGNet’s promising performance for predicting future facts in temporal knowledge graphs.The authors propose improving sequential copying and extending the learned representations to related temporal-text tasks.
A Hyper-parameter Analysis
CyGNet’s performance depends on balancing Copy and Generation modes through α. Generation-only and Copy-only settings are ineffective compared with intermediate weighting.
- α = 0 uses only Generation mode, whereas α = 1 uses only Copy mode.
- Performance increases as α rises within a certain range, showing that incorporating historical facts can improve predictions.
- Excessive reliance on historical facts decreases performance, with Copy-only inference ignoring de novo facts.
B Descriptions of Baseline Methods
The baselines include static knowledge-graph embedding methods and more recent temporal models. Static methods represent entities and relations without capturing temporal facts, while temporal methods model evolving facts over time.
- Static KG Embeddings: TransE, DistMult, ComplEx, and related methods represent static entity-relation interactions using translational, bilinear, or convolutional embeddings.
- Static KG Embeddings: RotatE models predicates as rotations from subject entities to object entities in complex space, but static methods do not capture temporal facts.
- Temporal KG Embeddings: TTransE extends TransE by incorporating temporal information into its score function.
- Temporal KG Embeddings: HyTE uses a time-related normal vector, while Know-Evolve learns non-linearly evolving entity representations and models fact occurrence as a temporal point process.