Source-linked AI summary
TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion
Jiapeng Wu, Meng Cao, Jackie Chi Kit Cheung, William L. Hamilton
TL;DR
TKGC seeks to infer missing facts despite limited use of multi-hop structure, recent temporal evidence, and uneven entity activity. TeMP combines message passing, temporal dynamics, imputation, and frequency-based gating, achieving a 10.7% relative Hits@10 improvement over state-of-the-art methods across three benchmarks. The analysis also identifies dataset variability and strong simple baselines in some settings.
Problem
TKGC methods do not explicitly combine multi-hop structural information and nearby temporal facts, while temporal sparsity and variability in entity activity remain insufficiently addressed.
Method
TeMP jointly models multi-hop structural information and nearby temporal facts using neural message passing, temporal dynamics, data imputation, and frequency-based gating.
Results
10.7% relative improvement in Hits@10 over state-of-the-art methods is reported across three benchmark datasets.
Takeaways & Limitations
The analysis identifies temporal sparsity variability within and across datasets, while simple baselines can outperform prior state of the art in certain settings.
Takeaways & Limitations
Future work includes extending TeMP to continuous TKGC and improving imputation for infrequent and inactive entities.
Abstract
from arXiv · showhide
Inferring missing facts in temporal knowledge graphs (TKGs) is a fundamental and challenging task. Previous works have approached this problem by augmenting methods for static knowledge graphs to leverage time-dependent representations. However, these methods do not explicitly leverage multi-hop structural information and temporal facts from recent time steps to enhance their predictions. Additionally, prior work does not explicitly address the temporal sparsity and variability of entity distributions in TKGs. We propose the Temporal Message Passing (TeMP) framework to address these challenges by combining graph neural networks, temporal dynamics models, data imputation and frequency-based gating techniques. Experiments on standard TKG tasks show that our approach provides substantial gains compared to the previous state of the art, achieving a 10.7% average relative improvement in Hits@10 across three standard benchmarks. Our analysis also reveals important sources of variability both within and across TKG datasets, and we introduce several simple but strong baselines that outperform the prior state of the art in certain settings.
1 Introduction
TKGC must infer missing time-stamped facts while using structural and temporal context that prior methods often overlook. TeMP combines temporal message passing with sparsity-aware techniques, achieving strong benchmark performance.
- Inferring missing facts in temporal knowledge graphs supports event prediction, question answering, social network analysis, and recommendation systems.
- Existing TKGC methods do not properly model multi-hop structure or explicitly use facts from nearby temporal snapshots.For example, earlier visits or related agreements can inform a later visit query.
- Temporal variability and sparsity complicate TKGC because queries expose different amounts of nearby reference information and few entities are active at each time step.Previous methods often assign inactive entities the same embedding across time steps.
- TeMP combines neural message passing, temporal dynamic models, frequency-based gating, and data imputation to address these challenges.
- 7.3% average relative improvement in Hits@10 over the next-best model is reported on ICEWS14, ICEWS05-15, and GDELT.The paper also reports state-of-the-art performance on standard TKGC benchmarks.
2 Related Work
Related work extends static knowledge-graph representation learning with temporal scoring, while other approaches use message passing or temporal mechanisms for different graph-reasoning settings. These lines of work leave several distinctions and limitations motivating TeMP.
- Static KG representation learning: Static KG methods learn entity and relation embeddings and score candidate facts, while graph neural networks can incorporate multi-hop neighborhood information.
- Temporal KG representation learning: Temporal KG methods typically add time-sensitive decoding to shallow embedding models.
- Temporal KG representation learning: These temporal methods do not explicitly capture entity-level temporal patterns such as event periodicity.
- Temporal graph reasoning: Other temporal graph methods combine neighborhood message passing with recurrence or attention, while temporal point-process work focuses on continuous TKGC.
- Temporal graph reasoning: Several closely related approaches target graph extrapolation, so they are not directly compatible with the TKGC setting.
3 Proposed Approach
TeMP encodes temporal knowledge graphs by combining multi-relational structural message passing with temporal modeling over recent snapshots. It further uses imputation and frequency-based gating to address inactive entities and heterogeneous temporal information.
- TKGC ranks subject and object entities for object and subject queries at a specified time.
- TeMP maps each entity to time-dependent embeddings, which a decoder uses to score temporal facts.
- 3.2 Temporal Encoder: The temporal encoder integrates structural representations across recent snapshots using either recurrent modeling with decay or self-attention over active representations.
- 3.1 Structural Encoder: The structural encoder applies multi-relational message passing within each snapshot to produce embeddings summarizing an entity’s L-hop neighborhood.
- 3.3 Tackling Temporal Heterogeneities: TeMP optionally imputes representations for inactive entities by combining stale structural representations with temporal information.
- 3.3 Tackling Temporal Heterogeneities: Frequency-based gating learns to fuse structural and temporal embeddings according to how many recent temporal facts involve an entity.
4 Experiments
Experiments evaluate TeMP and diverse baselines on three standard temporal knowledge graph completion benchmarks, using filtered ranking metrics. Results show strong performance overall, while component effects vary with dataset characteristics and temporal-pattern frequency.
- Experimental Setup: Experiments evaluate TeMP and baselines on three standard TKGC datasets using filtered Hits@1, Hits@3, Hits@10, and MRR metrics.The datasets are ICEWS14, ICEWS05-15, and GDELT; subject and object queries are evaluated separately.
- Comparative Study: TeMP-GRU and TeMP-SA achieve state-of-the-art Hits@10 results on all three datasets.Compared with prior state-of-the-art models, TeMP is 8.0% and 10.7% higher on ICEWS14 and ICEWS05-15, and 3.7% higher on GDELT than DE.
- Comparative Study: The TED rule-based baseline performs better than all existing neural models on GDELT by all reported measures.TED predicts missing facts by copying temporally nearby facts according to overlap and temporal distance.
- Comparative Study: SRGCN achieves the next-best Hits@10 results behind TeMP, consistent with leveraging multi-hop structural information specific to each time step.The paper presents this mechanism as a hypothesis explaining SRGCN’s strong performance.
- Exploration of Model Variations: 3.3% improvement results when ICEWS14 combines TeMP variants with both imputation and frequency-based gating, while GDELT shows no significant benefit from either component.On ICEWS05-15, gating improves performance by more than 1%, whereas adding imputation does not; GDELT has all entities active at every time step and many nearby reference facts.
- Fine-grained Error Analysis: TeMP’s temporal models show positive performance correlation with subject-relation frequency, whereas static models show relatively negative correlation.Gating slightly improves the vanilla TeMP model across frequencies and helps close SRGCN’s advantage on low-frequency queries.
5 Conclusion
TeMP jointly models multi-hop structural information and nearby temporal facts, adding imputation and frequency-based gating for temporal variability and sparsity. It achieves superior benchmark performance, while future work targets continuous TKGC and improved imputation for infrequent and inactive entities.
- TeMP jointly models multi-hop structural information and temporal facts from nearby time steps for temporal knowledge graph completion.
- Frequency-based gating and data imputation address temporal variability and sparsity in TKGC.
- 10.7% relative improvement over the state of the arts is reported across three benchmark datasets.
- Future work includes generalizing TeMP to continuous TKGC and improving imputation for infrequent and inactive entities.
A.1 Architecture Details Temporal Edge Dropout.
Temporal edge dropout is introduced to reduce TeMP’s overfitting to nearby temporal facts. The method randomly removes facts from the temporal window used to induce entity representations, with two dropout variants described.
- TeMP can increasingly copy temporal facts as temporal pattern frequencies rise, an effect described as overfitting to temporal facts.
- Temporal edge dropout randomly removes facts from the defined time window used to induce entity representations.
- The method drops a percentage of quadruples within each temporal snapshot or uses probabilities based on quadruple characteristics.
- A temporal edge dropout rate of 0.2 is used in all experiments.
A.2 Extended Imputation Formulation
Extended imputation defines future-based decay using the next time step when an entity is active, then combines decay rates for imputed representations. TeMP-SA treats representations for active and inactive entities as intrinsic imputation.
- For future imputation, t+ denotes the next time step at which entity ei becomes active after t.
- The bidirectional temporal recurrent model defines future-imputed representations analogously to Equations (9) and (10).
- At time t, both exponential decay rates are divided by two and renormalized to calculate the imputed representation of ei.
- TeMP-SA derives entity representations for active and inactive entities using Equations (4)–(7), treating this as intrinsic imputation.
A.3 Analogous Definition of Frequency Based Gating and Training Loss
For subject queries, the model derives entity representations analogously to the object-query formulation, uses object-relation frequencies for gating, and trains with subject and object query losses.
- Subject-query entity representations are derived analogously to Equations (11) and (12).
- The subject-query gating uses frequencies Fo = [ft_o,r] to combine static and temporal entity representations.
- The gating coefficients αss and αso are produced by an MLP and constrained to [0, 1].
- Negative subject entities are defined as entities s′ for which (s′, r, o) is not in D(t).
- The final training loss sums the losses for subject and object queries: L = Lsub + Lobj.
A.4 Detailed TED Formulation and Analysis
The TED baseline ranks reference entities by combining recurrence patterns, overlap with the query, and exponential temporal decay. Its analysis shows a dataset-dependent trade-off between recency and frequency, while recurring-event bias motivates pointer-style temporal completion.
- TED formulation: TED predicts missing facts by copying nearby temporal facts, weighting candidates by query overlap and temporal distance.Reference entities are collected from facts sharing elements with the queried quadruple and ranked using exponential decay.
- Reference sets: Reference entities are divided into subject and object sets according to shared relation-object, object, relation, or subject-relation patterns.The method prioritizes candidates sharing the same subject-relation pair over broader overlap patterns.
- Candidate priority: TED prioritizes same subject-relation candidates over candidates sharing only an object or relation, based on their presumed suitability for the current query.The priority order is (1) > (2) > (3), with overlapping tuples removed from lower-priority groups.
- Sensitivity analysis: TED performance peaks at σ = 0.1 on ICEWS datasets and σ = 1 on GDELT, revealing a trade-off between recency and frequency.Performance is low when σ is extremely large or small.
- Implications: TED results expose recurring-event bias in political event datasets, particularly GDELT, and suggest pointer-style TKGC as a future direction.Pointer-style completion would choose between copying historical entities and selecting an entity from the current snapshot.
A.5 Dataset Statistics and Characteristics
The datasets differ substantially in entity counts, temporal sparsity, and the amount of temporal information associated with each entity. ICEWS is much sparser than GDELT, making sparsity a more severe issue there.
- Dataset statistics: The three datasets contain 7,128, 10,488, and 500 entities respectively, with temporal sparsity severe on ICEWS and trivial on GDELT.The counts correspond to ICEWS14, ICEWS05-15, and GDELT.
- Temporal characteristics: Average associated temporal facts per entity are much lower in the ICEWS datasets than in GDELT.This difference demonstrates substantial temporal variability across the datasets.
A.6 Definitions for Evaluation Metrics
The evaluation section defines ranking metrics and describes the experimental analysis of temporal-frequency effects. It also reports implementation settings, dataset statistics, and evidence that TeMP and its gating mechanism exploit temporal information differently across datasets.
- Evaluation metrics: MRR and Hits@1, Hits@3, and Hits@10 evaluate ranking quality, with Hits@k measuring whether the correct entity appears among the k highest-ranked predictions.The reported values use filtered evaluation settings, and the Hit metrics in Table 3 are multiplied by 100.
- Model configuration: TeMP uses recurrent or self-attentive temporal models, two relational-convolution layers, temporal edge dropout, and ComplEx decoding.The implementation uses 128-dimensional embeddings and hidden states, with eight attention heads for TeMP-SA.
- Analysis results: TeMP-GRU shows more positive replication trends than static models, but performance drops when temporal patterns become large in reference-effect analysis.Frequency-based gating improves TeMP-GRU-Vanilla on ICEWS datasets, while its benefit is less obvious on GDELT.
- Conclusions: TeMP models are more advantageous in utilizing temporal facts, and frequency-based gating improves performance across different temporal-frequency patterns.The benefit of gating is less apparent on GDELT because that dataset is less affected by temporal sparsity and variability.