PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Knowledge Graph Reasoning with Historical Contrastive Learning

Yi Xu, Junjie Ou, Hui Xu, Luoyi Fu

arXiv:2211.10904v2cs.AIcs.LG

TL;DR

Temporal knowledge graph forecasting must handle future events that lack historical interaction, because existing methods often depend on recurring or periodic patterns. CENET jointly models historical and non-historical dependency, uses historical contrastive learning to identify relevant entities, and applies mask-based inference. Across five benchmark graphs, it outperforms existing methods in most metrics, with especially strong Hits@1 results on event-based datasets.

  • Problem

    Existing temporal knowledge graph reasoning methods depend heavily on recurring or periodic events, making future events involving entities with little historical interaction difficult to infer.

  • Method

    CENET learns historical and non-historical dependency, trains contrastive query representations, and uses a binary mask during inference to identify related entities.

  • Results

    CENET outperforms existing methods in most metrics across five benchmark graphs, especially for Hits@1.

  • Takeaways & Limitations

    Jointly investigating historical and non-historical information enables CENET to target both repetitive or periodic events and potential new events.

  • Takeaways & Limitations

    A poor second-stage historical contrastive-learning classifier can wrongly mask expected object entities and deteriorate performance.

Abstract

from arXiv · show

Temporal knowledge graph, serving as an effective way to store and model dynamic relations, shows promising prospects in event forecasting. However, most temporal knowledge graph reasoning methods are highly dependent on the recurrence or periodicity of events, which brings challenges to inferring future events related to entities that lack historical interaction. In fact, the current moment is often the combined effect of a small part of historical information and those unobserved underlying factors. To this end, we propose a new event forecasting model called Contrastive Event Network (CENET), based on a novel training framework of historical contrastive learning. CENET learns both the historical and non-historical dependency to distinguish the most potential entities that can best match the given query. Simultaneously, it trains representations of queries to investigate whether the current moment depends more on historical or non-historical events by launching contrastive learning. The representations further help train a binary classifier whose output is a boolean mask to indicate related entities in the search space. During the inference process, CENET employs a mask-based strategy to generate the final results. We evaluate our proposed model on five benchmark graphs. The results demonstrate that CENET significantly outperforms all existing methods in most metrics, achieving at least $8.3\%$ relative improvement of Hits@1 over previous state-of-the-art baselines on event-based datasets.

1 Introduction

Temporal knowledge graphs represent evolving facts and support forecasting future events, but existing approaches struggle with new events lacking historical interaction. CENET addresses this by jointly modeling historical and non-historical dependencies with contrastive learning.

  • Temporal Knowledge Graphs: Temporal knowledge graphs extend triples with timestamps, forming snapshots whose facts co-occur and evolve over time.TKG facts are represented as (s, p, o, t).
  • Forecasting Task: Event forecasting predicts missing entities in future queries such as (s, p, ?, t) for timestamps absent from training data.This task is also called graph extrapolation.
  • Challenge: Existing methods often rely on historical events, making repetitive or periodic events easier to predict than new events.New events account for about 40% of events in an event-based crisis early-warning dataset and have fewer temporal interaction traces.
  • CENET: CENET models both historical and non-historical events to mine potential temporal patterns beyond directly observed history.The model treats visible historical events as only part of the information influencing current events.
  • CENET: CENET applies historical contrastive learning to identify significant entities and uses the resulting approach for event forecasting.The paper presents contrastive learning as a framework for distinguishing highly correlated entities.
  • Results: CENET outperforms state-of-the-art temporal knowledge graph models in event forecasting on the reported benchmark evaluation.The introduction states that experiments were conducted on five public benchmark graphs.

2 Related Work

Temporal knowledge graph reasoning distinguishes interpolation from extrapolation, while contrastive learning provides a way to separate related representations. Existing forecasting methods differ in their temporal modeling, explainability, scalability, and event-frequency assumptions.

  • Temporal Knowledge Graph Reasoning: Interpolation completes missing events within observed timestamps, whereas extrapolation predicts possible events after the final observed timestamp.Extrapolation infers a missing subject or object for a query at t > t_n.
  • Temporal Knowledge Graph Reasoning: Interpolation-oriented models such as HyTE, TeMP, and ChronoR are not designed to predict future events outside the observed interval.Their task is to infer missing relations within observed data.
  • Forecasting Methods: Forecasting approaches use diverse mechanisms, including evolving embeddings, explainable evidence, neural ordinary differential equations, copy generation, reinforcement learning, and graph or recurrent architectures.The related work characterizes different assumptions and application scopes across these methods.
  • Contrastive Learning: Contrastive learning distinguishes instances or representations using positive and negative examples, and its learned representations can improve downstream classification.The cited formulation uses augmented minibatch examples and a temperature parameter for learning from hard negatives.
  • CENET Architecture: CENET combines historical and non-historical dependency modeling with contrastive entity identification and mask-based inference.Its architecture uses learned distributions and a boolean mask to generate final reasoning results.

3 Method

CENET forecasts future TKG events by modeling both historical and non-historical entity dependencies, then uses historical contrastive learning to select the relevant search space. Its inference combines entity probabilities with a learned mask, while soft-mask substitution provides a more conservative alternative when classification is unreliable.

  • Historical and Non-historical Dependency: CENET models historical and non-historical dependencies to capture both recurring events and potential new events.It uses query–entity similarity and frequency information with a copy mechanism, emphasizing historical or non-historical entities through separate context vectors.
  • Historical and Non-historical Dependency: Historical dependency scores are strengthened for previously associated entities, while non-historical dependency scores focus on entities without such historical interactions.Historical entities are identified from prior subject–predicate interactions; non-historical entities are those outside that set.
  • Historical and Non-historical Dependency: CENET trains separate dependency objectives and combines their softmax outputs to obtain predicted probabilities over object entities.The cross-entropy-like objective separates the ground-truth object from alternatives, and the maximum-valued entity is selected by each component.
  • Historical Contrastive Learning: Historical contrastive learning groups query representations by whether the missing object is historical, then trains a binary classifier to predict that category.The first stage learns supervised contrastive representations, while the second freezes the corresponding parameters and trains a classifier with cross-entropy loss.
  • Parameter Learning and Inference: During inference, a boolean mask reweights entity probabilities before selecting the highest-probability object.The hard-mask reduces the search space, whereas the soft-mask yields a more conservative distribution; an inaccurate classifier can otherwise mask expected entities and hurt performance.

4 Experiments

CENET is evaluated on five benchmark datasets against static and temporal reasoning models, with results reported for event-based and public knowledge graphs. It outperforms baselines in most metrics, particularly Hits@1.

  • Experimental Settings: CENET is evaluated on five benchmarks comprising three event-based TKGs and two public KGs.The event-based datasets are ICEWS18, ICEWS14, and GDELT; the public KGs are WIKI and YAGO.
  • Baselines: CENET is compared with 15 up-to-date static and temporal knowledge graph reasoning models.
  • Evaluation Metrics: CENET’s evaluation reports filtered MRR and Hits@1/3/10, with Hits metrics measuring correct predictions ranked within the corresponding top-k positions.
  • Results on Event-based TKGs: 8.25%, 8.48%, and 20.80% are CENET’s Hits@1 improvements on ICEWS18, ICEWS14, and GDELT, respectively.CENET is on par with HIP in Hits@10 across the three event-based datasets.
  • Results on Public KGs: 23.68% MRR, 25.77% Hits@1, and 7.08% Hits@3 are CENET’s improvements over SOTA on public KGs.CENET outperforms baselines in all metrics on WIKI and YAGO.

4.3 Ablation Study

The ablation study tests CENET’s historical and non-historical dependencies, contrastive learning, mask-based inference, and hyper-parameter choices on ICEWS18 and YAGO. Results support using both dependencies and the full contrastive-learning pipeline, while α and λ behave differently across dataset types.

  • Dependency Ablation: CENET-his outperforms CENET-nhis, but the authors conclude that both historical and non-historical dependencies are necessary.CENET-his uses only historical dependency, whereas CENET-nhis retains only non-historical dependency.
  • Contrastive Learning Ablation: Removing stage 1 contrastive learning reduces YAGO results by about 7%.The CENET-Lce variant retains Lce but removes Lsup; its ICEWS18 results remain close to CENET.
  • Mask-based Inference Ablation: Removing stage 2 and the binary classifier causes Hits@1 drops of 1.7% on ICEWS18 and 3.8% on YAGO.This variant discards mask-based inference.
  • Hyper-parameter Analysis: α balances Lce and Lsup, with Lce contributing more to event-based TKGs and Lsup being more favorable for public KGs.The authors set α to 0.2 because α = 0 prevents obtaining the final inference probabilities.
  • Hyper-parameter Analysis: Increasing λ improves YAGO results but worsens ICEWS18 results, so λ is set to 2.

5 Conclusion and Future Work

CENET is a temporal knowledge graph representation learning model for event forecasting that uses historical and non-historical dependencies. It significantly outperforms existing methods on most metrics, especially Hits@1.

  • CENET is a temporal knowledge graph representation learning model for event forecasting.
  • CENET identifies significant entities by learning a distribution over the whole entity set from historical and non-historical dependencies.
  • CENET significantly outperforms existing methods on most metrics, especially Hits@1.
  • Future work includes exploring contrastive learning for knowledge graphs, including the discovery of more reasonable contrastive pairs.

A Historical Contrastive Learning

Historical contrastive learning in CENET has two stages: contrastive representation learning followed by binary-classifier training. The stages use contrastive loss and cross-entropy loss, respectively.

  • Historical contrastive learning consists of learning contrastive representations and training a binary classifier.
  • Stage 1 learns representations using a contrastive loss.
  • Stage 2 trains a binary classifier using cross-entropy loss.

B Complexity Analysis

The historical contrastive learning framework has training complexity O(|M||E| + |M|2), combining dependency learning with supervised contrastive similarity computation.

  • Learning historical and non-historical dependency for minibatch M has time complexity O(|M||E|).
  • Training contrastive representations has time complexity O(|M|2) for computing the similarity matrix.
  • The overall training complexity is O(|M||E| + |M|2).

C Details of Datasets

The evaluation uses five benchmark datasets: three event-based temporal knowledge graphs and two public knowledge graphs. New events are prevalent in the event-based datasets but occur at lower rates in WIKI and YAGO.

  • The benchmark suite contains five datasets: ICEWS18, ICEWS14, GDELT, WIKI, and YAGO.
  • ICEWS18, ICEWS14, and GDELT are event-based temporal knowledge graphs where a single event may happen at any time.
  • WIKI and YAGO contain temporally associated facts that last a long time and hardly occur in the future.
  • New events form a large proportion of event-based datasets, while their rates are below 20% in WIKI and YAGO.

D More Baseline Results

CENET is evaluated against static and temporal baseline models, including nine additional approaches spanning both categories.

  • CENET is compared with nine additional models comprising static and temporal approaches.The listed static baselines include RotatE and CompGCN, while temporal baselines include HyTE, Know-Evolve, and TTransE.
  • The static comparison includes RotatE and CompGCN.
  • The temporal comparison includes HyTE, Know-Evolve, TTransE, TA-DistMult, DySAT, DyRep+MLP, and R-GCRN+MLP.

E Case Study

The case study examines three North Korea queries to show how CENET combines historical dependency with masking to make predictions, including a query without historical events.

  • Case Study: The case study selects three representative queries with North Korea as the subject entity.
  • Case Study: For Halt negotiations, frequent historical links to the United States support the correct prediction, while masking suppresses unrelated entities such as Russia and Singapore.
  • Case Study: For Intent to cooperate, CENET selects South Korea despite equally infrequent historical occurrences involving other entities.
  • Case Study: For Express accord, CENET correctly predicts the result despite no historical events for the query from timestamp 0.
Loading 2211.10904v2…