Source-linked AI summary
Integrate Temporal Graph Learning into LLM-based Temporal Knowledge Graph Model
He Chang, Jie Wu, Zhulin Tao, Yunshan Ma, Xianglin Huang, Tat-Seng Chua
TL;DR
Existing embedding-based methods for temporal knowledge graph forecasting are limited in modeling temporal patterns and aligning graph representations with language. TGL-LLM combines temporal graph learning, hybrid graph tokenization, and two-stage training; the authors report that it outperforms all baselines in TKGF.
Problem
Existing embedding-based TKGF methods insufficiently model temporal patterns and align graph representations with language ineffectively.
Source: §1 · p. 2Method
TGL-LLM combines temporal graph learning and temporally ordered hybrid graph tokens with two-stage fine-tuning on high-quality and diverse data.
Source: §1 · p. 2Results
Takeaways & Limitations
TGL-LLM provides a framework integrating temporal graph learning into an LLM-based model for temporal knowledge graph forecasting.
Source: §6 · p. 8The paper identifies broader forecasting subtasks, including relation and time-interval prediction, as areas for future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Temporal Knowledge Graph Forecasting (TKGF) aims to predict future events based on the observed events in history. Recently, Large Language Models (LLMs) have exhibited remarkable capabilities, generating significant research interest in their application for reasoning over temporal knowledge graphs (TKGs). Existing LLM-based methods have integrated retrieved historical facts or static graph representations into LLMs. Despite the notable performance of LLM-based methods, they are limited by the insufficient modeling of temporal patterns and ineffective cross-modal alignment between graph and language, hindering the ability of LLMs to fully grasp the temporal and structural information in TKGs. To tackle these issues, we propose a novel framework TGL-LLM to integrate temporal graph learning into LLM-based temporal knowledge graph model. Specifically, we introduce temporal graph learning to capture the temporal and relational patterns and obtain the historical graph embedding. Furthermore, we design a hybrid graph tokenization to sufficiently model the temporal patterns within LLMs. To achieve better alignment between graph and language, we employ a two-stage training paradigm to finetune LLMs on high-quality and diverse data, thereby resulting in better performance. Extensive experiments on three real-world datasets show that our approach outperforms a range of state-of-the-art (SOTA) methods.
1 INTRODUCTION
The introduction identifies limitations in temporal graph and LLM-based forecasting, then presents TGL-LLM as an approach combining temporal graph learning, hybrid graph tokenization, and two-stage fine-tuning. It reports experiments on three real-world datasets and says the method outperforms other approaches.
1 INTRODUCTION
- Existing embedding-based LLM methods insufficiently model temporal patterns and align graph representations with language effectively.The introduction also notes that temporal graph neural networks can be constrained by training-data dependence, extraction noise, and long-tail entities.Source: §1 · pp. 1–2
- TGL-LLM uses temporal graph learning and recent historical graph embeddings, then orders hybrid graph tokens alongside entity descriptions.The design aims to preserve temporal information from evolving facts and represent temporal patterns within the LLM.Source: §1 · p. 2
- A two-stage fine-tuning paradigm uses influence scores to select high-quality graph data and diversity sampling to expose varied temporal and relational patterns.The introduction says this training strategy is intended to improve graph-language alignment.Source: §1 · p. 2
- TGL-LLM outperforms state-of-the-art methods in experiments on three real-world datasets.The authors describe the results as demonstrating the framework’s effectiveness.Source: §1 · p. 2
2 RELATED WORK
The related-work section reviews conventional and LLM-based approaches to temporal knowledge graph forecasting. It describes rule-based and graph-neural methods, followed by text-based and embedding-based LLM methods.
2.1 Temporal Knowledge Graph Forecasting
- Rule-based methods learn or use temporal rules, while TGNN-based methods aggregate neighborhood information and model temporal evolution; both lines face limitations.Rule-based approaches can require substantial manual effort at scale, while graph-neural methods face noise and long-tail entities.Source: §2.1 · pp. 2–3
2.2 Large Language Models for TKGF
- Text-based LLM methods retrieve and flatten historical facts into prompts, whereas embedding-based methods align graph embeddings with language-model tokens.The review names GPT-NeoX-ICL, GENTKG, and CoH among text-based approaches, and GraphTranslator, TEA-LLM, and KoPA among embedding-based approaches.Source: §2.2 · p. 3
- The reviewed embedding-based approaches focus on static graph reasoning and can overlook temporal information and representation problems from noisy or long-tail training data.Text-based approaches are also limited by the information available in natural-language prompts.Source: §2.2 · p. 3
3 PRELIMINARY
The preliminary section formulates temporal knowledge graphs as timestamp-indexed graphs and forecasting as predicting a missing object from history. For LLM-based methods, it recasts prediction as a multiple-choice task over sampled candidate entities.
3 PRELIMINARY
- A fact is a subject-relation-object-timestamp quadruple, and facts at each timestamp form a graph G_t within the temporal graph G.The passage denotes the temporal graph as a collection of timestamp-specific graphs.Source: §3 · p. 3
- Classical TKGF predicts the missing object in a query (s, r, ?, t) using historical graphs preceding t.Source: §3 · p. 3
- LLM-based TKGF ranks a sampled candidate set as a multiple-choice question because the entity set and token limits make inputting every entity impractical.The passage presents this setting as supporting consistent evaluation and resembling scenarios with a limited number of likely candidates.Source: §3 · p. 3
4 METHOD
TGL-LLM combines temporal graph learning, hybrid graph tokenization, and two-stage training to model temporal patterns and align graph representations with language. Its pipeline builds recent historical graph embeddings, maps them into language-token space, and trains on selected high-quality and diverse data.
4.1 Temporal Graph Learning
- Recent entity embeddings from the graph model preserve evolving facts, while RGCN message passing captures relations and a GRU models temporal patterns.The graph model uses ConvTransE as its decoder and cross-entropy loss for optimization.Source: §4.1 · pp. 3–4
4.2.1 Temporal Graph Adapter.
- The Temporal Graph Adapter uses entity and relation adapters to map graph embeddings into language-token space for the LLM.The adapters project recent entity and relation representations into language-token space.Source: §4.2.1 · p. 4
4.2.2 Hybrid Prompt Design.
- Hybrid prompts place temporal graph tokens in temporal order for the subject and candidates, with a learned feature token distinguishing graph tokens from text.The prompts use temporal graph tokens rather than textual historical facts or static graph representations as subject and candidate descriptions.Source: §4.2.2 · p. 5
4.3.1 Data Pruning.
- Data pruning uses influence scores and stratified sampling to form a high-quality subset, then random sampling creates a smaller diversity subset.The diversity subset is kept significantly smaller to avoid affecting representation learning on the high-quality data.Source: §4.3.1 · p. 5
4.3.2 Prompt Tuning.
- Two-stage tuning first applies LoRA to the high-quality subset, then continues fine-tuning on the diversity subset to improve general reasoning ability.The paper states that this sequence better aligns temporal and global graph representations with token space and enhances robustness compared with random-sample training.Source: §4.3.2 · p. 5
5 EXPERIMENTS
Experiments on three POLECAT subsets show that TGL-LLM outperforms the evaluated conventional and LLM-based baselines. Ablations and model studies examine its graph-token design, training data selection, long-tail performance, history length, and training cost.
5.1.1 Datasets.
- Experiments use three country-specific POLECAT subsets, with test data extending beyond Llama2’s July 2023 training-data cutoff.POLECAT is a live dataset updated weekly; the test split covers November 2023 to April 2024.Source: §5.1.1 · p. 6, Table 1 · p. 6
5.1.2 Evaluation Protocols.
- Models are evaluated by accuracy in candidate sets of 4, 6, or 10 entities, against conventional, zero-shot, and fine-tuned LLM baselines.LLM-based methods use Llama2-7B-chat; the reported TGL-LLM training time is approximately 10 hours and may vary with dataset and candidate-set size.Source: §5.1.2 · p. 6, §5.1.3 · p. 6, §5.1.4 · p. 6
5.2 Performance Comparison (RQ1)
- TGL-LLM outperforms all evaluated Non-LLM and LLM-based baselines across three datasets, with all reported ANOVA p-values below 0.01.Fine-tuned LLM methods generally outperform closed commercial LLMs and Non-LLM methods; embedding-based methods outperform text-based ones across the datasets.Source: §5.2 · p. 7, Table 2 · p. 7
5.3.1 The impact of hybrid graph tokenization.
- Recent historical graph embeddings as graph tokens achieve the best performance among the tested Raw, Static, GRU, ConvTransE, and TGL-LLM variants.Static graph representations underperform those with temporal information, while deeper graph representations are harder for LLMs to comprehend and align with token embeddings.Source: §5.3.1 · p. 7, Table 3 · p. 7
5.3.2 The impact of two-stage training paradigm.
- The two-stage training paradigm outperforms the tested sampling variants, and removing either the high-quality or diversity subset performs worse than random sampling.Performance without diversity sampling remains strong on POLECAT-IS, which the authors suggest may reflect greater similarity between training and test patterns.Source: §5.3.2 · pp. 7–8, Table 4 · p. 8, Figure 3 · p. 8
5.4.1 Long-tail Issue.
- TGL-LLM has the strongest overall performance and smaller performance gaps across the evaluated long-tail settings.The test data are split into dense and sparse subsets; dense queries have objects appearing over 100 times in training, while remaining facts form the sparse subset.Source: §5.4.1 · p. 8, Figure 3 · p. 8
5.4.2 The impact of historical length.
- History length 5 performs best on POLECAT-IR and POLECAT-IS, while length 7 performs best on POLECAT-EG.Performance converges as history length increases, which the authors associate with added noisy information.Source: §5.4.2 · p. 8, Figure 4 · p. 8
5.4.3 Training Costs.
- Embedding-based methods have lower training costs than text-based methods, while KoPA has the lowest cost and TGL-LLM offers enhanced TKG reasoning capabilities.The cost comparison is reported for POLECAT-IR; the authors attribute the embedding-based cost reduction to integrating graph embeddings.Source: §5.4.3 · p. 8, Table 5 · p. 8
6 CONCLUSION AND FUTURE WORK
TGL-LLM combines hybrid graph tokenization and two-stage training to address temporal-pattern modeling and graph-language alignment. The authors report that it significantly outperforms all baselines in TKGF and identify several directions for future work.
6 CONCLUSION AND FUTURE WORK
- Hybrid graph tokenization and two-stage training target insufficient temporal-pattern modeling and ineffective graph-language alignment, respectively.The framework integrates temporal graph learning into an LLM-based temporal knowledge graph model.Source: §6 · p. 8
- TGL-LLM significantly outperforms all baselines in TKGF, according to the reported empirical results.The conclusion describes the results as demonstrating the framework’s effectiveness and superior performance.Source: §6 · p. 9
- Future work includes replacing the simple MLP adapter, enabling generative forecasting, and adding local graph representations alongside global ones.The authors also plan to improve performance on relation, time-interval, long-horizon, and event-distribution prediction.Source: §6 · p. 9
A.1 Details on Dataset Construction
The benchmark dataset is constructed from POLECAT news events spanning January 2018 to April 2024 to address possible information leakage from LLM pretraining. Its construction filters countries and incomplete records, expands multi-entity events, removes duplicates, and screens entities.
A.1 Details on Dataset Construction
- The benchmark is based on POLECAT, a publicly available event dataset extracted from internet news sources starting in 2018, including post-training-cutoff events, to address potential test-set information leakage in LLM event prediction.Source: §A.1 · p. 10
- Events from January 1, 2018, to April 21, 2024, were retained when Actor Country, Recipient Country, or Country named Iran, Israel, or Egypt.Source: §A.1 · p. 10
- Records were removed when both actor-name fields or both recipient-name fields were missing or empty.The raw-name fields provide fallback entity text when mapped names are unavailable.Source: §A.1 · p. 10
- Events with multiple actors or recipients were split into separate rows for each actor-recipient combination, using raw names as fallbacks when needed.Source: §A.1 · p. 10
- Duplicate events were removed using Actor Name, Recipient Name, Event Type, Event Mode, and Event Date as the identifying fields.Source: §A.1 · p. 10
- GPT-3.5-Turbo screened each unique entity; entities marked irrelevant were removed only when they appeared in five or fewer events.The model judged whether entities were meaningful, and entities marked relevant were kept.Source: §A.1 · p. 10, Figure 5 · p. 10
A.2 Dataset Statistics
The dataset statistics describe its time span and the distributions of events, articles, entities, and countries. The section reports visual summaries of these distributions without giving their specific values in the supplied passages.
A.2 Dataset Statistics
- The dataset spans January 2018 to April 2024 and is based on filtered events extracted from news articles.Source: §A.2 · p. 10
- Event and news-article counts are visualized by year and month, with the distinction that one article may contain multiple events.The heatmaps are intended to show temporal patterns such as peaks or trends in event occurrences.Source: §A.2 · p. 11, Figure 6 · p. 11, Figure 7 · p. 11
- A bar chart displays the 20 most frequent entities, counted across the Actor Name and Recipient Name fields.The section describes these entities as individuals, organizations, or groups involved in events in Iran, Israel, and Egypt.Source: §A.2 · p. 11, Figure 8 · p. 11
- A bar chart counts events involving Egypt, Iran, and Israel using entities identified in the actor and recipient fields.Entity expansion can introduce other related entities despite the country filtering.Source: §A.2 · p. 11, Figure 9 · p. 11
A.3 Statistical Analysis of Generative vs. MCQ Settings.
The analysis finds that LLM predictions in the generative setting largely reuse entities present in the prompt, and performance remains strongly influenced by that context. The authors conclude that changing the setting alone does not yield genuine generation, which remains challenging without context.
A.3 Statistical Analysis of Generative vs. MCQ Settings
- In both ICEWS and GDELT, most entities generated by GenTKG already appeared in the prompt, so entirely new entity generation was rare.The analysis compares generative predictions with ground-truth entities and context-listed entities.Source: §A.3 · p. 11, Table 6 · p. 12
- The model’s performance remains heavily influenced by context prompts, making the generative setting no different from the MCQ setting in this respect.The passage relates the dataset performance difference to the proportion of correct answers included in context prompts.Source: §A.3 · p. 12
- Pure generation without supplied context remains extremely challenging for LLMs, and changing the problem setting alone does not produce genuine generation.Source: §A.3 · p. 12
A.4 Baseline Implementation Details
The baseline implementations cover methods for relational and temporal graph modeling, as well as zero-shot and fine-tuned LLM approaches.
A.4 Baseline Implementation Details
- DistMult models entity relationships multiplicatively, ConvTransE applies convolution to entity–relation embeddings, and RGCN uses relation-specific transformations with decomposition.RGCN uses basis and block diagonal decomposition to handle many relations and prevent overfitting.Source: §A.4 · p. 12
- RENET, REGCN, and HisMatch represent temporal baselines through sequential and graph modeling, recurrent temporal dependencies, and query-candidate matching, respectively.REGCN also uses a static graph to improve initial embeddings.Source: §A.4 · p. 12
- Zero-shot methods use CoH prompts with GPT-3.5-turbo and GPT-4o-mini, while fine-tuned baselines include CoH, GenTKG, and KoPA.CoH uses schema- and entity- or relation-augmented historical facts; GenTKG uses temporal logic rules; KoPA turns structural embeddings into textual prefixes.Source: §A.4 · p. 12
A.5 Instruction
The instruction for hybrid graph tokenization specifies the task definition and the formats for the query and input.
A.5 Instruction
- The hybrid graph tokenization instruction includes a textual task definition and specifies the query and input formats.Figure 10 presents the instruction.Source: §A.5 · p. 12