PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models

Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, Dongsheng Li

arXiv:2402.16568v2cs.CL

TL;DR

TKGQA is challenging because questions can contain hidden temporal constraints, and LLM applications to this task remain relatively unexplored. GenTKGQA retrieves subgraphs using LLM knowledge and integrates graph signals with language representations for answer generation. The paper reports improved inference on complex temporal questions.

  • Problem

    TKGQA must answer questions with explicit or implicit temporal constraints over dynamic structured knowledge, while LLM applications to the task remain relatively unexplored.

    Source: §1 · pp. 1–2

  • Method

    GenTKGQA uses LLM knowledge to retrieve temporally and structurally relevant subgraphs, then uses virtual knowledge indicators and instruction tuning to integrate graph and text representations.

    Source: §1 · p. 2

  • Results

    The paper reports that GenTKGQA improves inference performance on complex temporal question types.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The conclusion presents retrieval followed by answer generation as a way to guide LLMs through temporal order and structural dependencies in TKG facts.

    Source: §6 · p. 9

    Most current TKGQA datasets involve single-hop facts; the authors identify multi-hop complex temporal inference as future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM's intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results on two widely used datasets demonstrate the superiority of our model.

1 Introduction

TKGQA asks models to answer questions with explicit or implicit temporal constraints using dynamic structured facts, a task complicated by the need to find relevant facts across temporal and structural dimensions. GenTKGQA addresses these challenges with two stages: subgraph retrieval using LLM knowledge and answer generation that integrates graph and text representations.

1 Introduction

  • TKGQA answers natural-language questions with explicit or implicit temporal constraints over dynamic structured knowledge.Real-world facts change over time; for example, Obama's presidency is valid only for [2009, 2016].Source: §1 · p. 1
  • Retrieving relevant facts is difficult because questions may leave both structural links and temporal bounds implicit, making broad entity-based searches noisy.For “after Obama,” the relation and the restriction after 2016 must both be identified.Source: §1 · p. 2
  • GenTKGQA uses LLM knowledge without extra training to mine question relations and time constraints, reducing subgraph search in both dimensions.Its retrieval phase decomposes the task into relation ranking and time mining.Source: §1 · p. 2
  • Virtual knowledge indicators bridge temporal-subgraph GNN signals and LLM text representations, supporting deeper structural and temporal reasoning through instruction tuning.This is the framework's answer-generation phase for complex temporal questions.Source: §1 · p. 2
  • GenTKGQA consistently outperforms embedding-based methods on Hits@1 across two widely used datasets.The introduction reports this as the framework's experimental result.Source: §1 · p. 2

2 Related Work

Prior work addresses temporal KGQA, temporal reasoning over language, and integration of language models with knowledge graphs. The paper distinguishes its focus on retrieving structured temporal knowledge for LLM-based TKGQA, an area it describes as little explored.

2.1 TKGQA Methods

2.2 LMs for Temporal Question Answering

2.3 LMs for KG Question Answering

3 Preliminaries

The paper defines temporal knowledge graphs as timestamped relational graphs and frames TKGQA as extracting an entity or timestamp that answers a natural-language question. It distinguishes in-context learning, which uses demonstrations without extra training, from instruction tuning, which trains a model to follow task instructions.

3 Preliminaries

  • A temporal knowledge graph is a directed, multi-relational graph whose timestamped facts are quadruples (subject, relation, object, timestamp).TKGQA extracts an entity or timestamp from the graph to answer a natural-language question.Source: §3 · p. 3
  • In-context learning concatenates instruction, demonstrations, and question without extra training, while instruction tuning uses instruction, question, and answer as its input sequence.The paper denotes these sequences as S = I : D : Q and S = I : Q : A, respectively.Source: §3 · p. 3

4 Method

GenTKGQA divides temporal question answering into subgraph retrieval and answer generation, using LLM knowledge to narrow retrieval and graph signals to support structure-aware temporal inference. Its generation stage connects temporal GNN representations to LLM text through virtual knowledge indicators.

4 Method

  • GenTKGQA uses an in-context subgraph retrieval phase followed by instruction-tuned answer generation, drawing on internal LLM knowledge first and external graph knowledge for temporal inference second.The two phases are described as addressing retrieval and answer generation, respectively.Source: §4 · pp. 3–4

4.1 Subgraph Retrieval

  • Relation ranking and time mining split retrieval into structural and temporal subtasks, allowing a small number of samples to retrieve the subgraph.The structural scope is identified through relations, while time mining addresses temporal scope.Source: §4.1 · p. 4

4.1.1 Relation Ranking

  • The LLM ranks candidate relations and returns the top k relevant relations, which can bridge entities identified in the question.Candidate relations are provided as a linearized set.Source: §4.1.1 · p. 4

4.1.2 Time Mining

  • Time mining uses explicit question times or infers implicit constraints from question wording and fact validity, then retrieves facts within the resulting temporal scope.For example, “after Obama” implies a restriction after 2016 given the validity period of the Obama-president fact.Source: §4.1.2 · p. 4

4.2 Answer Generation

  • For answer generation, the model extracts structural and temporal information with pretrained temporal GNN embeddings, bridges graph and text representations with virtual indicators, and fine-tunes an open-source LLM.The approach targets understanding temporal order and structural dependencies among retrieved facts.Source: §4.2 · p. 4

4.2.1 Temporal GNN

  • The temporal GNN initializes entity, relation, and time representations, then uses neighboring-node message passing with relation and timestamp embeddings to encode temporal and structural information.Its distinguishing feature is incorporating temporal embeddings when capturing correlations between neighboring nodes.Source: §4.2.1 · pp. 4–5

4.2.2 Virtual Knowledge Indicators

  • Three virtual indicators, [SUB], [REL], and [OBJ], link graph signals to prompt text; pooled structural representations and time embeddings enrich them before projection into the LLM’s text space.The time embeddings encode the subgraph’s minimum and maximum times.Source: §4.2.2 · pp. 5–6

5 Experiments

Experiments evaluate GenTKGQA on CronQuestions and TimeQuestions using Hits@1 and Hits@10, comparing it with embedding-based and language-model baselines. Results cover benchmark performance, component ablations, parameter sensitivity, and qualitative comparisons with ChatGPT using retrieved TKG facts.

5.1 Datasets, Metrics and Baselines

  • CronQuestions contains 410K question-answer pairs, while TimeQuestions has 13.5K; evaluation uses Hits@1 and Hits@10 against embedding and language-model baselines.CronQuestions includes simple and complex reasoning types; TimeQuestions includes Explicit, Implicit, Temporal, and Ordinal types.Source: §5.1 · p. 6

5.2 Main Results

  • On CronQuestions, GenTKGQA outperforms baselines overall, with Hits@1 gains of 11.3% on Complex and 9.6% on Time over the second-best method.It also achieves nearly 100% on Simple questions.Source: §5.2 · p. 6, Table 1 · p. 7
  • On TimeQuestions, GenTKGQA improves Hits@1 by 1.3% on Implicit and 11.6% on Ordinal questions, and performs significantly on complex questions across datasets.The paper reports gains over baseline methods, but does not give a numerical overall comparison here.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Ablation Study

  • Removing relevant subgraphs sharply reduces performance, and random fact selection also hurts results, supporting the retrieval stage’s importance.GenTKGQA retains some structured knowledge after training and can improve inference even when subgraphs are omitted at inference.Source: Table 3 · p. 7, §5.3 · p. 7
  • Replacing the T-GNN with temporal embeddings slightly lowers results, while removing virtual knowledge indicators degrades performance.The paper attributes the T-GNN result to its structural-information perception and says the indicators bridge distinct representations.Source: §5.3 · pp. 7–8, Table 3 · p. 7

5.5 Qualitative Results

  • With retrieved facts, ChatGPT handles simple questions but struggles with complex temporal order and implicit time; GenTKGQA performs well on both types.GenTKGQA can still generate irrelevant answers when producing multiple answers, as can other LLMs.Source: Table 4 · p. 8, §5.5 · pp. 8–9

5.4 Sensitivity Analysis

  • At 20% of training data, GenTKGQA exceeds the second-best model by 32% on complex-question Hits@1 and consistently leads as training data expands.The paper attributes this few-shot result to the model’s intrinsic knowledge.Source: §5.4 · p. 8, Figure 3 · p. 8
  • With n=3 retrieved facts, performance is poor; performance rises slightly at n=10, which the authors select as the setting.They note that too few facts provide insufficient context, while more facts may introduce noise.Source: §5.4 · p. 8, Figure 3 · p. 8

6 Conclusion

GenTKGQA uses subgraph retrieval and answer generation for temporal question answering over knowledge graphs. It mines temporal constraints and structural links from questions, then uses virtual knowledge indicators to connect graph signals with LLM text representations.

6 Conclusion

  • GenTKGQA is a two-stage framework for temporal question answering on temporal knowledge graphs.Its stages are subgraph retrieval and answer generation.Source: §6 · p. 9
  • Subgraph retrieval mines temporal constraints and structural links from questions, reducing the search space in both dimensions.The paper uses in-context learning with a small number of samples to complete retrieval for the dataset.Source: §6 · p. 9
  • At answer generation, instruction tuning and virtual knowledge indicators connect retrieved-fact structure with LLM text representations.The indicators bridge subgraph neural information and text representations to support understanding of temporal order and structural dependencies.Source: §6 · p. 9
  • Experiments show that GenTKGQA can use LLMs to solve complex TKGQA question types.The authors state that the results validate the adequacy of their motivation.Source: §6 · p. 9

Limitations

The evaluated CronQuestions questions involve multiple facts, but each fact connects entities through a single hop. The authors identify multi-hop inference and ChatGPT template consistency as scope considerations.

Limitations

  • CronQuestions questions may involve multiple facts, but each fact’s inter-entity connection is single-hop, so the model uses k=1.The authors say this setting achieves the best results.Source: Limitations · p. 9
  • Most current TKGQA datasets involve single-hop facts, leaving multi-hop complex temporal inference for future exploration.Source: Limitations · p. 9
  • Different in-context templates for ChatGPT may produce results incompletely consistent with manual results, though the authors say this does not affect their conclusions.They also cite prior results showing poor ChatGPT temporal question answering (Tan et al., 2023; Chen et al., 2023a).Source: Limitations · p. 9

Ethics Statement

The work uses publicly available datasets and open-source language models developed for temporal QA and uses them with their initial intention. The authors state that the work presents no potential risk under ACL’s ethics policy.

Ethics Statement

  • The experiments use publicly available datasets and language models from open sources.Source: Ethics Statement · p. 9
  • The dataset and language models are used according to their initial intended purposes, and answer generation involves no toxic content.Source: Ethics Statement · p. 9
  • The authors consider the work consistent with ACL’s ethics policy and state that it presents no potential risk.Source: Ethics Statement · p. 9

A Dataset Statistics and Metrics

The experiments use CronQuestions and TimeQuestions, with TimeQuestions converted into a temporal knowledge graph format and its times restricted to years. Evaluation uses Hits@1 and Hits@10, and the appendix gives training, validation, and test question counts.

A Dataset Statistics and Metrics

  • The experiments use the CronQuestions and TimeQuestions datasets.TimeQuestions was reformatted with facts represented as subject, relation, object, and start/end times, with times restricted to years.Source: A Dataset Statistics and Metrics · p. 12
  • Hits@K measures the share of test questions whose answer ranks at most K in the candidate list; the reported metrics are Hits@1 and Hits@10.Source: A Dataset Statistics and Metrics · p. 12
  • The training, validation, and test sets contain 6,970, 3,236, and 3,237 questions, respectively.The total exceeds the number of questions because some questions belong to multiple categories.Source: A Dataset Statistics and Metrics · p. 12

B Baselines and Implementation Details

The baselines span temporal-graph embedding and language-model approaches, while GenTKGQA uses ChatGPT for retrieval-stage mining and instruction-tunes Llama 2-7B for answer generation. Experiments average three runs.

Implementation details

  • GenTKGQA uses ChatGPT to mine temporal constraints and entity links, then instruction-tunes Llama 2-7B with up to 10 relevant facts for answer generation.The setup uses LoRA with rank 64 and four epochs.Source: B Baselines and Implementation Details · p. 12
  • The experiments use three-run averages, with Llama 2-7B fine-tuned using LoRA and AdamW on NVIDIA A100 GPUs.The fixed batch size is 8 and learning rate is 3e-4.Source: B Baselines and Implementation Details · p. 12

Baselines

  • EmbedKGQA ignores timestamps during pretraining and uses random time embeddings for QA, while EaE enhances question representations with TKG embeddings.CronKGQA uses language-model question embeddings and a TKG embedding-based scoring function; EntityQR and TempoQR build on EaE, with TempoQR adding temporal information.Source: B Baselines and Implementation Details · p. 12
  • TMA enhances fact retrieval and adaptive fusion, TSQA uses contrastive learning for sensitivity to time-relation words, and TwiRGCN processes TKGQA with an RGCN.BERT and RoBERTa baselines use either a [CLS]-based prediction head or question embeddings combined with annotated entity and time embeddings.Source: B Baselines and Implementation Details · p. 13

ChatGPT baseline

C Prompt Template

The appendix provides separate prompts for relation ranking and time mining, alongside templates for instruction tuning and the ChatGPT baseline.

Prompt templates

Loading 2402.16568v2…