PaperlayerView exact revision on arXiv

Source-linked AI summary

Question Answering Over Temporal Knowledge Graphs

Apoorv Saxena, Soumen Chakrabarti, Partha Talukdar

arXiv:2106.01515v1cs.LG

TL;DR

Temporal KGQA lacks broad-coverage datasets and must handle temporally scoped facts, while existing KGQA methods largely target non-temporal settings. The paper introduces CRONQUESTIONS and CRONKGQA, combining a large temporal KGQA dataset with temporal KG embeddings. CRONKGQA outperforms all baselines, while complex reasoning remains an area for improvement.

  • Problem

    Temporal KGQA is relatively unexplored, with limited broad-coverage datasets and existing KGQA methods largely designed for non-temporal KGs.

  • Method

    The paper introduces CRONQUESTIONS and CRONKGQA, an enhancement of EmbedKGQA that uses temporal KG embeddings with transformer-based language modeling.

  • Results

    CRONKGQA outperforms all baselines on CRONQUESTIONS and achieves high accuracy on simple temporal reasoning questions.

  • Takeaways & Limitations

    KG embeddings can be effectively used for Temporal KGQA, although substantial improvement remains possible for complex reasoning questions.

  • Takeaways & Limitations

    CRONQUESTIONS has lower-than-natural linguistic variety because its questions are generated automatically, and complex Temporal KGQA remains difficult.

Abstract

from arXiv · show

Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by providing temporal scopes (start and end times) on each edge in the KG. While Question Answering over KG (KGQA) has received some attention from the research community, QA over Temporal KGs (Temporal KGQA) is a relatively unexplored area. Lack of broad coverage datasets has been another factor limiting progress in this area. We address this challenge by presenting CRONQUESTIONS, the largest known Temporal KGQA dataset, clearly stratified into buckets of structural complexity. CRONQUESTIONS expands the only known previous dataset by a factor of 340x. We find that various state-of-the-art KGQA methods fall far short of the desired performance on this new dataset. In response, we also propose CRONKGQA, a transformer-based solution that exploits recent advances in Temporal KG embeddings, and achieves performance superior to all baselines, with an increase of 120% in accuracy over the next best performing method. Through extensive experiments, we give detailed insights into the workings of CRONKGQA, as well as situations where significant further improvements appear possible. In addition to the dataset, we have released our code as well.

1 Introduction

Temporal KGQA extends KGQA with time-scoped facts, temporal answers, and potentially complex temporal reasoning. The paper introduces CRONQUESTIONS and CRONKGQA to address limited datasets and evaluate temporal KGQA methods.

  • 1 Introduction: Temporal KGs annotate graph edges with time durations, unlike regular KGs without temporal annotations.Temporal scopes may include start and end times and can be non-contiguous.
  • 1 Introduction: Temporal KGQA may require answering with entities or time durations while performing complex temporal reasoning.
  • 1 Introduction: Temporal KG embeddings represent entities, relations, and timestamps in a low-dimensional vector space for temporal KG tasks.The paper investigates their use for Temporal KGQA compared with non-temporal embeddings and methods without KG embeddings.
  • 1 Introduction: CRONQUESTIONS provides a temporal KG with 125k entities and 328k facts, plus 410k natural-language questions requiring temporal reasoning.Its design targets training as well as evaluation and follows principles requiring temporal annotations, temporal reasoning, and sufficient labeled instances.
  • 1 Introduction: CRONKGQA is proposed as an enhancement of EmbedKGQA after language-model and hybrid baselines proved unsuitable for temporal reasoning.The supplied results state that CRONKGQA achieves high accuracy on simple temporal reasoning but falls short on more complex questions.
  • 1 Introduction: CRONQUESTIONS leaves ample scope for improving complex Temporal KGQA.

2 Related work

Existing KGQA datasets and methods largely target non-temporal settings, while temporal QA resources are limited or built on non-temporal KGs. The paper positions Temporal KGQA as requiring methods and datasets that directly represent temporal scopes and reasoning.

  • 2 Related work: Existing KGQA datasets cover single-fact, multi-hop, conjunction, and comparison questions but are based on non-temporal KGs.
  • 2 Related work: TORQUE studies temporal reading comprehension with a passage and five-choice answers, unlike KGQA without context and with potentially hundreds of thousands of entity answers.
  • 2 Related work: TempQuestions filters temporal questions from existing datasets using trigger words and other constraints, but contains only 1271 questions and uses a non-temporal FreeBase subset.
  • 2 Related work: Recent KGQA algorithms generally handle facts represented as subject-relation-object triples, whereas Temporal KGs add start and end times.The paper describes extending these algorithms to temporal KGs as non-trivial.
  • 2 Related work: TEQUILA decomposes questions into non-temporal sub-questions and temporal constraints before retrieving answers and reasoning over temporal intervals.Its limitations include prespecified decomposition templates, assumptions about temporal constraints on entities, and no direct application to temporally scoped facts.

3 CRONQUESTIONS: The new Temporal KGQA dataset

CRONQUESTIONS combines a temporally annotated knowledge graph with natural-language questions requiring temporal reasoning, using structured templates and paraphrases to create a large training and evaluation dataset. Questions are categorized by reasoning complexity and answer type, while the automatically generated question distribution remains less natural linguistically.

  • Dataset composition: CRONQUESTIONS contains a temporal KG and natural-language questions requiring temporal reasoning.The dataset includes temporal annotations in the KG and questions designed around temporal reasoning.
  • Temporal KG construction: The final KG contains 328k facts, 125k entities, and 203 relations after temporal filtering and event augmentation.World-event facts, including WWII’s 1939–1945 interval, were added to address missing temporal knowledge.
  • Dataset splits: Train/test construction excludes paraphrase and entity overlap, but permits event overlap, to reduce leakage from memorized questions or entities.This design responds to overlap problems reported for other KGQA datasets, where baseline performance may appear suspiciously high.
  • Limitation: A key limitation is that automatically generated question-answer pairs produce an artificial semantic distribution with lower linguistic variety than natural questions.The authors nevertheless position the dataset as useful for training and evaluation because of its large size.
  • Reasoning complexity: The dataset separates simple single-fact questions from complex questions involving before/after, first/last, or time-join reasoning.Answers may be entities or time instances, and complex questions require multiple facts.

4 Temporal KG Embeddings

The paper investigates temporal KG embeddings as representations for Temporal KGQA, building on embedding methods that represent entities, relations, and timestamps in low-dimensional spaces. It reviews ComplEx extensions that incorporate temporal information and uses embedding-derived scores for training representations.

  • Role of embeddings: Temporal KG embeddings represent entities, relations, and timestamps in a low-dimensional vector space.Their primary prior application has been temporal KG completion.
  • ComplEx: ComplEx represents entities and relations as complex vectors and assigns a score to claimed facts using their interactions.The score uses the real part of a complex-valued expression, with complex conjugation explicitly defined.
  • Temporal extensions: TComplEx extends ComplEx by representing each timestamp as a complex vector for scoring temporal facts.It scores tuples containing subject, relation, object, and timestamp.
  • Temporal extensions: TNTComplEx separates time-sensitive and time-insensitive relation representations and sums their contributions in the scoring function.The two relation representations distinguish temporal effects from relations that are not sensitive to time.
  • Temporal extensions: TimePlex incorporates discretized time instants through timestamp embeddings and three relation representations.Its base score is defined for subject-relation-object-time tuples.

5 CRONKGQA: Our proposed method

CRONKGQA enhances EmbedKGQA for Temporal KGQA by combining temporal KG embeddings with BERT-derived question representations. It predicts entity and time answers using separate scores, then normalizes their concatenated scores into answer probabilities.

  • Embedding inputs: CRONKGQA applies a temporal KG embedding algorithm to obtain entity, relation, and timestamp embeddings for scoring entity or timestamp answers.ComplEx, TComplEx, or TimePlex can provide the embeddings, with complex vectors expanded into real vectors when needed.
  • Baseline adaptation: The method adapts EmbedKGQA, whose original form uses ComplEx and supports non-temporal KGs with single-entity questions.For Temporal KGQA, the first entity in the question is used as the head entity, while timestamp embeddings are introduced.
  • Question representation: CRONKGQA uses BERT to produce a question embedding that is projected into separate entity- and time-prediction embeddings.These are denoted qeent and qetime in the method description.
  • Entity prediction: The entity scorer extracts a subject and timestamp, substitutes dummy values when either is absent, and scores every candidate entity.The resulting scores represent candidate entities’ suitability as answers.
  • Time prediction: The time scorer extracts subject and object entities, uses dummy entities when necessary, and scores candidate timestamps.Entity and time scores are produced by separate scoring functions.
  • Answer prediction: Entity and time scores are concatenated, passed through softmax to obtain answer probabilities, and trained with cross-entropy loss.This supports answers drawn from the combined entity-and-time candidate space.

6 Experiments and diagnostics

The experiments compare language-model, KG-embedding, and hybrid approaches on CRONQUESTIONS, while probing reasoning type, training-set size, and temporal embedding effects. CRONKGQA leads overall and on simple reasoning, but complex temporal reasoning remains difficult.

  • Experimental setup: The experiments evaluate baselines and CRONKGQA across overall performance, reasoning types, training-set size, and temporal versus non-temporal embeddings.The study includes language-model-only and KG-embedding-based approaches, with additional diagnostic experiments.
  • Main results: Language-model-only methods perform significantly worse than methods augmented with temporal or non-temporal KG embeddings.T5-3B is the strongest language-model baseline, possibly because of its parameter count and pre-training.
  • Main results: CRONKGQA performs best among KG-embedding methods on all metrics, while non-temporal EmbedKGQA performs very poorly when the answer is a time.Its time-answer performance is comparable to BERT, the language model used in the EmbedKGQA implementation.
  • Main results: CRONKGQA reaches close to 0.99 hits@1 on simple reasoning questions, compared with 0.329 for T-EaE.The authors attribute this partly to CRONKGQA’s TComplEx scoring-function inductive bias, while noting that T-EaE is more extensible across KG embeddings.
  • Performance across question types: Before/after questions are hardest across methods, with the best method achieving only 0.288 hits@1; time-join questions are easier because they typically have multiple answers.Among complex question types, time joins are strongest except for EmbedKGQA.
  • Effect of training dataset size: Increasing training data from 10% to 100% steadily improves T-EaE-add on simple and complex questions, whereas CRONKGQA benefits mainly for complex reasoning.The authors relate this difference to T-EaE’s larger number of trainable parameters and conclude that large synthetic datasets are useful for temporal reasoning models.
  • Temporal embedding analysis: Using temporal TComplex embeddings significantly boosts both CRONKGQA and T-EaE-replace relative to non-temporal ComplEx embeddings.CRONKGQA receives the larger boost in the reported comparison.

7 Conclusion

CRONQUESTIONS provides a large temporal KGQA benchmark with varied reasoning structures, enabling model training rather than evaluation alone. CRONKGQA outperforms all baselines, while complex reasoning remains an area for improvement.

  • CRONQUESTIONS combines a temporal KG with a large set of temporal questions requiring varied reasoning structures.The dataset was created because prior datasets used non-temporal KGs and contained relatively few questions.
  • Table 6 reports Hits@1 separately for simple entity, simple time, and complex reasoning question types.
  • Increasing the training dataset size steadily improves certain methods’ performance on TKGQA.
  • CRONKGQA outperforms all baselines on the Temporal KGQA task.
  • CRONKGQA performs very well on simple temporal reasoning questions but falls short on more complex reasoning.

A.1 Entities as Experts (EaE)

Entities as Experts combines Transformer processing with an entity memory layer. The model is trained end to end using objectives for entity linking, mention detection, and masked language modeling.

  • The Entities as Experts architecture interleaves a Transformer with an entity memory layer.It uses separate embedding matrices for tokens and entities.
  • The model has separate embedding matrices for tokens and entities.
  • The whole model is trained end to end with losses for entity linking, mention detection, and masked language modeling.

A.2 EaE for Temporal KGQA

The temporal KGQA adaptation uses BERT representations together with pretrained entity and timestamp embeddings. It assumes grounded entity and time mentions are marked, then predicts answers by scoring the final representation against candidate entity embeddings.

  • The temporal KGQA model uses BERT for Transformer0 and TokenEmbed, plus pretrained TComplEx/TimePlex entity and timestamp embeddings for EntityMemory.
  • For each input token, the model uses a contextual BERT embedding and adds an entity or timestamp embedding when the token is a marked mention.Non-mention tokens receive a zero vector for the second representation.
  • The answer head uses the final [CLS] representation, scores it against entity embeddings with a dot product, and applies softmax to obtain answer probabilities.Training uses cross-entropy loss on the QA dataset.
  • T-EaE-add combines BERT and entity/time embeddings through element-wise summation.
  • T-EaE-replace substitutes entity/time embeddings for BERT embeddings at entity and time mentions while leaving the rest of the model unchanged.

A.3 Examples

The examples present validation questions, question types, gold answers, and model predictions. They illustrate before/after, time-join, and first/last reasoning cases alongside top-five predictions from several models.

  • Examples: Tables 8 to 12 show validation questions from CRONQUESTIONS together with the top five model predictions.T5-3B has a single prediction because it is a text-to-text model.
  • Before/After: The examples include before/after reasoning questions, such as where John Hubley worked before Industrial Films.
  • Simple time: One example asks when Man on Wire received the Oscar for Best Documentary Feature, illustrating a simple time-answer question.
  • Reasoning types: The examples also include time-join and first/last reasoning types.
  • Predictions: For the John Hubley example, the gold answer is The Walt Disney Studios, while the listed model predictions include that answer among other entities.
Loading 2106.01515v1…