PaperlayerView exact revision on arXiv

Source-linked AI summary

UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph

Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2212.00959v2cs.CL

TL;DR

Multi-hop KGQA must identify answers several hops from question-linked entities despite the vast search space and the separation of retrieval from reasoning. UniKGQA unifies both stages with shared architecture and parameter learning, and outperforms existing baselines on three benchmark datasets, including 8.1% higher WebQSP Hits@1 and 2.0% higher CWQ Hits@1.

  • Problem

    Existing KGQA systems commonly separate subgraph retrieval and answer reasoning despite their shared semantic-matching essence, limiting relevance information sharing.

  • Method

    UniKGQA combines PLM-based question-relation matching, directed-edge matching-information propagation, abstract retrieval subgraphs, shared pre-training, and stage-specific fine-tuning.

  • Results

    8.1% improvement of Hits@1 on WebQSP and 2.0% improvement of Hits@1 on CWQ over existing state-of-the-art baselines.

  • Takeaways & Limitations

    The approach tightly relates retrieval and reasoning through a unified architecture and shared relevance information.

Abstract

from arXiv · show

Multi-hop Question Answering over Knowledge Graph~(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question on a large-scale Knowledge Graph (KG). To cope with the vast search space, existing work usually adopts a two-stage approach: it first retrieves a relatively small subgraph related to the question and then performs the reasoning on the subgraph to find the answer entities accurately. Although these two stages are highly related, previous work employs very different technical solutions for developing the retrieval and reasoning models, neglecting their relatedness in task essence. In this paper, we propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning. For model architecture, UniKGQA consists of a semantic matching module based on a pre-trained language model~(PLM) for question-relation semantic matching, and a matching information propagation module to propagate the matching information along the directed edges on KGs. For parameter learning, we design a shared pre-training task based on question-relation matching for both retrieval and reasoning models, and then propose retrieval- and reasoning-oriented fine-tuning strategies. Compared with previous studies, our approach is more unified, tightly relating the retrieval and reasoning stages. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our method on the multi-hop KGQA task. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/UniKGQA}.

1 INTRODUCTION

Multi-hop KGQA must search large knowledge graphs to connect question-linked topic entities with answers several hops away. UniKGQA unifies retrieval and reasoning through shared architecture and learning, improving benchmark performance.

  • Task motivation: Multi-hop KGQA finds answer entities multiple hops from topic entities in large-scale knowledge graphs.For example, answering a question about The Jeff Probst Show requires following a nominee-to-spouse reasoning path.
  • Existing pipeline: Prior systems retrieve a small question-relevant subgraph and then reason over it because the full KG creates a vast search space.Both stages evaluate semantic relevance, particularly through direct relations or composite relation paths.
  • Research gap: Previous pipelines pass retrieved triples between independently treated stages while neglecting other useful semantic-matching signals.The paper frames this separation as potentially sub-optimal for a task requiring multiple kinds of relevance information.
  • UniKGQA: UniKGQA uses a PLM-based question-relation matching module and propagates matching information along directed KG edges.It also introduces abstract subgraphs to reduce retrieval-stage node scale and shares parameters through pre-training and stage-specific fine-tuning.

2 PRELIMINARY

The paper formalizes knowledge graphs and multi-hop KGQA, then introduces abstract subgraphs as a reduced representation of question-related graph structure. Abstract nodes represent one or more original entities grouped by shared relational context.

  • Knowledge Graph: A knowledge graph is a set of head-relation-tail triples connecting entities through typed relations.The formulation also defines neighborhood triples, inverse relations, and entity and relation embedding matrices.
  • Multi-hop KGQA: Multi-hop KGQA maps a natural-language question and linked topic entities to answer entities several hops away in the KG.The task assumes that entities mentioned in the question are marked and linked to KG entities.
  • Abstract Subgraph: An abstract subgraph merges tail entities sharing the same head-and-relation prefix into an abstract node representing their entity set.The same operation can be applied to head entities, while unmerged original nodes are represented by singleton sets.

3 APPROACH

UniKGQA unifies retrieval and reasoning through a shared architecture that matches questions with relations and propagates matching information across directed KG edges. It uses shared question-relation pre-training followed by stage-specific fine-tuning, with retrieval operating on abstract subgraphs and reasoning on retrieved subgraphs.

  • Unified model architecture: UniKGQA combines semantic matching and matching information propagation in one architecture for both retrieval and reasoning.The semantic matching module uses a PLM, while propagation aggregates matching features along KG edges to update entity representations and scores.
  • Unified model architecture: The general input formulation represents retrieval with abstract subgraphs and reasoning with subgraphs derived from retrieval.Abstract nodes merge entities from the same relation during retrieval; reasoning operates without abstract nodes on the retrieved subgraph.
  • Semantic matching: The semantic matching module encodes questions and relations with a PLM and produces question-relation matching features through projection layers.The [CLS] representations of the question and relation are used to construct matching features with sigmoid activation and Hadamard products.
  • Matching information propagation: The propagation module aggregates neighboring triple features, updates entity representations, and iteratively produces final match scores as a probability distribution over subgraph entities.After T steps, the scores measure entities’ possibilities of being answers and are used in both retrieval and reasoning.
  • Model training: Training first pre-trains shared PLM parameters with question-relation matching, then fine-tunes retrieval and reasoning parameters for their respective stages.Retrieval fine-tuning uses abstract subgraphs and top-K selection; reasoning is initialized from the retrieval model and fine-tuned on retrieved subgraphs.
  • Model training: The reasoning model is initialized from the learned retrieval model so information between the two stages can be shared and reused.This transfer is part of the unified training procedure and supports reasoning on the smaller retrieved subgraph.

4 EXPERIMENT

Experiments evaluate UniKGQA on benchmark KGQA datasets, retrieval quality, ablations, and fine-tuning efficiency. UniKGQA substantially outperforms competitive baselines on WebQSP and CWQ, while its transferred knowledge supports efficient adaptation.

  • Experimental Setting: UniKGQA is evaluated on MetaQA, WebQSP, and CWQ using answer coverage for retrieval and ranking-based QA metrics for reasoning.MetaQA performance is already near saturation, so WebQSP and CWQ receive primary evaluation emphasis.
  • Evaluation Results: Most baselines achieve 100% Hits@1 on the three MetaQA datasets because they use few templates and only nine relation types.The authors attribute the high performance to the simplicity and ample training data of MetaQA.
  • Evaluation Results: UniKGQA substantially outperforms all competitive baselines on WebQSP and CWQ.The authors associate this result with unified question-relation matching and relevance transfer between retrieval and reasoning.
  • Evaluation Results: Updating the PLM encoder improves performance, while updating it only for questions can match or exceed updating it for both questions and relations.The question-only variant offers better performance with relatively less additional computation, whereas updating both may overfit downstream tasks.
  • Further Analysis: Under equal retrieved-subgraph sizes, UniKGQA and SR achieve higher answer coverage rates than PPR-based retrieval.This comparison supports using a learnable retrieval model to obtain smaller subgraphs with stronger answer coverage.
  • Further Analysis: After two fine-tuning epochs, UniKGQA achieves good performance, and before fine-tuning it matches NSM's best final-epoch performance.The comparison uses the same retrieved subgraphs and tests transfer from pretraining and retrieval initialization.

5 RELATED WORK

Related work frames multi-hop KGQA as retrieval followed by reasoning, alongside approaches based on structured query generation and dense retrieval. These methods differ in how they reduce search space, model relation paths, and represent relevance.

  • Multi-hop Knowledge Graph Question Answering: Multi-hop KGQA commonly retrieves a question-relevant subgraph before performing multi-hop reasoning to reduce the search space.This retrieval-and-reasoning paradigm is motivated by efficiency and accuracy on large-scale KGs.
  • Multi-hop Knowledge Graph Question Answering: Heuristic retrieval methods collect nearby entities and filter them with Personalized PageRank, but neglecting question semantics can limit retrieval efficiency and accuracy.Later work introduces question-aware retrieval to address this limitation.
  • Multi-hop Knowledge Graph Question Answering: Reasoning-focused methods use specialized architectures such as Key-Value Memory Networks and Graph Convolutional Networks to model multi-hop traversal.Other work strengthens these models through mechanisms including intermediate supervision.
  • Multi-hop Knowledge Graph Question Answering: Question-to-SPARQL approaches generate structured queries with encoder-decoder models and require annotated structured queries for training.T5 is identified as a commonly adopted encoder-decoder architecture in this line of work.
  • Dense Retrieval: Dense retrieval maps queries and documents into low-dimensional vectors with bi-encoders and measures relevance using vector distance.This contrasts with traditional sparse term-based methods such as TF-IDF and BM25.

6 CONCLUSION

The paper concludes that UniKGQA unifies retrieval and reasoning through shared architecture, abstract subgraphs, and joint parameter-learning strategies. Experiments on three benchmark datasets show advantages over competitive methods, especially on WebQSP and CWQ.

  • Conclusion: UniKGQA uses PLM-based semantic matching and matching-information propagation for both retrieval and reasoning.Abstract subgraphs reduce the retrieval-stage node search space, while pretraining and oriented fine-tuning support parameter learning.
  • Conclusion: Experiments on three benchmark datasets show that UniKGQA outperforms competitive methods, especially on WebQSP and CWQ.The conclusion attributes this outcome to sharing and transferring relevance information between the two stages.

A DATASETS

The experiments use three multi-hop KGQA datasets with different domains, sizes, and reasoning depths, while comparing UniKGQA against reasoning and retrieval baselines. MetaQA is treated as nearly saturated, making WebQSP and CWQ the primary evaluation targets.

  • Datasets: MetaQA contains more than 400k movie-domain questions with answers up to three hops away, split into one-, two-, and three-hop subsets.Its subsets are organized by the number of reasoning hops.
  • Datasets: WebQSP contains 4,737 Freebase questions requiring up to two-hop reasoning and uses the GraftNet train, validation, and test splits.The dataset is one of the primary evaluation targets.
  • Datasets: CWQ extends WebQSP with expanded question entities or answer constraints and requires up to four-hop reasoning on Freebase.CWQ is emphasized because it presents more complex questions than WebQSP.
  • Dataset Analysis: MetaQA is considered training-sufficient and simple because it has more training cases, simpler questions, and few relation types, producing very high baseline performance.The analysis examines templates, training cases per template, and relations used to construct questions.
  • Baselines: The comparison includes reasoning-focused methods and retrieval-augmented methods such as GraftNet, NSM, PullNet, SR+NSM, and SR+NSM+E2E.The baselines represent different combinations of retrieval and multi-hop reasoning architectures.
  • Implementation: For WebQSP and CWQ, retrieval and reasoning use maximum hop counts of two and four, respectively, with neighborhood subgraphs reserved around topic entities.The preprocessing retains entities within four hops of each sample's topic entities.

D IMPLEMENTATION DETAILS.

UniKGQA pre-trains its RoBERTa-base PLM on question–relation pairs derived from shortest relation paths, then initializes retrieval and reasoning with the resulting model.

  • Pre-training uses question–relation pairs constructed from shortest relation paths between topic and answer entities.The pairs train the RoBERTa-base model with a contrastive learning objective.
  • The pre-trained RoBERTa model initializes UniKGQA’s PLM module for both retrieval and reasoning.
  • The temperature τ is set to 0.05, and the best model is selected using validation-set Hits@1.

E ONE-SHOT EXPERIMENT FOR METAQA

The one-shot MetaQA experiment addresses possible performance saturation by training methods with one example per question template; UniKGQA consistently outperforms TransferNet and NSM across all three subsets.

  • One-shot setup: The one-shot experiment samples one training case for each question template to test performance under limited supervision.
  • Motivation: MetaQA’s standard results may be saturated because methods exceed 98% Hits@1 with abundant training samples.
  • Results: UniKGQA consistently outperforms TransferNet and NSM across all three MetaQA one-shot subsets.

F ABLATION STUDY OF OUR UNIFIED MODEL ARCHITECTURE

The ablation study finds that using the unified model architecture in both retrieval and reasoning is central to UniKGQA’s improvement.

  • Ablation results: All variants that combine UniKGQA with only an existing retrieval or reasoning model underperform the full UniKGQA system.The compared models are SR for retrieval and NSM for reasoning.
  • Interpretation: Removing the unified architecture limits sharing of pre-trained question–relation matching and transferring retrieval relevance information to reasoning.
  • Experimental design: The ablation compares UniKGQA with existing strong retrieval and reasoning models through combinations summarized in Table 7.

G ANALYSIS OF THE PRE-TRAINING STRATEGY

The analysis shows that pre-training materially improves performance, while fixing the pre-trained PLM can make fine-tuning more parameter-efficient.

  • Pre-training effect: Removing pre-training drops performance by 10.4% (2.1%) on WebQSP and 5.1% (3.3%) on CWQ when fixing (not fixing) the PLM.
  • Pre-training effect: Pre-training is identified as an important component of UniKGQA.
  • PLM updating: After pre-training, the PLM can be fixed for more efficient parameter optimization during fine-tuning.
  • Experimental design: Table 8 compares variants with or without pre-training and PLM updating.

H PARAMETER SENSITIVITY ANALYSIS

UniKGQA is relatively insensitive to hidden-layer size, while increasing the number of retrieved nodes improves answer coverage until gains become marginal. The best performance is reached with only a few pre-training steps.

  • Pre-training Steps: 2,800 pre-training steps are sufficient for UniKGQA to reach its best performance, outperforming the best TransferNet baseline.The authors conclude that extensive pre-training is unnecessary for their approach.
  • Parameter Tuning: Performance remains consistently near 77.0 across different hidden sizes, with 768 slightly outperforming the alternatives.The PLM uses an embedding size of 768.
  • Parameter Tuning: Increasing K consistently improves answer coverage, but gains become relatively small at K=15 or K=20.The authors interpret this saturation as further retrieved nodes providing only marginal improvement.
Loading 2212.00959v2…