PaperlayerView exact revision on arXiv

Source-linked AI summary

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering

Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng

arXiv:2110.04330v2cs.CLcs.LG

TL;DR

FiD-based open-domain question answering processes many retrieved passages, creating a computational bottleneck, while independent passage encoding leaves relationships among passages unused. KG-FiD uses graph-based reranking and pruning, and experiments on Natural Questions and TriviaQA report improved exact match alongside comparable performance at substantially lower computation cost.

  • Problem

    FiD processes about 100 retrieved passages per question at high computational cost, while independently encoding passages leaves their semantic relationships unused.

    Source: §1 · p. 2

  • Method

    KG-FiD uses a knowledge graph and GNNs for two-stage passage reranking and pruning, reusing DPR and reader-encoder representations.

    Source: §1 · p. 2

  • Results

    1.5% exact match improvement over original FiD, with on-par performance while reducing computation cost by over 60% on Natural Questions and TriviaQA.

    Source: §5 · p. 8

  • Takeaways & Limitations

    KG-based reranking improves passage cross-modeling and can substantially reduce computation while retaining performance comparable to FiD on the two evaluated benchmarks.

    Source: §5 · p. 8

    The computation-saving analysis assumes answers are much shorter than passages, making decoding cost negligible compared with encoding.

    Source: §3.5 · p. 6

Abstract

from arXiv · show

Current Open-Domain Question Answering (ODQA) model paradigm often contains a retrieving module and a reading module. Given an input question, the reading module predicts the answer from the relevant passages which are retrieved by the retriever. The recent proposed Fusion-in-Decoder (FiD), which is built on top of the pretrained generative model T5, achieves the state-of-the-art performance in the reading module. Although being effective, it remains constrained by inefficient attention on all retrieved passages which contain a lot of noise. In this work, we propose a novel method KG-FiD, which filters noisy passages by leveraging the structural relationship among the retrieved passages with a knowledge graph. We initiate the passage node embedding from the FiD encoder and then use graph neural network (GNN) to update the representation for reranking. To improve the efficiency, we build the GNN on top of the intermediate layer output of the FiD encoder and only pass a few top reranked passages into the higher layers of encoder and decoder for answer generation. We also apply the proposed GNN based reranking method to enhance the passage retrieval results in the retrieving module. Extensive experiments on common ODQA benchmark datasets (Natural Question and TriviaQA) demonstrate that KG-FiD can improve vanilla FiD by up to 1.5% on answer exact match score and achieve comparable performance with FiD with only 40% of computation cost.

1 Introduction

KG-FiD adds graph-based passage reranking to FiD to model relationships among retrieved passages and reduce how many passages receive further computation. On Natural Questions and TriviaQA, it reports comparable or better answer prediction than FiD at 40% of FiD’s computation cost.

1 Introduction

  • FiD is a DPR-retriever, generative-reader framework that encodes retrieved passages independently rather than concatenating them.The paper presents KG-FiD as a further improvement to this ODQA paradigm.Source: §1 · p. 1
  • FiD independently encodes retrieved passages, so it cannot use their semantic relationships even when available, although those connections can provide clues for answering.The paper identifies this independent-passage assumption as one limitation of FiD.Source: §1 · pp. 1–2
  • FiD processes about 100 passages per question at high cost; simply sending fewer passages to the reader can significantly reduce performance.The introduction gives inference cost as more than 6 trillion floating-point operations per question.Source: §1 · p. 2
  • KG-FiD reranks passages in two stages: DPR embeddings support broader initial reranking, then reader-encoder passage-question embeddings rerank and prune a smaller set.The first stage aims to enhance answer coverage; the second aims for more accurate reranking and passage pruning.Source: §1 · p. 2
  • The second-stage GNN uses intermediate reader-encoder representations, passing only a few top-ranked passages to higher encoder layers and the decoder for joint reranking and answer generation.The paper couples this strategy with joint training of passage reranking and answer generation.Source: §1 · p. 2
  • KG-FiD achieves comparable or better answer prediction than FiD on Natural Questions and TriviaQA with only 40% of FiD’s computation cost.The introduction also reports that the reranking strategies significantly reduce computation while maintaining good QA performance.Source: §1 · p. 2

2 Related Work

Related work improves ODQA retrieval and reading, while prior KG approaches use graph information in different ways. The paper identifies applying KG to the FiD framework as an unstudied direction.

ODQA with text corpus

  • Text-corpus ODQA work has advanced from sparse to dense retrieval and from recurrent readers to pretrained-language-model readers, with further work on training, reranking, and distillation.The cited passage notes that these approaches do not model relationships among different passages.Source: §2 · p. 2

ODQA with knowledge graph

  • Some KG-based ODQA methods restrict answers to graph entities, whereas this work targets answers that may be any words or phrases.Other approaches use KG information for ODQA, including converting KG triples to text or building relationships among passages.Source: §2 · pp. 2–3
  • UniK-QA converts KG triples into text, while KAQA and Graph-Retriever use passage graphs; applying KG to the FiD framework remained unstudied.The passage describes KAQA as propagating retrieval and answer-span scores and Graph-Retriever as iteratively retrieving related passages.Source: §2 · p. 3

3 Method

KG-FiD uses knowledge-graph links to connect retrieved passages, then applies two graph-based reranking stages to select passages for answer generation. Its intermediate-layer reranking is reported to reduce computation while maintaining comparable benchmark performance.

3.1 Construct Passage Graph using KG

  • The passage graph connects passages when their mapped article entities are directly connected in the knowledge graph, and is built on the fly for each question.A passage maps to the entity corresponding to its article; one entity may correspond to multiple passages.Source: §3.1 · p. 3

3.2 Passage Retrieving & Stage-1 Reranking

  • Stage-1 reranking applies a graph attention network to retrieved candidates, reusing static DPR passage embeddings as node representations before sending the top-ranked passages to the reader.The method first retrieves N0 candidates, where N0 > N1, then reranks them to return N1 passages.Source: §3.2 · pp. 3–4

3.3 Joint Stage-2 Reranking and Answer Generation

  • Stage-2 reranking uses the passage graph to rerank N1 input passages during encoding and sends only the top-N2 (N2 < N1) reranked passages into the decoder.Passage reranking and answer generation are jointly trained.Source: §3.3 · p. 5

3.4 Improving Efficiency via Intermediate Representation in Stage-2 Reranking

  • Using an intermediate encoder layer for stage-2 reranking lets only the top-N2 passages continue through the remaining encoder layers and into the decoder.The stated rationale is that passage reranking only predicts whether a passage contains the answer, while answer generation is more difficult.Source: §3.4 · pp. 5–6
  • 60% lower computation cost than original FiD with on-par performance on two benchmark datasets is reported for intermediate-layer stage-2 reranking.The complexity analysis gives L1 = L/4 and N2 = N1/5 as an example setting for about 60% savings.Source: §3.4 · p. 6, §3.5 · p. 6

3.5 Analysis on Computational Complexity

  • The complexity analysis attributes savings to processing all N1 passages through only L1 encoder layers, then processing N2 passages through the remaining layers and decoder.It assumes answer sequences are much shorter than passages, making decoding negligible relative to encoding in the approximation.Source: §3.5 · p. 6

4 Experiment

Experiments on Natural Questions and TriviaQA compare KG-FiD with FiD and other baselines, assess efficiency and accuracy, and ablate both reranking stages. KG-FiD improves on FiD while using substantially less computation, and both reranking stages contribute to performance.

4.1 Implementation Details

  • Experiments use NQ and TriviaQA, evaluate answer prediction by exact match, and report scores averaged over five random-seed runs.The setup uses DPR, three-layer GAT rerankers, pretrained T5-base and T5-large readers, and N0=1000, N1=100, N2=20.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Baseline Methods

  • Comparisons center on FiD and include baselines without external knowledge, reranking methods, graph or knowledge-graph methods, and methods with additional large-scale pretraining.The listed baselines include T5, GPT-3, RIDER, RECONSIDER, Graph-Retriever, Path-Retriever, KAQA, UniK-QA, REALM, RAG, and Joint Top-K.Source: §4.2 · p. 7

4.3 Main Results

  • For the large model, KG-FiD improves FiD by 1.5% on NQ and 1.1% on TriviaQA, and consistently improves FiD across both model sizes and datasets.KG-FiD outperforms the baselines except UniK-QA, which uses additional Wikipedia-Tables retrieval, making direct comparison unfair.Source: Table 1 · p. 7, §4.3 · p. 7
  • KG-FiD achieves performance on par with FiD at 38% of its computation cost; at equal computation (L1=24), it significantly outperforms FiD on both datasets.Reducing vanilla FiD's NQ reader input lowers performance from 51.9 to 50.3; KG-FiD trades intermediate-layer depth against efficiency and accuracy.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 Ablation Study

  • Removing either reranking stage lowers performance; stage 1 is more effective for the base model, while stage 2 is more effective for the large model.The authors relate stage 2's stronger large-model result to its reliance on the reader encoder, which is usually better in the large model.Source: Table 3 · p. 8, §4.4 · p. 8
  • DPR+stage-1 reranking beats DPR for Hits@K at K=10, 20, 50, and 100; two-stage reranking further improves results at K=10 and 20.The authors say reranking can raise gold passages ranked lower by DPR and improve passage pruning.Source: Figure 2 · p. 8, §4.4 · p. 8

5 Conclusion

KG-FiD applies GNN-based reranking to passage representations from DPR and the reader encoder, using intermediate encoder layers to reduce computation. On Natural Questions and TriviaQA, it improves FiD’s exact match score and reports comparable performance with lower computation cost.

5 Conclusion

  • KG-FiD improves original FiD by 1.5% exact match score on Natural Questions and TriviaQA.It also achieves on-par performance with FiD while reducing computation cost by over 60%.Source: §5 · p. 8
  • The two-stage reranking method reuses passage representations from the DPR retriever and reader encoder, then applies graph neural networks to compute reranking scores.Source: §5 · p. 8
  • Using an intermediate encoder layer for reranking reduces computation cost while maintaining good performance.Source: §5 · p. 8

A.1 Dataset

The experiments use Natural Questions and TriviaQA, with dataset-specific answer handling and a validation split drawn from training data.

A.1 Dataset

  • The study uses Natural Questions (NQ) and TriviaQA, with NQ answers longer than five tokens discarded and TriviaQA used in its unfiltered version.Source: §A.1 · p. 11
  • For NQ, the answer target is sampled from the provided answers; for TriviaQA, generation targets the unique human-generated answer.Source: §A.1 · p. 11
  • The original validation data is used as test data, and 10% of each training set is kept for validation.Source: §A.1 · p. 11

A.2 Preliminary Analysis

The passage graph is question-specific because it is built only among passages retrieved for that question. Preliminary statistics examine how often two sparse or concentrated graph conditions occur.

A.2 Preliminary Analysis

  • Edges are built only among retrieved passages, making each passage graph specific to its question.Source: §A.2 · p. 11
  • The analysis uses 100 retrieved passages per question to assess the passage graphs.Source: §A.2 · p. 11
  • Questions where all retrieved passages come from one article or the graph has very few edges occur only in a small portion of cases.Source: §A.2 · p. 11

A.3 Training Process

Training proceeds in separate stages: DPR first, stage-1 reranking second, and joint training of stage-2 reranking and the reader last.

A.3 Training Process

  • To avoid out-of-memory issues, training first fits DPR, then freezes it while training stage-1 reranking, and finally jointly trains stage-2 reranking and the reader.Source: §A.3 · p. 11
  • Stage-1 reranking uses AdamW with a 1e-3 learning rate, weight decay 0.01, batch size 64, and 15k total training steps.Source: §A.3 · p. 11
  • Reader training uses learning rates of 1e-4 for the base model and 5e-5 for the large model, with 1000 warm-up steps.Source: §A.3 · p. 11

A.4 Additional Experiment Results

The additional experiments tune the passage-reranking GNN and stage-2 filtering and loss weight. They identify GAT with three layers and N2 = 20 as choices, while showing that the reranking-loss weight must be balanced against answer-generation training.

GNN Model Design

  • GAT outperforms vanilla GCN on passage retrieval, and three GNN layers are the best tested choice.The authors attribute GAT's advantage to attention reweighting neighbor passages; other GNN variants are left for future exploration.Source: §A.4 · p. 12

Hyper-parameter Search

  • N2 = 20 performs better than N2 = 10, while increasing N2 further brings no performance gain and reduces efficiency.The authors choose N2 = 20.Source: §A.4 · p. 12
  • As the reranking-loss weight λ increases, performance first rises and then drops significantly, indicating a need to balance the two training losses.The balance is between passage reranking and answer-generation learning signals.Source: §A.4 · p. 12

A.5 FLOPs Computation

The FLOPs analysis compares FiD and KG-FiD across base and large models. KG-FiD reduces reader decoding computation to about one-fifth of FiD's, while the reported totals and stage costs quantify the comparison.

FLOPs Computation

  • The analysis assumes 20-token average questions, 5-token average answers, 250-token passage-question pairs, and N1 = 100 reader inputs.It excludes passage similarity search because its cost depends on corpus size and search method and is usually efficient.Source: §A.5 · pp. 12–13
  • With N0 = 1000 and N1 = 100, KG-FiD's stage-1 GAT costs 3.5 GFLOPs and stage-2 reranking costs 0.4/0.6 GFLOPs (base/large).The retrieval computation is the same as vanilla FiD.Source: §A.5 · p. 13
  • KG-FiD reader decoding costs 143.9/510.0 GFLOPs (base/large), about one-fifth of FiD's, while FiD totals 6491.0/20022.0 GFLOPs.Tables 7 and 8 report stage-wise GFLOPs for base and large models.Source: §A.5 · p. 13, Table 7 · p. 14, Table 8 · p. 14

Related papers

Loading 2110.04330v2…