Source-linked AI summary
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William W. Cohen
TL;DR
Open-domain QA systems have generally used text or knowledge bases separately, although incomplete KBs and text corpora are often jointly available. The paper introduces GRAFT-Net, which performs early fusion over a question-specific graph of KB facts, entities, and text, and finds competitive single-source performance and superior combined-setting performance.
Problem
Open-domain QA needs to combine text and incomplete KBs because neither source alone answers all questions.
Method
GRAFT-Net performs early fusion by classifying answer nodes in a question-specific heterogeneous subgraph containing KB facts and entity-linked text.
Results
GRAFT-Nets are competitive with state-of-the-art methods in text-only and KB-only settings and outperform baselines when text is combined with an incomplete KB.
Takeaways & Limitations
Early fusion approaches perform better than late fusion approaches for QA using text combined with an incomplete KB.
Takeaways & Limitations
In the KB-only WebQuestionsSP setting, GRAFT-Net trails Neural Symbolic Machines by 6.2% F1 points, partly because subgraph retrieval recall is 90.2%.
Abstract
from arXiv · showhide
Open Domain Question Answering (QA) is evolving from complex pipelined systems to end-to-end deep neural networks. Specialized neural models have been developed for extracting answers from either text alone or Knowledge Bases (KBs) alone. In this paper we look at a more practical setting, namely QA over the combination of a KB and entity-linked text, which is appropriate when an incomplete KB is available with a large text corpus. Building on recent advances in graph representation learning we propose a novel model, GRAFT-Net, for extracting answers from a question-specific subgraph containing text and KB entities and relations. We construct a suite of benchmark tasks for this problem, varying the difficulty of questions, the amount of training data, and KB completeness. We show that GRAFT-Net is competitive with the state-of-the-art when tested using either KBs or text alone, and vastly outperforms existing methods in the combined setting. Source code is available at https://github.com/OceanskySun/GraftNet .
1 Introduction
The paper studies open-domain QA when neither a large text corpus nor an incomplete KB suffices alone, and proposes early fusion through GRAFT-Net. It evaluates this setting with benchmark tasks and reports strong performance across conditions.
- Motivation: Existing open-domain QA models usually rely on either encyclopedia text or a single knowledge base.Text offers broad coverage but varied language patterns, whereas KBs are easier to query but incomplete and schema-restricted.
- Motivation: Questions may require text or KB evidence, yet prior work has paid little attention to combining both sources.The paper targets settings where a large-scale KB and text corpus are available but neither answers every question alone.
- Approach: Early fusion trains one model to extract answers from a question-specific subgraph containing relevant KB facts and text sentences.This contrasts with late fusion, which heuristically aggregates predictions from separate source-specific QA systems.
- Approach: GRAFT-Net is a graph-convolution neural network designed for heterogeneous graphs of KB facts and text sentences.It uses heterogeneous updates for KB and text nodes and directed propagation from question-linked seed nodes.
- Evaluation: The benchmark suite varies training supervision, KB completeness, and question complexity, with baselines including Key Value Memory Networks.The authors report superior performance across a wide range of conditions.
2 Task Setup
The task is to extract entity answers from a question-specific graph built from an incomplete or complete KB, text documents, and entity links. The system retrieves relevant KB entities and text, combines them into a graph, and classifies its nodes with GRAFT-Net.
- Task Definition: Given a natural-language question, the task extracts one or more entity answers from a graph containing KB entities, documents, and entity links.The KB may range from highly incomplete to complete for answering the questions.
- Overall Procedure: The system first retrieves a high-recall question subgraph and then classifies each node as an answer or non-answer using question-conditioned representations.The second-stage training data is generated with distant supervision.
- Subgraph Retrieval: Parallel KB and text-retrieval pipelines provide entities and documents that are combined through entity links into a fully connected graph.KB retrieval begins from question-linked seed entities, while text retrieval selects Wikipedia articles and sentences relevant to the question.
- Subgraph Representation: The question subgraph contains retrieved entities and documents, KB relations among entities, and a special linking relation for document-entity connections.Its edge types are represented as R+ = R ∪ {rL}.
3 GRAFT-Nets
GRAFT-Net performs question-conditioned classification over heterogeneous subgraphs containing KB entities and text documents. It combines heterogeneous updates, relation attention, and directed propagation from question-linked seed entities.
- Model formulation: GRAFT-Net reduces QA to binary classification over nodes in a question-specific graph containing KB entities and text documents.Answers are labeled positive and all other graph nodes negative.
- Heterogeneous updates: Entity nodes use fixed-size vectors, whereas document nodes retain variable-length word representations initialized from document sequences.Document representations preserve token positions because multiple entities may link to different positions.
- Heterogeneous updates: Entity updates combine entity and question states, relation-specific neighbor aggregation, and token states from linked document mentions.Relation vectors replace large relation-specific projection matrices, reducing the batched memory burden.
- Heterogeneous updates: Document updates first aggregate linked entity states at each token position, then aggregate token states within the document using an LSTM.This creates document representations that incorporate incoming entity information while retaining sequential structure.
- Conditioning on the question: Question conditioning uses relation attention and personalized propagation, with attention assigning greater weight to edges relevant to the question.The question representation is initialized from its words and updated across layers.
- Conditioning on the question: Directed propagation scales embedding transfer with PageRank scores so information spreads outward from question-linked seed entities along increasingly distant paths.At layer 1 propagation starts from seeds; at layer 2 it reaches seeds and their one-hop neighbors.
- Prediction and training: Final node representations are converted to answer probabilities with a sigmoid classifier trained using binary cross-entropy.Fact-dropout randomly removes graph edges during training to encourage use of all available information sources.
4 Related Work
Related work includes early-fusion, symbolic, and representation-learning approaches for combining KB facts with text. The paper distinguishes GRAFT-Net through question-conditioned graph propagation over heterogeneous structures.
- QA over KBs and text: Prior early-fusion QA used Key-Value Memory Networks with independently represented KB triples and text snippets, but ignored their rich relational structure.GRAFT-Net instead models the combined sources as a graph.
- QA over KBs and text: Earlier non-deep approaches combined text assertions and KB facts through semantic parsing or pipelined evidence aggregation.These systems represent alternative strategies for integrating structured and unstructured sources.
- Related representation learning: Work on joint KB-text representations for relation extraction and knowledge-base completion differs from QA because inference must be conditioned on the question.Different questions therefore induce different knowledge-source representations.
- Graph-based methods: GRAFT-Net builds on graph representation learning and message passing while operating on heterogeneous nodes and retrieving question-specific subgraphs.Unlike random-sampling approaches such as GraphSAGE, it uses retrieval to obtain the subgraph.
- KB and text QA: The paper extends KB-focused neural QA toward text integration, whereas text-only reading-comprehension systems commonly assume the answer-containing passage is known in advance.GRAFT-Net addresses a setting where relevant information is drawn from combined KB and text structures.
5 Experiments & Results
Experiments evaluate GRAFT-Net across datasets, KB completeness levels, fusion strategies, and model-component ablations. GRAFT-Net consistently outperforms KV-MemNNs, benefits from early fusion and heterogeneous updates, and approaches or exceeds specialized state-of-the-art systems in most settings.
- Datasets: WikiMovies-10K uses 10K training questions, while WebQuestionsSP includes 3,098 training and 1,639 test questions with varying reasoning difficulty.WebQuestionsSP requires aggregating two KB facts for approximately 30% of questions and reasoning over constraints for approximately 7%.
- Subgraph construction: 99.6% overall answer recall is achieved in the constructed question subgraphs using retrieved entities and text sentences.The subgraphs retrieve top entities around linked seeds and top sentences from the text corpus.
- Main results: GRAFT-Nets consistently improve over KV-MemNNs in KB-only, text-only, and early-fusion settings across both datasets.The comparison reports Hits@1 and F1, with GRAFT-Net attention able to assign high weights to multiple correct answers.
- Main results: Early fusion consistently outperforms late fusion, while ensembling the two produces the best performance across the evaluated models.The benefit of adding text decreases as the KB becomes more complete, although GN-EF with text improves over KB-only across the reported settings.
- Comparison to specialized methods: GRAFT-Net matches or outperforms specialized state-of-the-art models in three of four text-only or KB-only comparisons.The exception is KB-only WebQuestionsSP, where it performs 6.2% F1 points worse than Neural Symbolic Machines.
- Effect of model components: Heterogeneous updates outperform non-heterogeneous updates, and moderate fact-dropout rates improve performance on both datasets.Without heterogeneous updates, entities mentioned in the same document receive identical updates; fact-dropout helps until the model cannot learn the KB inference chain.
6 Conclusion
The paper investigates QA over text combined with an incomplete KB, introducing benchmarks and early-fusion approaches for this setting. GRAFT-Net is competitive in text-only and KB-only settings and outperforms baselines when combining text with an incomplete KB.
- The paper introduces benchmark problems for QA using text combined with an incomplete KB, a setting that has received limited prior attention.
- GRAFT-Net is a novel early-fusion model that classifies nodes in subgraphs containing KB entities and text documents.
- GRAFT-Nets achieve competitive performance in text-only and KB-only settings while outperforming baseline models with text combined with an incomplete KB.