Source-linked AI summary
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, Scott Yih
TL;DR
Open-domain question answering spans text, tables, lists, and knowledge bases, while prior systems often rely on specialized approaches for particular sources. UniK-QA converts heterogeneous sources to text for a retriever-reader model. It improves benchmark results, including new state of the art on NaturalQuestions and WebQuestions.
Problem
Open-domain QA must handle answers across heterogeneous sources, but prior approaches commonly assume either structured data or unstructured text.
Source: §1 · p. 1Method
UniK-QA linearizes structured sources into text and uses a retriever-reader architecture to process them alongside text passages.
Source: §1 · p. 2, §3.1 · p. 4Results
UniK-QA consistently improves strong TextQA baselines across source combinations and advances state of the art by 3.5 points on NaturalQuestions and 2.6 points on WebQuestions.
Source: §1 · p. 2Takeaways & Limitations
The unified approach also improves KBQA itself, showing that TextQA methods can be applied to structured knowledge sources.
Source: §1 · p. 2Flattening sources into text loses the knowledge base’s ability to return all answers corresponding to a query.
Source: §8 · p. 9
Abstract
from arXiv · showhide
We study open-domain question answering with structured, unstructured and semi-structured knowledge sources, including text, tables, lists and knowledge bases. Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only. Our approach greatly improves the results on knowledge-base QA tasks by 11 points, compared to latest graph-based methods. More importantly, we demonstrate that our unified knowledge (UniK-QA) model is a simple and yet effective way to combine heterogeneous sources of knowledge, advancing the state-of-the-art results on two popular question answering benchmarks, NaturalQuestions and WebQuestions, by 3.5 and 2.6 points, respectively. The code of UniK-QA is available at: https://github.com/facebookresearch/UniK-QA.
1 Introduction
Open-domain QA must draw answers from heterogeneous sources, but most systems assume either structured data or unstructured text. UniK-QA flattens structured sources into text for a retriever-reader model, improving KBQA and multi-source benchmark results.
1 Introduction
- Prior systems often assume structured data or unstructured text; combining specialized systems is complex and makes cross-source reasoning difficult.TextQA methods also exclude tables and structured knowledge bases as inputs.Source: §1 · pp. 1–2
- UniK-QA linearizes structured sources and applies TextQA methods, supporting lists, tables, and multiple knowledge bases without specially handling KB schemas or ontologies.The proposed model incorporates about 27 million text and list passages.Source: §1 · p. 2
- 11% improvement in WebQSP exact match over prior state of the art validates modeling KBQA as a pure TextQA task.The approach represents KB relations by their textual surface forms and trains a retriever-reader as if they were text documents.Source: §1 · p. 2
- 3.5 points on NaturalQuestions and 2.6 points on WebQuestions advance the state of the art, while UniK-QA consistently beats strong TextQA baselines across evaluated source combinations.The experiments combine lists, tables, and knowledge bases with text using simple heuristics.Source: §1 · p. 2
- A single model trained on combined benchmark data outperforms single-source baselines when a question’s source is unknown in advance.This setting is presented as realistic for a practical system.Source: §1 · p. 2
2 Background & Related Work
KBQA, TextQA, and table QA each address distinct sources, while earlier fusion methods retain structural constraints or combine specialized systems. UniK-QA instead drops structure and applies a retriever-reader framework across diverse sources and benchmarks.
2.1 Knowledge-base question answering (KBQA)
- KBQA converts natural-language questions into logical forms for querying knowledge bases, whose graph-specific methods are not directly applicable to other sources.Knowledge bases encode facts as subject-predicate-object triples.Source: §2.1 · pp. 2–3
2.2 Open-domain question answering from text (TextQA)
- TextQA uses retrieval with neural machine reading, but remains limited by excluding semi-structured sources such as tables and by underperformance on KB-origin benchmarks such as WebQuestions.KBQA itself is limited in fact coverage and the question types it can answer.Source: §2.2 · p. 3
2.3 Question answering from tables
- Tables provide factual knowledge, but their inconsistent formatting, ambiguous cells, and lack of formal schema make them semi-structured; prior table-QA work often omitted text or retrieval.This work targets simple natural questions and tests whether adding tables improves popular TextQA and KBQA benchmarks.Source: §2.3 · p. 3
2.4 Fusion of text and knowledge-base
- Earlier fusion methods preserve entity-relation structure, limiting their ability to capture TextQA’s richness; UniK-QA discards structure and applies retriever-reader methods across more sources and datasets.Compared with concurrent work (Agarwal et al., 2021), it covers tables, lists, multiple KBs, stronger baselines, and a single-model multi-dataset setting.Source: §2.4 · p. 3
3 Modeling
UniK-QA uses a retriever-reader architecture to process text and linearized structured sources together. It represents KB relations, lists, and tables as text documents that can be retrieved and jointly read.
3.1 UniK-QA architecture
- A dense DPR retriever selects heterogeneous documents, and a FiD reader independently encodes each question-document input before fusing them through decoder attention.The reader uses T5-large and 100 context documents; iterative hard-negative mining uses two iterations.Source: §3.1 · p. 4
3.2 Unified representations for KBs
- KB triples are serialized from the surface forms of subject, predicate, and object; n-ary relations become sentences with a comma-separated clause for each predicate.Simple heuristics achieve 94.7% WebQSP dev retrieval recall @100, versus 87.9% for the tested WebNLG model; heuristics were retained for all experiments.Source: §3.2 · p. 4
- DPR indexes individual KB relations, can combine multiple KBs in one index, and retrieves from a high-recall two-hop neighborhood when searching billions of relations directly poses engineering challenges.Retrieved short relations are grouped into passages of at most 100 tokens for FiD; representing complex relations as one text piece avoids treating them as multi-hop in traditional graph-based KBQA.Source: §3.2 · pp. 4–5
3.3 Unified representations for lists & tables
- Lists remain in text documents without special preprocessing, while tables are linearized by joining row cell values with commas and combining rows with newlines.The table corpus contains 455,907 tables; chunks contain 100 tokens and repeat the first non-empty row as header, a heuristic that adds 4–6 points to top-20 passage accuracy.Source: §3.3 · p. 5
4 KBQA as TextQA: A Motivating Experiment
The experiment tests whether UniK-QA’s retriever-reader approach can treat KBQA as TextQA using Freebase relations. It outperforms prior state-of-the-art KBQA methods on WebQSP.
4 KBQA as TextQA: A Motivating Experiment
- 79.1% exact match accuracy outperforms previous state-of-the-art methods on WebQSP using the same KB setup and preprocessing procedure.The authors attribute the improvement to the UniK-QA model.Source: §4 · p. 6
5 Multi-Source QA Experiments
The experiments test whether a unified model can combine text, tables, lists, and knowledge bases across four open-domain QA datasets. Adding structured sources improves results, including in a setting where the question source is unknown.
5.1 Datasets
- The four evaluation datasets are NaturalQuestions, WebQuestions, TriviaQA, and CuratedTREC, spanning questions associated with Wikipedia, Freebase, web trivia, and open-domain text.NQ includes answer spans in text and some tables; WebQ uses Freebase as its answer source.Source: §5.1 · p. 6
5.2 Combinations of sources
- The five variants range from Text-only, Tables-only, and KB-only baselines to Text + tables and Text + tables + KB.Text and table chunks share a dense index; retrieved KB relations are separately indexed and merged with retrieved text and table passages using a fixed quota.Source: §5.2 · p. 6
5.3 A multi-dataset model
- The multi-dataset setting trains one model on all four datasets and evaluates it without telling the model the question source.WebQ and TREC are upsampled five and eight times, respectively, during training.Source: §5.3 · p. 6
5.4 Results
- On WebQ, KB-only performs 5% better than text-only, and adding structured sources yields state-of-the-art results on NQ, WebQ, and TREC.Table 2 reports exact-match results on the test set.Source: §5.4 · pp. 6–7, Table 2 · p. 7
- 5.4 points on average: in the multi-dataset setting, Text + tables + KB outperforms the Text-only baseline.Performance is generally lower than for per-dataset models, especially on WebQ and TREC; the authors suggest the model may exploit less source-specific prior knowledge.Source: §5.4 · p. 7, Table 2 · p. 7
6 Analysis
The analysis examines whether adding sources improves retrieval and whether retrieved evidence from those sources relates to better answers. It also tests whether gains remain when the reader receives text alone.
Retriever
- Adding knowledge sources improves retrieval recall on the NQ development set.Table 4 reports retrieval recall across settings, with Tables-only results restricted to the NQ development subset whose answers are in tables.Source: §6 · p. 7, Table 4 · p. 8
Reader
- For questions improved by the expanded model, positive contexts from added sources correlate with better outcomes on NQ and WebQ.Figure 3 compares the share of positive contexts from added sources in all questions and in the improvement set; the added sources are lists and tables for NQ, and KB for WebQ.Source: §6 · p. 8, Figure 3 · p. 8
Other factors
- 48.7 EM: the Text + tables model given text-only input performs similarly to a text-only model on the same input on the NQ test set.The authors interpret this as evidence that varied sources and additional table training samples are not major contributors, and that gains primarily reflect added structured knowledge.Source: §6 · p. 8
7 Implementation Details
The implementation converts structured sources into retrievable passages and combines retrieved KB and text evidence for the reader. It uses DPR for retrieval and FiD with T5-large as the reader.
7 Implementation Details
- UniK-QA code, data, and trained model checkpoints are available in the project repository.Source: §7 · p. 8
7.1 DPR Training
- DPR is trained on Wikipedia text, lists, and linearized tables, while KB relations are retrieved without retraining DPR.The setup uses BERT-base encoders, 100-token passages, one negative document per question, 40 epochs, batch size 128, and learning rate 10^-5.Source: §7.1 · p. 8
7.2 FiD Training
- The reader uses FiD with T5-large, 100 context documents, and the original FiD hyperparameters where possible.Training runs for 10,000 steps with batch size 64 on 64 V100 GPUs; no hyperparameter search was performed.Source: §7.2 · p. 9
7.3 Merging KB and Text
- KB passage quotas are tuned on development-set retrieval recall, and text and KB passages are interleaved in the final context.The quota is the number of KB passages among 100 total context passages; it is relatively stable across KB choices and highest for WebQuestions, where KBs also bring the most gains.Source: §7.3 · p. 9, Table 5 · p. 9
8 Discussion
UniK-QA applies a retriever-reader framework to structured and unstructured sources and reports improved QA results. Flattening sources into text sacrifices some structured-KB capabilities, though the authors outline possible extensions.
8 Discussion
- UniK-QA uses a general retriever-reader framework for structured and unstructured sources, improving over traditional KBQA and achieving new state-of-the-art results on two open-domain QA benchmarks.The discussion describes the approach as effective for structured sources and for combining sources.Source: §8 · p. 9
- Flattening sources into text loses the ability to return all answers to a query and to infer multi-hop paths for more complex questions.The authors focus on exact match (equivalent to Hits@1), which largely ignores multiple answers, and evaluate datasets composed mostly of simple questions answerable from one piece of information.Source: §8 · p. 9
- The authors suggest merging outgoing edges with the same relation or generating multiple reader answer candidates to address the lost ability to return all answers.They also note that dense retrieval methods for multi-hop QA could be applied within this framework, but comparison with traditional structured methods remains to be seen.Source: §8 · p. 9