HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data

Wenhu Chen; Hanwen Zha; Zhiyu Chen; Wenhan Xiong; Hong Wang; William Wang

Source-linked AI summary

HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data

Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, William Wang

arXiv:2004.07347v3Apr 15, 2020cs.CLcs.AI

TL;DR

Existing QA datasets do not guarantee that answering requires combining structured and unstructured evidence, motivating a benchmark for heterogeneous reasoning. The paper constructs HybridQA from Wikipedia tables and linked passages, then evaluates homogeneous and hybrid models. Homogeneous models score below 20% EM, whereas HYBRIDER exceeds 40% EM but remains far behind human performance.

Problem
Existing QA datasets were designed around either structured or unstructured information and do not guarantee that answers require aggregating heterogeneous evidence.
Method
The paper constructs HybridQA by crowdsourcing questions over Wikipedia tables and their hyperlinked passages, and introduces HYBRIDER to combine both information forms for multi-hop reasoning.
Results
EM below 20% for the two homogeneous models contrasts with EM over 40% for HYBRIDER, while HYBRIDER remains far behind human performance.
Takeaways & Limitations
HybridQA provides a challenging benchmark for studying question answering with heterogeneous information.
Takeaways & Limitations
HYBRIDER's staged design causes nonreversible error propagation because mistakes made earlier cannot be corrected in later stages.

Abstract

from arXiv · show

Existing question answering datasets focus on dealing with homogeneous information, based either only on text or KB/Table information alone. However, as human knowledge is distributed over heterogeneous forms, using homogeneous information alone might lead to severe coverage problems. To fill in the gap, we present HybridQA https://github.com/wenhuchen/HybridQA, a new large-scale question-answering dataset that requires reasoning on heterogeneous information. Each question is aligned with a Wikipedia table and multiple free-form corpora linked with the entities in the table. The questions are designed to aggregate both tabular information and text information, i.e., lack of either form would render the question unanswerable. We test with three different models: 1) a table-only model. 2) text-only model. 3) a hybrid model that combines heterogeneous information to find the answer. The experimental results show that the EM scores obtained by two baselines are below 20\%, while the hybrid model can achieve an EM over 40\%. This gap suggests the necessity to aggregate heterogeneous information in HybridQA. However, the hybrid model's score is still far behind human performance. Hence, HybridQA can serve as a challenging benchmark to study question answering with heterogeneous information.

1 Introduction

HybridQA addresses the gap in datasets that do not require combining structured and unstructured evidence. It introduces a Wikipedia-table-based benchmark and shows that heterogeneous reasoning substantially outperforms homogeneous baselines, while remaining difficult for current systems.

Motivation: Existing QA datasets generally use either free-form text or structured data, without guaranteeing that answers require heterogeneous evidence.Free-form corpora offer broader coverage, while structured data supports compositional multi-hop reasoning.
Dataset: HybridQA is a crowdsourced dataset built from Wikipedia tables and their hyperlinked passages, with questions requiring aggregation across both information forms.Each question is aligned with a table and multiple linked corpora.
Dataset: Approximately 70K question-answer pairs are aligned with 13,000 Wikipedia tables.The complementary information in WikiTables and text makes this setting suitable for hybrid question answering.
Dataset: The benchmark is both multi-hop and hybrid: reasoning hops may use tabular or textual information, and answers may come from either source.The annotation process was calibrated to reduce questions answerable through single-hop or homogeneous reasoning.
Results: EM below 20% for the Table-only and Passage-only models contrasts with EM over 40% for HYBRIDER.The authors report this gap as evidence for heterogeneous multi-hop reasoning, while noting that HYBRIDER remains far behind human performance.

2 Dataset

HYBRIDQA is constructed from manageable Wikipedia tables and linked passages through crowdsourced annotation, with explicit criteria and filtering intended to enforce hybrid multi-hop questions. The dataset also uses bias controls and difficulty-aware splits to support evaluation.

Table/Passage Collection: Tables contain 5–20 rows, 3–6 columns, and no more than 35% hyperlinked cells; linked Wikipedia pages contribute up to 12 introductory sentences.These collection rules aim to make tables viewable for crowd-workers while supplying textual information.
Question/Answer Collection: Each Mechanical Turk HIT presents one table with linked passages and asks a worker to write six questions and their answers.The annotation interface provides examples and explanations because questions must rely on both tabular and textual information.
Annotation Criteria: Accepted questions require multiple reasoning steps over both information forms, with table operations, passage selection, and minimum-span answers.Table operations include filtering, superlatives, cell hops, and extraction; text reasoning selects passages and extracts answer spans.
Annotation De-biasing: Annotation de-biasing addresses table-position, passage-position, and fake-hybrid-question biases through region highlighting, answer-span checks, and human-expert review.Fake-hybrid questions can appear to require a table hop even when a passage alone identifies the answer.
Filtering and Splits: The released dataset filters out answers absent from the table or passages, answers longer than 20 words, and questions whose answer passage is directly retrievable without tabular information.It further samples development and test sets evenly from simple and hard categories, classified using statistical features.

3 Data Analysis

HYBRIDQA contains syntactically diverse questions and a wide range of answer types, with most multi-hop questions requiring two or three reasoning hops across tables and passages.

3.1 Question Types: Question types are identified from the central question word and its neighboring three tokens, revealing syntactic diversity in HYBRIDQA.
3.2 Answer Types: The dataset covers a wide range of answer types, including more number-related and date-related questions than Yang et al. (2018).
3.3 Inference Types: Most questions require two or three hops to find the answer, combining table operations with linked-passage retrieval or extraction.
3.3 Inference Types: 23.4% use Table →Passage, while 20.3% use Passage →Table to connect table cells and linked passages.
3.3 Inference Types: 35.1% of questions use the Passage →Table→Passage chain, making it the most common inference pattern.
3.3 Inference Types: The remaining patterns include joint passage-table reasoning, parallel chains with comparison, and multiple chains with superlatives.

4 Model

HYBRIDER performs heterogeneous multi-hop question answering by linking questions to table cells and passages, then ranking cells, selecting hops, and extracting answers. Its three-stage architecture decomposes reasoning into ranking, hopping, and reading comprehension.

HYBRIDER: HYBRIDER addresses heterogeneous question answering with a three-stage architecture for multi-hop reasoning over tables and passages.The stages are ranking, hopping, and reading comprehension, each modeled with a separate neural network.
Linking: Question linking combines cell matching with passage retrieval to identify table cells relevant to the question.Cell matching uses explicit values, comparisons, or column extrema; passage retrieval maps relevant passages back to linked cells.
Cell Encoding: Retrieved cells encode content, table location, passage description, source, and a normalized linking score.The passage description is the evidence sentence most similar to the question, while source records how the entry was obtained.
Reasoning: The ranking model scores retrieved cells, and the hop model scores transitions to neighboring cells or the current cell.Cell representations support ranking over linked cells, while concatenated hop-pair representations support selecting the next cell.
Reasoning: The reading-comprehension model extracts answers from a hopped cell or its linked passage.For non-hyperlinked cells it outputs the cell text; otherwise it prepends the cell text to the linked passage before predicting answer boundaries.
Training & Inference: The three models are trained independently because direct marginal-likelihood optimization incurs a large computation cost.Weak supervision derives valid retrieved cells and hop targets from answer locations, while passages containing the ground-truth answer train the reading-comprehension model.

5 Experiments

Experiments compare homogeneous and heterogeneous QA models, analyze retrieval and reasoning behavior, and decompose HYBRIDER’s errors. HYBRIDER benefits from combining structured and unstructured information, but remains well below human performance and suffers from cascading stage errors.

Experimental Results: In-Table questions are roughly 8–10% more accurate than In-Passage questions because they do not require the reading-comprehension step.BERT-large-uncased is roughly 2% better than BERT-base-uncased, but performance remains far below human performance.
Heterogeneous Reasoning: Table-only and Passage-only models achieve accuracy below 20%, whereas HYBRIDER achieves up to a 50% EM increase by leveraging both data forms.The comparison supports heterogeneous reasoning over HYBRIDQA.
Retriever Threshold: Increasing the retrieval threshold raises correct-passage recall but burdens filtering, while decreasing it improves precision but can miss evidence for later reasoning.Experiments find the model relatively stable across threshold values.
Error Analysis: HYBRIDER errors are categorized as linking, ranking, hop, and reading-comprehension errors across its retrieval and reasoning stages.The analysis evaluates these categories on the full development set using BERT-large-uncased with τ=0.8.
Error Analysis: The product of step-wise accuracies, 87.4% × 87.9% × 89.2% × 61.9% ≈42%, matches the overall accuracy and highlights error cascading across reasoning steps.The reading-comprehension step is slightly more erroneous than the others.
Error Analysis: HYBRIDER’s staged reasoning improves explainability but makes earlier mistakes nonreversible in later stages.The authors suggest end-to-end reasoning as a possible way to alleviate this propagation problem.

6 Related Work

Related QA work largely uses either textual or structured evidence, while hybrid systems aggregate heterogeneous information. HYBRIDQA combines tables and text to address the complementary coverage and compositionality of these representations.

Text-Based QA: Text-based QA datasets study machine reading across settings including single-paragraph, open-domain, long-context, and multi-hop questions.Examples include SQuAD, DrQA, TriviaQA, SearchQA, and DROP.
KB/Table-Based QA: KB- and table-based QA exploits structured knowledge for its unambiguous and compositional properties.Datasets include WebQuestion, ComplexWebQuestions, WebQuestionSP, WikiTableQuestions, WikiSQL, SPIDER, and TabFact.
Information Aggregation: KBs and tables suffer from low coverage, motivating HYBRIDQA’s use of text as complementary information for answering natural questions.The paper positions heterogeneous evidence as a way to combine the strengths of both forms.
Information Aggregation: Earlier hybrid QA systems aggregate heterogeneous information through early fusion or learned retrieval and reasoning over corpus sentences, entities, and knowledge-base facts.GRAFT uses heuristics for question-specific subgraphs, while PullNet dynamically learns retrieval and reasoning.

7 Conclusion

The paper presents HYBRIDQA as a dataset for question answering over both tabular and textual data. It releases the benchmark, introduces HYBRIDER as a baseline, and frames the task as a challenging problem for future research.

Conclusion: HYBRIDQA is presented as the first hybrid question-answering dataset spanning tabular and textual data.The dataset is released to support research on answering questions with heterogeneous information.
Conclusion: The paper designs HYBRIDER as a strong baseline and offers insights into its behavior and limitations.The authors characterize HYBRIDQA as an interesting and challenging next problem for the research community.