PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Question Answering over Tables and Text

Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, William W. Cohen

arXiv:2010.10439v2cs.CLcs.AI

TL;DR

Open QA systems had largely retrieved answers from unstructured text, motivating open QA over tables and text. The paper introduces OTT-QA and combines fused-block retrieval with cross-block reading, reaching 28% EM; it also identifies dataset and supervision boundaries.

  • Problem

    The paper addresses open-domain question answering when relevant evidence is distributed across tabular and textual data.

  • Method

    It constructs OTT-QA and uses fusion retrieval, fused-block pretraining, and a sparse-attention cross-block reader to retrieve and jointly analyze heterogeneous evidence.

  • Results

    28% EM is achieved by combining the two strategies, an 18% absolute improvement over the baseline; individually, fusion retrieval reaches 14% EM and cross-block reading 17% EM.

  • Takeaways & Limitations

    The combined fusion retriever and sparse reader increase effectiveness and efficiency by a large margin within open question answering over tables and text.

  • Takeaways & Limitations

    OTT-QA questions initially covered only 13K HybridQA tables, leaving the newly crawled 400K tables without questions in that evaluation.

Abstract

from arXiv · show

In open question answering (QA), the answer to a question is produced by retrieving and then analyzing documents that might contain answers to the question. Most open QA systems have considered only retrieving information from unstructured text. Here we consider for the first time open QA over both tabular and textual data and present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task. Most questions in OTT-QA require multi-hop inference across tabular data and unstructured text, and the evidence required to answer a question can be distributed in different ways over these two types of input, making evidence retrieval challenging -- our baseline model using an iterative retriever and BERT-based reader achieves an exact match score less than 10%. We then propose two novel techniques to address the challenge of retrieving and aggregating evidence for OTT-QA. The first technique is to use "early fusion" to group multiple highly relevant tabular and textual units into a fused block, which provides more context for the retriever to search for. The second technique is to use a cross-block reader to model the cross-dependency between multiple retrieved evidence with global-local sparse attention. Combining these two techniques improves the score significantly, to above 27%.

1 INTRODUCTION

The paper introduces open question answering over both tables and text, addressing a gap left by systems focused on a single data type. It presents OTT-QA and methods for retrieving and jointly reasoning over heterogeneous evidence.

  • Motivation: Open QA retrieves documents and analyzes evidence, but prior systems primarily handled free-form text rather than semi-structured tables.Tables compactly aggregate related facts, especially numeric information, and collections of homogeneous entities.
  • Contribution: The paper presents the first study to jointly exploit tables and text for open-domain question answering.It constructs the Open Table-and-Text Question Answering (OTT-QA) dataset for this setting.
  • Dataset: OTT-QA contains 45K human-annotated questions requiring retrieval and multi-hop aggregation over tables and text from all of Wikipedia.Questions are decontextualized so answers can be determined from the question alone, without provided context.
  • Baseline: The baseline combines an iterative retriever with a BERT reader, using beam search over document subsets before predicting an answer span.The iterative retriever reformulates queries using previously retrieved evidence.
  • Proposed Methods: Fusion retrieval groups aligned table segments and passages into fused blocks, while cross-block reading jointly models the retrieved blocks with sparse attention.The fused blocks provide richer multimodal context for retrieval.

2 BACKGROUND

Open QA uses a retriever-reader pipeline: retrieval selects evidence from a large corpus, and reading extracts the answer. The background describes sparse, dense, and iterative retrieval mechanisms.

  • Open QA Pipeline: An open QA system extracts an answer from a large corpus through retrieval followed by reading.The retriever selects passages, and the reader extracts the answer from the retrieved evidence.
  • Retrieval Functions: Sparse retrieval scores evidence units with BM25, whereas dense retrieval uses separate Transformers to encode the query and passage and compares their pooled representations.The dense score is the dot product between the query and passage [CLS] vectors.
  • Iterative Retrieval: Iterative retrieval appends previously retrieved blocks to the query at each round and uses beam search during testing.This formulation supports multi-hop retrieval by conditioning later retrieval on earlier evidence.

3 TASK AND DATASET

OTT-QA evaluates open-domain answering over independently retrieved table and passage candidates. Its questions and supervision combine decontextualized multi-hop data with noisy distant signals and in-domain/out-domain splits.

  • Task Setting: OTT-QA answers questions by extracting strings from either table candidates or passage candidates, using exact match and F1 for evaluation.Unlike HybridQA, its web-table candidates do not provide hyperlinks.
  • Corpus: The passage corpus contains over 5 million Wikipedia passages averaging 94 words, while the table collection includes newly crawled tables.Passages are limited to at most 12 introduction-section sentences.
  • Annotation: The dataset de-contextualizes questions in two annotation steps: adding context phrases only, then making each sentence more concise and natural.The first step restricts annotators to phrases from the context.
  • Supervision: Distant supervision for roughly 8K in-domain tables uses cell hyperlinks and approximate exact-match traversal, with a manually observed error rate of roughly 15%.The resulting fine-grained information is treated as weak supervision for training.

4 MODEL

The OTT-QA model replaces standard iterative retrieval and single-block reading with fusion retrieval and cross-block reading. Fused blocks provide multimodal context for retrieval, while sparse cross-block attention aggregates evidence across retrieved blocks efficiently.

  • Fusion Retriever: Unlike iterative retrieval, fusion retrieval retrieves fused blocks in one step, reducing encoding and retrieval cost.Iterative retrieval requires repeated query re-encoding, lacks stepwise supervision, and can propagate early retrieval errors.
  • Fusion Retriever: Fusion retrieval groups relevant table segments and passages into self-contained fused blocks before retrieval.Early fusion supplies richer multimodal context, especially for table segments with incomplete context.
  • Fusion Retriever: ICT pretraining creates pseudo-queries by combining a partially corrupted table segment with a sampled passage sentence, paired with the original fused block.The pretrained retriever is then fine-tuned on OTT-QA and used to retrieve top-K fused blocks.
  • Cross-Block Reader: The cross-block reader jointly processes the top-k retrieved fused blocks to model dependencies between them.It uses a long-range sparse-attention transformer that supports sequences exceeding 4K tokens.
  • Results: Fusion-Retriever and Cross-Block Reader are combined to obtain the highest score in the main-results evaluation.The table reports dev-set selection of sparse or dense retrievers followed by test-set results, with daggered entries denoting ablations.
  • Cross-Block Reader: Sparse attention reduces complexity from quadratic O(N^2) to linear O(N|R|), with N = 4096 and |R| = 84 in the experiments.Global states assigned to local blocks enable cross-block interaction through multiple layers of global-local structures.

5 EXPERIMENTS

Experiments evaluate retrievers and readers for OTT-QA, showing that fusion retrieval and cross-block reading substantially improve answer accuracy and retrieval recall. Additional results examine entity linking, ablations, and generalization to HotpotQA.

  • Main Results: 28% EM results from combining fusion retrieval and cross-block reading, exceeding the roughly 10% iterative-retriever and single-block-reader baseline.The combined 18% absolute improvement exceeds the sum of the individual improvements from replacing either component alone.
  • Main Results: 14% EM results from replacing iterative retrieval with the sparse fusion retriever, while 17% EM results from replacing the single-block reader with the cross-block reader.These correspond to 4.5% and 7% absolute improvements, respectively.
  • Ablations: Removing ICT pre-training and query augmentation reduces Dev-EM to 24.6%, while removing GPT-2 query augmentation reduces it to 22.1%.Replacing predicted hyperlinks with oracle links increases fusion-model performance by 7% EM, indicating remaining room for improvement.
  • Retriever Results: 48.1% HITS@4K results for sparse fusion retrieval versus 35.8% for iterative retrieval, while dense retrieval rises from 27.2% to 52.4%.HITS@4K measures whether the ground-truth block appears within the retrieved 4096 subword tokens.
  • Generalization Results: On HotpotQA, the model achieves competitive performance with similarly sized state-of-the-art models without an additional reranking step.Retrieved blocks are fed directly to the cross-block reader, simplifying and accelerating the system.

6 RELATED WORK

Related work spans web-table retrieval, joint tabular-text pre-training, and long-range transformer architectures. The paper differs by addressing open-domain QA over both tables and text and by using a sparse-attention cross-block reader.

  • Table Retrieval: Prior web-table retrieval studies use smaller private datasets or restrict evidence to tables, whereas this work releases a large-scale dataset with table and text evidence.The paper positions OTT-QA as a more realistic and challenging open-domain setting.
  • Tabular-Text Modeling: TAPAS and TABERT investigate joint pre-training over textual and tabular data, while this method also encodes spatial and logical table operations with special tokens and embeddings.The method draws inspiration from these models.
  • Long Range Transformer: Long-range transformer variants reduce O(n^2) attention costs through hierarchical, hashing, or global-local architectures, and the cross-block reader is based on ETC.ETC provides the sparse-attention foundation for modeling dependencies across retrieved blocks.

7 CONCLUSION

The paper studies open QA over tables and text and proposes fusion retrieval with a sparse reader. It reports large gains in effectiveness and efficiency and identifies extension to more modalities as future work.

  • The paper focuses on open question answering over tables and text.
  • Fusion retrieval and a sparse reader increase the model’s effectiveness and efficiency by a large margin.
  • Extending open question answering to modalities such as images or audio is identified as a future question.

A.1 DATASET ANNOTATION

Dataset annotation begins with filtering HybridQA examples and applies manual quality control to preserve semantics, multi-hop reasoning, and fluency.

  • Filtering: Filtering reduces HybridQA from over 72k questions and 13k tables to 46k questions paired with 9k in-domain tables.The heuristics remove tables lacking sufficient meta-information or containing excessive non-textual information, along with questions longer than 30 words.
  • Quality Control: Manual quality checks require that decontextualized questions retain their original semantics and answer, require multi-hop reasoning, and remain concise and fluent.Quality checking is performed over batches distributed to the same annotator.

A.2 DATASET EXAMPLES

OTT-QA contains diverse questions requiring retrieval and reasoning across tables and text, including single-hop, two-hop, multi-hop, and multiple-path cases. The paper’s examples illustrate how table decomposition, fusion, and retrieval support these reasoning chains.

  • Question types: 57% of sampled questions require reading one passage and one table, covering table bridge → answer text and text bridge → answer table cases.Single-hop questions account for 13% of the sample.
  • Question types: 30% of sampled questions require reading two passages and one table, mainly following a text bridge → table bridge → answer text chain.The analysis also identifies 57% as two-hop questions requiring one passage and one table.
  • Question types: 9% of questions are answerable from one text passage, 18% from two text passages, and 4% from two tables because Wikipedia contains redundant information.These multiple reasoning paths can provide alternative routes to an answer.
  • Table and block representation: Table decomposition adds titles, row-position and max/min tokens, plus column embeddings, producing compact table-segment representations.After alignment, related units are grouped into fused blocks separated by [SEP] tokens.
  • Retrieval strategies: Fusion retrieval groups relevant heterogeneous data before retrieval, while the iterative retriever conditions each round on previous retrieval results.The sparse fused retriever searches over enriched blocks instead of repeatedly retrieving standalone units.
  • Performance by question type: The best model shows a smaller gap over the baseline for 1-hop questions than for 2-hop and 3-hop questions.The comparison uses dense fusion retrieval with a cross-block reader versus dense iterative retrieval with a single-block reader.

C.2 RETRIEVER ERROR ANALYSIS

Retriever errors arise from lexical mismatch, failed table-passage fusion, numerical reasoning, and distracting near-duplicate evidence. Quantitative analysis indicates retrieval is the larger source of errors than reader span selection.

  • Retriever errors: The main retriever error types are low lexical overlap, fusion errors, numerical reasoning, and distracting passages or tables.These categories were identified through manual error analysis.
  • Retriever errors: Fusion errors occur when entity linking fails to fuse all hyperlinked passages, with the entity-linker error reported as F1=50%.This failure can prevent the fused block from containing all relevant information.
  • Retriever errors: Low-lexical-overlap errors mainly involve abbreviations and rephrased table metadata, such as ‘New York University’ versus ‘NYU’.The paper notes that this error is also common in other open QA datasets.
  • Dataset-specific errors: Fusion and distraction errors are especially specific to OTT-QA compared with the reported general low-lexical-overlap problem.The paper links distraction to similar information appearing across tables or passages.
  • Dataset-specific errors: Questions linked to over ten passages or containing over ten similar tables are difficult because relevant evidence is hard to fuse or distinguish.The examples involve the “Team Record” and “List of RMIT University people” pages.
  • Error attribution: 49% of examples cannot find the correct supporting block, while the reader fails to select the correct span for 23% of the remaining examples.The analysis attributes errors to both retriever and reader stages.

C.3 LENGTH SENSITIVITY ANALYSIS OF RETRIEVAL/READER

Increasing the retrieved-evidence length improves recall for both iterative and fusion retrievers, with a steeper improvement for fusion. Longer inputs also favor cross-block reading over independent single-block reading.

  • Retriever sensitivity: Recall@K significantly improves for both fusion and iterative retrievers as the retrieved-evidence limit increases from 400 to 4096.At low token budgets, the performance gap is smaller because single-hop questions dominate the dataset.
  • Retriever sensitivity: Fusion retriever improvement is steeper than iterative retriever improvement as the evidence-length limit increases.The paper attributes this to contextualized fusion blocks being easier to retrieve than standalone table segments or passages.
  • Reader sensitivity: At low token budgets, single-block and cross-block readers perform comparably.Their behavior changes as the available input length increases.
  • Reader sensitivity: At an input limit of 4000, the cross-block reader achieves better scores because sparse attention can digest long inputs.The single-block reader truncates information and can obtain lower exact-match scores because of introduced noise.
  • Reader sensitivity: Modeling cross-attention between retrieved evidence units is important for consistent answers in this dataset.Reading different blocks independently can lead to suboptimal predictions.
Loading 2010.10439v2…