Source-linked AI summary
Open Domain Question Answering over Tables via Dense Retrieval
Jonathan Herzig, Thomas Müller, Syrine Krichene, Julian Martin Eisenschlos
TL;DR
Open-domain QA over tables requires retrieving relevant tables rather than assuming the table is given, but tables differ structurally from textual passages. The paper introduces a table-aware dense retriever with pre-training and mined hard negatives, and evaluates it on a new NQ-TABLES dataset. The retriever improves recall@10 from 72.0 to 81.1 and end-to-end exact match from 33.8 to 37.7 over a BERT-based retriever.
Problem
Open-domain table QA lacks suitable retrieval methods and datasets because existing QA systems usually assume that the relevant table is provided at test time.
Method
The paper introduces a table-aware dense retriever, pre-trains it on text-table pairs, improves it with mined hard negatives, and constructs NQ-TABLES from Natural Questions.
Results
Recall@10 improves from 72.0 to 81.1 and end-to-end exact match improves from 33.8 to 37.7 over a BERT-based retriever.
Takeaways & Limitations
A retriever designed for tabular context can outperform textual retrievers for open-domain QA over tables, with pre-training and hard negatives further improving retrieval.
Takeaways & Limitations
In a manual analysis, 44 of 100 examples were wrong, usually because they required table reasoning or common-sense knowledge.
Abstract
from arXiv · showhide
Recent advances in open-domain QA have led to strong models based on dense retrieval, but only focused on retrieving textual passages. In this work, we tackle open-domain QA over tables for the first time, and show that retrieval can be improved by a retriever designed to handle tabular context. We present an effective pre-training procedure for our retriever and improve retrieval quality with mined hard negatives. As relevant datasets are missing, we extract a subset of Natural Questions (Kwiatkowski et al., 2019) into a Table QA dataset. We find that our retriever improves retrieval results from 72.0 to 81.1 recall@10 and end-to-end QA results from 33.8 to 37.7 exact match, over a BERT based retriever.
1 Introduction
The paper addresses open-domain QA over tables, where systems must retrieve a relevant table before extracting an answer. It proposes a dense table retriever, a reader pipeline, and a dataset derived from Natural Questions.
- Open-domain table QA requires searching a large table corpus, parsing the relevant table, and extracting the answer.
- The approach retrieves candidate tables with a dense retriever and then uses a QA reader to select the answer.
- The paper extracts 11K examples from Natural Questions where the answer resides in a table.
2 Setup
Open-domain extractive QA over tables requires finding the relevant table in a large corpus before extracting an answer span. The system therefore separates table retrieval from question answering.
- Task definition: Open-domain extractive QA over tables asks a model to return answer spans from a relevant table in a large corpus.The training examples pair questions with answer-containing tables, while the corpus contains many candidate tables.
- Task definition: Unlike document-based open-domain QA, tables are semi-structured, so it is unclear whether passage-oriented methods transfer directly.Document QA typically retrieves passages and then uses a machine reader to select the answer.
- Two-step pipeline: The system first retrieves a small candidate set of tables from the corpus and then applies a QA model to extract the answer.This decomposition avoids applying the QA model independently to every table in the large corpus.
3 Dense Table Retrieval
The dense table retriever represents questions and tables in a shared vector space using table-aware TAPAS encoders. It is trained with in-batch negatives, pre-trained on text-table pairs, and strengthened with mined hard negatives.
- Retriever architecture: The dense table retriever selects K = 10 candidate tables by comparing question and table representations.At inference, table representations are computed offline and the closest tables are retrieved from the corpus.
- Retriever architecture: Dense encoding is used because answer-containing tables may not share tokens with the question.This allows similarity to reflect table contents rather than only lexical overlap.
- Retriever architecture: DTR uses TAPAS table embeddings to represent tabular structure, with separate question and table encoders linked by a learned similarity metric.TAPAS captures structure through features such as row and column identifiers.
- Training: Training uses in-batch negatives so each gold question-table pair is contrasted with the other tables in its batch.The model learns embeddings in which relevant pairs have smaller distance, corresponding to a larger dot product.
- Pre-training: Pre-training uses 21.3M text-table pairs extracted from 6.2M Wikipedia tables, based on text occurring near each table.The procedure adapts the Inverse Cloze Task intuition that relevant context should be semantically similar to the text span.
- Hard negatives: Mined hard negatives are the highest-scoring retrieved tables that do not contain the reference answer.The retriever is first used to identify difficult non-answer tables, which are then included in subsequent training.
- Inference: Fast maximum inner product search can replace exhaustive search when retrieval must scale to larger corpora.The paper identifies FAISS and SCANN as possible tools for this purpose.
4 Question Answering over Tables
The QA system uses a reader that jointly scores candidate tables and extracts an answer span, after retrieval supplies a small candidate set.
- The reader jointly encodes each question and candidate table with TAPAS, scoring candidates while extracting a suitable answer span.Candidate scoring uses the CLS token and a logistic loss.
- Answer extraction applies a softmax over possible spans, masking spans outside cells or crossing cell boundaries.The span representation concatenates contextual representations of the span’s first and last tokens.
- The system retrieves K = 10 candidate tables and adds the reference table during training when it is absent.
- At inference, the answer from the candidate table with the highest reader score is returned.
5 Dataset
The paper introduces NQ-TABLES, an English table-QA dataset extracted from Natural Questions, with tables collected and normalized from Wikipedia articles.
- NQ-TABLES is a new English dataset created from Natural Questions for open-domain question answering over tables.The source questions were originally designed for QA over Wikipedia articles.
- The retrieval and QA results are reported on the NQ-TABLES test set, with reported model numbers averaged over five random runs.
- 12K Natural Questions examples were identified where the answer resides in a table, forming question-table-answer triplets.Tables were extracted from article HTML and infobox tables were normalized by transposition.
- The corpus contains all tables appearing in articles across the Natural Questions splits, with train and development data split from the original training set.The original development questions serve as the test set.
- Near-duplicate tables from different versions of the same Wikipedia page were merged using token-based cosine similarity.
6 Experiments
Experiments compare table-specific and textual retrievers on retrieval and end-to-end QA, showing gains from tabular representations, pre-training, and mined hard negatives while exposing reader and data limitations.
- Retrieval: Retrieval recall@K measures whether the highest-scoring K tables contain the reference table.
- Retrieval: Mined hard negatives improve DTR retrieval by more than 5 R@10 points, and DTR mining outperforms BM25 mining by 0.6 R@10.
- Retrieval: All pre-trained dense models outperform BM25 by a large margin in table retrieval.
- End-to-End QA: DTR+hn significantly outperforms all baselines, while a TAPAS-based reader exceeds a BERT reader by more than 3 EM points.The significance result uses a McNemar’s test with p<0.05.
- Analysis: The best model answers 29% of questions correctly; 12% lack a candidate containing the answer, and 34% involve reader span-extraction failures.
- Analysis: Among 100 manually analyzed reader failures, 44 require table reasoning such as column maxima or common-sense knowledge.Other cases involved partial correctness, ambiguity, or missing table context.
7 Conclusion
The paper concludes that a retriever designed for tabular context can outperform textual retrievers for open-domain table QA, with further gains from pre-training and hard negatives.
- A tabular-context retriever outperforms textual retrievers for open-domain QA over tables.The retriever can also be effectively pre-trained and improved with hard negatives.
A Experimental Setup
The experimental setup specifies training configurations for the retrieval and QA models, including tuning ranges and computational scale.
- A Experimental Setup: The retrieval model uses 1M pre-training steps and up to 200,000 fine-tuning steps with early stopping based on recall@10.Training uses a batch size of 256, learning rate 1.25e-5, Adam, linear warm-up scheduling, and dropout rate 0.2.
- A Experimental Setup: The QA reader is trained for 50,000 steps from a public TAPAS checkpoint without early stopping.It uses a batch size of 512, learning rate 1e-6, dropout rate 0.2, and a maximum answer length of 10 word pieces.
- A Experimental Setup: The QA model’s hyper-parameter ranges are tuned using a black box Bayesian optimizer.Table 3 reports the hyper-parameter ranges used for tuning.
- A Experimental Setup: All models are trained on 32 Cloud TPU v3 devices, with retrieval pre-training taking approximately 6 days.Retrieval training takes approximately 4–5 hours, while QA training takes approximately 10 hours.
- A Experimental Setup: The model has 340M parameters, matching the parameter count of a BERT large model.
B Results
The evaluation reports table-retrieval results and end-to-end QA results on the NQ-TABLES development and test sets.
- B Results: Table retrieval results are reported on the NQ-TABLES development and test sets.Table 4 distinguishes hard negatives, BM25-derived hard negatives, and pre-training settings.
- B Results: The reported result layout includes dev and test splits and separate retriever, reader, EM, F1, and oracle metrics.
- B Results: End-to-end QA results are reported on the NQ-TABLES development and test sets.Table 5 varies the number of candidates and reports settings with hard negatives or BM25-derived hard negatives.