Source-linked AI summary
FeTaQA: Free-form Table Question Answering
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Nick Schoelkopf, Riley Kong, Xiangru Tang, Murori Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev
TL;DR
Existing table QA datasets emphasize short-form factual answers, leaving complex reasoning and explanation underrepresented. FeTaQA introduces a Wikipedia-based free-form table QA dataset and evaluates pipeline and end-to-end approaches, finding that the end-to-end model substantially outperforms the pipeline model while the task remains challenging.
Problem
Existing table QA datasets are limited to short-form answers, despite users asking complex questions that require elaborate explanations.
Method
FeTaQA defines generative table QA over Wikipedia-based tables and benchmarks a semantic-parsing pipeline alongside an end-to-end model integrating comprehension, reasoning, and generation.
Results
The end-to-end model achieves much higher scores than the pipeline model, while human evaluation indicates that FeTaQA remains challenging.
Takeaways & Limitations
FeTaQA introduces new table QA challenges requiring innovative model designs beyond table semantic parsing.
Abstract
from arXiv · showhide
Existing table question answering datasets contain abundant factual questions that primarily evaluate the query and schema comprehension capability of a system, but they fail to include questions that require complex reasoning and integration of information due to the constraint of the associated short-form answers. To address these issues and to demonstrate the full challenge of table question answering, we introduce FeTaQA, a new dataset with 10K Wikipedia-based {table, question, free-form answer, supporting table cells} pairs. FeTaQA yields a more challenging table question answering setting because it requires generating free-form text answers after retrieval, inference, and integration of multiple discontinuous facts from a structured knowledge source. Unlike datasets of generative QA over text in which answers are prevalent with copies of short text spans from the source, answers in our dataset are human-generated explanations involving entities and their high-level relations. We provide two benchmark methods for the proposed task: a pipeline method based on semantic-parsing-based QA systems and an end-to-end method based on large pretrained text generation models, and show that FeTaQA poses a challenge for both methods.
1 Introduction
FeTaQA addresses the gap between short-form table QA benchmarks and users’ complex questions by requiring free-form answers that retrieve, reason over, and integrate multiple table facts. It introduces a dataset and two benchmark modeling approaches, while human evaluation shows the task remains challenging.
- Motivation: Existing table QA datasets primarily retrieve short-form facts, limiting their coverage of complex questions requiring explanations.Table QA commonly uses semantic parsing to generate executable logical forms, with answers typically extracted entities or facts.
- Dataset: FeTaQA provides long, informative, free-form answers that integrate multiple retrieved entities, their relations, and associated information into coherent explanations.The dataset targets retrieval, relational reasoning, information aggregation for abstract or ambiguous queries, and faithful answer generation.
- Methods: The paper evaluates generative table QA as sequence-to-sequence learning with an end-to-end T5 model and a pipeline model using TAPAS and separate generation.The end-to-end approach integrates query and table comprehension, logical reasoning, and language generation, whereas the pipeline separates content selection and surface realization.
- Evaluation: Human studies assess fluency, correctness, adequacy, and faithfulness, finding that FeTaQA is challenging and that substantial room for improvement remains.The evaluation covers both model-generated answers and reference answers, and the dataset is released online.
2 Dataset
FeTaQA is constructed to pair diverse Wikipedia tables with questions requiring retrieval and reasoning over multiple table facts, producing coherent free-form answers. Its statistics and analyses indicate substantial question complexity, broad topical coverage, and high-quality annotations.
- Dataset Construction: FeTaQA combines semi-structured Wikipedia tables, questions, free-form answers, metadata, and supporting table-cell annotations.The dataset is designed around moderately sized, intelligible tables and answers that integrate facts and inferences into coherent sentences.
- Dataset Construction: The dataset uses ToTTo table-grounded sentences and highlighted cells as answers, weak supervision, and retrieval labels.ToTTo provides sentences supported directly or inferentially by highlighted table regions.
- Dataset Construction: 10,330 instances were collected, with 3,039 from internal annotators and 7,291 from MTurk workers.The annotation process used internal and external phases with evaluation and filtering procedures.
- Data Analysis and Statistics: FeTaQA has a median of 6 highlighted cells and 18-token answers, indicating multi-entity reasoning with concise, table-grounded explanations.The highlighted-cell median is twice the corresponding number for ToTTo, while human evaluations indicate correctness, adequacy, and grounding.
- Data Analysis and Statistics: Topics are evenly distributed despite coverage being limited to ToTTo topics, with many instances in biography, sports, geography, media, politics, and government.The topic analysis uses Wikimedia Foundation’s topic categorization model.
- Data Analysis and Statistics: FeTaQA includes diverse questions asking about plural or abstract entities and has more non-quantitative how questions than existing table QA datasets.Examples of abstract entities include outcomes, results, margins, and percentages.
3 Models
The paper benchmarks generative table QA with an end-to-end model and a pipeline that separates retrieval from generation. The pipeline adapts TAPAS and T5, while the end-to-end approach adapts T5 to jointly comprehend tables, reason, and generate answers.
- Models: Two benchmark approaches evaluate generative table QA: an end-to-end T5 model and a pipeline using TAPAS with a separate generator.The approaches adapt established table QA and text-generation models to FeTaQA.
- Pipeline Model: The pipeline separates table semantic parsing from data-to-text generation because FeTaQA targets generated explanations rather than retrieved denotations.The parser retrieves table content, while the generator integrates it into a cogent sentence.
- Pipeline Model: TAPAS is fine-tuned on highlighted FeTaQA denotations, whose predicted cells are converted into triples for the Data-to-Text module.The pipeline begins from a TAPAS-base checkpoint fine-tuned on WikiTableQuestions.
- Pipeline Model: The Data-to-Text module fine-tunes T5 on DART and ToTTo to realize highlighted cells and metadata as natural-language answers.Triples represent table context, column headers, cell values, and selected metadata fields.
- End-to-End Model: The end-to-end model linearizes the table with [SEP]-separated rows and space-separated cells, prepending the question to the source sequence.This simple scheme addresses transformer input-length constraints by incorporating the question before the table representation.
4 Experiments
The experiments compare pipeline and end-to-end approaches for FeTaQA using automatic and human evaluations. End-to-end T5 performs best overall, while fine-tuning improves pipeline retrieval but substantial gaps remain.
- Experiment setup: The study evaluates generated answers with automatic metrics and human judgments of fluency, correctness, adequacy, and faithfulness.Automatic evaluation includes n-gram and semantic metrics, while human evaluation assesses four answer-quality criteria.
- Automatic evaluation: T5-large with end-to-end modeling achieves the highest scores in almost all evaluation metrics.The result is reported on the FeTaQA test set and exceeds pipeline performance despite using simple table linearization.
- Pipeline results: Fine-tuning the pipeline on FeTaQA with denotations as weak supervision improves its performance by almost 2 BLEU points.The same analysis reports improved retrieval capability after fine-tuning the table semantic parser.
- Pipeline results: The parser predicts two table cells on average before FeTaQA fine-tuning and three afterward, compared with a reference median of six highlighted cells.The authors identify low denotation prediction and lost relational information as likely sources of pipeline inadequacy.
- Human evaluation: Human evaluation shows that end-to-end outputs significantly outperform pipeline outputs in correctness, adequacy, and faithfulness.The comparison covers 100 samples and also indicates room for improvement relative to human references.
5 Related Work
Related work situates FeTaQA among generative QA, table semantic parsing, multimodal table QA, and data-to-text generation. FeTaQA distinguishes itself by framing table QA as generation requiring reasoning and integration over structured sources.
- Generative QA: Existing generative QA datasets use contexts such as Wikipedia articles, books, scripts, dialogues, or web documents and often contain copied source spans.ELI5 instead pairs complex questions with paragraph-length answers and relevant web documents.
- Generative QA: FeTaQA is presented as the first dataset for generative question answering over tables.Unlike multi-document generative QA, it emphasizes reasoning and integration given a structured knowledge source.
- QA over tables: Earlier table QA datasets commonly use semantic parsing to produce executable logical forms whose answers are extracted facts or entities in short form.Examples include WikiTableQuestions, SequentialQA, WikiSQL, and Spider.
- QA over tables: HybridQA, OTT-QA, and MultiModalQA combine tables with other modalities or text but retain extractive, short-form answers, whereas FeTaQA uses generation.This distinction defines a different target for table question answering.
- Data-to-text generation: Data-to-text work studies content planning, surface realization, pretraining, and table-grounded generation, including ToTTo and DART.The paper draws on these datasets and methods when developing its FeTaQA models.
6 Conclusion
The paper introduces FeTaQA for complex table questions requiring free-form explanations and presents pipeline and end-to-end modeling approaches. Its results favor end-to-end modeling and identify further challenges for table QA.
- Conclusion: FeTaQA introduces generative table question answering with complex questions requiring free-form, elaborate answers.The dataset extends table QA beyond short retrieved denotations.
- Conclusion: The paper proposes a pipeline combining table semantic parsing with a Data-to-Text generator and an end-to-end model combining comprehension, reasoning, and generation.The end-to-end model uses a simple table encoding strategy.
- Conclusion: The end-to-end model achieves much higher scores than the pipeline model requiring table semantic parsing.This conclusion is based on the reported experimental results.
- Conclusion: FeTaQA introduces new table question answering challenges that call for innovative model designs.The conclusion frames these challenges as a direction for future work.
A Appendix
The appendix provides supplementary materials on ToTTo sampling, FeTaQA annotation, and question-type distributions across table QA datasets.
- Appendix contents: Figure 7 presents a flowchart of the ToTTo instances sampling process.It documents supplementary dataset-construction procedures.
- Appendix contents: Figure 8 provides a screenshot of the FeTaQA annotation interface.It illustrates the interface used during annotation.
- Appendix contents: Figure 9 compares question-type distributions between FeTaQA and other table QA datasets.The comparison supplies supplementary context for dataset composition.