Source-linked AI summary
TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chua
TL;DR
QA over closely linked tables and text is underexplored despite the prevalence of such hybrid data in financial reports. The paper introduces the TAT-QA dataset and TAGOP, which extracts evidence from both modalities and applies symbolic reasoning; TAGOP achieves 58.0% F1, improving on the best baseline by 11.1% but remaining below human performance.
Problem
QA over hybrid data containing closely linked tables and text is largely neglected, although this form is pervasive in financial reports and often requires numerical reasoning.
Method
The paper builds TAT-QA from real financial reports and proposes TAGOP to extract relevant table cells and text spans before symbolic reasoning with aggregation operators.
Results
58.0% F1 is an 11.1% absolute increase over the previous best baseline on TAT-QA, while human experts achieve 90.8% F1.
Takeaways & Limitations
TAT-QA can serve as a challenging benchmark and TAGOP as a baseline for developing QA models over hybrid data requiring numerical reasoning.
Takeaways & Limitations
TAGOP remains far below expert human performance, and about 84% of errors arise from failures to extract supporting evidence from tables and paragraphs.
Abstract
from arXiv · showhide
Hybrid data combining both tabular and textual content (e.g., financial reports) are quite pervasive in the real world. However, Question Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and the compositions. We further propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text. It adopts sequence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. TAGOPachieves 58.0% inF1, which is an 11.1% absolute increase over the previous best baseline model, according to our experiments on TAT-QA. But this result still lags far behind performance of expert human, i.e.90.8% in F1. It is demonstrated that our TAT-QA is very challenging and can serve as a benchmark for training and testing powerful QA models that address hybrid form data.
1 Introduction
TAT-QA addresses the scarcity of QA research on tightly linked tabular and textual data by introducing financial-report contexts that require numerical reasoning. TAGOP reasons over both modalities, but its performance remains well below expert human performance.
- Research gap: Hybrid-data QA research has largely focused on text, knowledge bases, or tables, while work combining closely linked tables and text remains rare.Existing simulated links between table cells and Wiki pages are described as relatively loose.
- Dataset: TAT-QA is a dataset of real-world financial-report contexts containing tables, associated paragraphs, questions, answers, and labeled derivations.Its contexts include tables with headers and numbers plus at least two paragraphs describing, analyzing, or complementing the table.
- Dataset: 16,552 questions are associated with 2,757 hybrid contexts from 182 financial reports.A large portion of the questions demand numerical reasoning, and answer derivations are labeled to support explainable models.
- Method: TAGOP extracts relevant table cells and text spans, then applies symbolic reasoning with aggregation operators to produce answers.The model also predicts numerical scales such as thousand, million, and billion.
- Results: 58.0% F1 is an 11.1% absolute improvement over the best baseline, but remains below the 90.8% F1 achieved by human experts.The experiments position TAT-QA as a challenging benchmark for hybrid-data QA involving numerical reasoning.
2 Dataset Construction and Analysis
TAT-QA is constructed from financial reports by selecting tables, linking relevant surrounding paragraphs, and having finance-trained annotators create question-answer pairs with derivations. Quality control and context-level splitting produce 2,757 hybrid contexts and 16,552 questions from 182 reports.
- Dataset scope: TAT-QA contains tables and relevant text extracted from real-world financial reports.The tables generally contain numbers and are paired with associated textual content.
- Table selection: About 500 reports yielded approximately 20,000 candidate tables after table detection and PDF extraction, restricted to 3–30 rows and 3–6 columns.Candidate tables had no standard schema and contained many numbers.
- Table selection: Candidate tables could contain extraction errors, so annotators manually deleted or corrected problematic tables.Examples include incorrect row or column counts and mis-detected numbers.
- Hybrid-context construction: Valid hybrid contexts pair a table with at least two relevant surrounding paragraphs.Annotators first checked that at least two paragraphs surrounded the table and then assessed their relevance.
- Question-answer creation: Annotators created financially useful questions using common wording, with at least six questions generated per hybrid context.Generated answers included labeled derivations such as executable arithmetic expressions or counted item sets.
- Quality control: About 30 finance-trained students annotated the data after qualification testing, followed by verifier-based quality control.A second verifier reviewed annotations after the first verifier’s confirmation.
- Dataset statistics: 2,757 hybrid contexts and 16,552 question-answer pairs were produced from 182 reports and split by context into training, development, and test sets.The split proportions were 80%, 10%, and 10%, ensuring questions from one context stayed in one split.
3 TAGOP Model
TAGOP extracts evidence from both tables and associated paragraphs, then applies learned aggregation and scale prediction to produce answers for hybrid-context questions.
- 3.1 Sequence Tagging: TAGOP sequence-tags the flattened table and associated paragraphs to identify relevant cells and text spans as supporting evidence.Each token receives an Inside–Outside label, and positive sub-tokens mark table cells or paragraph words for answer construction.
- 3.2 Aggregation Operator: TAGOP applies an operator classifier to route extracted evidence through ten aggregation operators, including selection, summation, counting, averaging, multiplication, division, difference, and change ratio.Some operators rank candidate numerical evidence before operating on the top two values.
- 3.2 Aggregation Operator: For Difference, Division, and Change ratio, TAGOP uses an auxiliary number-order classifier because reversing the two input numbers changes the result.The classifier represents the top two evidence tokens and predicts their order relative to the input sequence.
- 3.3 Scale Prediction: TAGOP predicts answer scale separately because financial numerical answers may require None, Thousand, Million, Billion, or Percent.The predicted numerical or string answer is multiplied or concatenated with the corresponding scale before comparison with the ground truth.
- 3.4 Training: TAGOP is trained with a joint objective that sums the losses for its four classification tasks.The supervision includes tagged evidence, the aggregation operator, answer scale, and number order where applicable.
4 Experiments and Results
Experiments compare textual, tabular, and hybrid QA baselines with TAGOP on TAT-QA. TAGOP performs best, but substantial errors remain, especially in extracting supporting evidence and handling domain knowledge.
- Comparison with Baselines: 50.1/58.0 vs. 37.0/46.9 in EM/F1 on test set, TAGOP outperforms the second-best baseline.The comparison reports EM/F1 values for TAGOP and the second-best model on TAT-QA’s test set.
- Comparison with Baselines: NumNet+ V2 outperforms BERT-RC among textual baselines, while Tapas and HyBrider show limited performance on TAT-QA’s hybrid data.The paper attributes these differences respectively to numerical reasoning capability, training on tabular data alone, and insufficient focus on table-paragraph interdependence and numerical reasoning.
- Answer Type and Source Analysis: TAGOP performs better on table-reliant questions than text-reliant questions, plausibly because table cells have clearer boundaries than text spans.The analysis compares performance by answer source and links the difference to evidence extraction difficulty.
- Error Analysis: 84% of sampled errors arise from wrong or missing supporting evidence, while wrong calculation accounts for 9% and scale error for 3%.The error analysis samples 100 test-set errors and also identifies unsupported calculation as 4%.
- Error Analysis: Some TAT-QA questions require domain knowledge, such as computing gross profit margin from gross profit and revenue.The paper identifies integrating finance knowledge into QA models as an open direction.
5 Related Work
Existing QA datasets largely focus on text or knowledge bases/tables, while hybrid QA combining tables and text has received limited attention. HybridQA connects tables to Wikipedia pages through hyperlinks, but the connection is relatively loose.
- Existing QA datasets primarily target textual data or knowledge bases and tables, whereas QA over hybrid data has received limited effort.
- HybridQA uses tables connected to Wikipedia pages through hyperlinks as its hybrid context.
- The table–text connection in HybridQA is relatively loose because it relies on manually added hyperlinks between table cells and Wiki pages.
6 Conclusion
The paper introduces TAT-QA and TAGOP for QA over real-world financial hybrid contexts whose tables depend closely on accompanying text. The dataset is challenging and is intended to support more advanced hybrid QA models, especially for numerical reasoning.
- TAT-QA contains real-world financial hybrid contexts with numeric tables and comprehensive dependencies on text.
- TAGOP aggregates information from tables and accompanying text and performs numerical reasoning with predefined operators to compute answers.
- Experiments show that TAT-QA is very challenging and requires further effort to address hybrid-data QA tasks.
- TAT-QA and TAGOP are intended to serve as a benchmark and baseline for developing QA technologies on more complex hybrid data requiring numerical reasoning.
A.1 Table Analysis
TAT-QA preserves the semi-structured nature of financial tables, which makes their structures difficult to interpret. Manual analysis found that around 79% of sampled tables contain at least two row headers.
- TAT-QA almost preserves the original table structures from financial reports to retain their semi-structured nature.
- The manual evaluation sampled 100 hybrid contexts from the training set to assess table-structure complexity.
- Around 79% of sampled tables have two or more row headers, indicating difficulty in interpreting financial table structures.
A.2 Operator Classifier
The operator-classifier analysis compares the ground-truth distribution of questions across aggregation operators with TAGOP’s prediction accuracy on the development and test sets.
- Table 8 reports the ground-truth proportion of questions assigned to each aggregation operator.
- Operator classification is assessed by comparing which aggregation operator questions should use with the classifier’s predictions.
- The analysis evaluates TAGOP’s operator-classifier performance on both development and test sets.
A.3 Scale Prediction
This section reports ground-truth answer-scale proportions and evaluates TAGOP’s scale predictor on the TAT-QA development and test sets.
- A.3 Scale Prediction: The analysis measures the proportion of the ground-truth scale appearing in an answer.
- A.3 Scale Prediction: The section also reports TAGOP’s scale-predictor accuracy on both dataset splits.
- A.3 Scale Prediction: Table 9 reports ground-truth scale proportions for the TAT-QA development and test sets.