Source-linked AI summary
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang
TL;DR
Existing hybrid-data QA benchmarks lack multi-step numerical reasoning across multiple hierarchical tables. The paper introduces the annotated MULTIHIERTT benchmark and MT2Net, which retrieves supporting facts before reasoning over them; MT2Net reaches 38.43% F1 versus 87.03% for human experts, showing the benchmark remains challenging for current QA models.
Problem
Existing hybrid-data QA benchmarks use a single flat table per document and lack examples requiring multi-step reasoning across multiple hierarchical tables and paragraphs.
Method
The paper constructs MULTIHIERTT with annotated reasoning processes and supporting facts, and introduces MT2Net to retrieve facts from tables and text before generating executable reasoning programs.
Results
MT2Net achieves 38.43% F1, while human experts achieve 87.03% F1, demonstrating that MULTIHIERTT strongly challenges existing QA models.
Takeaways & Limitations
MULTIHIERTT provides a benchmark for numerical reasoning over documents with multiple hierarchical tables and paragraphs, motivating further QA research.
Takeaways & Limitations
Models perform significantly worse on questions requiring multiple tables, multi-step reasoning, complex hierarchical tables, or external financial knowledge.
Abstract
from arXiv · showhide
Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multi-step numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data. MultiHiertt is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain multiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each question is more complex and challenging than existing benchmarks; and 4) fine-grained annotations of reasoning processes and supporting facts are provided to reveal complex numerical reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reasoning over retrieved facts. We conduct comprehensive experiments on various baselines. The experimental results show that MultiHiertt presents a strong challenge for existing baselines whose results lag far behind the performance of human experts. The dataset and code are publicly available at https://github.com/psunlpgroup/MultiHiertt.
1 Introduction
MULTIHIERTT addresses the lack of hybrid-data QA benchmarks with multiple hierarchical tables and develops MT2Net for retrieving facts and reasoning over them. Experiments show that the benchmark remains substantially harder for QA models than for human experts.
- Motivation: Existing hybrid-data QA datasets contain only one flat table per document, leaving multi-step reasoning across multiple paragraphs and hierarchical tables underrepresented.Hierarchical headers require multi-level, bidimensional cell indexing, and ignoring them can produce incorrect selections.
- Dataset: MULTIHIERTT provides 10,440 expert-annotated QA examples with reasoning-process and supporting-fact annotations.The dataset applies strict quality control to support meaningfulness, diversity, and correctness.
- Results: 38.43% F1 is achieved by MT2Net, compared with 87.03% F1 for human experts, while baseline models still lag far behind human performance.The results identify MULTIHIERTT as a strong challenge for existing QA systems.
- Dataset: Each MULTIHIERTT document contains multiple hierarchical tables and longer unstructured text, requiring more complex reasoning across tables and paragraphs.The benchmark is designed for numerical reasoning over financial-report documents.
- Method: MT2Net retrieves relevant supporting facts from hierarchical tables and text before generating executable reasoning programs to answer questions.This separates evidence selection from symbolic reasoning over the retrieved facts.
2 Related Work
Prior QA benchmarks span text, tables, knowledge bases, and hybrid data, but MULTIHIERTT targets numerical reasoning across multiple hierarchical tables and paragraphs.
- Existing QA datasets cover textual data, structured tables, knowledge bases, and hybrid text-table settings.
- Numerical and multi-hop reasoning have become important directions across recent QA benchmarks.
3 MULTIHIERTT Dataset
MULTIHIERTT is constructed from filtered financial-report documents and expert-guided annotations, with quality controls targeting diverse, meaningful, and correct multi-source numerical reasoning.
- 3.1 Data Collection and Preprocessing: MULTIHIERTT uses FinTabNet financial-report pages with detailed HTML annotations for table hierarchies and cell information.
- 3.1 Data Collection and Preprocessing: Documents are filtered to contain 1–4 pages, 2–6 tables, substantial text, and sufficient numerical information before hierarchical preprocessing.
- 3.2 Annotation: Annotators compose complex numerical questions using textual content, tabular content, or multiple tables, then provide detailed reasoning annotations.
- 3.3 Quality Control: Quality control combines expert feedback, annotator training, multi-round validation, and human evaluation of 100 samples.
- 3.4 Dataset Analysis: MULTIHIERTT contains 10,440 QA pairs and is described as the first dataset for numerical reasoning over hybrid documents with multiple hierarchical tables.
- 3.4 Dataset Analysis: 48.74% of questions require both text and table information, while 23.20% require information from more than one table.
- 3.4 Dataset Analysis: The average numerical reasoning program contains 2.47 steps, with 18.10% of annotated programs requiring more than three steps.
4 MT2Net Model
MT2Net retrieves supporting facts from hierarchical tables and paragraphs, then predicts answers through arithmetic-program or span reasoning modules.
- MT2Net first retrieves relevant supporting facts from hierarchical tables and paragraphs, then reasons over the retrieved facts.
- Fact Retrieving Module: The retrieval module addresses documents exceeding 3,000 tokens by selecting supporting facts before reasoning.
- Fact Retrieving Module: A BERT-based bi-classifier ranks candidate facts, and the top-n retrieved sentences are reordered by their original document positions.
- Reasoning Module: The reasoning module separates answers into arithmetic programs and spans, with a distinct sub-module for each type.
- Program sub-module: The program sub-module generates executable programs from retrieved numbers, operators, and previously generated tokens, then executes them.
- Span sub-module: The span sub-module selects an answer span from retrieved sentences using start- and end-position probabilities.
5 Experiments
Experiments show that MT2Net’s retrieving–reasoning design and hierarchical-table handling outperform baselines, while performance declines on increasingly complex reasoning and remains below human experts.
- Model Performance: MT2Net outperformed other baselines, while all models performed far behind human experts.
- 76.4% top-10 recall and 80.8% top-15 recall were achieved by the fact retrieving module.
- Ablation Analysis: The end-to-end Longformer baseline fell far behind because irrelevant numerical information in longer inputs makes reasoning difficult to learn.
- Baseline Comparison: TAGOP and FinQANet performed worse than MT2Net because row-wise flattening loses hierarchical table structure during retrieval.
- Reasoning Module: Reasoning-program models generally outperformed direct answer generation, while NumNet and TAGOP underperformed because they support limited symbolic reasoning.
- Further Analysis: Model accuracy deteriorated with more than two numerical reasoning steps and with questions requiring cross-table supporting facts.
6 Conclusion
The paper introduces MultiHiertt and MT2Net for numerical reasoning over documents with multiple hierarchical tables and paragraphs. Current QA models remain substantially below human performance, motivating further research.
- MultiHiertt is a large-scale QA dataset for complicated numerical reasoning over documents containing multiple hierarchical tables and paragraphs.
- MT2Net retrieves supporting facts from financial reports and generates executable reasoning programs to answer questions.
- 38.43% best F1 lagged far behind human expert performance of 87.03% F1.
7 Ethics Considerations
The dataset combines FinQA and FinTabNet under licenses permitting additional annotations, with annotation work conducted by paid experts and graduate students.
- MULTIHIERTT data was collected from FinQA and FinTabNet, whose licenses permit composing, modifying, publishing, and distributing additional annotations.
- External annotation hired 23 graduate students in finance or similar disciplines at around $1.1 per unit task.
- Annotation required approximately 1500 working hours and lasted about 70 days.
A Dataset Annotation
The annotation materials define the operators used for annotators, providing an operational reference for constructing reasoning annotations.
- Table 7 provides definitions of all operations used for annotators.