PaperlayerView exact revision on arXiv

Source-linked AI summary

HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation

Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, Dongmei Zhang

arXiv:2108.06712v3cs.CLcs.IR

TL;DR

Existing table-reasoning methods largely overlook hierarchical tables, whose multi-level indexing and implicit numerical and semantic relationships complicate QA and NLG. HiTab addresses this gap with a cross-domain dataset, fine-grained alignments, hierarchy-aware logical forms, and partially supervised training; experiments show strong benchmark difficulty and improved task performance. The dataset is bounded by manual handling of some complex formulas and precise-cell alignment that neglects textual rounding.

  • Problem

    Existing table-reasoning work mainly focuses on flat tables, despite hierarchical tables requiring multi-level indexing and containing implicit calculation and semantic relationships.

  • Method

    HiTab combines hierarchical tables from statistical reports and Wikipedia with revised descriptions, QA pairs, entity and quantity alignments, hierarchy-aware logical forms, and partially supervised training.

  • Results

    HiTab is a challenging benchmark; hierarchy-aware reasoning and alignment-guided methods improve QA, while aligned cells and calculation types help generate meaningful NLG texts.

  • Takeaways & Limitations

    HiTab provides a broad benchmark for future research on QA and NLG over complex hierarchical tables.

  • Takeaways & Limitations

    Some XLOOKUP or IF cases provide answer-cell references instead of explicit formulas, and alignment uses precise cell quantities rather than rounded descriptions.

Abstract

from arXiv · show

Tables are often created with hierarchies, but existing works on table reasoning mainly focus on flat tables and neglect hierarchical tables. Hierarchical tables challenge existing methods by hierarchical indexing, as well as implicit relationships of calculation and semantics. This work presents HiTab, a free and open dataset to study question answering (QA) and natural language generation (NLG) over hierarchical tables. HiTab is a cross-domain dataset constructed from a wealth of statistical reports (analyses) and Wikipedia pages, and has unique characteristics: (1) nearly all tables are hierarchical, and (2) both target sentences for NLG and questions for QA are revised from original, meaningful, and diverse descriptive sentences authored by analysts and professions of reports. (3) to reveal complex numerical reasoning in statistical analyses, we provide fine-grained annotations of entity and quantity alignment. HiTab provides 10,686 QA pairs and descriptive sentences with well-annotated quantity and entity alignment on 3,597 tables with broad coverage of table hierarchies and numerical reasoning types. Targeting hierarchical structure, we devise a novel hierarchy-aware logical form for symbolic reasoning over tables, which shows high effectiveness. Targeting complex numerical reasoning, we propose partially supervised training given annotations of entity and quantity alignment, which helps models to largely reduce spurious predictions in the QA task. In the NLG task, we find that entity and quantity alignment also helps NLG models to generate better results in a conditional generation setting. Experiment results of state-of-the-art baselines suggest that this dataset presents a strong challenge and a valuable benchmark for future research.

1 Introduction

HiTab addresses the neglect of hierarchical tables in table reasoning by introducing a dataset for QA and NLG, with annotations and methods targeting hierarchical indexing, numerical reasoning, and semantic relationships.

  • Motivation: Existing table-reasoning work mainly targets flat tables, while hierarchical tables use multi-level headers and are widespread in statistical and scientific settings.Hierarchical tables require compositional, two-dimensional cell selection and contain implicit relationships among quantities and entities.
  • Dataset: HiTab builds QA and NLG data from expert-authored statistical-report descriptions, revising them into meaningful, diverse sentences and questions.The construction process preserves descriptions grounded in real table understanding rather than asking annotators to create questions from scratch.
  • Annotations: Fine-grained entity and quantity alignment annotations expose numerical relationships, including composite quantities computed from multiple cells.These annotations support reasoning over entities, quantities, and formulas in both tasks.
  • Methods: HiTab introduces a hierarchy-aware logical form and partially supervised training to address hierarchical reasoning and spurious QA programs.The approach targets both structural selection and complex numerical reasoning in hierarchical tables.
  • Results: 45.1% accuracy is achieved after partially supervised training, while NLG conditioning on aligned cells and calculation types produces meaningful texts.The reported QA result accompanies findings that HiTab remains challenging for existing methods.

2 Dataset Construction and Analysis

HiTab combines hierarchical tables from statistical reports and Wikipedia with expert-oriented annotations, quality checks, and analyses showing broad domain coverage and challenging numerical reasoning.

  • 2.1 Hierarchical Table Collection: HiTab collects tables from StatCan, NSF, and Wikipedia, using HTML statistical reports to extract cell information and balancing report and Wikipedia sources.The construction includes 1,083 StatCan reports, 208 NSF reports, and 1,851 Wikipedia tables from ToTTo.
  • 2.2 Sentence Extraction and Revision: Annotators extract and revise table descriptions, align entity mentions and quantities to cells, and record composite quantities with spreadsheet formulas.Questions are formed by converting declarative descriptions into QA pairs whose answers are represented by formulas.
  • 2.3 Entity and Quantity Alignment: Some complex XLOOKUP or IF cases omit explicit formulas and instead provide manually checked answer-cell references.This scope boundary applies when reasoning logic is not fully covered by spreadsheet formulas, such as discontinuous candidate cells.
  • 2.3 Entity and Quantity Alignment: Rounded descriptions are mapped to precise table-cell quantities, intentionally neglecting rounding during alignment.The alignment procedure therefore uses the exact quantities in cells rather than rounded values in text.
  • 2.4 Hierarchy Extraction: 94% of 100 randomly sampled tables have precisely extracted hierarchical structures using formatting-based heuristics.The extracted top and left header trees model the table hierarchy.
  • 2.7 Dataset Statistics and Comparison: HiTab contains 98.1% hierarchical tables, broad domain coverage, fine-grained alignments, and richer statistical-report descriptions than comparable sources.Statistical reports contain 5.0 real descriptions per table, compared with 1.4 for ToTTo and 3.8 for scientific papers.
  • 2.7 Dataset Statistics and Comparison: The dataset covers 28 statistical-report domains, and many questions involve complex cell selection and numerical operations.Figure 2 analyzes distributions across domains and operations in StatCan and NSF.

3 Hierarchical Table QA

Hierarchical Table QA extends table reasoning beyond flat structures by handling multi-level indexing and implicit numerical relationships. HiTab addresses these challenges with hierarchy-aware logical forms and partially supervised training, while results show substantial but incomplete progress.

  • Challenges: Hierarchical tables require multi-level, bi-dimensional indexing and reasoning over implicit calculation and semantic relationships.Existing logical forms are designed for flat or database tables and do not adequately support these structures.
  • Logical form: The hierarchy-aware logical form treats rectangular data regions indexed by left and top headers as operating objects.Execution separates hierarchical region selection from region operations, enabling arithmetic over cells that may share rows or columns.
  • Logical form: Two hierarchical filters select regions by header subtrees and tree levels before operators produce answers.The logical form defines 19 operators, including operators such as difference rate for hierarchical tables.
  • Results: MAPO with the hierarchy-aware logical form improves execution accuracy by 11.5% over MAPO with its original logical form.MAPO reaches 40.7% execution accuracy with a 19% spurious program rate, while more than half of questions remain incorrect.
  • Results: Partial supervision improves execution accuracy and lowers spurious programs by using entity links, quantity links, and calculation annotations.For MML, execution accuracy rises from 36.7% to 45.1%; error analysis identifies missing entities, region selection, operations, and logical-form coverage as failure categories.
  • Results: QA accuracy decreases as combined left and top header levels increase, reflecting the greater complexity of deeper table hierarchies.Tables without hierarchies are an exception, performing relatively worse while representing only 1.9% of HiTab tables.

4 Hierarchical Table-to-Text

HiTab frames hierarchical table-to-text as controlled generation: models receive selected cells and operators, then generate faithful descriptions despite compositional indexing and numerical reasoning challenges.

  • Task Definition: The task generates a faithful description S from a hierarchical table T, highlighted cells C, and specified operators O.
  • Controlled Generation: Operators clarify numerical intent because SUM, AVERAGE, and COUNT over identical cells should produce different texts.
  • Controlled Generation: Precomputed operation results are added as inputs because sequence-to-sequence models struggle to perform arithmetic reliably.
  • Input Construction: The subtable-selection heuristic expands highlighted cells through indexed data cells, corresponding headers, and parent headers in the hierarchy.
  • Results: Adding operators to cell highlights substantially improves both BLEU and PARENT, while pretrained sequence-to-sequence models yield rises of +6.5 BLEU and +11.3 PARENT.
  • Results: For hierarchical depths 2, 3, and 4+, BLEU scores are 31.7, 26.5, and 21.3, while PARENT scores are 40.9, 36.5, and 31.6.

5 Related Work

Related work largely targets flat or domain-specific tables, whereas HiTab emphasizes hierarchical tables and operator-aware generation. This positions HiTab between more open-ended and more tightly specified table-to-text settings.

  • Table-to-Text: ToTTo contains only 5% hierarchical tables and does not indicate or use table hierarchies.
  • Table-to-Text: HiTab differs from ToTTo by conditioning generation on operators that reflect symbolic operations on cells.
  • Challenges: Figure 4 is identified as a meaningful but challenging HiTab case.
  • Table QA: Prior table QA datasets mainly cover database, flat web, domain-specific, or table-and-text settings, while hierarchical tables remain understudied.

6 Discussion

HiTab exposes cross-domain and complex-calculation challenges: performance decreases under domain-based splits, and difficult cases require jointly reasoning over quantities, header semantics, and hierarchies.

  • Cross-Domain Generalization: Cross-domain train/dev/test splits produce decreases across all QA and NLG metrics.
  • Complex Calculation: Complicated calculations require jointly considering quantity relationships, header semantics, and table hierarchies.

7 Conclusion

The conclusion presents HiTab as a cross-domain benchmark for QA and NLG over hierarchical tables, with fine-grained entity and quantity alignment. Experiments characterize it as challenging for research on complex tables.

  • Dataset: HiTab supports both question answering and natural language generation on hierarchical tables collected across statistical reports and Wikipedia domains.
  • Annotations: The dataset provides fine-grained annotations aligning entities and quantities in text with corresponding table cells.
  • Conclusion: Experiments with strong baselines suggest that HiTab is a challenging and valuable benchmark for future research on complex tables.

8 Ethical Considerations

HiTab is a free and open English dataset built from public statistical and Wikipedia sources, with annotation procedures addressing licensing and sensitive-content concerns.

  • HiTab combines processed tables, QA pairs, target text, entity–quantity mappings, recognized hierarchies, and source code.
  • The dataset draws on public statistical organizations and Wikipedia tables under the MIT license.
  • Annotators checked for names, unique identifiers of individuals, offensive content, and sensitive information, finding none.
  • Eighteen students or graduates from computer science, finance, and English were recruited, and each was paid $7.8 per hour.

A.1 Dataset Preprocessing

Dataset preprocessing filters tables by size, cell-content, hierarchy-parsing, and hierarchy-depth constraints; 85% of tables satisfy all constraints.

  • Tables must contain more than 2 and fewer than 64 rows and columns.
  • 85% of tables meet all preprocessing constraints.The constraints limit table dimensions, cell-string complexity, parsed hierarchies, and hierarchy depth.
  • Cell strings may contain at most one non-ASCII character and 20 tokens.
  • Hierarchies must be successfully parsed and contain no more than four levels on either side.

A.2 Annotation Artifacts

HiTab reduces potential annotation artifacts by revising report descriptions into diverse questions and testing whether tables, positions, or question text provide shortcuts.

  • Artifact mitigation: Annotators revise high-quality statistical-report descriptions into questions to promote diversity and naturalness.The source descriptions span 28 domains.
  • Artifact tests: 4.0% and 2.6% test accuracy were obtained when BERT+MAPO and TaPas used tables without questions.Both methods failed to converge in this setting.
  • Artifact tests: Shuffling table rows and columns produced similar performance within ±1% of the original tables.This indicates little correlation between answers and table-cell position in the reported experiment.
  • Artifact tests: Only 5.3% of answers occur in the questions.A model learning only to retrieve the question therefore cannot achieve high performance, according to the passage.
  • Dataset coverage: Figure 5 presents the distribution of samples across 29 domains.
  • Annotation workflow: Excel supports annotation because formulas can be written, executed, and checked for cell linking and calculation.

B.2 Baseline Implementation Details

The baselines use standard sequence and transformer architectures alongside hierarchy-aware logical forms and structured table linearization for hierarchical reasoning experiments.

  • Neural baselines: Pointer Generator uses a two-layer bidirectional LSTM encoder with 300-dimensional embeddings and 300 hidden units.Fine-tuning uses batch size 2, learning rate 0.05, and beam size 5.
  • Neural baselines: BART and T5 are fine-tuned with batch size 8 and learning rate 2e−4.BART uses facebook/bartbase and T5 uses t5-base.
  • Decoding: Decoded outputs are searched with beam size 5 and sequence lengths from 8 to 60 tokens.
  • Logical-form reasoning: The hierarchy-aware logical form supports union selection by allowing one or two headers in a filter tree.The implementation prunes each function’s search region using the selected region from the preceding function.
  • Logical-form reasoning: Order-relation functions and the argmax/argmin order argument k are deactivated because few questions use them and they increase spurious programs.
  • Logical-form coverage: 78.3% logical-form coverage was achieved after deactivation in 300 iterations of random exploration.Uncovered types include scale conversion, differently indexed operating data, and complex composite operations.
  • Table encoding: The question and table are concatenated, while table headers are linearized by level with [LEVEL] tokens and name–type representations.Headers with identical names gather token embeddings by mean pooling.
  • Table encoding: Figure 8 illustrates level representations and unique header-cell representations for a hierarchy-aware encoder input.
Loading 2108.06712v3…