PaperlayerView exact revision on arXiv

Source-linked AI summary

AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry

Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, Soumen Chakrabarti

arXiv:2106.12944v1cs.CLcs.AI

TL;DR

Table QA models are often evaluated on flat Wikipedia tables, while business and scientific tables may have hierarchical headers and specialized vocabulary. AIT-QA introduces a human-authored airline-industry dataset and preprocessing for complex tables; evaluation finds that strong performance on simpler tables does not guarantee similar performance on AIT-QA.

  • Problem

    Open-domain Table QA datasets often use flat tables and lack the hierarchical headers and specialized vocabulary found in domain-specific documents.

    Source: §1 · p. 2

  • Method

    The paper introduces AIT-QA, a human-authored dataset from airline financial reports, and preprocessing that maps complex tables to simpler layouts for existing models.

    Source: §1 · pp. 2–3

  • Results

    State-of-the-art Table QA models struggle on AIT-QA, and strong performance on open-domain datasets with simple tables does not guarantee similar performance on complex domain-specific tables.

    Source: §1 · p. 3, §4.4 · p. 8

  • Takeaways & Limitations

    AIT-QA explicitly captures complex table structures, and its preprocessing translates those structures into layouts resembling those used to train existing models.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent advances in transformers have enabled Table Question Answering (Table QA) systems to achieve high accuracy and SOTA results on open domain datasets like WikiTableQuestions and WikiSQL. Such transformers are frequently pre-trained on open-domain content such as Wikipedia, where they effectively encode questions and corresponding tables from Wikipedia as seen in Table QA dataset. However, web tables in Wikipedia are notably flat in their layout, with the first row as the sole column header. The layout lends to a relational view of tables where each row is a tuple. Whereas, tables in domain-specific business or scientific documents often have a much more complex layout, including hierarchical row and column headers, in addition to having specialized vocabulary terms from that domain. To address this problem, we introduce the domain-specific Table QA dataset AIT-QA (Airline Industry Table QA). The dataset consists of 515 questions authored by human annotators on 116 tables extracted from public U.S. SEC filings (publicly available at: https://www.sec.gov/edgar.shtml) of major airline companies for the fiscal years 2017-2019. We also provide annotations pertaining to the nature of questions, marking those that require hierarchical headers, domain-specific terminology, and paraphrased forms. Our zero-shot baseline evaluation of three transformer-based SOTA Table QA methods - TaPAS (end-to-end), TaBERT (semantic parsing-based), and RCI (row-column encoding-based) - clearly exposes the limitation of these methods in this practical setting, with the best accuracy at just 51.8\% (RCI). We also present pragmatic table preprocessing steps used to pivot and project these complex tables into a layout suitable for the SOTA Table QA models.

1 Introduction

AIT-QA addresses the gap between flat, open-domain tables and domain-specific tables with hierarchical headers and specialized vocabulary. It introduces a human-authored airline-industry dataset and preprocessing for complex tables, and evaluates state-of-the-art Table QA models on it.

1 Introduction

  • Unlike flat web tables, airline and other business tables can require reasoning over hierarchical row and column headers; ignoring a row-header hierarchy can return a value for the wrong operations.The example distinguishes mainline Revenue Passenger Miles from a value covering both mainline and regional operations.Source: §1 · p. 2
  • Open-domain datasets often lack specialized business vocabulary, such as “Revenue Passenger Miles,” found in domain-specific documents.The introduction identifies this vocabulary gap alongside the difference in table structure.Source: §1 · p. 2
  • AIT-QA is a human-authored Table QA dataset built from major airlines’ 10-K reports and questions about table contents and key performance indicators.The dataset targets domain-specific use cases and includes complex tables with numerical values and airline technical vocabulary.Source: §1 · pp. 2–3
  • State-of-the-art Table QA models struggle on domain-specific tables, and strong performance on simple open-domain tables does not guarantee similar performance on complex tables.The authors identify the lack of a domain-specific Table QA dataset as a partial explanation for this gap.Source: §1 · pp. 2–3
  • AIT-QA adds preprocessing that translates complex tables, including hierarchical row and column headers, into simpler layouts resembling those used to train existing models.The authors describe this preprocessing as a technique to improve model performance on complex tables.Source: §1 · p. 3

2 Related Work

Prior work includes table retrieval datasets and Table QA datasets, with many Table QA resources based on Wikipedia tables. AIT-QA distinguishes itself by explicitly encoding hierarchical row and column headers, enabling analysis of complex table structures.

2.1 Table Retrieval Datasets

  • Table retrieval datasets include keyword-query collections built over table corpora, as well as datasets derived from Natural Questions and other sources.Examples include WikiTables and its annotated query dataset, GNQtables, and other retrieval collections.Source: §2.1 · p. 3

2.2 Table QA Datasets

  • Common Table QA datasets include WikiTableQuestions, WikiSQL, TabMCQ, and FeTaQA, while HybridQA, OTT-QA, and TAT-QA address questions requiring reasoning across tables and associated text.TAT-QA is based on financial reports and is described as the most similar of these multi-hop datasets to AIT-QA.Source: §2.2 · p. 4
  • AIT-QA explicitly annotates hierarchical row and column headers, unlike other datasets that omit row headers and use flat column headers.The paper notes that TAT-QA may include complex structures but does not explicitly encode the headers or their hierarchies.Source: §2.2 · p. 4

3 Dataset

AIT-QA was constructed from airline 10-K filings to capture KPI- and table-driven questions about complex tables. Annotators recorded question types, paraphrases, and header-hierarchy information, while the dataset preserves identified row and column header hierarchies.

Data acquisition

  • The dataset uses 10-K filings from five S&P 500 airlines, retrieved through the SEC EDGAR system.The airlines are Alaska, American, Delta, Southwest, and United.Source: §3 · p. 5

Question annotation

  • Annotators created KPI-driven questions from 13 airline KPIs and their variants, and table-driven questions about other concepts in the tables.KPI questions use analyst-relevant metrics and common synonyms; the second question type broadens coverage beyond the limited KPI set.Source: §3 · p. 5
  • Annotators added paraphrases to selected original questions and recorded whether each question depends on row-header hierarchy.Questions are classified as hierarchy-dependent when the higher-level row headers are needed to identify an unambiguous answer.Source: §3 · pp. 5–6
  • The first dataset version focuses on lookup questions whose answers appear in table cells, leaving aggregate questions for future work.Annotators recorded the answer cell and question metadata, and other annotators reviewed the pairs for validity and minor issues.Source: §3 · p. 6

Data preparation and header identification

  • Table Understanding identified the row and column header hierarchies for body cells, and those annotations are included in the dataset for model use.Before annotation, HTML tables were cleaned to remove extraneous rows and columns and rejoin physical cells representing one logical cell.Source: §3 · pp. 5–6

4 Experimental Evaluation

The evaluation tests whether Table QA systems trained on WikiSQL transfer to AIT-QA’s complex airline tables, using table transformations to accommodate their headers. RCI and TaPaS outperform TaBERT overall, while performance varies with question type, header hierarchy, and phrasing.

4.1 Baseline methods

  • TaBERT predicts a logical form from table and question encodings, while TaPaS selects answer cells and RCI predicts the answer’s row and column.TaPaS and RCI treat Table QA as classification rather than generating logical forms.Source: §4.1 · pp. 6–7

4.2 Experimental Setup

  • All three models were trained on WikiSQL and tested on AIT-QA without hyperparameter tuning.The evaluation used the models’ original released code and pretrained weights.Source: §4.2 · p. 7

4.3 Transforming Table Structures

  • The models assume flat column headers and no row headers, so the authors transformed AIT-QA tables to make them more compatible.Base transformations add row headers as a column and flatten header hierarchies by concatenating parent and child text.Source: §4.3 · p. 7
  • Transposing tables benefits RCI and TaBERT but lowers TaPaS performance; the authors select each model’s best-performing version for later analysis.They test no transpose, all-table transpose, and partial transpose when row headers have more characters than column headers.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Analyzing Baseline Performance on AIT-QA’s Dimensions

  • 52% accuracy for RCI, 49% for TaPaS, and 34% for TaBERT on AIT-QA; RCI and TaPaS lead overall.The authors also report that this relative trend holds across dataset slices.Source: §4.4 · p. 8
  • KPI-driven questions score higher in accuracy than table-driven questions, with the largest gap—12%—for RCI.The authors connect the difference to the limited, frequently recurring KPI set and to distorted header or cell wording in table-driven questions.Source: §4.4 · pp. 8–9
  • Questions requiring row-header hierarchy lower all baselines’ performance by an average of 13%.The authors note that transformations can lose explicit hierarchy information or add noise to a row or column.Source: §4.4 · p. 9
  • Almost 30% of paraphrased question cases are answerable in one phrasing but not another, indicating sensitivity to question wording.The analysis groups paraphrase sets into all-correct, mixed-correctness, and all-wrong outcomes.Source: §4.4 · p. 9, Table 4 · p. 9

5 Conclusion

AIT-QA is presented as the first Table QA dataset explicitly capturing complex row- and column-header hierarchies. The authors report deficiencies across three model approaches and call for methods that support this complexity across scientific and business use cases.

5 Conclusion

  • AIT-QA is presented as the first Table QA dataset explicitly capturing complex structures, including column- and row-header hierarchies.The dataset focuses on financial documents in the airline industry, while the authors note such structures are common in other scientific and business documents.Source: §5 · p. 9
  • Benchmarking shows deficiencies in end-to-end, weakly-supervised, and row-column encoding Table QA methods on these complex tables.Source: §5 · p. 9
  • The authors hope the dataset encourages new Table QA approaches that support complex tables across a wider range of scientific and business use cases.Source: §5 · p. 9

A Annotation Tool

Figure 2 shows a page from the custom Table QA annotation tool. Annotators generate questions for a given table and annotate properties such as whether a question is KPI-driven or table-driven.

A Annotation Tool

Related papers

Loading 2106.12944v1…