PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPAS: Weakly Supervised Table Parsing via Pre-training

Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno, Julian Martin Eisenschlos

arXiv:2004.02349v2cs.IRcs.AIcs.CLcs.LG

TL;DR

Semantic parsing over tables traditionally requires expensive logical-form annotations, and weak supervision remains difficult because of spurious programs and sparse rewards. TAPAS instead selects table cells and an optional aggregation operator, using BERT-based table encoding, joint text–table pre-training, and end-to-end weakly supervised training. It outperforms or matches prior models across three datasets, including improving SQA accuracy from 55.1 to 67.2 and achieving 48.7 accuracy through WIKISQL-to-WIKITQ transfer.

  • Problem

    Weakly supervised table semantic parsing avoids expensive logical-form annotation but remains difficult because of spurious logical forms and reward sparsity.

  • Method

    TAPAS predicts table-cell selections and an optional aggregation operator with a BERT extension, Wikipedia text–table pre-training, and end-to-end weakly supervised training.

  • Results

    TAPAS performs better or on par across three datasets, improving SQA state-of-the-art accuracy from 55.1 to 67.2 and achieving 48.7 accuracy after WIKISQL-to-WIKITQ transfer.

  • Takeaways & Limitations

    TAPAS achieves competitive table-question-answering results with a simpler architecture that avoids logical-form generation and supports aggregation and conversational settings.

  • Takeaways & Limitations

    TAPAS handles single tables that fit in memory and therefore does not cover very large tables or databases containing multiple tables.

Abstract

from arXiv · show

Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

1 Introduction

TAPAS addresses the cost and difficulty of training semantic parsers from weak supervision by answering table questions without generating logical forms. It combines cell selection, optional aggregation, structured pre-training, and end-to-end training, achieving strong results across three datasets.

  • Motivation: Weak supervision reduces annotation requirements by providing question–denotation pairs instead of full logical forms, but spurious programs and sparse rewards make training difficult.Semantic parsing also uses logical forms only as an intermediate step before retrieving denotations.
  • Approach: TAPAS answers table questions without generating logical forms by selecting table cells and optionally applying an aggregation operation.The model learns these operations directly from natural language rather than requiring a formalism.
  • Approach: TAPAS extends BERT with tabular-structure embeddings and classification layers for cell selection and aggregation prediction.Its pre-training extends masked language modeling to text segments and tables crawled from Wikipedia.
  • Training: TAPAS trains end-to-end from weak supervision using direct cell-selection training or expected soft outcomes with regression when aggregation is required.The latter handles cases where the denotation does not reveal the selected cells or aggregation operator.
  • Results: Compared with prior logical-form-free approaches, TAPAS has a simpler single-encoder architecture, uses pre-training, supports aggregation, and handles conversational settings.It uses no autoregressive decoding.
  • Results: 67.2 accuracy improves SQA state of the art from 55.1, while TAPAS performs on par with state of the art on WIKISQL and WIKITQ.Transfer learning from WIKISQL to WIKITQ achieves 48.7 accuracy, 4.2 points above state of the art.

2 TAPAS Model

TAPAS encodes a question and flattened table jointly with table-aware embeddings, then predicts selected cells and an optional aggregation operator. Inference executes those predictions against the table to obtain the answer.

  • Architecture: TAPAS uses a BERT encoder, flattens the table into tokens, places question tokens before table tokens, and adds cell-selection and aggregation classifiers.The architecture is illustrated in Figure 1 and table encoding in Figure 2.
  • Additional embeddings: Table-aware embeddings encode token position, question-versus-table segments, row and column identity, numeric rank, and previous conversational answers.Rank embeddings support superlative questions, while previous-answer embeddings mark cells answering the prior question.
  • Cell selection: The cell-selection layer models cells as independent Bernoulli variables and averages token logits to produce each cell’s selection probability.Selected cells may be the final answer or inputs to an aggregation operation.
  • Cell selection: A column-selection bias averages cell embeddings within each column and assigns zero probability to cells outside the selected column.An additional option represents selecting no column or cells.
  • Aggregation and inference: TAPAS predicts aggregation operators such as SUM, COUNT, AVERAGE, or NONE using a softmax layer over the [CLS] representation.The operator is applied to the selected cells when producing the answer.
  • Aggregation and inference: Inference selects the most likely operator and all cells whose selection probability exceeds 0.5, then executes the predicted aggregation over those cells.Without aggregation, the selected cells themselves provide the answer.

3 Pre-training

TAPAS is pre-trained on Wikipedia text–table pairs using masked language modeling adapted to structured data. The corpus and masking procedure expose the model to relationships between text, headers, and cells before task fine-tuning.

  • Data: TAPAS pre-trains on Wikipedia tables to learn correlations between text and tables and between cells and their column headers.The pre-training data is intended to initialize the table-parsing task.
  • Data: The corpus contains 6.2M tables: 3.3M Infobox tables and 2.9M WikiTable tables, restricted to tables with at most 500 cells.The end-task datasets use horizontal tables with header rows, so the extracted WikiTables follow that format.
  • Data: The pre-training corpus includes 21.3M text snippets associated with the extracted tables, including captions, article metadata, section titles, and surrounding text.These snippets serve as proxies for questions in downstream tasks.
  • Objective: Pre-training applies masked language modeling to combined text-and-table inputs, predicting masked tokens from both textual and tabular context.The sequence length is budgeted, and snippets and table content are truncated to fit.
  • Objective: Whole-word masking is used for text, while whole-cell masking masks all word pieces of a table cell when any piece is masked.The authors report that whole-cell masking benefits end-task performance.
  • Data augmentation: Synthetic question–denotation data augmentation did not significantly improve end-task performance and was therefore omitted.The augmentation was generated over real tables using a grammar.

4 Fine-tuning

TAPAS fine-tunes from weak supervision by mapping denotations to cell selections and scalar answers, avoiding program search through differentiable aggregation. Its training combines selection, aggregation, and scalar-answer objectives, with special handling for ambiguous answers.

  • Supervision setup: TAPAS maps each denotation to cell coordinates and, when applicable, a scalar answer to guide fine-tuning.Cell-selection examples supervise selected cells, while scalar-answer examples supervise an aggregation producing the target scalar.
  • Cell selection: For cell-selection examples, the model first selects a column and then selects cells within that column, using column and cell cross-entropy losses.An additional aggregation loss assigns NONE because no aggregation occurs.
  • Scalar answers: Scalar-answer examples use differentiable operator execution and expected results instead of searching for programs that reproduce the denotation.The model computes soft outcomes for supported operators and trains against the scalar with Huber loss.
  • Scalar answers: The scalar-answer objective combines aggregation and scalar losses, ignores examples above a scalar-loss cutoff, and temperature-scales logits toward discrete inference.The total scalar-answer loss is JSA = Jaggr+βJscalar.
  • Ambiguous answers: For ambiguous answers, TAPAS dynamically chooses cell-selection or scalar supervision according to the current probability of the NONE operator.Cell-selection supervision is used when pa(op0) ≥ S; otherwise scalar-answer supervision is used.

5 Experiments

TAPAS is evaluated on three table-question-answering datasets using weak supervision, with results generally matching or exceeding prior systems. The experiments also examine transfer learning, model components, aggregation, pre-training, and error patterns.

  • Results: 83.6 WIKISQL denotation accuracy places weakly supervised TAPAS near the 83.9 state of the art, while gold cell and aggregation supervision reaches 86.4.
  • Results: TAPAS reaches 42.6 accuracy on WIKITQ using only the original training data.
  • Results: 11-point or greater gains across SQA metrics include average question accuracy rising from 55.1 to 67.2 and sequence accuracy from 28.1 to 40.4.
  • Ablations: Table pre-training and column and row embeddings are the most important tested features, while positional and rank embeddings improve quality less.
  • Ablations: Removing aggregation losses lowers WIKITQ accuracy from 29.0 to 23.1 and WIKISQL accuracy from 84.7 to 82.6.

6 Related Work

Prior work reduces logical-form annotation through weak supervision and explores differentiable table reasoning without explicit logical-form generation. TAPAS follows this direction with a simpler, broader model that supports aggregation over selected cells.

  • Weak supervision replaces annotated logical forms with question–denotation pairs, guiding searches for compatible logical forms.
  • End-to-end alternatives avoid explicit logical-form generation, but prior models use complex sequential operations, predefined or learned operators, or cannot predict aggregations.
  • TAPAS uses a single encoder that performs many operations implicitly, while supporting aggregation operators over selected cells.
  • Pre-training methods extend beyond text to paired modalities, including text-image data and, in this work, table-related representations.

7 Conclusion

The conclusion presents TAPAS as a weakly supervised table-question-answering model that avoids logical-form generation and trains end to end. It reports better or competitive results against state-of-the-art semantic parsers, while identifying larger and multi-table contexts as future extensions.

  • TAPAS avoids generating logical forms and can fine-tune on semantic parsing datasets using only weak supervision with an end-to-end differentiable recipe.
  • TAPAS pre-trains on large-scale text–table data and successfully restores masked words and table cells.
  • TAPAS achieves better or competitive results compared with state-of-the-art semantic parsers.
  • The model is limited to single tables that fit in memory, leaving multiple tables and very large tables for future work.

A WIKISQL Execution Errors

The WIKISQL execution analysis identifies discrepancies caused by numeric values stored as text and compares accuracies against both official and internally derived reference answers.

  • “REAL” numbers stored in “TEXT” format cause incorrect results for some comparison and aggregation examples.
  • WIKISQL execution accuracy can penalize systems that execute answers themselves rather than producing SQL queries.
  • Differences between TAPAS and WIKISQL results arise because numbers in the “Crowd” column are not represented numerically in the SQL table.
  • Table 9 reports WIKISQL development denotation accuracy, while Table 10 reports test denotation accuracy.

B Hyperparameters

The appendix lists hyperparameter search spaces and selected optimal settings for pretraining and the WIKISQL, WIKITQ, and SQA tasks.

  • Table 11 specifies hyperparameter values for WIKISQL and WIKITQ, constrained to ranges or enumerated lists.
  • Table 12 reports optimal hyperparameters for pretraining, SQA, WIKISQL, and WIKITQ.

C Pre-training Example

The model uses a soft accuracy measure for numerical predictions, rewarding values close to the target rather than only exact matches. It achieves higher soft accuracy overall and especially for numbers, although predictions can remain inconsistent across rows and columns.

  • The soft accuracy metric assigns full credit to exact numeric matches, zero to nonnumeric cases, and partial credit based on relative error otherwise.
  • 74.5 overall accuracy and 80.5 numerical accuracy are achieved with the soft metric, compared with 71.4 and 53.9 respectively under exact accuracy.
  • The example table comes from the Wikipedia page describing the 1997 Rugby World Cup Sevens, with correct and incorrect predictions marked explicitly.
  • The model correctly restores Draw and Loss for Spain but gives only close estimates for Points For and Points Against for Zimbabwe.
  • The predictions are not completely consistent because row and column arithmetic constraints are not always satisfied.

D The average of stochastic sets

The approach estimates aggregate values over stochastic cell subsets using independent inclusion probabilities and a latent operation choice. For averages, differentiable zero- and second-order approximations improve slightly over a weighted-average baseline, though average examples are rare.

  • Aggregates are estimated from conditionally independent Bernoulli cell-inclusion variables and a latent categorical choice among AVERAGE, SUM, and COUNT.
  • The WIKISQL example table contains a REAL-typed Crowd column whose values are stored as TEXT, alongside two training questions and derived answers.
  • A random subset assigns each cell c an inclusion probability p_c = P(c ∈ S), while expected sums can be computed directly from these probabilities.
  • For averages, the expected quotient differs from the probability-weighted average, which is insensitive to uniformly scaling all output probabilities.
  • The expected reciprocal is related to a Poisson binomial distribution, and Jensen’s inequality provides a lower bound for the relevant quantity.
  • Zero- and second-order Taylor approximations make the aggregate computation differentiable and yield small improvements over the weighted-average baseline.
  • Average examples are very uncommon in the dataset, so the authors expect the approximation method to matter more in broader settings.
Loading 2004.02349v2…