PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou

arXiv:2107.07653v3cs.CLcs.AI

TL;DR

Table pre-training lacks large-scale, high-quality tabular corpora. TAPEX learns a neural SQL executor from synthetic executable query–table examples, and it achieves new state-of-the-art results across four downstream datasets.

  • Problem

    Table pre-training lacks a large-scale, high-quality corpus and an efficient pre-training task for structured tabular data.

  • Method

    TAPEX synthesizes executable SQL queries and execution results, then pre-trains a BART encoder-decoder to mimic SQL execution over tables.

  • Results

    TAPEX achieves new state-of-the-art results on all four evaluated datasets, including 89.5% WikiSQL denotation accuracy and 57.5% WikiTableQuestions denotation accuracy.

  • Takeaways & Limitations

    Synthetic executable programs provide a supported route for exploiting structured data in table pre-training.

  • Takeaways & Limitations

    TAPEX cannot ideally handle large tables because table flattening becomes infeasible when tables do not fit in memory.

Abstract

from arXiv · show

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

1 INTRODUCTION

TAPEX addresses the scarcity and quality challenges of table pre-training by learning to execute synthetic SQL queries over tables, using a sequence-generation language model.

  • 1 INTRODUCTION: Table pre-training faces shortages of large-scale, high-quality corpora and challenges in designing efficient pre-training tasks.Existing approaches commonly use web-mined or synthesized NL–table pairs, but web data can be noisy and reconstruction-based tasks require very large corpora.
  • 1 INTRODUCTION: TAPEX pre-trains language models to mimic a SQL execution engine over tables.It samples executable SQL queries, obtains their execution results, and trains the model to generate those results.
  • 1 INTRODUCTION: Systematically sampled SQL queries yield a diverse, large-scale, and high-quality synthetic pre-training corpus.The approach uses executable programs and their outputs rather than relying solely on naturally occurring NL–table pairs.
  • 1 INTRODUCTION: TAPEX uses an encoder-decoder sequence-generation architecture and reformulates downstream table tasks for generation.The paper implements the approach with BART and evaluates table question answering and fact verification across four benchmark datasets.

2 FINE-TUNING ON DOWNSTREAM TASKS

TAPEX formulates table question answering and fact verification within a shared encoder-decoder framework that encodes an NL sentence with a flattened table and generates task outputs.

  • 2.1 DOWNSTREAM TASK FORMULATION: Downstream examples pair an NL sentence with a semi-structured table, while outputs differ between question answering and fact verification.TableQA returns cell values or aggregated numbers, whereas TableFV returns whether the sentence is entailed or refused.
  • 2.2 GENERATIVE FINE-TUNING: TAPEX treats TableQA and TableFV as sequence-generation tasks using autoregressive decoding.For TableQA, answers are generated word by word; the architecture is implemented with BART in the experiments.
  • 2.2 GENERATIVE FINE-TUNING: The model flattens each table into a marked sequence and prefixes it with the NL sentence before encoding.Special tokens identify table headers and rows, allowing the structured input to be fed directly into the model.
  • 2.2 GENERATIVE FINE-TUNING: The decoder generates comma-separated TableQA answers, while TableFV uses a binary classifier over the decoder’s final-token representation.This formulation supports answer operators and their compositions and can be extended to other table-related tasks.
  • 2.2 GENERATIVE FINE-TUNING: The shared architecture supports both individual-task and multi-task fine-tuning.The authors discuss flexibility, convenience, and transferability from using a common sequence-generation formulation.

3 TABLE PRE-TRAINING VIA EXECUTION

TAPEX uses SQL execution as its sole table pre-training task, generating executable query–table examples from public tables and SQL templates to train result prediction.

  • 3 TABLE PRE-TRAINING VIA EXECUTION: Existing reconstruction-based table pre-training tasks are less efficient because they generally require extremely large corpora.TAPEX instead exploits discrete operations that structured tables support through programming languages such as SQL.
  • 3 TABLE PRE-TRAINING VIA EXECUTION: TAPEX concatenates an executable SQL query with a flattened table and trains the model to produce the query’s execution result.SQL execution is used as the only pre-training task, paralleling the generative downstream fine-tuning procedure.
  • 3.2 PRE-TRAINING CORPUS: The synthetic corpus is built around two factors: the table source and the SQL query sampling strategy.These choices determine the scale, diversity, and quality of the automatically generated examples.
  • 3.2 PRE-TRAINING CORPUS: TAPEX uses high-quality public semi-structured tables and works with only a few thousand tables rather than millions of raw web tables.The supplied passage specifies that nearly 1,500 training tables are randomly selected, though the sentence is truncated.
  • 3.2 PRE-TRAINING CORPUS: SQL templates from SQUALL are instantiated by sampling headers and cell values, while empty-result queries are discarded.This produces concrete executable queries and a large-scale corpus with high quality.

4 EXPERIMENTS

TAPEX is evaluated on four table reasoning benchmarks using synthetic pre-training with up to 5 million SQL-query execution pairs. It achieves state-of-the-art results across WikiSQL-WEAK, WikiTableQuestions, SQA, and TABFACT, with especially broad gains in low-resource settings.

  • Dataset and Evaluation: TAPEX is evaluated on WikiSQL-WEAK, WikiTableQuestions, SQA, and TABFACT, using denotation accuracy for TableQA and accuracy for TABFACT.The SQA table reports sentence- and conversation-level denotation accuracies.
  • Main Results: 89.5% WikiSQL-WEAK denotation accuracy surpasses the previous best by 2.3%.This result is also 3.7% higher than BART and is reported as a new state-of-the-art result.
  • Main Results: 57.5% WikiTableQuestions denotation accuracy surpasses the previous best by 4.8%.TAPEX improves over BART by 19.5% on this more challenging benchmark.
  • Main Results: 74.5% sentence-level and 48.4% conversation-level SQA denotation accuracy establish new state-of-the-art results.The paper notes these gains despite SQA being conversational while TAPEX’s pre-training task is context-free.
  • Main Results: TAPEX achieves new state-of-the-art results on all TABFACT subsets, including a 4.0% gain over the previous best on Testcomplex.The authors interpret this as evidence that the learned table understanding can adapt across downstream tasks.
  • Overall Findings: Across four datasets, TAPEX broadly improves table understanding, especially in the low-data regime.Multi-task fine-tuning provides significant gains from BART initialization but only marginal gains from TAPEX initialization.

5 ANALYSIS

The analysis shows that TAPEX learns effective table selection, aggregation, and numerical reasoning, improves operator-level reasoning, benefits from larger synthetic corpora, and remains limited on large tables and text-to-SQL.

  • SQL Execution by Pre-training: 89.6% of nearly 20,000 held-out SQL queries over unseen tables were correctly executed by TAPEX after pre-training.Performance was especially strong for Filter, Aggregate, and Superlative operators, with good results on Arithmetic and Comparative operators.
  • Table Understanding by Pre-training: TAPEX’s self-attention focuses on the row and header associated with relevant table cells in sampled WikiTableQuestions examples.The example associates “adrian lewis” with the first column “player” and the entire third row.
  • Table Reasoning by Pre-training: TAPEX significantly boosts BART’s performance across all analyzed operators, indicating improved joint reasoning over text and tables.The comparison uses 500 randomly selected WikiTableQuestions development questions and reports operator-level performance for BART and TAPEX.
  • The Scale of Pre-training Corpus: Scaling up TAPEX’s synthetic pre-training corpus generally improves downstream performance across datasets.The observed trend is described as analogous to language-model scaling, where larger pre-training corpora yield better downstream performance.
  • The Efficiency of Pre-training: TAPEX surpasses existing table pre-training approaches on WikiTableQuestions with a much smaller pre-training corpus.This comparison is presented as evidence of TAPEX’s high pre-training efficiency.
  • Limitations: TAPEX cannot ideally handle large tables because table flattening becomes infeasible when tables do not fit in memory.Removing unrelated rows or columns can make large tables fit but decreases downstream performance.
  • Limitations: TAPEX shows no significant advantage over BART on text-to-SQL, possibly because its synthetic corpus lacks grounding and learned table reasoning may not be necessary for SQL generation.The paper gives “total” versus “sum” as an example where semantic understanding may not require knowing the mathematical meaning of the aggregation function.

6 RELATED WORK

Related work on table pre-training largely uses NL-table corpora, while TableQA methods commonly formulate answering as weak semantic parsing with logic-form generation.

  • Table Pre-training: Previous table pre-training methods collect web-mined NL-table pairs, human-annotated parallel data, or synthesized data for their corpora.These approaches differ primarily in how they obtain the pre-training corpus.
  • Joint Understanding on Table and Text: Prior TableQA systems commonly formulate answering as weak semantic parsing and use reinforcement learning to optimize semantic parsers over tables.Although these systems can produce logic forms such as SQL, training is difficult because of the large search space.

7 CONCLUSION

TAPEX learns neural SQL execution from automatically synthesized SQL queries and execution results, achieves state-of-the-art results across four downstream datasets, and motivates synthetic executable-program pre-training.

  • Conclusion: TAPEX automatically synthesizes a diverse, large-scale, high-quality corpus by sampling SQL queries and their execution results.The approach addresses table-pre-training data scarcity by learning a neural SQL executor.
  • Conclusion: TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all four downstream datasets.The conclusion frames synthetic executable programs as a promising basis for extending pre-training to other areas such as knowledge bases.

ETHICS STATEMENT

The paper presents TAPEX as a pre-training approach for tabular data and evaluates it on question answering and fact verification, while SQA incorporates conversational history into the input.

  • Ethics Statement: TAPEX approximates formal-language structural reasoning over tables through pre-training on sampled SQL queries and execution results.The corpus uses 1,500 high-quality public tables rather than TABERT’s 26 million noisy web-crawled tables, reducing potential privacy and bias issues.
  • Ethics Statement: The evaluation covers table-based question answering and fact verification on four benchmark datasets, including conversational SQA.For SQA, the conversation history and current question are concatenated as the input sentence.

B MULTI-TASK RESULTS

The section introduces multi-task fine-tuning experiments and frames SQL query difficulty using four levels. Results are reported as denotation accuracy on a target development set.

  • Experimental setup: WIKISQL-WEAK and TABFACT were selected as transfer sources because their training data are relatively rich.
  • Experimental setup: Multi-task fine-tuning first uses a source dataset and then fine-tunes on a target dataset.The notation Source → Target denotes this sequential transfer procedure.
  • SQL difficulty: SQL queries are organized into four difficulty levels for experimental analysis.The section references four corresponding example SQL queries.

C.1 IMPACT OF SQL QUERY DIFFICULTY IN PRE-TRAINING

The analysis examines how SQL-query difficulty and pre-training language affect downstream performance. Harder SQL queries generally help through Medium difficulty, while translated natural-language pre-training is comparable or worse than SQL pre-training.

  • SQL difficulty: SQL difficulty is measured by the number of SQL elements, with categories defined using thresholds such as 6 and 14 elements.Elements include SQL keywords and table schema items such as headers or cell values.
  • Downstream performance: 10.6% improvement on WIKITABLEQUESTIONS is observed when expanding pre-training from ≤Easy to ≤Medium SQL queries.Adding harder queries helps in most cases, but gains become less significant beyond Medium; Extra-Hard slightly hurts TABFACT.
  • Downstream performance: Figure 7 compares downstream dev-set performance across pre-training difficulty levels using 0.5 Million examples.≤Extra Hard represents the use of all SQL query templates.
  • Fine-grained analysis: Figure 8 reports fine-grained performance across SQL difficulty levels and question difficulty levels on the WIKITABLEQUESTIONS dev set.The analysis divides questions into four difficulty levels using SQUALL SQL annotations.
  • Natural-language generation: 1.4% drop on WIKITABLEQUESTIONS occurs when translated NL sentences replace SQL queries in pre-training.The NL corpus is produced by translating SQL with a SQL-to-NL model, and the authors attribute the drop to translation noise.

D FINE-GRAINED ANALYSIS OF SQL EXECUTION

The fine-grained execution analysis breaks down held-out SQL-query behavior by operator type. It combines operator statistics, example queries, operator proportions, and execution accuracies.

  • Execution accuracy: Figure 9 analyzes SQL execution accuracies separately for each operator type.
  • Operator analysis: The analysis reports fine-grained statistics for typical SQL operators.
  • Evaluation statistics: The held-out evaluation contains 20,000 SQL queries and includes example SQL queries and operator percentages.
Loading 2107.07653v3…