Source-linked AI summary
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Pengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian Riedel
TL;DR
Pretrained language models are not tailored to jointly reason over free-form natural-language utterances and structured database tables. TaBERT jointly represents both, and parsers using it achieve state-of-the-art WikiTableQuestions performance while remaining competitive on Spider.
Problem
Existing language models encode free-form text but lack general-purpose pretrained representations for structured database tables and joint reasoning over both modalities.
Method
TaBERT builds on BERT to jointly learn contextual representations for natural-language utterances and structured table schemas using table linearization, content snapshots, and vertical attention.
Results
TaBERT-augmented parsers achieve state-of-the-art performance on WikiTableQuestions and competitive performance on Spider.
Takeaways & Limitations
TaBERT functions as a drop-in encoder replacement that provides contextual representations of natural-language utterances and database tables across supervised and weakly supervised semantic parsing.
Takeaways & Limitations
Training TaBERT from scratch produced inferior results, potentially because the web-scraped tables were lower quality than purely textual corpora.
Abstract
from arXiv · showhide
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .
1 Introduction
TaBERT addresses the challenge of jointly understanding free-form NL utterances and structured table schemas, which existing text-only pretrained LMs do not directly support. It provides a general-purpose encoder and performs strongly across supervised and weakly supervised semantic parsing.
- Motivation: Structured semantic parsing requires aligning NL utterances with table schemas, including column names, data types, and stored values.The alignment between terms such as “GDP” and schema columns is essential for inferring the correct query.
- Motivation: Existing LMs are trained on free-form text, while database tables contain strongly structured information and require domain-specific encoding strategies.These limitations motivate pretraining approaches tailored to both utterances and structured tables.
- Approach: TaBERT jointly learns contextual representations for utterances and table schemas by linearizing tables, selecting relevant content snapshots, and applying vertical attention.It is pretrained on a parallel corpus of 26 million tables and English paragraphs.
- Approach: TaBERT supplies representations for utterances and tables as a drop-in encoder for neural semantic parsers across domains.The approach targets shared representational needs despite domain-specific parser architectures.
- Results: TaBERT-based systems outperform BERT counterparts, reach state-of-the-art performance on WikiTableQuestions, and perform competitively on Spider.The two evaluations cover weakly supervised and supervised semantic parsing settings.
2 Background
The paper frames table semantic parsing as translating NL utterances into executable structured queries over relational or semi-structured tables. The background emphasizes table schemas, schema–utterance alignment, and masked-language-model pretraining.
- Semantic Parsing over Tables: Semantic parsing translates an NL utterance u into a formal meaning representation z, such as an executable SQL query over relational tables.The tables are represented as collections of rows and columns.
- Table Structure: Tables may be fully structured with typed, normalized contents or semi-structured with unnormalized textual cell values.The underlying representation affects the table and query setting.
- Semantic Parsing over Tables: Table representations must capture schema information such as columns, data types, and cell values for downstream query generation.The paper defines a schema as the set of columns and represents it with column vectors.
- Masked Language Models: Masked language models recover randomly masked input tokens from corrupted contexts and are pretrained on large textual corpora.BERT later serves as an encoder whose parameters are tuned with task-specific components.
- TaBERT Overview: TaBERT extends this representation-learning setting to utterances and table schemas, producing token and column representations for semantic parsers.Figure 1 summarizes content selection, row encoding, and vertical attention.
3 TABERT: Learning Joint Representations over Textual and Tabular Data
TaBERT constructs content-sensitive representations of utterances and table schemas by encoding relevant table rows with a Transformer and sharing information across rows. It pretrains these representations using text and table-specific objectives.
- Content Snapshot: TaBERT creates a content snapshot of rows relevant to the input utterance before encoding the table.For K > 1 it selects top-K rows by n-gram overlap; for K = 1 it forms a synthetic row from relevant cell values.
- Row Linearization: Each snapshot row is linearized from the utterance, column names, data types, and cell values, then encoded by a Transformer.Cell fields are concatenated with separators into the row sequence.
- Content Snapshot: A simple matching heuristic correctly identifies the best-matched rows in 40 out of 50 sampled WikiTableQuestions examples.The experiments use n ≤3 for the matching heuristic.
- Vertical Self-Attention: Vertical self-attention aggregates information across aligned cells in different rows, enabling cross-row dependencies on cell values.The mechanism operates over vectors corresponding to the same question token or column.
- Representations: Final utterance-token and column representations are obtained by pooling vertically aligned vectors for use by downstream parsers.Column vectors pool cell representations from the last vertical layer.
- Pretraining: TaBERT pretraining combines masked language modeling with masked column prediction and cell value recovery objectives.These objectives target NL context, column metadata, and representative cell values.
4 Example Application: Semantic Parsing over Tables
The paper evaluates TaBERT as an encoder in supervised text-to-SQL and weakly supervised table semantic parsing. It replaces or augments existing parser encoders while preserving task-specific decoding and learning setups.
- Supervised Semantic Parsing: SPIDER provides a supervised text-to-SQL setting with 10,181 examples across 200 databases and annotated SQL queries.Examples commonly require joining multiple tables.
- Supervised Semantic Parsing: The SPIDER parser pairs each utterance with every database table, encodes them with TaBERT, and uses an LSTM decoder with hierarchical attention.The decoder constructs an intermediate meaning representation under a predefined SemQL grammar.
- Weakly Supervised Semantic Parsing: Weakly supervised parsing infers queries from execution results without labeled queries and must explore a large search space using noisy binary rewards.This setting is described as significantly more challenging than supervised learning.
- Weakly Supervised Semantic Parsing: WikiTableQuestions contains 22,033 utterances and 2,108 semi-structured Web tables for weakly supervised semantic parsing.Its examples typically require compositional, multi-hop reasoning over entries in a single table.
- Weakly Supervised Semantic Parsing: TaBERT replaces MAPO’s LSTM-derived utterance and column representations in the weakly supervised parsing setup.MAPO supplies a strong base system with a domain-specific language for compositional single-table questions.
5 Experiments
Experiments evaluate TaBERT as a general-purpose encoder for semantic parsing on WikiTableQuestions and Spider, including effects of content snapshots, linearization, and pretraining choices. TaBERT outperforms BERT-based counterparts, reaches state-of-the-art performance on WikiTableQuestions, and performs competitively on Spider.
- Main Results: TaBERT is evaluated on weakly supervised WikiTableQuestions and supervised Spider semantic parsing tasks.The models use TaBERT as an utterance and table encoder within task-specific parsers.
- Main Results: 52.3% single-model exact-match accuracy on WikiTableQuestions surpasses the previous best ensemble’s 46.9% by 5.4% absolute.This result uses MAPO augmented with TaBERTLarge and three-row content snapshots.
- Main Results: 65.2% exact-match accuracy on Spider improves 2.8% over the base parser using BERTBase.The result is achieved by the TranX and SemQL parser augmented with TaBERTLarge using three-row content snapshots.
- Content Snapshots: Content snapshots improve column selection on Spider, with TABERTBase at K = 1 reaching 87.4% versus 86.4% without encoded content.Sampled cell values help align utterance entities and relational phrases with relevant table columns.
- Content Snapshots: Using three content-snapshot rows generally outperforms using one, enabling TaBERTLarge to capture two target rows in an example that K = 1 misses.The comparison supports using multiple relevant rows for questions requiring reasoning across rows.
- Linearization: Adding type information and content snapshots improves row-linearization performance by providing more hints about column meaning.A strategy without cell contents underperforms, while pretraining on tabular data makes it slightly better.
6 Related Works
Prior work adapted semantic parsers to structured databases and semi-structured Web tables, while knowledge-enhanced pretraining incorporated structured semantic information into language models. TaBERT addresses the related need for representations jointly suited to textual and tabular data.
- Semantic parsers have been adapted to operate over structured database tables and open-domain, semi-structured Web tables.
- Knowledge-enhanced pretraining incorporates knowledge-base entities and relations into contextual word representations.
- TaBERT is broadly relevant to knowledge-enhanced pretraining while targeting joint representations of natural-language utterances and structured tables.
7 Conclusion and Future Work
The paper presents TaBERT as a pretrained encoder for joint understanding of textual and tabular data, with semantic parsers using it as a general-purpose representation layer achieving strong results on two benchmarks. Future work extends evaluation, table linearization, pretraining, and cross-lingual support.
- Conclusion: TaBERT is a pretrained encoder for joint understanding of textual and tabular data.
- Conclusion: Semantic parsers using TaBERT as a general-purpose feature representation layer achieve strong results on two benchmarks.
- Future Work: Future work will evaluate TaBERT on table retrieval and table-to-text generation.
- Future Work: Future work will explore table linearization strategies, improved pretraining corpora, novel unsupervised objectives, and cross-lingual content snapshots.
A.1 Training Data
The training corpus pairs tables with surrounding English natural-language contexts from Wikipedia and the WDC WebTable Corpus. Because arbitrary Web tables are noisy, preprocessing applies heuristics to remove malformed, sparse, repetitive, or unsuitable content.
- Data Sources: Training data pairs tables with surrounding natural-language sentences collected from English Wikipedia and the WDC WebTable Corpus.
- Data Sources: Wikipedia tables are paired with the preceding three paragraphs as their natural-language context.
- Data Sources: The WDC corpus contributes 50.8 million English relational tables and surrounding natural-language contexts from the 2015 Common Crawl subset.
- Preprocessing: Preprocessing removes long column names, oversized or non-ASCII-heavy cells, empty or repetitive rows and columns, and tables below minimum dimensions.
- Preprocessing: Column data types are identified as text or real value by majority voting over table cells.
A.2 Pretraining Setup
Pretraining creates synthetic utterance–table instances from paired natural-language contexts and tables, then optimizes three pretraining objectives. The setup includes 37.6 million training instances per epoch over 10 epochs.
- Each epoch contains 37.6M training instances, and TaBERT is trained for 10 epochs.
- Training instances pair synthetic natural-language utterances with content snapshots sampled from tables.
- The training objective sums three pretraining objectives, including masked language modeling for utterance tokens.
- Pretraining hyper-parameters are listed in Table 6.
B.1 Supervised Parsing on SPIDER
The SPIDER parser adapts TranX to generate SemQL-based abstract representations of SQL queries, extending the action space to select tables and columns. It integrates TaBERT representations through hierarchical attention and adds entity-linking features.
- Parser framework: TranX translates utterances into tree-structured abstract meaning representations before deterministically converting them into SQL queries.The adaptation uses SemQL, a simplified SQL grammar, with transition-based generation following production rules.
- Parser framework: SemQL decoding expands grammar nodes depth-first and left-to-right, then emits terminal values when the current node is a leaf.The root production rule is select stmt → SelectStatement.
- Parser framework: The action space adds SELECTTABLE(Ti) and SELECTCOLUMN(Ti, cj) actions for selecting tables and columns from the database schema.These actions support table references such as FROM targets and column references such as SELECT results.
- TaBERT integration: For each table, TaBERT supplies table, utterance-token, and column representations that TranX combines using hierarchical attention.The decoder first attends over tables, then attends over utterance tokens and columns within each table before updating its LSTM state.
- TaBERT integration: Entity-linking features append trainable table- or column-match embeddings when utterance tokens match schema names.A zero-vector is appended for representations without a corresponding match.
- Configuration: The SPIDER configuration uses separate Adam optimizers for TaBERT and remaining parameters, freezes TaBERT initially, and trains for 25,000 iterations.The TaBERT optimizer uses learning rate 3e-5, while remaining parameters use 1e-3.