Source-linked AI summary
TABBIE: Pretrained Representations of Tabular Data
Hiroshi Iida, Dung Thai, Varun Manjunatha, Mohit Iyyer
TL;DR
Prior methods jointly pretrain tables with associated text but underperform on table-only tasks. TABBIE instead learns from tables alone using corrupt cell detection, achieves competitive or better performance across three benchmarks, and requires less training compute.
Problem
Joint table-and-text pretraining underperforms on practical tasks that operate over tables without associated text.
Method
TABBIE pretrains exclusively on tables using corrupt cell detection and separate row- and column-oriented Transformers to produce cell, row, and column representations.
Results
Across three downstream table-based tasks, TABBIE achieves competitive or better performance than existing methods, with state-of-the-art results in most configurations.
Takeaways & Limitations
TABBIE provides semantically meaningful representations of cells, rows, and columns while reducing the computational cost of table representation pretraining.
Takeaways & Limitations
Training TABBIE has a negative environmental impact, with estimated emissions of 149.19 kg of CO2 per run.
Abstract
from arXiv · showhide
Existing work on tabular representation learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model's learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.
1 Introduction
TABBIE targets tabular understanding without associated text, addressing limitations of joint table-text pretraining. It uses corrupt cell detection and achieves strong downstream performance while providing representations for cells, rows, and columns.
- Approach: TABBIE provides accessible representations for cells, rows, and columns.Its architecture independently encodes rows and columns, pooling their representations to reduce sequence-length complexity.
- Motivation: Table-text pretraining improves paired table-text tasks but underperforms on applications operating over tables without external text.Examples include retrieving similar columns or cells and filling missing information.
- Approach: TABBIE learns exclusively from tabular data using a corrupt cell detection objective adapted from ELECTRA.The model predicts whether each cell is real or corrupted, using simple corruption processes without a separate generator.
- Results: On three table-based benchmarks, TABBIE achieves state-of-the-art performance on most configurations and remains competitive with TaBERT on others.The benchmarks evaluate column population, row population, and column type prediction.
- Results: TABBIE required 8 V100 GPUs for just over a week, whereas TaBERT used 128 V100 GPUs for six days.The comparison used similarly sized models pretrained on the same number of tables.
2 Model
TABBIE contextualizes table cells through separate row and column Transformers, combines their representations, and trains with corrupt cell detection. It also exposes row and column embeddings and evaluates multiple corruption strategies for learning table semantics.
- 2.1 Model Architecture: TABBIE encodes each cell with separate row and column Transformers, then averages their representations across layers.This preserves row- and column-level context while reducing each Transformer’s sequence length.
- 2.2 Pretraining: A binary classifier predicts whether each final-layer cell representation is genuine or corrupted.The corruption probability uses a sigmoid classifier, and training minimizes binary cross entropy averaged across table cells.
- 2.1 Model Architecture: TABBIE produces embeddings for every cell as well as individual rows and columns.Prepended [CLSROW] and [CLSCOL] tokens provide extractable representations for entire rows and columns in downstream tasks.
- 2.3 Cell corruption process: Corruption strategies include frequency-based sampling and intra-table cell swapping.Swapping cells within the same row or column creates more challenging distinctions between topically similar data.
- 2.3 Cell corruption process: Intra-table swapping underperforms frequency-based sampling downstream but yields more semantically similar nearest neighbors for column and row embeddings.The result indicates a trade-off between downstream task performance and qualitative embedding similarity.
3 Experiments
TABBIE is evaluated on three table-centric benchmarks measuring column population, row population, and column type prediction. It generally outperforms TaBERT and other baselines, while showing a compute advantage in pretraining.
- Evaluation setup: TABBIE is evaluated on column population, row population, and column type prediction benchmarks measuring semantic table understanding.The experiments use the same fine-tuning strategy for TABBIE and TaBERT, selecting task-specific final-layer representations for classification.
- Caveat: The column population comparison may favor TaBERT because its pretraining data likely includes the test-set tables.The paper notes this possible data overlap as an advantage for TaBERT in the comparison.
- Column Population: TABBIE outperforms all methods on column population, with the largest improvement when given a single seed column (N = 1).The FREQ corruption strategy performs better than MIX on this task.
- Row Population: TABBIE outperforms baselines on row population with more seed rows, although TaBERT is superior when given only one seed row.With more seed rows, TABBIE shows small improvements over TaBERT; the FREQ strategy again outperforms MIX.
- Column Type Prediction: TABBIE and TaBERT significantly outperform SATO on column type prediction, with no clear difference between the two models.Both models retain higher F1 scores than other baselines even with only 1,000 training examples.
4 Analysis
TABBIE’s analysis finds stronger corrupt-cell detection than TaBERT, especially for intra-column swaps, and reveals representations that capture formatting-independent semantics, company-level table similarity, and numerical trends.
- Corrupt cell detection: TABBIE outperforms TaBERT on all corrupt-cell types, including intra-column swaps where it achieves 68.8 F1 versus TaBERT’s 23.7 F1.Intra-column swaps are the most difficult corruption type for both models.
- Corrupt cell detection: The MIX corruption strategy is substantially better than FREQ for challenging corruptions, although it performs worse on the three downstream tasks.MIX is nearly equivalent to FREQ for detecting randomly sampled cells.
- Numerical trends: Only TABBIE MIX reliably identifies cells that violate numerical trends in columns, such as swapped ascending jersey numbers.TABBIE FREQ and TaBERT struggle with the same swap.
- Nearest neighbors: TABBIE’s nearest-neighbor columns show greater tolerance for formatting differences and less dependence on matching headers than TaBERT’s neighbors.Examples include semantically related date columns with different formatting and columns headed “nom,” “nombre,” and “name.”
- Clustering: TABBIE embeddings cluster tables into reasonable semantic types and group tables from the same company across different financial years.The clustering experiment uses [CLS] embeddings on approximately 110K FinTabNet tables.
5 Related work
Prior tabular representation methods jointly model tables with associated text and use language-model objectives, while related work also explores structured table pretraining and semantic clustering.
- Table-text pretraining: TaBERT and TaPas jointly model tables with text, making them suited to tasks such as question answering over tables.TaBERT uses masked column prediction and cell value recovery, whereas TaPas uses objectives encouraging tabular textual entailment.
- Embedding analysis: TABBIE’s clustering analysis provides qualitative evidence that table embeddings can organize tables by semantic type and company across financial years.The analysis is illustrated with sample table clusters.
- Structured-table pretraining: TUTA is a concurrent structured-table model that uses masked language modeling for pretraining.It is presented as related work alongside TaBERT and TaPas.
6 Conclusion
The paper introduces TABBIE, a self-supervised method trained exclusively on tables, combining corrupt-cell detection with separate row and column Transformers.
- Contribution: TABBIE is a self-supervised pretraining method for tables without associated text.It is designed for tabular representation learning independent of paired textual input.
- Method: TABBIE repurposes the ELECTRA objective for corrupt-cell detection and uses separate Transformers for rows and columns to reduce sequence-length complexity.This design also produces representations for cells, rows, and columns.
- Results: Across three downstream table-based tasks, TABBIE achieves competitive or better performance than existing methods, while its representations capture semantic information about cells, rows, and columns.The authors publicly release pretrained models and code.
7 Ethics Statement
The ethics statement acknowledges TABBIE’s environmental impact from GPU computation while reporting a lower estimated cost than TaBERT under a similar-platform assumption.
- Environmental impact: Training TABBIE used 1,344 Tesla V100 GPU-hours and produced an estimated 149.19 kg of CO2 per run.The paper reports two runs in total.
- Environmental impact: The reported emissions are estimated to be more than ten times lower than TaBERT’s cost per run, assuming a similar computing platform.The paper also equates the emissions to approximately 17 gallons of fuel consumed by an average motor vehicle.