TUTA: Tree-based Transformers for Generally Structured Table Pre-training

• Most table-understanding work focuses on relational tables, overlooking entity and matrix tables despite their widespread use in real-world data.Relational tables account for 0.9% of commonly crawled web tables and 22.0% of spreadsheet tables; labeling structural information is time-consuming and labor-intensive.Source: §1 · pp. 1–2
• Prior relational-table pre-training methods do not fit other structures, where a cell may depend on headers across multiple rows and columns and relational treatment loses structural information.Self-supervised pre-training on unlabeled tables is proposed as a way to address the shortage of structural labels.Source: §1 · p. 2
• TUTA introduces a bi-dimensional coordinate tree to define cell locations and distances while representing both spatial and hierarchical structure.Its tree-based positional encodings combine tree coordinates with rectangular Cartesian coordinates, while structure-aware attention supports top-down, bottom-up, and peer-to-peer information flow.Source: §1 · p. 2
• TUTA is the first transformer-based method applied to Cell Type Classification and Table Type Classification, achieving state-of-the-art across five widely studied datasets.Source: §1 · p. 2
• TUTA progressively pre-trains token-, cell-, and table-level representations using masked language modeling, multichoice Cloze, and context retrieval on unlabeled tables.These objectives are MLM, CLC, and TCR, respectively.Source: §1 · p. 2

2.1 Dataset construction

• The corpus spans relational, entity, and matrix tables, excludes tiny non-data layout tables, and totals 57.9 million web and spreadsheet tables.Web sources include Wikipedia and the WDC WebTable Corpus; spreadsheet data comes from more than 13.5 million web-crawled files.Source: §2.1 · p. 3

2.2 Bi-dimensional coordinate tree

• The bi-dimensional coordinate tree uses orthogonal top and left trees whose paths jointly give each cell a unique coordinate and encode header hierarchy.Each cell maps to a node in each tree; the two positions combine into its bi-dimensional coordinate.Source: §2.2 · p. 3
• Tree distance sums shortest-path steps in the top and left trees; in flat tables, coordinates become Cartesian and cell distances are 2 in the same row or column and 4 otherwise.Distances from titles and surrounding natural-language descriptions to cells are set to 0.Source: §2.2 · p. 3
• Tree extraction detects header regions with a CNN-based method, then uses formatting heuristics and SUM or AVERAGE formulas to infer header hierarchies.The framework can also use other header- and hierarchy-extraction methods.Source: §2.2 · p. 3

3.1 Vocabulary construction

• Because table-cell strings use compact forms such as abbreviations, TUTA builds a dedicated WordPiece vocabulary with 9,754 new tokens and merges it with the NL vocabulary.Sub-tokenization is then applied to table contexts and cell strings.Source: §3.1 · p. 4

3.2 Embedding layer

• Each token embedding sums token semantics, numerical features, in-cell position, in-table position, and cell-format features.In-table position combines bi-tree coordinates with row and column indexes; in-cell position marks the token's location within its cell.Source: §3.2 · p. 4

3.3 Tree attention

• Tree attention masks token visibility to structurally neighboring cells within a distance threshold, enabling top-down, bottom-up, and peer-to-peer information flow.The threshold controls locality; with a sufficiently large threshold, tree attention behaves like global attention.Source: §3.3 · p. 5

3.4 Pre-training objectives

• TUTA combines token-level masked language modeling with cell-level Cloze retrieval and table-context retrieval to train representations at three progressive levels.Cell-Level Cloze maps blanked cell positions to candidate strings, while table-context retrieval trains [CLS] to select relevant text segments.Source: §3.4 · p. 5

3.5 Pre-training details

• TUTA uses a BERT_BASE-aligned Transformer initialized with BERT token embeddings and encoder weights, and pre-trains on Wiki, WDC, and spreadsheet tables.The reported pre-training used about 16M tables over 2M steps.Source: §3.5 · p. 5

4.1 Cell Type Classification (CTC)

• 88.1% averaged macro-F1 across four CTC datasets, exceeding every baseline by 3.8%+.On WebSheet, TUTA also achieves the highest F1-score for each cell type, including its fine-grained header types.Source: Table 3 · p. 7, §4.1 · p. 7

4.2 Table Type Classification (TTC)

• 4.0%+ higher macro-F1 than all baselines on TTC, evaluated on five WCC table types.TaBERT outperforms TAPAS on TTC despite trailing it on CTC; DWTC's List-type errors reflect a taxonomy mismatch, while TabNet struggles on WCC's 386 annotated tables.Source: Table 4 · p. 7, §4.2 · p. 7

4.3 Ablation studies

• Reducing tree-attention distance improved CTC: TUTA-base rose from 79.9% macro-F1 to 85.3% at distance 2, beating TAPAS's 82.2%.TUTA-implicit reached 88.1% macro-F1, 2.7% above TA-2 and 0.3% above TUTA-explicit; implicit embeddings also beat both variants on TTC.Source: §4.3 · pp. 7–8, Table 5 · p. 8
• Removing any pre-training objective reduced accuracy on both tasks, with CLC most important for CTC and CLC or TCR for TTC.Removing CLC lowered CTC macro-F1 by 1.3%; removing MLM lowered TTC performance by 3.6%, while removing CLC or TCR dropped it by around 6%. TCR removal also caused a 2% SAUS drop.Source: §4.3 · p. 8

4.4 Case studies

• TUTA correctly classified all header cells in a challenging WebSheet case, but a SAUS case exposed errors in both its prediction and the ground truth.D9 was labeled Data but predicted Derived, while E9 was labeled Data but should have been Derived.Source: §4.4 · p. 8

Table representation learning

• Prior transformer pre-training targeted relational tables, while other approaches learned table or cell representations using bag-of-words, skip-gram, or LSTM-based token embeddings.Table-BERT linearized tables for BERT; TAPAS and TaBERT jointly pre-trained tables and text for question answering; TURL supported table knowledge matching and augmentation, with cells aggregating information from their row and column.Source: §5 · p. 8

Neural networks for table understanding

• Neural table-understanding methods combined spatial and semantic information using CNNs, bidirectional RNNs or LSTMs, hybrid CNN-RNNs, and graph neural networks; conditional GANs were used for formatting recommendation.Source: §5 · pp. 8–9

• TUTA is the first transformer-based method for general table structure understanding and achieves state-of-the-art results on two representative tasks.Source: §5 · p. 9

• TUTA achieves large-margin improvements over all baselines on five widely studied datasets in downstream tasks.Source: §6 · p. 9
• TUTA uses tree attention, tree position embeddings, and three pre-training objectives to learn token-, cell-, and table-level representations.The tree mechanisms capture spatial and hierarchical table information.Source: §6 · p. 9
• The authors aim to extend table QA and entity-linking datasets beyond relational tables, which are the only structures current datasets consider.They also state that TUTA’s performance on CTC and TTC supports the effectiveness of structure-aware pre-training.Source: §6 · p. 9

Data collection

• 2.62 million WikiTable tables and 50.82 million WDC WebTable Corpus tables [27] were collected with their titles, captions, and natural-language contexts.Source: A DATASET CONSTRUCTION · p. 10

Spreadsheet tables

• 13.5 million public spreadsheets from over 1.75 million websites yielded about 115 million detected tables using TableSense.Source: A DATASET CONSTRUCTION · p. 10

Pre-processing

• 52.34% of spreadsheet tables were filtered; after deduplication, 23.49% remained, 31.76% were English, and 4.49 million were retained for TUTA.Filters excluded tables below 4 rows or columns, above 512 rows or 128 columns, with more than 5 top/left header rows or columns, or without headers.Source: A DATASET CONSTRUCTION · p. 10

Feature extraction

• ClosedXML parsed spreadsheet files, while web-table JSON files were loaded and parsed using a unified feature-extraction schema.Source: A DATASET CONSTRUCTION · p. 10

• The rule-based extraction method builds separate hierarchies from detected top and left headers, then uses them to construct TUTA’s bi-dimensional coordinate tree.Source: B TREE EXTRACTION · p. 10
• Merged regions help identify header parent-child relationships because cells beneath or beside a merged parent are often its children.This inter-level association is used to form a prototype hierarchical header tree.Source: B TREE EXTRACTION · p. 10
• Indentation cues in left headers—including spaces or tabs, separate columns, and spreadsheet cell formatting—are converted into hierarchical levels for a left-header tree.Source: B TREE EXTRACTION · p. 10
• Aggregation formulas such as SUM and AVERAGE can establish parent-child relationships among referenced rows or columns, and formula-based extraction takes priority over indentation.Source: B TREE EXTRACTION · p. 10

C.1 Data pre-processing

• Data-region sampling removes 50% of text-dominant cells and 90% of value-dominant cells to limit redundant numerics and unhelpful information.The procedure aims to control sequence length and retain semantic content.Source: §C.1 · pp. 10–11
• Cell contents are capped at 8 tokens and text segments at 64 tokens, a condition met in 99% of cases in empirical studies.These limits control sequence length.Source: §C.1 · p. 11
• For MLM, TUTA selects 15% of table cells, masking one random token with 70% probability or the entire cell with 30% probability.This combines token-wise and whole-cell masking.Source: §C.1 · p. 11
• CLC selects around 20% of cells per table, favoring headers and sampling either within one bi-tree sub-tree or across top header rows and left header columns.The two selection strategies target structural information.Source: §C.1 · p. 11
• TCR supplies each table with up to three positive context segments from its own title or surrounding description and three negative segments from other contexts.One positive segment follows [CLS], while the remaining segments receive [SEP] tokens and are padded after the table tokens.Source: §C.1 · p. 11
• In TCR, [CLS] can attend to all table cells, while each context segment’s [SEP] can attend only to that segment’s internal tokens.This visibility setup gives [CLS] table-level representations and [SEP] text-segment representations.Source: §C.1 · p. 11

C.2 Model Configuration

• The bi-dimensional tree supports L=4 layers with degrees [32, 32, 64, 256]; large tables are split on detected headers, and tree/row-column embeddings use d_TL=72/d_RC=96.Each tree-coordinate index maps to a 72-dimensional embedding, while row and column indices use 96 dimensions.Source: §C.2 · p. 11
• Internal position embeddings retain 8 tokens per cell and 64 per text segment, covering over 99% of corpus strings and texts; the <1% longer cases are trimmed.The paper notes trimming avoids noise and computational inefficiency.Source: §C.2 · p. 11
• Numeric embeddings represent magnitude, precision, first digit, and last digit in [0, 10] ranges; hidden size is H=768, and 11 integer format features are linearly projected.Format features are transformed by a weight matrix and bias.Source: §C.2 · p. 11
• Pretraining runs 1M steps at 256 tokens/batch 12, then 1M at 512/batch 4; TUTA variants train on 64 Tesla V100 GPUs.Training uses PyTorch distributed; each variant takes nine days on four Tesla V100 GPUs.Source: §C.2 · p. 11

Cell Type Classification (CTC)

• CTC represents each cell with its leading [SEP] token and applies a two-layer GELU classification head to predict cell types.The head computes logits from the cell representation.Source: Cell Type Classification (CTC) · p. 11
• The CTC head is trained with cross-entropy loss against cell-type labels.The number of output classes is the number of cell types.Source: Cell Type Classification (CTC) · p. 11
• CTC datasets are split by table rather than cell, ensuring no cells from test tables are used for training.Source: Cell Type Classification (CTC) · p. 11
• CTC fine-tuning uses 4 epochs on WebSheet and 100 epochs on five random-split folds for DeEx, SAUS, and CIUS, averaging macro F1.All use batch size 4 and learning rate 8e−6.Source: Cell Type Classification (CTC) · p. 11
• For TTC, a dedicated head classifies each table from its leading [CLS] representation, and TAPAS and TaBERT use the same head and loss for comparison.The task uses cross-entropy loss and five table types.Source: Cell Type Classification (CTC) · p. 11
• TTC tables are randomly split 8/1/1 within classes to keep type distributions comparable across train, validation, and test sets.Source: Cell Type Classification (CTC) · p. 11
• TTC fine-tuning runs TUTA, TAPAS, and TaBERT for 4 epochs with batch size 2 and learning rate 8e−6.Source: Cell Type Classification (CTC) · p. 11