PaperlayerView exact revision on arXiv

Source-linked AI summary

TUTA: Tree-based Transformers for Generally Structured Table Pre-training

Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, Dongmei Zhang

arXiv:2010.12537v4cs.IRcs.AIcs.DB

TL;DR

Table-understanding research has largely centered on relational tables despite the prevalence of other structures and the difficulty of labeling their structural information. TUTA introduces a bi-dimensional coordinate tree with tree-based attention and position embeddings, plus progressive token-, cell-, and table-level pre-training. It achieves state-of-the-art performance across five widely studied datasets, while numerical computation remains outside its current scope.

  • Problem

    Most table-understanding methods focus on relational tables, overlooking common matrix and entity tables; labeling structural information is time-consuming and labor-intensive.

  • Method

    TUTA uses a bi-dimensional coordinate tree, tree-based attention and position embeddings, and three progressive objectives for token-, cell-, and table-level representations.

  • Results

    TUTA achieves state-of-the-art performance across five widely studied datasets for cell type classification and table type classification.

  • Takeaways & Limitations

    TUTA demonstrates the effectiveness of structure-aware pre-training for understanding generally structured tables.

  • Takeaways & Limitations

    Numerical computations exceed the paper’s current scope of table understanding and are left for future work.

Abstract

from arXiv · show

Tables are widely used with various structures to organize and present data. Recent attempts on table understanding mainly focus on relational tables, yet overlook to other common table structures. In this paper, we propose TUTA, a unified pre-training architecture for understanding generally structured tables. Noticing that understanding a table requires spatial, hierarchical, and semantic information, we enhance transformers with three novel structure-aware mechanisms. First, we devise a unified tree-based structure, called a bi-dimensional coordinate tree, to describe both the spatial and hierarchical information of generally structured tables. Upon this, we propose tree-based attention and position embedding to better capture the spatial and hierarchical information. Moreover, we devise three progressive pre-training objectives to enable representations at the token, cell, and table levels. We pre-train TUTA on a wide range of unlabeled web and spreadsheet tables and fine-tune it on two critical tasks in the field of table structure understanding: cell type classification and table type classification. Experiments show that TUTA is highly effective, achieving state-of-the-art on five widely-studied datasets.

1 INTRODUCTION

TUTA addresses the gap between relational-table-focused table understanding methods and the diverse structures of real-world tables through structure-aware pre-training. It represents spatial and hierarchical information with a bi-dimensional coordinate tree and progressively learns token-, cell-, and table-level representations.

  • Motivation: Most existing table-understanding research focuses on relational tables, although matrix and entity tables are also common in real-world data.Relational tables account for 0.9% of commonly crawled web tables and 22.0% of spreadsheet tables.
  • Motivation: Labeling structural information is time-consuming and labor-intensive, creating a data-shortage challenge for data-hungry machine-learning methods.
  • Motivation: Hierarchical layouts help people interpret tables, whereas treating every table as relational loses valuable structural information.A matrix table can jointly describe a value through multiple row and column headers.
  • Approach: TUTA introduces a bi-dimensional coordinate tree that systematically represents cell location and header hierarchy in generally structured tables.The model uses tree-based positional encodings and tree-based attention to incorporate structural information.
  • Approach: Three progressive pre-training tasks learn token-, cell-, and table-level representations, followed by fine-tuning for cell and table type classification.The tasks are masked language modeling, multichoice Cloze, and context retrieval; experiments cover five widely studied datasets.

2 PRELIMINARIES

The paper builds a diverse pre-training corpus and represents generally structured tables with a unified bi-dimensional coordinate tree. Tree coordinates encode positions across top and left header hierarchies, while tree distance supports spatial and hierarchical information flow.

  • Pre-training corpus: The pre-training corpus includes relational, entity, and matrix tables from both web pages and spreadsheets.Extremely small non-data tables are excluded to ensure data quality.
  • Pre-training corpus: 57.9 million tables are collected from Wikipedia, the WDC WebTable Corpus, and more than 13.5 million web-crawled spreadsheet files.
  • Bi-dimensional coordinate tree: The bi-dimensional coordinate tree jointly describes cell location and header hierarchy using orthogonal top and left sub-trees.Each cell maps to nodes in both sub-trees, whose paths combine into a unique coordinate.
  • Bi-dimensional coordinate tree: A cell’s tree distance is the sum of its shortest-path distances in the top and left trees.For example, the distance between A6 and C2 is 6 when movement requires three steps in each tree.
  • Bi-dimensional coordinate tree: In flat tables, tree coordinates reduce to rectangular Cartesian coordinates, while global titles and descriptions have distance zero from cells.The resulting distance supports spatial and hierarchical data flow through tree attention.

3 TUTA MODEL

TUTA adapts BERT for generally structured tables by combining table-specific embeddings, tree-based attention, and progressive objectives across token, cell, and table levels.

  • Architecture: TUTA adapts BERT with a dedicated table vocabulary, tree-based position embeddings, structure-aware attention, and three progressive pre-training objectives.The objectives target token-, cell-, and table-level representations.
  • Vocabulary construction: Because table cell strings use abbreviations and distributions unlike natural-language text, TUTA builds a dedicated vocabulary with 9,754 new tokens.The vocabulary is merged with the natural-language vocabulary to model table descriptions as well.
  • Embedding layer: Tokens combine semantic, numerical, in-cell, in-table, and formatting embeddings to represent table content and structure.The combined representation is E = E_tok + E_num + E_c_pos + E_t_pos + E_fmt.
  • Embedding layer: In-table position embeddings concatenate projected top-tree, left-tree, column, and row coordinates.Top and left coordinates receive level-specific sub-embeddings, while row and column indexes receive separate sub-embeddings.
  • Tree attention: Tree-based attention masks token visibility to structural neighborhoods defined by a distance threshold, reducing attention to unrelated contexts.Neighbor cells are selected using tree distance, while neighbor tokens include tokens in structurally related cells.
  • Pre-training details: The pre-training objective sums masked language modeling, cell-level contrastive learning, and table context retrieval losses.Table context retrieval uses positive or negative text segments and binary cross-entropy classification.

4 EXPERIMENTS

TUTA is evaluated on cell- and table-level structure understanding, with experiments, ablations, and case studies examining its effectiveness and failure boundaries.

  • Cell Type Classification: CTC identifies fine-grained structural cell types across datasets spanning financial, business, agricultural, and healthcare domains.The evaluation uses WebSheet, DeEx, SAUS, and CIUS, with Macro-F1 as the overall metric.
  • Cell Type Classification: 88.1% averaged macro-F1 on four CTC datasets surpassed every baseline by at least 3.8%.The comparison used the same train/validation/test splits, with no test tables or cells seen during training.
  • Cell Type Classification: TUTA achieved the highest F1-score for every WebSheet cell type, including challenging fine-grained header categories.WebSheet’s header types have diverse semantics under complicated hierarchies because the task supports relational data extraction.
  • Table Type Classification: TTC evaluates table-level structural classification over five WCC categories: relational, entity, matrix, list, and non-data.The study compares TUTA with DWTC, TabNet, TabVec, TAPAS-large, and TaBERT-large.
  • Table Type Classification: TaBERT outperformed TAPAS on TTC despite performing worse on CTC, while representation-learning methods generalized better across table taxonomies than expert-engineered features.DWTC performed poorly on the List type because its definitions differed from WCC; TabNet also struggled on the small 386-table dataset.
  • Ablation Studies: Reducing tree-attention distance and adding tree position embeddings improved TUTA, while removing any pre-training objective reduced performance on both tasks.TA-2 improved the base CTC score by 5.4%; removing CLC reduced CTC macro-F1 by 1.3%, while removing CLC or TCR reduced TTC performance by about 6%.

5 RELATED WORK

Prior table representation work largely pre-trained transformers for relational tables, while other neural architectures jointly model tables’ spatial and semantic information.

  • Table representation learning: Earlier transformer methods such as Table-BERT, TAPAS, TaBERT, and TURL primarily target relational tables.Table-BERT linearizes tables; TAPAS and TaBERT jointly pre-train tables with text; TURL learns representations for table knowledge matching and augmentation.
  • Table representation learning: TURL aggregates information from each cell’s located row and column, whereas other approaches use continuous bag-of-words, skip-gram, or LSTM-based models.The related work also mentions Table2Vec and TabNet as alternative representation-learning approaches.
  • Neural networks for table understanding: Neural architectures for table understanding aim to capture spatial and semantic information jointly, including CNNs and conditional GANs for table formatting.The passage also notes neural networks for table understanding and question answering.
  • Neural networks for table understanding: TUTA is presented as the first transformer-based method for general table structure understanding and achieves state-of-the-art results on two representative tasks.This distinguishes TUTA from prior transformer work focused on relational tables.

6 CONCLUSION AND DISCUSSION

TUTA is a structure-aware pretraining model for generally structured tables, combining tree-based mechanisms with progressive representation learning. It improves over baselines on five widely studied datasets and performs strongly on cell and table structure classification.

  • Conclusion: TUTA is a structure-aware pretraining model designed to understand generally structured tables.The paper describes it as the first transformer-based method for table structure understanding.
  • Conclusion: Tree attention and tree position embeddings capture spatial and hierarchical information in tables.These are the model’s two core techniques for representing table structure.
  • Conclusion: Three pre-training objectives learn representations progressively at the token, cell, and table levels.The objectives extend representation learning beyond the token level.
  • Conclusion: TUTA shows a large margin of improvement over all baselines on five widely studied datasets in downstream tasks.The reported downstream tasks include cell type classification and table type classification.
  • Discussion: TUTA’s superior performance on cell type classification and table type classification supports the effectiveness of structure-aware pre-training.The authors also aim to extend question answering and entity linking datasets beyond relational tables.

A DATASET CONSTRUCTION

TUTA’s pre-training data combines large web-table and spreadsheet-table corpora, with spreadsheet tables detected from crawled files and additional preprocessing used to address noise.

  • Data collection: The web-table corpus contains 2.62 million WikiTable tables and 50.82 million WDC WebTable tables.Titles, captions, and natural-language contexts are retained for the web tables.
  • Data collection: The spreadsheet corpus comes from about 13.5 million public files collected across more than 1.75 million websites.Table detection yields about 115 million spreadsheet tables.
  • Pre-processing: Spreadsheet tables are noisy because they are crawled from varied websites, so the corpus undergoes data cleansing and language filtering.The passage introduces these steps as preprocessing for building a clean training corpus.
  • Feature extraction: Web and spreadsheet tables use a unified feature-extraction schema after parsing their respective source formats.ClosedXML parses spreadsheet files, while JSON files are loaded for the two web-table corpora.

B TREE EXTRACTION

Tree extraction builds TUTA’s bi-dimensional coordinate tree from detected top and left header regions using structural cues such as merged cells, indentation, and formulas.

  • Tree extraction: A rule-based strategy extracts tree hierarchies from merged cells, indentation levels, and formulas in data cells.The strategy is applied separately to detected top and left headers.
  • Tree extraction: The extracted top and left header hierarchies are combined into the bi-dimensional coordinate tree used by TUTA.This construction follows header-region detection.
  • Feature extraction: Table cells are represented with features extracted through a unified schema for web and spreadsheet tables.The feature set is summarized in Table 7.
  • Structural cues: Merged regions provide spatial alignment and grouping information that helps identify hierarchical parent-child relationships among headers.Cells beneath or beside a merged region are often treated as children of the merged parent.

C.1 Data pre-processing

The preprocessing pipeline reduces redundant numerical content and limits sequence length before model training. It samples cells by content type and truncates cells and text segments to empirically supported lengths.

  • Cells are heuristically sampled because similar numerics often provide limited semantics.Text-dominant cells are sampled out by 50%, while value-dominant cells are sampled out by 90%.
  • The preprocessing retains at most 8 tokens per cell and 64 tokens per text segment.The paper reports that 99% of cases satisfy these limits.

C.2 Model Configuration

TUTA uses bounded tree and token representations for table positions, while separately embedding numerical magnitude, precision, boundary digits, and cell-format features. Pre-training proceeds in two stages with progressively longer sequences.

  • Tree configuration: The coordinate tree uses at most 4 layers with degrees of 32, 32, 64, and 256.This supports up to four top or left header rows or columns and up to 256 cells at the deepest level.
  • Tree configuration: Large tables can be split into smaller tables that share the same top or left header.
  • Token representation: Internal position embeddings keep the first 8 tokens of each cell and 64 tokens of each text segment.The corpus study found that over 99% of cell strings and texts satisfy these limits, while longer strings are trimmed.
  • Feature embeddings: Number embeddings represent magnitude, precision, first digit, and last digit using separate embedding weights.Each numerical range parameter is set to 10, and the hidden size is 768.
  • Feature embeddings: Format embeddings transform 11 integer cell features into the hidden representation with a weight matrix and bias.
  • Pre-training schedule: Pre-training uses 256-token sequences for 1M steps, then 512-token sequences for another 1M steps.The batch sizes are 12 and 4 in the two stages, respectively.

C.3 Downstream tasks

The downstream evaluation fine-tunes TUTA for cell type classification and table type classification using dedicated transformer representations and classification heads. Dataset splitting and shared hyperparameters are used to support evaluation and comparisons.

  • Cell Type Classification: Cell type classification represents each cell with its leading [SEP] output and predicts cell-type logits through a two-layer GELU head.Training uses cross-entropy loss against cell-type labels.
  • Evaluation protocol: Datasets are split table-wise so cells from test tables are not used during training.
  • Fine-tuning: WebSheet is fine-tuned for 4 epochs, while DeEx, SAUS, and CIUS use 100 epochs on five random-split folds and report average macro F1.All these experiments use batch size 4 and learning rate 8e−6.
  • Table Type Classification: Table type classification represents each table with its leading [CLS] output and predicts five table types using a dedicated classification head.The objective is cross-entropy loss against ground-truth types.
  • Table Type Classification: TAPAS and TaBERT use the same table-type classification head and loss as TUTA for unbiased comparison.
  • Evaluation protocol: Table type datasets are split within classes in an 8/1/1 train, validation, and test proportion.The type distribution is kept comparable across the three splits.
  • Fine-tuning: TUTA, TAPAS, and TaBERT use identical downstream hyperparameters: batch size 2, learning rate 8e−6, and 4 epochs.
Loading 2010.12537v4…