PaperlayerView exact revision on arXiv

Source-linked AI summary

TableFormer: Robust Transformer Modeling for Table-Text Encoding

Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, Shachi Paul

arXiv:2203.00274v2cs.CL

TL;DR

Prior table encoders introduce order biases and incompletely model table structure and table-text alignment. TableFormer replaces these weaknesses with learnable structural attention biases, remaining invariant to row and column order while outperforming strong baselines across three table reasoning datasets and achieving state-of-the-art SQA performance.

  • Problem

    Existing table-text encoders introduce spurious row and column order biases and insufficiently model structural relations and table-text alignment.

  • Method

    TableFormer encodes table-text structure with learnable task-independent attention biases while excluding row and column order information from its input.

  • Results

    TableFormer outperforms strong baselines on three table reasoning tasks and achieves state-of-the-art performance on SQA, especially under row and column perturbations.

  • Takeaways & Limitations

    TableFormer’s structural biases support robust table reasoning without dependence on absolute row or column order.

  • Takeaways & Limitations

    TableFormer increases training time by around 20% and cannot directly answer questions depending on absolute row order; only 0.2% of examined SQA questions had such answers.

Abstract

from arXiv · show

Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer, where tabular structural biases are incorporated completely through learnable attention biases. TableFormer is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases. Our evaluations showed that TableFormer outperforms strong baselines in all settings on SQA, WTQ and TabFact table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models' performance drops by 4% - 6% when facing such perturbations while TableFormer is not affected.

1 Introduction

Existing table-text models can fail under row or column perturbations and inadequately encode structural relations. TableFormer addresses both issues with structural attention biases and achieves stronger table-reasoning performance.

  • Robustness Examples: TABLEFORMER remains correct under the illustrated row perturbation, predicting 5:02 where TAPAS gives 5:00 on the original table.After moving the first row to the end, TAPAS also predicts 5:02.
  • Structural Understanding: TABLEFORMER correctly predicts “Spain, Ukraine,” while TAPAS returns only “Spain” because of limited cell grounding.Removing the proposed structural bias causes TABLEFORMER to return the incomplete answer “Spain.”
  • Limitations of Existing Models: Prior models introduce spurious row and column order biases through table linearization, making predictions vulnerable to answer-invariant perturbations.TAPAS’s SQA performance drops 4%–6% under such perturbations.
  • TABLEFORMER: TABLEFORMER uses 13 task-independent table↔text attention biases to represent table structure and align cells with text.The same-row bias avoids spurious positional effects, while the cell-to-sentence bias improves grounding.
  • Evaluation: Experiments on three table reasoning datasets show that TABLEFORMER consistently outperforms original TAPAS across pretraining and intermediate-pretraining settings with fewer parameters.The model achieves state-of-the-art performance on SQA, and ablations support the effectiveness of its inductive biases.

2 Preliminaries: TAPAS for Table Encoding

TAPAS adapts BERT-style Transformer encoding to table-text pairs by flattening tables and adding embeddings that represent token, segment, rank, row, and column information. Its self-attention then operates over the resulting sequence.

  • Input representation: TAPAS uses linearized tables and text for pretraining and finetuning on table question answering and table-text entailment tasks.The text and flattened table are separated by [SEP] and prefixed by [CLS].
  • Input representation: TAPAS represents each sequence token by combining token, segment, global positional, rank, column, and row embeddings.For token i, the segment, column, row, and rank identifiers correspond to its location and numerical context in the table-text sequence.
  • Self-attention: TAPAS applies BERT’s multihead self-attention architecture, allowing every token to attend to all tokens in each Transformer layer.The layer input is a sequence of hidden representations, one for each table-text token.
  • Self-attention: For a single attention head, the hidden states are projected into queries, keys, and values before attention outputs are computed.The projections use W_Q, W_K, and W_V, producing Q, K, and V representations.

3 TABLEFORMER: Robust Structural Table Encoding

TABLEFORMER replaces ordered row and column information with learnable, relation-specific attention biases that encode table structure and table-text alignment. This design targets both structural understanding and invariance to row and column perturbations.

  • Architecture: TABLEFORMER uses 13 task-independent attention biases to encode relations between table and text tokens.The relations are applied through a function φ(v_i, v_j) that maps token pairs to bias types.
  • Order invariance: TABLEFORMER removes row and column embeddings and replaces global positional embeddings with per-cell positional embeddings.These changes eliminate inter-cell order information while retaining token positions within each cell.
  • Order invariance: The model encodes same-row and same-column relations instead of absolute row and column order, making its structural representation invariant to row and column permutations.Its stated intuition is that cells need relational information about shared rows, columns, and headers rather than absolute table positions.
  • Table-text alignment: Additional biases connect cells with headers and paired sentences to support column grounding, cell grounding, schema understanding, and sentence interpretation.A same-cell bias also supports cell-content understanding, while other relations remain available rather than being strictly masked.
  • Attention mechanism: Each attention head and layer learns a scalar for every bias type, and these scalars are added to the self-attention scores.The attention-bias matrix is defined by the relation between each query and key token pair.
  • Relation to prior encoding: Unlike ETC’s relative positional embeddings, TABLEFORMER uses scalar attention biases and pairwise attention rather than global memory with local attention.The authors report that relative positional embeddings increased training time by more than 7x in their observation.

4 Experimental Setup

The experiments cover table question answering, table-text entailment, and robustness to answer-invariant row and column perturbations. Models use staged pretraining and are evaluated with task-specific accuracy measures plus prediction variation after perturbation.

  • Datasets: Experiments use WTQ and SQA for table question answering and TABFACT for table-text entailment.WTQ contains complex questions over Wikipedia tables, SQA contains decomposed question sequences, and TABFACT contains entailed or non-entailed sentences.
  • Perturbation evaluation: New SQA and TABFACT test sets randomly perturb row and column orders while preserving table content to evaluate robustness during inference.The perturbations are answer-invariant and applied to all tables in the standard test sets.
  • Training: All models are pretrained on Wikipedia text-table data, optionally receive intermediate synthetic pretraining, and are then fine-tuned on the target dataset.WTQ models additionally use SQA pretraining in the inter-sqa setting.
  • Evaluation metrics: SQA evaluation reports ALL, SEQ, and ALLd accuracy, while WTQ uses denotation accuracy and TABFACT uses binary classification accuracy.SQA robustness also includes V_P, a lower bound on the percentage of example predictions that change after perturbation.
  • Baselines and models: The study compares TAPAS_BASE and TAPAS_LARGE with corresponding TABLEFORMER_BASE and TABLEFORMER_LARGE models initialized from BERT checkpoints.TABLEFORMER attention-bias scalars are initialized to zero.
  • Augmentation comparison: Data augmentation tests whether randomly shuffling training rows and columns can mitigate ordering biases without changing the model architecture.The study creates 1, 2, 4, 8, or 16 perturbed versions per training table and adjusts selected answer-cell positions for question answering.
  • Reporting: Table 1 reports SQA results before and after inference-time perturbation using median performance over 5 runs.Its metrics include ALL, SEQ, ALLd, and V_P; missing values indicate results not reported in the original paper.

5 Experiments and Results

Experiments evaluate TABLEFORMER across table reasoning benchmarks, perturbation robustness, model size, augmentation, and submodule ablations. TABLEFORMER generally outperforms TAPAS baselines and maintains stable predictions under row and column-order perturbations.

  • 5.1 Main Results: TABLEFORMER outperforms corresponding TAPAS baselines in all settings on SQA and WTQ, and achieves state-of-the-art SQA performance with intermediate pretraining.
  • 5.2 Perturbation Results: TABLEFORMER is strictly invariant to row and column-order perturbations, producing no performance drop while TAPAS drops by 3.7%–6.5% on SQA ALL accuracy.
  • 5.2 Perturbation Results: 6.2% and 2.4% improvements over TAPAS baselines occur on perturbed SQA and TABFACT test sets, respectively.
  • 5.3 Model Size: TABLEFORMER adds 13 attention-bias scalar parameters per head per layer, removes two large embedding matrices, and outperforms TAPAS with fewer parameters.
  • 5.4 Analysis of TABLEFORMER Submodules: Soft attention biases outperform hard masking, adding biases after scaling outperforms adding them before scaling, and the same-row bias is the most important ablated bias.
  • 5.3 Perturbation Augmentation: Perturbation-based augmentation improves TAPAS robustness, but its best prediction variation remains 7.0%, compared with nearly 0.1% for TABLEFORMER.
  • 5.7 Limitations of TABLEFORMER: TABLEFORMER increases training time by around 20% and cannot directly answer questions whose answers depend on absolute row order.

6 Other Related Work

Prior table models incorporated table information through headers, row/column encoders, or attention mechanisms, but structural modeling remained incomplete or costly. Learnable attention biases for tabular structures were comparatively under-explored.

  • Prior methods prepended column headers or used them as cell features, duplicating headers and increasing computing overhead.
  • Sequential row and column encoders introduced computational costs while still not fully incorporating tabular structures such as same-row information.
  • Modified attention scores have been used for relative positions, long documents, and graphs, extending structural modeling beyond ordinary sequences.
  • A manual analysis identified four questions whose predictions changed from correct with TAPASLARGE to incorrect with TABLEFORMERLARGE.
  • Learnable attention biases for modeling tabular structures had received limited exploration.

7 Conclusion

The paper identifies structural-bias capture and perturbation robustness as vulnerabilities in prior table encoders, then proposes TABLEFORMER to address both. Across three table reasoning tasks, TABLEFORMER outperforms strong baselines and reaches state-of-the-art performance on SQA, particularly under row and column perturbations.

  • TABLEFORMER targets two prior-model vulnerabilities: capturing structural bias and robustness to row and column perturbations.
  • TABLEFORMER introduces learnable, task-independent structural attention biases while remaining invariant to row and column order.
  • Across three table reasoning tasks, TABLEFORMER outperforms strong baselines and achieves state-of-the-art performance on SQA.
  • Its SQA advantage is especially pronounced under row and column order perturbations because the model is invariant to those orders.

Ethical Considerations

The authors report no ethical concerns regarding the research presented in this paper.

  • The authors foresee no ethical concerns with the research presented in this paper.
Loading 2203.00274v2…