PaperlayerView exact revision on arXiv

Source-linked AI summary

MATE: Multi-view Attention for Table Transformer Efficiency

Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, William W. Cohen

arXiv:2109.04312v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Large tables and associated passages can exceed the sequence lengths typical Transformer models handle, creating a challenge for table reasoning. MATE uses row- and column-aware sparse attention to model table structure; the paper reports state-of-the-art results on three datasets, including a 19-point improvement on HybridQA.

  • Problem

    Large tables and linked passages can form documents too long for typical Transformer models, while many table benchmarks have used moderate-sized tables.

    Source: §1 · p. 1

  • Method

    MATE uses table-structure-aware attention patterns and row- or column-oriented heads to apply sparse attention while preserving global access for query tokens.

    Source: §3 · p. 3

  • Results

    MATE reports state-of-the-art results on TABFACT, SQA, and HYBRIDQA, with HYBRIDQA improved by 19 points.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The paper concludes that restricting attention improves the inductive bias for tabular data and enables scaling training to longer sequences through linear complexity.

    Source: §8 · p. 9

    Asymptotic efficiency gains may not materialize for short sequences, and the breakeven point depends on accelerator, batch size, and other factors.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HybridQA (Chen et al., 2020b), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.

1 Introduction

Large tables challenge standard Transformers, while table-cell representations may not need information from unrelated rows and columns. The paper introduces MATE’s row- and column-aware sparse attention and reports improved results on table-reasoning tasks and HybridQA.

1 Introduction

  • More than 20% of web relational tables have at least 20 rows, a scale that poses a problem for typical Transformer models.The introduction notes that many semi-structured documents are large, although widely used benchmarks have often used moderate-sized tables.Source: §1 · p. 1
  • The motivating intuition is that a cell’s contextual representation is unlikely to require information from a completely different row and column.The paper focuses on QA and entailment tasks on tables; HybridQA is highlighted because it combines tables with long passages about linked entities.Source: §1 · p. 1
  • MATE exploits table structure by letting each attention head traverse input row-wise or column-wise, enabling sparse, context-aware attention.The architecture is designed to give each head its own data-dependent notion of locality.Source: §1 · p. 1
  • MATE constrains attention to table rows or columns, and the paper reports new state-of-the-art results on SQA and TABFACT.The authors report that table Transformers naturally focus attention according to rows and columns, and that enforcing this improves accuracy.Source: §1 · p. 2
  • MATE scales linearly in sequence length, unlike traditional self-attention, supporting training and inference on longer sequences.The paper presents this as a way to exploit table structure when modeling larger inputs.Source: §1 · p. 2
  • 19 points: POINTR, a two-phase framework using MATE, improves the state of the art on HYBRIDQA.The task requires multi-hop reasoning over tabular and textual data.Source: §1 · p. 2

2 Related Work

Related work spans Transformer models for tables and methods for reducing attention cost. The paper distinguishes MATE by using table structure to define token locality and by combining its approach with selection methods.

2 Related Work

  • MATE is orthogonal to heuristic column selection and differentiable cell selection, and the paper reports that the methods can be combined.The cited discussion points to Table 4 for the combined approach.Source: §2 · p. 2
  • Unlike SAT’s non-head-dependent attention mask, MATE’s head-dependent design supports efficient scaling to longer sequences; the paper reports it is faster and more accurate than SAT under controlled comparisons.The comparison controls for model size and pre-training, with speed and accuracy results referenced in Tables 4 and 6.Source: §2 · p. 2
  • MATE uses input structure to cluster tokens, unlike REFORMER, which reorders tokens using locality-sensitive hashing of contextual embeddings.REFORMER was originally defined for auto-regressive training, though the paper notes this limitation can be circumvented by adapting the architecture.Source: §2 · p. 3
  • ETC and BIGBIRD use global-memory and local sparse attention, while ETC does not support dynamic or static data reordering and relative positional attention adds training overhead in the authors’ observations.BIGBIRD adds attention to random tokens to a similar approach.Source: §2 · p. 3

3 The MATE model

MATE assigns attention heads row-wise or column-wise to exploit table structure, with non-table tokens available to both. Its efficient implementation reorders tokens by head, while the paper also establishes a universal-approximation result under stated model-size conditions.

3 The MATE model

  • Row heads attend within the same row and column heads within the same column, while both can attend to and from non-table tokens.MATE uses the same row, column, and rank embeddings as TAPAS.Source: §3 · p. 3

3.1 Efficient implementation

  • MATE reorders tokens by head-specific row or column traversals, then groups them into local buckets around a global section for efficient sparse attention.When the global section contains the question and the bucket radius fits a full row or column, this implementation matches the mask-based version.Source: §3.1 · p. 4
  • Asymptotic efficiency gains may not materialize for short sequences because reshaping and reordering add overhead, with breakeven depending on accelerator and batch size.The experiments choose between two functionally equivalent MATE implementations for each use case.Source: §3.1 · p. 4

3.2 Compatibility with BERT weights

  • MATE adds no parameters, so its checkpoints are compatible with BERT or TAPAS; beyond 512 tokens, positional indices reset at each cell’s beginning.This avoids learning embeddings for larger positions and the computational cost of relative positional embeddings.Source: §3.2 · p. 4

3.3 Universal approximators

  • MATE is a universal approximator for sequence-to-sequence functions; the theorem states conditions of at least 3 heads and feed-forward hidden size at least 4.The proof reduces to prior results by ensuring at least two heads share the same type.Source: §3.3 · p. 4

4 The POINTR architecture

POINTR addresses HYBRIDQA’s long inputs with a two-stage pipeline that selects an expanded table cell and then reads an answer span from it. Its selector uses MATE and query-relevant linked-entity descriptions to fit large examples within a 2048-token budget.

4 The POINTR architecture

  • HYBRIDQA examples average 70 table cells and 44 linked entities, while covering more than 90% of examples can require over 11,000 word pieces.About 50% of examples have ambiguous answer sources, and many standard table QA datasets use tables limited to 512 tokens.Source: §4 · pp. 4–5
  • POINTR is a two-stage framework: it points to an expanded cell and then reads a span from that cell’s content and linked text.The cell content is expanded with descriptions of linked entities.Source: §4 · p. 5, Figure 3 · p. 5

4.1 POINTR: Cell Selection Stage

  • POINTR selects an expanded answer-containing cell with MATE, expanding cells using the top-k entity-description sentences most relevant to the query.With k = 5, 97% of examples fit in 2048 tokens; remaining examples are truncated uniformly. The model scores cells from token logits and uses maximum marginal likelihood to handle multiple answer-containing candidates.Source: §4.1 · pp. 5–6

4.2 POINTR: Passage Reading Stage

  • The passage reader receives the selected cell plus linked-entity sentences, truncates this expanded content to 512 tokens, and predicts an answer span conditioned on the query.At inference, it evaluates the highest-ranked cell from the selector.Source: §4.2 · p. 6

5 Experimental Setup

The experiments compare MATE on HYBRIDQA with existing systems and efficient-Transformer alternatives, then evaluate it on three other table tasks. Results are reported using the median over five seeds, with uncertainty estimated from the interquartile range.

5 Experimental Setup

  • The setup compares MATE’s HYBRIDQA performance with existing systems, reporting error bars as the interquartile range.Experiments use five random seeds and report median results.Source: §5 · p. 6

5.1 Baselines

  • HYBRIDQA comparisons include table-only, passage-only, and HYBRIDER baselines, plus LINFORMER-, ETC-, TAPAS-, and SAT-based alternatives for cell selection.The LINFORMER and ETC comparisons preserve row, column, and rank embeddings; ETC uses a 256-slot global memory divided between row and column summaries.Source: §5.1 · pp. 6–7

5.2 Other datasets

  • MATE is also evaluated on SQA, WikiTQ, and TabFact, with and without Counterfactual + Synthetic intermediate pretraining.These cover sequential question answering, table question answering that can require aggregation, and binary entailment, respectively.Source: §5.2 · p. 7

6 Results

MATE improves HYBRIDQA end-to-end over the previous state of the art and the next-best efficient-transformer system. The results also report linear scaling, faster inference than TAPAS at 2048 tokens, and gains on shorter table-reasoning tasks.

6 Results

  • MATE and LINFORMER show linear scaling as sequence length increases, but LINFORMER pretraining is 6 times slower.LINFORMER pretraining must be done at the final sequence length of 2048.Source: §6 · p. 7
  • Over 19 points over HYBRIDER and over 2.5 points over LINFORMER on both exact-match accuracy and F1, MATE leads HYBRIDQA end-to-end.The comparison is against the previous state-of-the-art and the next-best efficient-transformer system, respectively.Source: §6 · p. 7
  • At a sequence length of 2048, MATE is nearly twice as fast as TAPAS in inference speed on a cloud VM with 64GB.Source: Figure 4 · p. 7
  • On shorter table-reasoning tasks, MATE improves accuracy; with Counterfactual + Synthetic pretraining, it achieves state of the art on SQA and TABFACT.For WIKITQ, it closes the gap with TABERT’s 51.8 mean test accuracy.Source: Table 4 · p. 7, §6 · p. 7

7 Analysis

HYBRIDQA analysis finds that many sampled errors are lexical near-misses, while other misses often involve numerical or entity reasoning. Attention weights also favor table tokens in the same row or column relative to a uniform baseline.

HYBRIDQA Error analysis

  • 55% of sampled incorrect HYBRIDQA development answers were lexical near-misses; among those, around 30% involved numerical answers and 58% redundant or missing auxiliary words.The authors identify inconsistent gold-answer formatting and unavailable multiple gold answers as potential causes of the auxiliary-word discrepancies.Source: §7 · pp. 7–8
  • Among non-near-misses, most predictions were numerically incorrect or named an incorrect entity, especially on questions involving more than two hops.For a huge majority (> 80%), predicted entity type matched the gold answer’s type.Source: §7 · p. 8

Ablation Study

  • HYBRIDQA cell-selection results dropped when sparse pre-training was removed or ambiguous training examples were discarded; TAPAS was comparable to MATE on this dataset.The authors note that TAPAS lacks MATE’s theoretical and practical improvements.Source: §7 · p. 8

Observed Attention Sparsity

  • 23% of table-token attention weights went to tokens in different rows and columns, about one third of the uniform-attention baseline, on WIKITQ development data.The table reports average attention flow from table tokens to other token types.Source: Table 7 · p. 9, §7 · p. 9

8 Conclusion

The conclusion presents MATE as an efficient Transformer method that restricts attention flow for tabular data. It reports improved inductive bias, linear complexity, and state-of-the-art results on three table reasoning datasets.

8 Conclusion

  • MATE restricts attention flow in Transformers applied to tabular data and scales linearly in sequence length.The conclusion says the method allows training on larger sequence lengths.Source: §8 · p. 9
  • MATE improves the inductive bias for tabular data in both theory and practice.Source: §8 · p. 9
  • MATE achieves state-of-the-art results on TABFACT, SQA, and HYBRIDQA, improving the HYBRIDQA result by 19 points.Source: §8 · p. 9

Ethical Considerations

The paper notes that large-model experiments can carry monetary and carbon costs, and describes the experimental settings, pre-training, and appendices provided for reproducibility.

Ethical Considerations

  • Some experiments used relatively smaller scale to reduce training costs and carbon emissions; the authors supplied all annotations for the error analysis.The authors also planned to release their trained models to support reproduction and extension without retraining.Source: Ethical Considerations · p. 9

Appendix

  • The appendix provides experimental setup details, development-set results, the proof of Theorem 1, and code blocks for implementing MATE.Source: Appendix · p. 12

A.1 Pre-training

  • MATE pre-training used sequence length 128, batch size 512, and one million steps over two days.LINFORMER used sequence length 2048, batch size 128, and two million steps over 12 days; TABLEETC also used two million steps, with batch size 32.Source: §A.1 · p. 12

A.2 Fine-tuning

  • Fine-tuning experiments used Large models over five seeds and reported medians; HYBRIDQA cell selection trained for 80,000 steps at sequence length 2048.The appendix compares execution-graph FLOPs and CPU inference peak memory; it describes MATE’s memory footprint as linear, and reports reader training took around six hours.Source: §A.2 · p. 12, Figure 5 · p. 12, Figure 6 · p. 12

B Development set results for SQA, WIKITQ and TABFACT

The appendix reports development-set results for the attempted datasets and says they are consistent with the main paper’s test-set results.

B Development set results for SQA, WIKITQ and TABFACT

C Proof of Theorem 1

The proof establishes that MATE can approximate functions in F to arbitrary accuracy when it has at least three heads and a feed-forward hidden size of at least four. The section also presents an approximate TensorFlow implementation and identifies omitted implementation steps.

C Proof of Theorem 1

  • With at least 3 heads and feed-forward hidden size at least 4, MATE can approximate any function in F to within any positive error.The theorem states that an approximating function exists in TMATE for every such function and error tolerance.Source: C Proof of Theorem 1 · pp. 12–13
  • The proof restricts MATE to two same-type heads, then applies the universal-approximation result for sparse transformers with 2 heads, hidden size 4, and head size 1.The proof notes that the shared attention pattern satisfies Assumption 1 of Yun et al. (2020b).Source: C Proof of Theorem 1 · p. 13

D TensorFlow Implementation

  • The approximate TensorFlow implementation omits neighbor-bucket masking and tensor reordering and reshaping; the full implementation is slated for open-source release.The figure also omits MultiViewEmbedding creation and additional masking by input, row, and column indices; the tensor views facilitate attention across global or long parts.Source: D TensorFlow Implementation · pp. 13–14, Figure 7 · p. 14

Related papers

Loading 2109.04312v1…