PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepTable: Structural Attention Biases and Tree Path Encoding for Hierarchical Table Understanding

Jyun-Ying Yen, Cheng-Kuan Lin, Yu-Chee Tseng

arXiv:2609.07707v1cs.CL

TL;DR

Linearized tables and flat row and column coordinates do not explicitly represent the pairwise and hierarchical relationships encoded by multi-level headers. DeepTable adds learnable row- and column-aware attention biases and header-path encodings to TableLoRA, improving its results across three benchmarks and three LLM backbones.

  • Problem

    Linearized table sequences and existing parameter-efficient methods do not explicitly capture the pairwise relationships and hierarchical header paths of multi-level tables.

    Source: §3.3 · p. 3, §3.4 · p. 4

  • Method

    DeepTable extends TableLoRA with learnable same-row and same-column attention biases and token encodings based on row- and column-header ancestor paths.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 4

  • Results

    DeepTable improves the corresponding TableLoRA baselines by 7.42 points on HiTab, 3.23 points on WikiTQ, and 2.01 BLEU points on FeTaQA, averaged across three LLM backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported gains across three benchmarks and three backbones support the effectiveness of DeepTable’s structural biases within the evaluated setting.

    Source: §1 · p. 2

    On HiTab arithmetic queries, all evaluated methods, including DeepTable and TableLoRA, achieve 0% accuracy; the structural biases do not improve the base LLM’s numerical reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated strong performance in table understanding. However, they typically process table content and headers as linearized token sequences. This representation weakens the two-dimensional and hierarchical structural relationships encoded by multi-level row and column headers. Existing parameter-efficient fine-tuning methods incorporate basic row and column information but do not explicitly capture the rich structural dependencies induced by hierarchical table headers. We propose DeepTable, a structure-aware approach for table understanding with LLMs. DeepTable comprises two complementary components. Structural Attention Bias (SAB) introduces learnable biases into the attention logits to explicitly represent whether pairs of table tokens share the same row or column. Tree Path Encoding (TPE) represents each table token using the ancestor paths of its row and column headers, preserving its position within the multi-level table structure. We integrate DeepTable with TableLoRA (He et al., 2025) to inject structural information into parameter-efficient adaptation. Across three LLM backbones, DeepTable consistently improves the corresponding TableLoRA baselines on three table question answering benchmarks, achieving average gains of 7.42 points on HiTab, 3.23 points on WikiTQ, and 2.01 BLEU points on FeTaQA. These results demonstrate the effectiveness of the proposed structural biases across different LLM backbones.

1 Introduction

Hierarchical headers encode relationships that linearized table sequences and flat coordinates do not explicitly represent. DeepTable adds learnable attention biases and header-path encodings to TableLoRA, improving results across three benchmarks and three LLM backbones.

1 Introduction

  • The work follows the parameter-efficient fine-tuning direction, which updates only a small number of parameters while keeping the pretrained model frozen.The authors focus on improving structural modeling within this adaptation setting.Source: §1 · p. 1
  • Linearizing tables does not explicitly encode two-dimensional relationships induced by multi-level row and column headers, leaving those relationships to be inferred from the sequence.These headers organize cells into nested groups and define relationships across table levels.Source: §1 · p. 1
  • TableLoRA uses flat row and column coordinates, but does not explicitly encode pairwise relationships or hierarchical header paths, with smaller reported gains on deeper hierarchies.The authors identify this limitation as motivation for additional structural signals.Source: §1 · p. 1
  • SAB adds learnable attention biases for same-row or same-column token pairs, while TPE encodes each token using the ancestor paths of its top and left headers.DeepTable is built on TableLoRA and remains parameter-efficient.Source: §1 · pp. 1–2
  • DeepTable improves TableLoRA by 7.42 points on HiTab, 3.23 points on WikiTQ, and 2.01 BLEU points on FeTaQA, averaged across three LLM backbones.The evaluation covers three table question answering benchmarks.Source: §1 · p. 2

2 Related Work

Related work spans structure-aware table models, parameter-efficient adaptation, and structural modeling in Transformers and other hierarchical inputs. DeepTable combines attention-based structural information and header-path encodings within parameter-efficient adaptation.

Structure-aware table understanding

  • Prior table models incorporate structure through specialized encoders, tree-based positional encodings, or row- and column-wise dependencies; DeepTable instead models structure within parameter-efficient adaptation.Examples include TAPAS, TABBIE, and TUTA.Source: §2 · p. 2

Parameter-efficient adaptation for table understanding

  • TableLoRA extends LoRA with learnable row and column coordinates, while DeepTable additionally models pairwise relationships and hierarchical header paths.LoRA, prompt tuning, and p-tuning reduce adaptation cost by updating only a subset of parameters.Source: §2 · p. 2

Structural modeling in Transformers

  • Transformer structural priors include learned relative-position biases in T5, distance-based attention biases in ALiBi, and table-layout biases or patterns in TableFormer, SAT, and MATE.The approaches differ in whether they use learnable biases or fixed attention patterns.Source: §2 · p. 2

Hierarchical representations

  • Tree-LSTM and HiBERT model tree-structured inputs and hierarchical documents, respectively; DeepTable combines structural attention and header-path encodings for hierarchical tables.The related-work review also identifies these broader hierarchical representations.Source: §2 · p. 2

3 Method

DeepTable extends TableLoRA with structural signals beyond flat coordinates: SAB adds learnable same-row and same-column attention biases, and TPE encodes hierarchical header paths. The method serializes tables as token sequences and incorporates these signals into parameter-efficient adaptation.

3.1 Background: TableLoRA

  • TableLoRA augments key and value projections with learnable row and column coordinate embeddings, yielding representations based on flat coordinates rather than explicit pairwise or hierarchical structure.The coordinate embedding sums the learned row and column embeddings for each table token.Source: §3.1 · p. 3

3.2 Table Content Tokenization

  • Tables are serialized row-major with [TAB], [ROW], and [CELL] markers; column-header levels occupy separate rows, while row-header ancestors are duplicated alongside their associated data cells.Header and ordinary cells are treated uniformly as textual tokens.Source: §3.2 · p. 3

3.3 Structural Attention Bias (SAB)

  • TableLoRA’s fixed same-cell and same-row/column biases scored 40.50 and 41.00 versus 48.94 for the original 2D-LoRA baseline on HiTab.The authors cite this comparison as evidence that manually designed fixed biases are insufficient for hierarchical tables.Source: §3.3 · pp. 3–4
  • SAB learns attention biases from row- and column-adjacency matrices, with separate learnable scalar weights for each layer and attention head.The matrices indicate whether table-token pairs share a row or column; special tokens receive zero structural bias.Source: §3.3 · p. 4

3.4 Tree Path Encoding (TPE)

  • Because row- and column-membership patterns do not explicitly encode header hierarchy, TPE builds row- and column-header trees and assigns each token paths from their roots to its headers.For example, the row and column paths for token 10 are (A → A1) and (years → 2014).Source: §3.4 · pp. 4–5
  • TPE averages learned embeddings along each row and column path, then adds both path embeddings to the coordinate embeddings before incorporating the structural signals into LoRA adaptation.The resulting token representation uses row and column coordinates plus row- and column-path embeddings.Source: §3.4 · p. 5

4 Experiments

DeepTable improves on TableLoRA across the three table question-answering benchmarks, while remaining comparable on flat-table TabFact. The experiments show that the relative benefits of SAB and TPE vary with table structure, task, and backbone.

4.1 Experimental Setup

  • Evaluation averages four seeds with greedy decoding across four benchmarks; HiTab alone has hierarchical headers, and its aggregation breakdown is reported only for DeepSeek.The study also evaluates DeepSeek, Llama-3, and Qwen2.5 backbones.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 Comparison with TableLoRA

  • DeepTable consistently improves over TableLoRA on HiTab, WikiTQ, and FeTaQA while maintaining comparable performance on TabFact.The comparison covers four benchmarks and three LLM backbones.Source: §4.2 · p. 6
  • Across datasets, SAB and TPE trade off: SAB leads on HiTab, TPE gains most on FeTaQA, and their relative contributions vary by backbone.DeepTable variants have their largest average gains on WikiTQ and FeTaQA, while DeepTable without TPE performs better on HiTab and TabFact.Source: §4.2 · p. 6, §4.4 · p. 8

4.3 Header-Depth Analysis

  • SAB’s lift rose +6.37→+7.03→+7.52 on DeepSeek and +1.27→+2.54→+6.29 on Qwen2.5 as header depth increased; TPE’s lift showed no monotonic trend.TPE’s lifts were +0.14, +1.11, and +0.41 on DeepSeek, and +0.28, −0.59, and +1.31 on Qwen2.5.Source: §4.3 · p. 7

4.4 Two-Axis Analysis: Structure vs. Hierarchy Complexity

  • SAB benefits structurally complex tables by supporting cell alignment; its largest aggregation gain is on single-cell lookup, while arithmetic queries remain at 0%.Comparison and ranking queries also benefit, but less than lookup queries.Source: §4.4 · p. 7
  • TPE contributes more on sentence-generation FeTaQA than extraction tasks, where its gains are marginal or sometimes negative.The authors suggest that header-path information can be redundant for extraction and may sometimes interfere with two-dimensional coordinates.Source: §4.4 · p. 7

5 Conclusion

DeepTable combines SAB and TPE to represent table structure and hierarchical header paths. It improves over TableLoRA across three question-answering benchmarks, while analyses show that the biases’ contributions vary by task, dataset, and backbone.

5 Conclusion

  • On DeepSeek-LLM-7B-Chat, DeepTable exceeds TableLoRA by +5.11 on HiTab, +2.66 on WikiTQ, and +3.03 BLEU on FeTaQA, with comparable TabFact performance.SAB encodes same-row and same-column relationships; TPE injects hierarchical header-path information into each cell representation.Source: §5 · p. 8
  • SAB gains increase with header depth, while TPE contributes more prominently on free-form generation; their relative effectiveness varies across backbones and datasets.TPE’s gains are limited or inconsistent across most aggregation types and header depths.Source: §5 · p. 8
  • Future work may explore adaptive mechanisms that select structural biases according to the task.Source: §5 · p. 8

Limitations

The evaluation is limited to three open-source 7–8B backbones, and several reported analyses or settings have narrower scope. Arithmetic queries and hyperparameter selection also expose constraints on the reported results.

  • DeepTable is evaluated on three open-source 7–8B LLMs; generalization to larger or closed-source models remains untested, and stronger models may capture structure implicitly.Source: Limitations · p. 9
  • Fine-grained analyses cover only selected backbones, and reported performance and gains may differ under decoding strategies other than greedy decoding.Source: Limitations · p. 9
  • All evaluated methods achieve 0% accuracy on arithmetic HiTab queries, leaving DeepTable’s arithmetic performance bounded by the base model’s reasoning ability.Source: Limitations · p. 9
  • The shared SAB/TPE learning-rate multiplier was selected on one HiTab seed with DeepSeek; per-model or per-dataset tuning may yield further improvements.Source: Limitations · p. 9

A Decoding-Protocol Audit

The decoding audit finds different HiTab gains under sampling and greedy decoding, while the learning-rate sweep selects a shared multiplier using one seed.

A Decoding-Protocol Audit

  • All reported results use greedy decoding because inherited stochastic sampling produces different absolute scores and performance gaps.Source: A Decoding-Protocol Audit · p. 11
  • 2.49 percentage points under stochastic sampling versus 1.50 percentage points under greedy decoding are DeepTable’s HiTab gains over TableLoRA.Source: A Decoding-Protocol Audit · p. 11

B Learning-Rate Multiplier Sweep

C Subspace-Overlap and Tree Path Encoding Variant Ablations

Ablations indicate that TPE retains effectiveness without row/column embeddings, while tested TPE variants do not consistently outperform the adopted configuration. The current design ties path-embedding capacity to a shared projection matrix.

C Subspace-Overlap and Tree Path Encoding Variant Ablations

D Per-Seed Results and Significance

The appendix provides per-seed benchmark breakdowns and a depth-stratified HiTab analysis. DeepTable improves on TableLoRA across most examined top/left header-depth combinations, with the largest fluctuations in sparse deepest buckets.

D Per-Seed Results and Significance

E Depth-Stratified Grid

  • Table 10 stratifies HiTab by top- and left-header depths; DeepTable improves over TableLoRA in most combinations, with the largest fluctuations in sparse deepest buckets.Source: E Depth-Stratified Grid · p. 12

F TabFact True/False Asymmetry

The TabFact analysis finds different effects from TPE and SAB on True and False examples, with TPE also shifting predictions toward True. The hyperparameter section describes shared settings and backbone-specific differences.

F TabFact True/False Asymmetry

G Hyperparameters

  • Experiments follow He et al. (2025), with hardware-driven batch-size and precision changes plus SAB/TPE learning-rate multipliers; DeepSeek and Qwen share settings apart from backbone-dependent SAB parameter counts.Source: G Hyperparameters · p. 13

H Implementation Details

Implementation assigns each cell’s structural metadata to its tokens for SAB and TPE, then adds SAB biases to attention logits. The implementation also verifies unchanged initial logits and notes a throughput cost from disabling FlashAttention.

  • Each serialized example carries per-cell row and column indices, text, top- and left-header paths, and a cell-type label.Source: H Implementation Details · p. 13
  • The TabFact table reports four-seed accuracy by gold label for 6,425 True and 6,354 False test examples.Source: Table 11 · p. 13
  • The collator assigns cell row and column indices to tokens for SAB and header-path IDs for TPE; SAB biases enter logits after causal masking or through the SDPA additive mask.Source: H Implementation Details · p. 13
  • The training-hyperparameter table indicates that Qwen2.5-7B and Llama-3-8B-Instruct differ from DeepSeek only in backbone and resulting SAB parameter count.Source: Table 12 · p. 14
  • Disabling FlashAttention because it cannot inject arbitrary additive attention biases reduces training throughput by approximately 22%.Source: H Implementation Details · p. 14
  • Before optimization, TableLoRA and the full SAB + TPE model produce bit-identical logits on a fixed validation batch.Source: H Implementation Details · p. 14

Related papers

Loading 2609.07707v1…