PaperlayerView exact revision on arXiv

Source-linked AI summary

InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

Hanqian Li, Sirui Huang, Chen Ling, Jungang Li, Yu Huang, Kening Zheng, Yonghua Hei, Xiangrong He, Shiyi Wang, Pengcheng Zhu, Dongnan Liu, Wei Zhou, Linjian Mo, Nai Ding, Xuming Hu

arXiv:2609.32660v1cs.CVcs.AI

TL;DR

Table reasoning requires locating relevant cells step by step, but current MLLMs rely on a single global image encoding that can fail to ground those steps. InterTab trains reasoning interleaved with structure-aligned crops, improving average accuracy across nine benchmarks from 68.28 to 73.17.

  • Problem

    Current MLLMs encode a table image once before reasoning, although multi-step questions require locating different rows, columns, and cells as reasoning unfolds.

    Source: §1 · pp. 1–2

  • Method

    InterTab interleaves chain-of-thought with structure-aligned crops and trains on aligned trajectories using supervised structure-aware alignment followed by active localization optimization.

    Source: §1 · p. 2

  • Results

    68.28 to 73.17 average accuracy: InterTab improves its backbone and achieves the best average performance among compared methods across nine table datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    InterTab shifts multimodal table reasoning from passive global encoding toward active, structure-aware evidence acquisition.

    Source: §1 · p. 2

    InterTab may produce incorrect reasoning or inaccurately grounded references, especially with ambiguous layouts or low-quality scans, and is intended as assistive rather than autonomous support in high-stakes domains.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Table images preserve structural information that are often lost in text serialization, and reasoning over them requires locating relevant rows, columns, and cells step by step. Current multimodal large language models (MLLMs) encode the whole image once before reasoning, so they cannot pick up row-, column-, and cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each reasoning step to that structure. We propose \textbf{InterTab}, an \textbf{Inter}leaved structure-aware framework for CoT reasoning over \textbf{Tab}le images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware alignment (SSA) on InterTab-22K teaches the model to interleave reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.

1 INTRODUCTION

InterTab addresses the limits of fixed global image perception by interleaving reasoning with structure-aligned table crops. It trains on aligned trajectories and improves its backbone’s average accuracy across nine table datasets.

1 INTRODUCTION

  • Current MLLMs encode the whole table image once, so reasoning steps that need different small regions can misread adjacent cells and propagate visual-grounding errors.Source: §1 · pp. 1–2
  • InterTab interleaves chain-of-thought with tool calls that crop structure-aligned regions, conditioning subsequent reasoning on the selected visual tokens.Source: §1 · p. 2
  • InterTab-22K contains 21.7K trajectories aligned through table metadata, and training combines supervised structure-aware alignment with active localization optimization.Source: §1 · p. 2
  • InterTab improves its backbone’s average accuracy from 68.28 to 73.17 and achieves the best average performance among compared methods across nine table datasets.Source: §1 · p. 2

2 RELATED WORKS

Prior multimodal table methods use OCR or end-to-end vision-language models, while table benchmarks extend evaluation to structured visual settings. InterTab combines interleaved reasoning with table-structure-aligned crops.

2 RELATED WORKS

  • Prior multimodal table approaches include OCR-based extraction and end-to-end vision-language models, while benchmarks extend reasoning evaluation to structured visual environments.Source: §2 · p. 2
  • Generic interleaved visual reasoning alternates textual and visual evidence, but table reasoning needs each step bound to discrete row, column, or cell structure.Source: §2 · p. 3
  • InterTab combines textual reasoning with structure-aligned regional crops and learns where to crop rather than only imitating demonstrations.Source: §2 · p. 3

3 PRELIMINARY

The preliminary setup contrasts reasoning from one global image encoding with interleaved reasoning that can add local visual evidence at each step. For tables, those crops must align precisely with referenced structure.

3 PRELIMINARY

  • In standard CoT, an MLLM encodes table image I into visual tokens and generates each output token conditioned on those tokens, the question, and prior output.Source: §3 · p. 3
  • Interleaved CoT lets reasoning steps end with bounding boxes whose crops provide local visual evidence, with boxes aligned to the rows, columns, or cells referenced.Source: §3 · p. 3
  • InterTab-22K construction uses tool-based multi-turn rollouts, judge filtering, then deduplication, length filtering, and balancing across eight source datasets.Source: Figure 2 · p. 4

4 METHODOLOGY

InterTab combines a structure-aware reasoning dataset with a two-stage training framework that interleaves reasoning and visual evidence. Its stages first align reasoning with table structure, then optimize tool-augmented reasoning and localization.

4 METHODOLOGY

  • InterTab is trained in two stages: Supervised Structure-aware Alignment (SSA), followed by Active Localization Optimization (ALO).The framework is built on InterTab-22K.Source: §4 · p. 4

4.1 STRUCTURE-AIDED REASONING DATA CONSTRUCTION

  • 21,721 unique trajectories make up InterTab-22K, constructed from eight table datasets through structure-aided rollouts, validation, and cleaning.For supervised fine-tuning, the eight source datasets are balanced by upsampling.Source: §4.1 · p. 4

4.2 INTERTAB

  • InterTab pairs SSA, which incrementally aligns visual and structural information, with ALO, which uses tool-augmented training to improve reasoning accuracy and visual localization.ALO can be applied after SSA or directly to the base model.Source: §4.2 · p. 4

4.2.1 SUPERVISED STRUCTURE-AWARE ALIGNMENT (SSA)

  • SSA trains on InterTab-22K examples in which reasoning steps needing visual evidence are paired with boxes aligned to referenced rows, columns, or cells.The corresponding visual subregions are encoded as tokens and appended to the reasoning.Source: §4.2.1 · p. 5

4.2.2 ACTIVE LOCALIZATION OPTIMIZATION (ALO)

  • ALO treats each tool call and its box as a model-sampled action, rewarding answer correctness, output format, and box IoU while penalizing missing or excessive calls.The tool penalty applies to calls beyond the first N0.Source: §4.2.2 · p. 5

5 EXPERIMENT

Experiments compare InterTab with existing models on in-domain benchmarks, test its design choices, and evaluate transfer to held-out Visual Table QA. InterTab leads across the in-domain benchmarks, while ablations examine how interleaving, training stages, reward terms, and tool-call allowance relate to performance.

5.1 EXPERIMENTAL SETTINGS

  • The experiments address four questions: comparative in-domain performance, interleaving versus text-only reasoning, design and allowance effects, and performance on a held-out benchmark.Evaluation uses nine benchmarks, including VisualTQA as out-of-domain; training uses Qwen3-VL-8B-Instruct and InterTab-22K data from eight in-domain datasets.Source: §5 · p. 5, §5.1 · p. 5

5.2 MAIN RESULTS

  • 73.17% versus 67.18% average accuracy across nine benchmarks, with InterTab best on all eight in-domain benchmarks.The results also report a mean improvement of 5.92 points across the in-domain benchmarks.Source: §5.2 · pp. 6–7, Table 1 · p. 6

5.3.1 INTERLEAVED REASONING

  • 73.17% versus 68.89% average accuracy: interleaved reasoning outperforms the matched text-only baseline across all nine benchmarks, including held-out VTQA.The gains are +8.39 points on HybridQA and +5.41 points on held-out VTQA.Source: §5.3.1 · p. 7, Table 2 · p. 7

5.3.2 TWO-STAGE TRAINING

  • SSA+ALO averages 73.17%, exceeding SSA-only at 69.69% and ALO-only at 69.63%, and is best on eight of nine benchmarks.TabMWP is the exception: ALO-only scores 97.34% versus 96.35% for SSA+ALO.Source: Figure 4 · p. 7, §5.3.2 · p. 8

5.3.3 REWARD COMPONENTS

  • Removing localization reward lowers average accuracy by 2.87 points to 70.30%, while removing the call penalty lowers it by 1.69 points to 71.47%.Both variants underperform the full reward on eight of nine benchmarks; TabMWP is the main exception for localization reward.Source: §5.3.3 · p. 8, Table 3 · p. 8

5.3.4 TOOL-CALL ALLOWANCE

  • N0 = 7 achieves the highest nine-benchmark average, 73.17%, among tested allowances of 3, 5, 7, and 9.The corresponding averages for N0 = 3, 5, and 9 are 71.30%, 71.58%, and 72.99%.Source: §5.3.4 · p. 8, Figure 5 · p. 9

5.4 OUT-OF-DOMAIN EVALUATION

  • On held-out Visual Table QA, InterTab scores 59.80%, ahead of Qwen3-VL-8B at 53.27% and GLM-4.1V-9B at 51.75%.Removing the call penalty raises VTQA to 62.75%, but the full setting is retained for the overall benchmark trade-off.Source: §5.4 · pp. 8–9, Figure 5 · p. 9

5.5 CASE STUDY

  • In the case study, DeepEyes and Qwen3-VL-8B-Instruct select the wrong row; InterTab identifies the unchecked July 27 entry, selects the correct July 26 entry, and returns Dylan McDermott, Josh Henderson.The task is to find the latest row with a checked “Kelly’s Inbox” entry and retrieve its corresponding guests.Source: §5.5 · p. 9

6 CONCLUSION

InterTab combines chain-of-thought reasoning with structure-aligned cropping for table reasoning, using InterTab-22K traces, SSA, and ALO. Across nine benchmarks it achieves strong overall performance, with ablations indicating the stages are complementary.

6 CONCLUSION

  • InterTab interleaves chain-of-thought with structure-aligned cropping, using InterTab-22K traces, SSA to establish the format, and ALO to optimize localization under a unified reward.Across nine benchmarks, the paper reports strong overall performance and complementary contributions from SSA and ALO.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI for writing assistance, coding support, and construction of InterTab-22K reasoning traces, with AI-assisted work reviewed and verified.

AI USE STATEMENT

  • Generative AI assisted with writing and coding, and GLM-5.3-Flash helped construct InterTab-22K reasoning traces; authors reviewed and verified the AI-assisted work.The authors state that AI was not used for research ideation, retrieving related work, or reviewing-related tasks.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work uses publicly available table-reasoning datasets and Visual Table QA only for out-of-domain evaluation. The authors caution that reasoning and grounding may be inaccurate, especially with ambiguous layouts or low-quality scans.

ETHICS STATEMENT

  • Models may produce incorrect reasoning chains or inaccurately grounded references, especially with ambiguous layouts or low-quality scans, so the authors recommend assistive rather than high-stakes autonomous use.The work uses public datasets and excludes human subjects, private data, and personally identifiable information.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents its dataset construction and training materials for reproducibility, including the statistics and preparation of InterTab-22K.

REPRODUCIBILITY STATEMENT

  • Anonymized code and data are slated for release, while prompts, filtering criteria, training objectives, reward definitions, and training settings are documented.The construction pipeline is described in Section 4.1, prompts in Appendix A.3, objectives and rewards in Section 4.2 and Appendix B.1–B.2, and settings in Section 5.1.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASET DETAILS

  • Appendix A provides InterTab-22K statistics and the prompts used in the dataset-generation pipeline.The dataset is described as a dual-aligned dataset.Source: A DATASET DETAILS · p. 14

A.1 DATASET STATISTICS

  • Dataset statistics cover 21,721 source records before upsampling; balanced upsampling yields 53,048 examples per SSA epoch.The ALO subset contains 3,200 training prompts and 80 validation prompts before prompt-length filtering; the SSA-initialized ALO run retains 3,196 training prompts after filtering.Source: §A.1 · p. 14

A.2 TRAINING HYPERPARAMETERS

The appendix records dataset and training configurations, the prompts used at inference and training, and the answer and format reward definitions.

A.2 TRAINING HYPERPARAMETERS

  • Table 4 reports SSA source-dataset statistics before upsampling, while Table 5 gives dataset composition before and after SSA upsampling.ALO samples are drawn without replacement from the SSA source pool, and its counts precede prompt-length filtering.Source: Table 4 · p. 15, Table 5 · p. 16
  • The SSA schedule is three epochs, but ALO starts from its epoch-one checkpoint; each ALO step uses 16 prompts and eight trajectories per prompt.A minibatch contains 64 trajectories, with one PPO epoch; 398 outer steps correspond to 796 optimizer updates under normal execution.Source: §A.2 · pp. 14, 16, Table 6 · p. 16

A.3 PROMPTS USED

  • The appendix includes separate inference, rollout, and LLM-as-a-judge prompts; rollout steps explain needed evidence before requesting a complementary crop.The judge checks semantic equivalence to the reference answer, and inference permits get_sub_table calls with the final answer in <answer> tags.Source: §A.3 · p. 16, Figure 8 · p. 17, Figure 7 · p. 17

B.1 ACCURACY REWARD (RACC)

  • The accuracy reward gives credit for an exact answer match or an LLM judge's binary correctness decision, with no partial-match credit.An exact match scores 1; otherwise the judge scores the response as correct or incorrect.Source: §B.1 · p. 18

B.2 FORMAT REWARD (RFMT)

  • The format reward scores required response tags: both <think> and <answer> earn 1, answer-only earns 0.5, and neither earns 0.The reward encourages explicit reasoning traces and final answers while allowing partial credit for simple questions answered with only the final answer.Source: §B.2 · p. 18

C.1 CONTRIBUTION OF TRAINING SOURCES

The analyses report benefits from a diverse training mixture and characterize ALO dynamics after base-model or SSA initialization, alongside deployment cautions.

C.1 CONTRIBUTION OF TRAINING SOURCES

  • Every source removal lowers nine-benchmark average accuracy by 3.94–15.99 points, while FeTaQA and TATQA removals cause the largest cross-task mean losses.Cross-task losses are 16.81 and 13.22 points, respectively; HybridQA removal improves FeTaQA by 5.42 points despite lowering the overall average by 5.40 points. These conditional removal effects do not establish additive contributions or isolate diversity from training exposure.Source: §C.1 · p. 19, Figure 10 · p. 19

D BROADER IMPACTS AND SOCIETAL CONCERNS

  • InterTab may improve reliability and interpretability, but errors in reasoning or grounding remain possible, especially on ambiguous layouts, low-quality scans, or adversarially designed tables.The paper recommends assistive rather than fully autonomous deployment and identifies robustness, fairness, privacy preservation, and uncertainty estimation as future work.Source: D BROADER IMPACTS AND SOCIETAL CONCERNS · pp. 19–20

E TRAINING DYNAMICS AND ROLLOUT ANALYSIS

E.1 OPTIMIZATION DYNAMICS AND STABILITY

  • SSA-initialized ALO maintains higher reward and rollout quality than Base, with shorter responses and comparable tool-call counts.Late-window mean reward is 0.832 versus 0.674; correctness is 87.61% versus 79.89%, IoU is 0.697 versus 0.492, and response length is about 2,044 versus 2,468 tokens. Policy-gradient-loss standard deviation is 0.0228 versus 0.0346, while entropy is 0.069 versus 0.039; these diagnostics do not establish greater reasoning diversity or measure end-to-end latency.Source: §E.1 · pp. 20–21, Figure 11 · p. 20, Table 7 · p. 20

E.2 REWARD COMPOSITION

  • SSA's late-window reward advantage reflects correctness, tool quality, and format contributions, rather than only tag compliance or reduced penalties.The approximately 0.158 reward gap decomposes into 0.046 correctness, 0.061 tool quality, 0.048 format, and 0.002 reduced penalties; correctness and localization together account for most of the gap.Source: Figure 12 · p. 21, §E.2 · pp. 21–22

E.3 QUALITY OF INDIVIDUAL ROLLOUTS

  • SSA retains advantages in correctness and localization after the initial phase, and its lower-tail rollout scores also improve, though difficult-example failures remain possible.Late-window correctness and IoU means are 0.876 and 0.697 for SSA versus 0.799 and 0.492 for Base; mean 10th-percentile score is 0.511 versus 0.234. Correctness bands overlap, and tool quality and IoU reflect the same localization signal in these logs.Source: Figure 13 · p. 22, §E.3 · p. 22

E.4 EVIDENCE ACQUISITION AND TOOL-CALL COUNTS

  • SSA rollouts cluster in the three-to-seven-call range and have higher correctness, tool quality, and reward there, but call-count bucket comparisons are descriptive.Three to seven calls account for 98.61% of SSA rollouts versus 81.84% for Base; the SSA ten-or-more-call bucket contains only three rollouts, so its low score is not a reliable general trend.Source: §E.4 · pp. 22–23, Figure 14 · p. 23

E.5 RELATIONSHIPS AMONG ROLLOUT METRICS

Rollout metrics distinguish evidence quality from call frequency: reward is more strongly associated with tool quality and correctness than with the number of calls.

E.5 RELATIONSHIPS AMONG ROLLOUT METRICS

  • For SSA, reward correlates with tool quality (ρ = 0.81) and correctness (ρ = 0.61), but weakly with call count (ρ = 0.10); Base shows the same qualitative pattern.Base correlations are 0.69 for tool quality, 0.73 for correctness, and 0.05 for call count; correctness and tool quality have weak rank correlations of 0.09 for Base and 0.11 for SSA.Source: Figure 15 · p. 23, §E.5 · pp. 23–24

Related papers

Loading 2609.32660v1…