PaperlayerView exact revision on arXiv

Source-linked AI summary

MERIT: Multimodal Evidence Reasoning over Image Tables

Haoyang Jiang, Zhuotong Fan, Mingyang Yi, Ju Fan, Lanqing Hu, Xiao Zan

arXiv:2610.17937v1cs.CVcs.AI

TL;DR

Table serialization can lose or corrupt evidence needed for table reasoning, motivating methods that work directly from table images. MERIT combines table-specific visual adaptation, evidence-grounded SFT, and verifiable RL; across evaluated tasks, it outperforms its backbone and table-specialized baselines, and images further improve results when paired with text.

  • Problem

    Table reasoning commonly assumes clean serialized tables, although OCR and parsing can corrupt content and structure or discard visual cues that remain in table images.

    Source: §1 · p. 1

  • Method

    MERIT adapts a visual encoder for tables, then uses verified-trajectory SFT and separate RL rewards for evidence, calculations, answers, and output format.

    Source: §1 · p. 2, §4.3 · p. 7

  • Results

    Across evaluated benchmarks, image-only MERIT outperforms its backbone and table-specialized models; adding images raises average accuracy from 83.42% with text-only input to 86.70%.

    Source: §5 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Ablations indicate encoder adaptation benefits performance after post-training, while SFT and GRPO are complementary and the approach transfers to unseen datasets.

    Source: §5 · p. 9, §6 · p. 10

    MERIT does not directly adopt MinerU 2.5 because connecting its encoder would incur comparable alignment costs, and its training emphasizes text extraction rather than visual attributes such as color.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Table reasoning with language models typically assumes access to clean textual tables. In practice, tables are often serialized into text, which can corrupt cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and reasoning grounded in the relevant cells. To address this, we propose MERIT (Multimodal Evidence Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and reasoning using verified trajectories. Finally, reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional datasets with different table structures and visual presentations.

1 INTRODUCTION

Text serialization can corrupt or discard table evidence, while table images preserve content, structure, and visual cues. MERIT combines table-specific visual adaptation with evidence-grounded supervised fine-tuning and reinforcement learning.

1 INTRODUCTION

  • OCR and serialization can misalign rows and columns, flatten headers, omit cells, and lose merged regions or visual cues, even when parsed tables remain syntactically valid.Table images preserve the original evidence that text conversion can lose.Source: §1 · p. 1, Figure 1 · p. 2
  • Table images retain content, two-dimensional structure, and visual semantics, serving as direct input when structured text is unavailable and auxiliary input when it is available.Source: §1 · p. 1
  • Generic visual encoders may struggle with dense text, multi-level headers, merged cells, and formatting, while encoder adaptation alone does not ensure correct evidence retrieval or computation.The paper identifies visual adaptation as necessary but insufficient for end-to-end Table QA.Source: §1 · p. 2
  • MERIT adapts the visual encoder with reconstruction, cell localization, and color recognition, then uses multimodal SFT and table-specific RL for evidence retrieval and reasoning.Experiments show improved perception, evidence retrieval, and answer accuracy, including when structured representations are unavailable or imperfect.Source: §1 · p. 2

2 RELATED WORK

Prior Table QA methods commonly assume accurate serialized tables, while image-based approaches and multimodal post-training extend reasoning to visual evidence. MERIT distinguishes evidence retrieval from computation through staged training and decomposed rewards.

Table Question Answering

  • Table QA methods use serialized tables for semantic parsing, pretraining, cell selection, aggregation, and executable programs, but cannot directly recover information lost or corrupted during serialization.Related extensions address fact verification, numerical reasoning, and explicit row or cell selection.Source: §2 · p. 2
  • Image-based Table QA and joint image-text studies use visual tables, while document-oriented models target general text extraction rather than table structure and visual attributes such as color.MERIT separates table-aware encoder adaptation from QA post-training to use visual evidence without assuming reliable structured input.Source: §2 · pp. 2–3

Post-Training for Multimodal Table Reasoning

  • Unlike prior post-training approaches, MERIT separates correct evidence retrieval from correct computation through explicit cell retrieval and calculation in SFT and decomposed RL rewards.The rewards cover evidence correctness, calculation execution, final-answer correctness, and output format, targeting retrieval and computational errors.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe VLM inputs and outputs, supervised fine-tuning, and group-relative policy optimization as foundations for multimodal table reasoning. In MERIT, SFT trains grounded responses and formatting before RL refines those components.

Vision-Language Modeling

  • A VLM combines a visual encoder and autoregressive language model to generate responses from an image and text context; MERIT uses a table image with a question and optional serialized table.When structured input is unavailable, the text context contains only the question.Source: §3 · p. 3

Supervised Fine-Tuning

  • SFT trains instruction-following and task-specific responses; in MERIT it teaches table-evidence retrieval, calculations, and parseable answers for later RL refinement.The SFT objective is token-level negative log-likelihood over reference response tokens.Source: §3 · p. 3

Group Relative Policy Optimization

  • GRPO estimates advantages from relative rewards among multiple responses sampled for the same input, avoiding a separate value model.Its objective uses clipping and a KL penalty relative to a reference policy.Source: §3 · p. 3

4 METHOD

MERIT links table-specific visual adaptation to grounded SFT and component-wise table-verifiable RL, using images as primary input and serialized text optionally.

4.1 TABLE-CENTRIC VISUAL ENCODER ADAPTATION

  • On FinQA, MinerU 2.5 raises image-based accuracy from 57.8% to 72.2%; with gold facts, scores are 79.8% versus 79.6%.The comparison uses the same Qwen3 decoder, and the gold-fact results motivate a visual-perception bottleneck diagnosis.Source: §4.1 · p. 4, Figure 3 · p. 5
  • RECON targets dense-content recognition, LOCATE maps cells to logical grid positions, and COLOR trains grounding in independently assigned cell colors.These tasks address errors in reading cell contents, aligning values with row–column positions, and detecting relevant visual attributes.Source: §4.1 · p. 5

4.2 MULTIMODAL SUPERVISED FINE-TUNING

  • Multimodal SFT trains on verified trajectories that identify evidence cells, include calculations when needed, and serialize evidence, reasoning, and answers in a parseable format.The adapted encoder is frozen during SFT, which initializes table-solving behavior and an output protocol for subsequent RL.Source: §4.2 · p. 6

4.3 TABLE-VERIFIABLE REINFORCEMENT LEARNING

  • Table-verifiable RL combines separate rewards for evidence retrieval, calculation, final-answer correctness, and parseable output; the calculation reward is disabled for non-arithmetic questions.The SFT checkpoint initializes the policy, while the adapted visual encoder remains frozen during RL.Source: §4.3 · pp. 6–7

4.1 TABLE-CENTRIC VISUAL ENCODER ADAPTATION

  • The adapted encoder raises the six-task macro-average from 46.89 to 53.51.The average covers six held-in table-structure benchmarks.Source: §4.1 · p. 5, Table 1 · p. 6

5 EXPERIMENTS

Experiments compare MERIT with text-based and table-specialized methods on held-out tasks, test transfer to five unseen datasets, and ablate its training stages. Results show strong image-based performance, complementary input and training benefits, and out-of-distribution gains.

Evaluation setup

  • Evaluation covers table QA and fact verification on five post-training datasets, plus five additional datasets for generalization without further fine-tuning.The comparisons include proprietary, general-purpose, table-specialized, and text-based models; OOD datasets vary in table structure and visual presentation.Source: §5 · pp. 7–8

Results

  • On held-out splits of five datasets, with image-only input, MERIT reaches 85.45% average accuracy, 9.40 points above its Qwen3-VL-8B backbone, and outperforms TURBO by 4.36 points.TURBO is the strongest table-specialized baseline in this comparison.Source: Table 2 · p. 8, §5 · p. 8
  • 86.70% average accuracy with image-plus-text input exceeds 83.42% for text-only input using the same checkpoint and is the best among open models.The image-only model also surpasses Claude Sonnet 5 and approaches GPT-5.5 and Gemini 3.5 Flash.Source: Table 2 · p. 8, §5 · p. 9

OOD generalization

  • 54.90% average accuracy across five unseen datasets is 2.63 points above initialization; MERIT exceeds text-only RL by 12.50 points.The evaluation uses no further fine-tuning; the largest gain is +15.65 on FinMMR, while HIPPO averages 25.48%.Source: Table 3 · p. 9, §5 · p. 9

Ablation study

  • The full pipeline beats SFT + GRPO without encoder adaptation by 0.95 points, while encoder adaptation alone leaves accuracy nearly unchanged at 76.07% versus 76.05%.The full pipeline also exceeds encoder adaptation followed by GRPO alone by 9.47 points and SFT alone by 2.11 points.Source: Table 4 · p. 9, §5 · p. 9

6 CONCLUSION

MERIT uses table images as primary evidence, with serialized text optional, and combines table-aware visual adaptation with evidence-grounded SFT and RL. The appendix describes the visual-adaptation, post-training, and evaluation datasets used in the study.

6 CONCLUSION

  • MERIT treats table images as primary evidence and serialized text as optional input; its stages adapt visual perception, then train evidence-grounded reasoning with SFT and RL.The conclusion reports that image-only MERIT outperforms its backbone, table-specialized models, and a text-only RL baseline, transfers to unseen datasets, and improves further with serialized text.Source: §6 · p. 10

A.1 DATASET DETAILS

  • Stage 1 adapts the visual encoder on a structured-table corpus using reconstruction, cell localization, and color-recognition objectives.The corpus construction and statistics are provided in Appendix A.2.Source: §A.1 · p. 14
  • SFT and GRPO use examples from TABMWP, WikiTableQuestions, TAT-QA, TabFact, and InfoTabs, spanning mathematical, compositional, financial, and fact-verification tasks.TabFact uses binary labels and InfoTabs uses three-way labels.Source: §A.1 · p. 14
  • Evaluation adds HiTab, FinQA, FinMMR, TableVista, and MMMU-T to assess generalization to unseen table-reasoning tasks and domains.Source: §A.1 · p. 14

A.2 VISUAL-ADAPTATION DATA DETAILS

The appendix details Stage 1 visual-adaptation tasks, verification of SFT trajectories, and the prompts used to generate and check grounded reasoning.

A.2 VISUAL-ADAPTATION DATA DETAILS

  • Visual adaptation uses MMTab for table reconstruction, position-based cell localization, and color-conditioned questions, alongside auxiliary table QA.Random cell background colors are sampled independently of cell values, so color questions cannot be answered from textual content alone.Source: §A.2 · p. 14

A.3 SFT DATA COMPOSITION

  • SFT trajectories are retained only after three verification checks; rejected examples include incorrect cell references, invalid calculations, or answers inconsistent with targets.Duplicates and examples overlapping evaluation splits are also removed.Source: §A.3 · p. 15

A.4 GENERATION AND VERIFICATION PROMPTS

  • Generated trajectories cite relevant cells by logical coordinates and normalized values, show intermediate calculations, and end with an answer; the verifier checks evidence, derivation, and answer correctness.The generation prompt requires explicit visual conditions and arithmetic steps, while the verifier retains only trajectories that pass all three checks.Source: §A.4 · pp. 15–16

A.5 RL DATA COMPOSITION

The appendix specifies GRPO prompt sampling, reward parsing, and the base model and hyperparameter reference for the three training stages.

A.5 RL DATA COMPOSITION

  • GRPO prompts are drawn from five datasets, and each input question yields G = 8 candidate responses from the current policy.Source: §A.5 · p. 16

A.6 REWARD AND PARSING DETAILS

  • Reward weights are λans = 1 and λevi = λcalc = λfmt = 0.5, with answer, cell evidence, and calculations checked against normalized or verified targets.Evidence matching uses both logical coordinate and cell value; calculation operands must come from cited cells or earlier intermediate results.Source: §A.6 · p. 16

A.7 TRAINING HYPERPARAMETERS

  • All experiments use Qwen3-VL-8B-Instruct, with the principal hyperparameters for all three training stages summarized in Table 9.Source: §A.7 · p. 16

B.1 EVALUATION PROTOCOL AND SCORING

The appendix specifies the evaluation splits, input settings, scoring, and comparison models for table perception and reasoning. It also distinguishes the visual, table-specialized, proprietary, and text-only baselines.

B.1 EVALUATION PROTOCOL AND SCORING

  • Five post-training datasets define the IID setting; HiTab, FinQA, FinMMR, TableVista, and MMMU-T are OOD-only, with no OOD fine-tuning.For FinMMR, TableVista, and MMMU-T, table images are converted to Markdown for text-based models; visual models receive both image and text.Source: §B.1 · pp. 16–17
  • Table QA reports accuracy and an unweighted per-dataset average, while fact verification reports accuracy for binary or three-way labels.Ambiguous cases are reviewed by GPT-5.5; Table 2 combines scores from published reports under original protocols with scores produced by the paper's evaluation pipeline.Source: §B.1 · p. 17

B.2 TABLE-STRUCTURE UNDERSTANDING EVALUATION

  • The six held-in MMTab tasks assess table size, cell extraction and locating, merged cells, row and column extraction, and table recognition.Metrics include accuracy, F1, and TEDS; Avg. macro-averages the six task scores after averaging within-task submetrics.Source: §B.2 · p. 17

B.3 GENERAL-PURPOSE VISION-LANGUAGE MODELS

  • InternVL2.5-8B and Qwen3-VL-8B are evaluated without table-specific training, and Qwen3-VL-8B-Instruct is MERIT's initialization reference.In IID comparisons, they receive table images, questions or statements, and accompanying passages when available, but no serialized table contents.Source: §B.3 · pp. 17–18

B.4 TABLE-SPECIALIZED MODELS

  • Table-specialized comparisons include Table-LLaVA, TabPedia, HIPPO, and TURBO, with OOD checkpoint evaluation excluding unreleased TURBO.HIPPO's reported scores use both images and structured tables, whereas TURBO receives images at inference; Table-LLaVA 7B and 13B are also included for perception evaluation.Source: §B.4 · p. 18

B.5 PROPRIETARY MODELS

  • GPT-5.5, Claude Sonnet 5, and Gemini 3.5 Flash are evaluated through their APIs using table images and original questions at temperature zero.Their outputs use the same scoring protocol as the other models.Source: §B.5 · p. 18

B.6 TEXT-BASED MODELS

  • Text-only Qwen3-8B and Qwen3-8B + RL use serialized tables, while MERIT's fixed checkpoint supports comparisons across text-only, image-only, and image-plus-text inputs.Qwen3-8B + RL receives the same SFT and GRPO settings as MERIT, with serialized tables replacing images.Source: §B.6 · p. 18

C.1 ERROR ANALYSIS

The error analysis finds that MERIT reduces errors across all reported categories, while perception remains a substantial source of mistakes. A representative case illustrates how missed row content can derail a calculation.

C.1 ERROR ANALYSIS

  • The analysis classifies errors on 1,495 stratified IID examples into recognition, locating, color, reasoning, and comprehension categories.A separate color probe covers 486 questions; format-only mismatches and other specified non-errors are excluded.Source: §C.1 · p. 18
  • MERIT reduces recognition errors 57→42, locating errors 27→18, color errors 83→68, and reasoning errors 199→133, with 290 total errors versus Qwen3-VL-32B's 328.Perception errors account for about two fifths of Qwen3-VL-8B's errors (167 of 405).Source: §C.1 · p. 18
  • In a representative failure, the unadapted encoder misses the Discovery Channel row; correct grounding finds both cells and computes 246 − 183 = 63.Color errors also remain a major source of mistakes, mostly from counting colored cells.Source: §C.1 · p. 19

C.2 VISUAL ENCODER OBJECTIVE ABLATION

The encoder-objective ablation compares reconstruction, localization, and color supervision before SFT and GRPO. The full objective yields the highest aggregate perception score, with localization and reconstruction contributing the largest reported gains.

C.2 VISUAL ENCODER OBJECTIVE ABLATION

  • RECON and LOCATE alone score 48.82 and 47.71, respectively; COLOR alone scores 40.25, while RECON plus LOCATE reaches 52.73.The ablation evaluates each objective and combinations before reasoning post-training.Source: §C.2 · p. 19
  • Removing LOCATE drops the full objective's average by 11.56 points, removing RECON by 7.04, and removing COLOR by 0.78.The reported results identify reconstruction and localization as the main contributors to the aggregate perception score, with a smaller incremental gain from COLOR.Source: §C.2 · p. 19
  • 53.51 is the full objective's average score, versus 46.89 for the base model.The score macro-averages six held-in benchmarks after averaging their submetrics.Source: Table 10 · p. 19, §C.2 · p. 19

C.3 REWARD COMPONENT ABLATION

The reward ablation finds modest average accuracy gains from intermediate feedback, with calculation feedback producing the largest individual gain. Additional cases show structural errors under text-only input and illustrate how grounded trajectories connect cells, calculations, and answers.

C.3 REWARD COMPONENT ABLATION

  • 85.45% is the Full reward variant's average accuracy, 0.44 points above answer-only supervision and 0.06 above the calculation-reward variant.The calculation-reward variant yields 85.39%, compared with 85.11% for evidence reward and 85.01% for answer-only supervision; the largest individual gain comes from calculation feedback.Source: §C.3 · pp. 19–20, Table 11 · p. 21

C.4 ADDITIONAL IMAGE–TEXT COMPARISON CASES

C.5 MORE GROUNDED REASONING CASES

  • Figures 8 and 9 show grounded trajectories linking logical cell coordinates to intermediate calculations and final answers.The format makes evidence selection, computation, and the answer individually checkable against the source table.Source: §C.5 · p. 21, Figure 8 · p. 22, Figure 9 · p. 23

Related papers

Loading 2610.17937v1…