PaperlayerView exact revision on arXiv

Source-linked AI summary

TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition

Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He

arXiv:2512.01248v2cs.CV

TL;DR

Table recognition needs large labeled datasets, which are costly, while open-source models remain behind proprietary systems and practical use of those systems can raise privacy concerns. TRivia fine-tunes VLMs on unlabeled table images using GRPO and QA-based rewards. Its TRivia-3B model establishes state-of-the-art performance across multiple benchmarks and can generate pseudo-labels that rival human and proprietary annotations.

  • Problem

    Large-scale labeled data for table recognition is costly, and open-source models remain constrained by limited datasets while proprietary APIs raise privacy and compliance concerns.

    Source: §1 · pp. 1–2

  • Method

    TRivia uses GRPO to fine-tune VLMs on unlabeled table images, selecting informative samples and using attention-guided table QA as proxy rewards.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    TRivia-3B establishes a new state of the art among TR models, outperforming specialized and proprietary models across multiple benchmarks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    TRivia also serves as a scalable annotation engine, generating pseudo-labels that rival human and proprietary annotations.

    Source: §6 · p. 8

Abstract

from arXiv · show

Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or Markdown. As a core component of document parsing, TR has long relied on supervised learning, with recent efforts dominated by fine-tuning vision-language models (VLMs) using labeled data. While VLMs have brought TR to the next level, pushing performance further demands large-scale labeled data that is costly to obtain. Consequently, although proprietary models have continuously pushed the performance boundary, open-source models, often trained with limited resources and, in practice, the only viable option for many due to privacy regulations, still lag far behind. To bridge this gap, we introduce TRivia, a self-supervised fine-tuning method that enables pretrained VLMs to learn TR directly from unlabeled table images in the wild. Built upon Group Relative Policy Optimization, TRivia automatically identifies unlabeled samples that most effectively facilitate learning and eliminates the need for human annotations through a question-answering-based reward mechanism. An attention-guided module generates diverse questions for each table image, and the ability to interpret the recognition results and answer them correctly provides feedback to optimize the TR model. This closed-loop process allows the TR model to autonomously learn to recognize, structure, and reason over tables without labeled data. Leveraging this pipeline, we present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems (e.g., Gemini 2.5 Pro, MinerU2.5) on three popular benchmarks. Model and code are released at: https://github.com/HKU-TASR/TRivia

1. Introduction

Table recognition requires both accurate text extraction and reconstruction of spatial and logical organization, but labeled data are costly and open-source systems remain constrained. TRivia uses unlabeled images to fine-tune VLMs, and TRivia-3B surpasses expert and general-purpose models across benchmarks.

1. Introduction

  • Table recognition must recover both table text and spatial and logical organization, making it more demanding than standard OCR.Outputs include semi-structured formats such as HTML or Markdown.Source: §1 · p. 1
  • Open-source TR models lag proprietary systems because existing labeled datasets are limited, while proprietary APIs raise privacy and compliance concerns.MinerU2.5 also inherits a performance ceiling from its proprietary teacher, Gemini 2.5 Pro.Source: §1 · p. 2
  • Synthetic tables lack real-world diversity, real-world annotation is costly, and proprietary-model pseudo-labeling remains costly and teacher-limited.The authors identify large-scale labeled-data curation as a major bottleneck and ask whether unlabeled images can improve TR.Source: §1 · p. 2
  • TRivia uses GRPO to select informative unlabeled images and generates diverse, verifiable table questions as proxy supervision for fine-tuning.The model’s recognition results are assessed by whether a language model can answer the generated questions correctly.Source: §1 · p. 2
  • The framework combines response-consistency sampling with attention-guided QA, while its reported contributions include TRivia-3B as an open-source model.The paper also describes an attention-guided mechanism for diverse, verifiable, and stable supervisory signals.Source: §1 · p. 2
  • TRivia-3B surpasses expert and general-purpose models, including MinerU2.5 and Gemini 2.5 Pro, establishing a new state of the art across multiple benchmarks.The model is fine-tuned from Qwen2.5-VL-3B using unlabeled table images.Source: §1 · p. 2

2. Related Work

Earlier table-recognition systems used modular structure and OCR pipelines or image-to-markup generation, while VLMs brought instruction-based recognition. TRivia departs from labeled or distilled data by using QA rewards and GRPO on unlabeled real-world tables.

2. Related Work

  • Modular structure-recognition and OCR pipelines can propagate errors, while image-to-markup methods face limits from context windows, resolution, and training-data quality.UniTable, for example, processes images up to 448 × 448 with a 512-token output cap.Source: §2 · p. 2
  • General-purpose VLMs can perform TR from natural-language instructions, while expert VLMs are typically fine-tuned on synthetic data with some manually or automatically labeled real tables.MinerU2.5 remains constrained by the performance ceiling of its Gemini 2.5 Pro teacher.Source: §2 · p. 2
  • TRivia departs from reliance on labeled or distilled data by using table QA-based rewards with GRPO to learn from unlabeled real-world images.Its pipeline prepares supervisory signals from unlabeled images before reinforcement-learning fine-tuning.Source: §2 · p. 3, Figure 2 · p. 3

3. Methodology

TRivia fine-tunes a VLM with GRPO using table QA as proxy rewards, and an adaptive data engine selects informative images and constructs diverse, verifiable questions. This design couples reward-based optimization to sample uncertainty and visual coverage.

3.1. Table QA-driven Self-supervised Fine-tuning

  • Table QA provides annotation-free reward signals because correct answers reflect whether recognition preserves textual and structural information.QA pairs are easier to generate and cross-check than full HTML markup, which requires explicit complex structure prediction.Source: §3.1 · p. 3
  • Each recognition response receives the average F1 score of LLM answers to the image’s QA set as its GRPO reward.The TR model produces a group of recognition responses for each image, and each response is paired with questions answered from the recognized table.Source: §3.1 · p. 4
  • Illegal recognition outputs receive zero reward and can destabilize relative advantages, so TRivia discards them before GRPO advantage computation.The filtering strategy is reported to stabilize QA-based rewards and make training smoother and more reliable.Source: §3.1 · p. 4

3.2.1. Response-Consistency Sampling

  • Response-consistency sampling uses pairwise TEDS among multiple outputs; lower consistency indicates greater response diversity and uncertainty, making an image more valuable for GRPO.Sampling is conducted offline because repeated online evaluation is computationally prohibitive, and the authors report offline sampling is stable and effective.Source: §3.2.1 · p. 4

3.2.2. Attention-based Diverse QA Generation

  • Single-pass question generation may cover only part of a table, while repeated sampling can produce overlapping QA pairs that limit reward diversity.The attention-guided approach uses attention distributions to diversify question sources and produce concise, comprehensive QA pairs.Source: §3.2.2 · p. 4, Figure 3 · p. 4
  • TRivia defines a QA pair’s visual source using answer-token-averaged attention and greedily selects valid questions with minimally overlapping visual sources.An external VLM checks that each question is correctly answerable with the image but not without it; an IOU threshold relaxes disjointness to account for attention sinks.Source: §3.2.2 · p. 5

4. TRivia-3B

TRivia-3B is built by adapting Qwen2.5-VL-3B through OTSL warm-up, real-world supervised fine-tuning, and then self-supervised GRPO training. The paper reports high TEDS and S-TEDS performance across four benchmarks against 12 existing methods.

4. TRivia-3B

  • TRivia-3B’s three stages are OTSL warm-up, supervised fine-tuning on real-world tables, and final TRivia optimization with GRPO and QA-based rewards.The model is fine-tuned from Qwen2.5-VL-3B-Instruct.Source: §4 · p. 5
  • The pipeline uses about 700K OTSL warm-up instances, about 50K real-world supervised samples, and about 50K selected unlabeled images for RL training.The unlabeled pool begins with approximately 100K cropped tables; roughly 30 diverse QA pairs are retained per selected image.Source: §4 · p. 5
  • Table 1 reports TEDS and S-TEDS performance across four benchmarks for TRivia-3B and 12 existing methods spanning expert TR, general-purpose VLM, and document-parsing systems.The caption characterizes TRivia-3B’s performance as consistently high across the four benchmarks.Source: Table 1 · p. 6

5. Experimental Evaluation

TRivia-3B is evaluated against expert, general-purpose, and document-parsing models, followed by analyses of its performance enablers and use as a data annotator. It outperforms leading systems across diverse benchmarks, while ablations examine QA supervision, sample selection, and filtering.

5. Experimental Evaluation

  • Evaluation covers OmniDocBench v1.5, CC-OCR, and OCRBench v2, with comparisons to expert TR, general-purpose VLM, and document-parsing models.The evaluated sets contain 512, 300, and 700 table images, respectively.Source: §5 · p. 6

5.1. Advancing TR Performance

  • TRivia-3B outperforms MinerU2.5 across all reported cases and beats Gemini 2.5 Pro on most benchmarks, including by 11.47 TEDS over PaddleOCR-VL on OCRBench.It also exceeds UniTable by 27.06 TEDS on CC-OCR and 23.03 TEDS on OCRBench v2; its exception against expert systems is PubTabNet S-TEDS.Source: §5.1 · pp. 6–7

5.2. Ablation Studies

  • Qwen2.5-VL-72B pseudo-labels degrade performance: SFT loses 8.37 TEDS on average, while GRPO still trails Stage-1 by 4.92 TEDS.TRivia’s QA-based supervision avoids the bias or semantic inconsistency attributed to those annotations; Figure 4 shows the teacher’s direct recognition errors.Source: §5.2 · p. 7, Figure 4 · p. 7, Table 2 · p. 7
  • Removing attention-guided QA generation significantly reduces TR performance, especially on structurally complex or visually ambiguous tables.Figure 5 also reports response-consistency sampling raising TEDS from 52.0 to 63.5 versus random sampling.Source: §5.2 · p. 7, Figure 5 · p. 8
  • Illegal-sample filtering stabilizes GRPO by removing invalid responses before advantage computation, reducing convergence steps by about 25% and improving test performance by 3 TEDS.Without filtering, reward noise leads to learning instability and fluctuating QA rewards.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3. TRivia as a Data Annotator

  • SFT on pseudo-labels generated by TRivia-3B yields performance nearly identical to TRivia-3B across benchmarks and slightly exceeds it on CC-OCR.Unlike Qwen2.5-VL-72B distillation, this supports TRivia-3B as a source of reliable pseudo-labels for unlabeled tables.Source: §5.3 · p. 8, Table 3 · p. 8

6. Conclusions

TRivia learns table recognition from unlabeled images, and TRivia-3B achieves state-of-the-art performance across multiple benchmarks. The framework also generates pseudo-labels that rival human and proprietary annotations.

6. Conclusions

  • TRivia-3B establishes a new state-of-the-art among TR models, outperforming specialized and proprietary models across multiple benchmarks.TRivia also serves as a scalable annotation engine whose pseudo-labels rival human and proprietary annotations.Source: §6 · p. 8

A. More Training Details

The appendix reports the three-stage training setup, backbone, image-token range, and compute requirements.

A. More Training Details

  • Qwen2.5-VL-3B is the backbone, and image-token counts range from 256 to 1280 across all stages.These correspond to image resolutions from 256 ∗28 ∗28 to 1280 ∗28 ∗28.Source: A. More Training Details · p. 12
  • Stage 3 uses GRPO with sampling temperature 1.2, G = 16 samples per step, and a constant learning-rate scheduler.All experiments use 8 A100 80GB GPUs.Source: A. More Training Details · p. 12
  • Stage 1 takes about one day, Stage 2 about two hours, and Stage 3 about two days.Table 4 summarizes training configurations and hyperparameters for the three stages.Source: A. More Training Details · p. 12, Table 4 · p. 12

B. Dataset Construction

Dataset construction filters training data and selects unlabeled images for response-consistency sampling before generating and validating QA pairs.

B. Dataset Construction

  • Stage 1 and 2 dataset construction removes samples with incomplete HTML tags; when only structure tags exist, Qwen2.5-VL-72B supplies OCR text for cell boxes.The cited training data for structure-only tags are [21] [49].Source: B. Dataset Construction · p. 12
  • Response-consistency sampling reduces 100K unlabeled images to 50K, selecting uniformly across consistency-score intervals from 0.4 to 1.0 and using distinct PDFs.Scores use pairwise TEDS among eight outputs per image.Source: B. Dataset Construction · p. 12
  • The final dataset contains 48,470 images averaging 28.3 QAs each, after QA validation, visual-source overlap filtering, and removal of images with fewer than three valid QAs.Sixteen QAs are generated per image before filtering; retained pairs pass image-conditioned F1 thresholds.Source: B. Dataset Construction · p. 12

C. More Details about Experiments

The experiments test TRivia across multi-task OCR, complex table layouts, QA noise, and Stage-2 data sizes. They also describe the experimental setup used for these evaluations.

C. More Details about Experiments

  • Experiments use eight A100 80GB GPUs; general-purpose VLM prompts are optimized, and inference uses a sampling temperature of 0.2 by default.Gemini 2.5 Pro uses thinking mode; Qwen models use 256–1280 image tokens, and PaddleOCR-VL has unwrapping and orientation classification disabled.Source: C. More Details about Experiments · p. 12

D.1. Extension to Multi-Task OCR Learning

  • Applying TRivia to table recognition improves TR while maintaining key information extraction accuracy; adding QA rewards for both tasks substantially improves KIE while preserving TR performance.The study uses a small-scale mixed training setup for TR and KIE.Source: Table 6 · p. 12, §D.1 · p. 13

D.2. Performance on Complex Table Types

  • +21.76 TEDS over the base model on rotated tables; TRivia improves across all four complex-table settings, with large gains also on large and borderless merged-cell tables.The evaluations cover formulas, rotated tables, large tables, and borderless tables with merged cells.Source: Table 5 · p. 12, §D.2 · p. 13

D.3. Robustness to QA Noise

  • Even with 20% incorrect QA pairs, TR degradation remains limited; incorrect pairs can contribute shared low scores that GRPO's within-group normalization largely removes.Noisy QA pairs mainly reduce the informative reward components available for estimating response quality.Source: §D.3 · p. 13, Figure 7 · p. 13

D.4. Effect of Stage-2 Data Size

  • Strong performance is achievable with relatively little Stage-2 data; Stage 2 mainly initializes the policy, while Stage-3 TRivia training contributes most of the improvement.The data-size analysis is shown in Figure 8.Source: §D.4 · p. 13, Figure 8 · p. 13

D.5. Results on Standard Large-Scale Benchmarks

TRivia-3B retains an advantage on complex real-world tables, while final supervised training with benchmark-style samples can restore standard-benchmark performance. The appendix also documents the prompts used for QA generation, cross-checking, and table recognition.

D.5. Results on Standard Large-Scale Benchmarks

  • TRivia-3B remains substantially stronger on complex real-world tables, while reintroducing PubTabNet and FinTabNet samples in the final supervised stage restores strong in-domain performance without losing the complex-table advantage.The comparison covers PubTabNet, FinTabNet, and complex tables from OmniDocBench, CC-OCR, and OCRBench.Source: Table 7 · p. 13, §D.5 · p. 14

E. Prompt Template

  • QA-generation prompts avoid direct row or column references and favor simple questions to limit dependence on the MQA model's structural parsing and reasoning abilities.Table 8 contains the QA-generation prompt.Source: E. Prompt Template · p. 15, Table 8 · p. 15
  • The prompts are bilingual for Chinese and English QA, include a VLM QA cross-checking prompt, and provide an optimized general-purpose VLM table-recognition prompt.The prompt tables cover LLM QA, VLM QA cross-checking, and table recognition.Source: E. Prompt Template · p. 15, Table 9 · p. 16, Table 10 · p. 16

Related papers

Loading 2512.01248v2…