Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Changjiang Jiang; Qiannian Zhao; Lei Xin; Jinxiang Xie; Preslav Nakov; Zhuohan Xie

Source-linked AI summary

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Changjiang Jiang, Qiannian Zhao, Lei Xin, Jinxiang Xie, Preslav Nakov, Zhuohan Xie

arXiv:2608.22429v1Aug 23, 2026cs.AI

TL;DR

MLLMs struggle with fine-grained reasoning over text-dense, spatially structured charts and visual tables, while external visual tools add inference latency. TwSG distills region-based tool use into full-image representations and combines cold-start SFT with TL-GRPO-driven reinforcement fine-tuning. Experiments report improved accuracy and robustness with reduced inference latency, although evaluation is limited to models of 8B parameters or fewer and depends on raw cold-start data quality.

Problem
MLLMs struggle with the Spatial-Structural Gap in text-dense charts and visual tables, while external tools introduce inference latency and standard models lack fine-grained visual reasoning.
Method
TwSG distills teacher-guided region perception, multi-turn reasoning, and tool-use capabilities into a one-pass model using cold-start SFT and TL-GRPO process-reward reinforcement fine-tuning.
Results
TwSG improves reasoning accuracy and robustness while reducing inference latency; TwSG-8B achieves 73.70% average accuracy and leads on 10 of 11 evaluated subsets.
Takeaways & Limitations
TwSG provides native fine-grained region description and flexible reasoning for chart and visual-tabular understanding without relying on external inference-time tools.
Takeaways & Limitations
The framework was trained and evaluated only on models with 8B parameters or fewer and depends on the accuracy of raw cold-start data.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.

1 Introduction

Charts and visual tables create a Spatial-Structural Gap because dense text is tied to strict spatial relationships, causing coordinate-grounding errors. TwSG internalizes tool capabilities through distillation and process-reward reinforcement learning, while TL-GRPO targets length-induced gradient bias.

Charts and visual-tabular data are text-dense and structurally relational, with textual elements bound by strict relative spatial arrangements.
Spatial grounding failures can cause logical collapse and spatial hallucinations, such as assigning a value to the wrong row or bar.
TwSG shifts interleaved visual reasoning from external inference-time tools into parameter internalization during training through data distillation.
The framework combines cold-start supervised fine-tuning with process reward-driven reinforcement learning to improve reasoning accuracy, robustness, and inference latency.
TL-GRPO uses Tag-level Importance Sampling, Clipped Group Sampling, and Interleaved Verifiable Reward to mitigate length-induced gradients and stabilize complex reasoning optimization.

2 Related Work

CvTR research has progressed from single-modality table understanding and basic chart tasks toward complex visual marks and multi-step arithmetic. Existing approaches often remain isolated by format or insufficiently optimized for the fine-grained, graph-like structure of charts and tables, motivating TwSG’s unified perspective.

CvTR encompasses reasoning over data-intensive visual structures, evolving from table understanding and text-based TableQA toward complex charts and arithmetic.
General-purpose and domain-specific MLLMs have achieved strong chart results through zero-shot capability or specialized fine-tuning, while tool-augmented methods add image operations to Chain-of-Thought reasoning.
Chart and visual-tabular reasoning are often treated as isolated tasks, and chart-specific optimization can degrade visual-tabular performance.
Thinking-with-images methods use cropping or enhancement tools to improve local visual grounding, but their application to CvTR remains underexplored.
CvTR requires precise extraction from small visual elements and high-level synthesis, yet existing frameworks are not optimized for charts’ and tables’ structured, graph-like nature.

3 Methodology

TwSG internalizes fine-grained, tool-assisted visual reasoning into one-pass MLLM inference through structured region-based distillation. Its TL-GRPO training combines tag-aware sampling, clipped group sampling, and multidimensional process rewards to stabilize grounded reasoning.

TwSG framework: TwSG distills teacher-generated region perception and multi-turn reasoning into a student model designed for one-pass reasoning without external tools.The pipeline proposes regions, crops high-resolution sub-images, generates structured reasoning trajectories, and refines them against hallucinations.
Structured trajectory: Structured trajectories interleave think, observation, and answer segments so each reasoning step is grounded in explicit visual evidence.The trajectory is represented as τ = {t1, o1, t2, o2, . . . , a}, with reasoning, intermediate evidence, and a final answer.
Tag-level Importance Sampling: Tag-level Importance Sampling normalizes importance weights across structured segments, reducing bias from unequal lengths of perceptual and reasoning tokens.The method computes ratios within tag segments and aggregates them across valid segments; this addresses sequence-length variance across functional tags.
Process rewards: TL-GRPO uses format, answer, and region-grounding rewards, with an expert MLLM verifying intermediate observations against the image.The verification reward is normalized to [0, 1], while the process signal penalizes ungrounded observations and suppresses hallucination propagation.
Clipped Group Sampling: Clipped Group Sampling trims the highest and lowest advantages within each sampled group, retaining central trajectories for more stable policy updates.For n trajectories, typically k = 1 trajectories at each extreme are removed, leaving n − 2k trajectories.

4 Experiment

TwSG is evaluated across chart and visual-tabular reasoning benchmarks, where it shows strong cross-task accuracy, generalization, throughput, and ablation-supported gains from TL-GRPO and visual verification.

Main Results: TwSG-8B achieves 73.70% average accuracy, outperforming Chart-R1 by 68.12% and leading on 10 of 11 evaluated subsets.The evaluation covers ChartQA, ChartQAPro, and TableVQA-Bench at comparable model scales.
Main Results: TwSG maintains balanced performance across challenging chart and visual-tabular tasks, with TwSG-4B reaching 68.16% versus 68.12% for 7B Chart-R1.The comparison highlights transfer across both modalities rather than performance on standard chart benchmarks alone.
Main Results: Chart-specific specialization can improve ChartQA-H from 74.24% to 80.08% while reducing FinTabNetQA from 77.76% to 70.16%, motivating unified treatment.The result illustrates a cross-format performance trade-off in Chart-RVR-Hard relative to Qwen2.5-VL-3B.
Main Results: TwSG achieves approximately 1.75–2.00 samples per second, exceeding Qwen3-VL throughput while avoiding external inference-time tool calls.The reported throughput accompanies improved CvTR performance, indicating a favorable accuracy–efficiency trade-off.
Ablation Study: TL-GRPO improves over GRPO by +6.20% on ChartQAPro and +8.99% on TableVQA-Bench, reaching 55.60% and 86.79%, respectively.The gains are attributed to combining tag-level importance sampling and clipped group sampling for disproportionately long reasoning trajectories.
Ablation Study: Visual verification raises performance to 55.60% and 86.79%, while removing ⟨think⟩ or ⟨observation⟩ causes the largest task-specific drops.The reported drops are -7.37% on ChartQAPro without ⟨think⟩ and -6.84% on TableVQA-Bench without ⟨observation⟩.

5 Conclusion

TwSG targets structured visual reasoning in charts and tables by combining structured distillation with TL-GRPO and two-stage training. The paper reports improved fine-grained reasoning and efficiency, while identifying model-size, cold-start-data, and arithmetic-reasoning boundaries.

Conclusion: TwSG combines structured data distillation, TL-GRPO, and cold-start SFT with reinforcement learning for complex CvTR tasks.The framework is presented as a tool-free pathway for structured visual reasoning.
Conclusion: The method enhances fine-grained OCR reasoning and perceptual capabilities while improving effectiveness and inference efficiency.The conclusion attributes these outcomes to ablation studies and inference-latency evaluations.
Limitations: Training and evaluation were limited to models with 8B parameters or fewer, and the method depends on the accuracy of raw cold-start data.The initial context provided to the large model is therefore not guaranteed to be entirely error-free.
Limitations: Pure CoT prompting remains insufficient for complex arithmetic calculations in chart understanding.The paper identifies programmatic visual reasoning as a direction for future work.
Broader Impact: The work is positioned for data analysis, document understanding, and accessibility-oriented applications rather than weapons, surveillance, biometric identification, or high-stakes decision-making.The authors do not anticipate direct safety risks such as physical harm or increased weapon lethality.

B Experimental Setup

TwSG training uses cold-start supervised fine-tuning followed by TL-GRPO reinforcement learning to refine reasoning trajectories.

Training Pipeline: Cold-start SFT initializes model reasoning with a learning rate of 1e-5 and a global batch size of 32.This phase precedes reinforcement fine-tuning.
Training Pipeline: TL-GRPO follows SFT and emphasizes final outcome correctness rather than dense chain-of-thought supervision.The supplied passage also specifies reward coefficient β = 0.0 and a dual-clip epsilon strategy.

C Compare with close-source MLLMs

TwSG-8B performs strongly on chart reasoning benchmarks, including against larger and proprietary models, without explicit test-time reasoning or tool use.

CharXiv-R: 67.8% on CharXiv-R is achieved by TwSG-8B, exceeding GPT-4.1 at 56.7% and GPT-4.5 at 55.4%.The result also exceeds Qwen3-VL-235B-A22B-Thinking at 66.1% despite TwSG-8B’s smaller parameter size.
CharXiv-R: TwSG-8B surpasses Qwen3-VL-235B-A22B-Thinking on CharXiv-R despite using substantially fewer parameters.The comparison is reported for the CharXiv-R subset in Table 3.
Benchmark Comparison: Table 3 compares ChartQA results for open-source MLLMs and CharXiv-R results for proprietary models, with tool availability explicitly indicated.Bold and underline identify the best and second-best results, respectively.

D Verify Reward

The verification-reward setup uses an expert MLLM to assess visually grounded intermediate reasoning, while ablations examine judge choice and reward weighting.

Verify Reward: An expert MLLM judge evaluates open-ended intermediate observations because string matching does not reliably measure semantic correctness or visual faithfulness.The assessment is framed as a multiple-choice task to distinguish visual hallucinations from acceptable phrasing variations.
Ablation Studies: Table 4 studies reward verification and reward weighting in TL-GRPO.The table’s ablations address both the judge model and the configuration of reward components.

D.1 Ablation Experiment

The ablations examine judge-model choice, TL-GRPO reward weighting, and decoding stability. Performance is largely robust to judge substitution and sampling variation, while adding format and verification rewards improves over answer-only supervision.

Effect of judge models: 54.44 on ChartQAPro and 85.32 on TableVQA-Bench are obtained with GPT-4o, while Qwen3.6-plus reaches 56.32 and 85.90, respectively.The default Qwen3-VL-72B-Instruct achieves 55.60 on ChartQAPro and the best TableVQA-Bench result in the comparison.
Effect of reward weights: 52.30 on ChartQAPro and 82.91 on TableVQA-Bench result from answer-only reward weighting.Introducing format and verification rewards raises performance to 54.83 and 85.74 with weights (0.05, 0.90, 0.05).
Effect of decoding hyperparameters: Performance remains stable across three random seeds, with only minor fluctuations on ChartQAPro and TableVQA-Bench.The reported results are therefore not sensitive to random sampling effects in the tested settings.
Effect of decoding hyperparameters: Sampling temperatures from 0.1 to 0.9 cause only slight performance changes, with higher temperatures producing a small degradation.The degradation is attributed to increased sampling variability in the supplied passage.
Verification setup: The verification prompt evaluates a generated observation against the provided image using an expert visual evaluator.The exact prompt template and option descriptions are referenced in Figure 4.

E Cold-Start SFT Data Construction

The cold-start SFT data construction distills interleaved visual reasoning trajectories from chart and table data. It combines ROI localization, region description, CoT refinement, hallucination checking, and a broad benchmark spanning 45 visualization categories and eight question categories.

Cold-Start SFT Data Construction: Cold-start SFT inputs contain an image, question, and ground-truth label, while outputs are teacher-elicited interleaved CoT trajectories.Training on these trajectories is intended to help the MLLM internalize the reasoning paradigm.
ROI Identification: ROI identification uses GPT-4o to locate query-relevant subregions and return their absolute coordinates.The full image and question are supplied to generate the relevant bounding boxes.
Region Description: Region description crops each ROI and prompts GPT-5.1 to produce text-focused descriptions containing visible chart or table content.The instruction is optimized for text recognition rather than standard global image description.
CoT Refinement and Structuring: CoT refinement removes redundant reasoning, preserves essential text-related descriptions, and structures observations and question-specific reasoning with tags.Visual observations use <observation> tags, while question-specific reasoning uses <think> tags.
Hallucination Detection: Each data entry undergoes hallucination detection to improve faithfulness, using a verification methodology detailed in Figure 4.The construction pipeline includes explicit verification after region and reasoning refinement.
Benchmark Construction: 12,674 samples span 45 chart/table categories and eight question categories after automatic classification, balancing, and manual quality filtering.The benchmark includes diverse charts, tables, multi-chart layouts, and question types such as mathematical, visual, conversational, and unanswerable QA.
Distillation Objective: The student is trained with cross-entropy on teacher-generated trajectories so it can internally simulate a “Reason-Observation-Reason” process.The approach is described as mitigating coordinate hallucination and CoT hallucinations through expert-teacher supervision.

F Qualitative Analysis

The qualitative comparison shows that TwSG more faithfully links localized visual regions to their textual values than existing CvTR-domain MLLMs. Longer reasoning traces alone do not ensure reliable visual grounding.

Qualitative Comparison: Existing models identify the relevant heat-map region but often misalign color-coded cells with their textual values.Chart-R1 extracts incorrect values from non-orange cells, while Chart-RVR repeats the error despite a longer reasoning trace.
Qualitative Comparison: TwSG first localizes the key region through its structured observation process, contrasting with the incorrect mappings shown for Chart-R1 and Chart-RVR.The supplied passage presents this localization as the qualitative distinction in the comparison.