START: Spatial and Textual Learning for Chart Understanding

1. Introduction

• Charts’ combination of spatial layout and textual data makes them challenging: MLLMs can misground chart elements, such as mapping “condition” to the wrong x-axis position.Source: §1 · p. 1
• START adds chart-element grounding and chart-to-code generation to chart question answering in both supervised fine-tuning and reinforcement learning.Source: §1 · p. 2
• The START-Dataset pipeline converts real chart images into executable Python code, then uses an LLM to derive chart-element locations.Source: §1 · p. 2
• CS-Bench evaluates MLLMs’ understanding of chart visual structure, addressing a gap in comprehensive chart-understanding evaluation.Source: §1 · p. 2
• START-RL-7B outperforms the previous best [4] by 42.7 on ChartMimic, 14.7 on CharXiv-descriptive, 1.5 on CharXiv-reasoning, 2.1 on ChartQAPro, and 35.7 on CS-Bench.Source: §1 · p. 2

2. Related Works

• Chart understanding moved from CNN–RNN and multi-stage or transformer systems to MLLMs, with work on instruction tuning, chart-to-text, reinforcement learning, and chart-to-code generation.Source: §2 · p. 2
• Chart-image synthesis progressed from parameterized templates to LLM-evolved Python code, alongside collection of real-world chart images to better capture natural chart distributions.Source: §2 · p. 2
• Related evaluation efforts include chart question-answering benchmarks [19, 20, 29, 36, 40, 56, 59] and chart-to-code/text generation benchmarks [60, 62, 64].Source: Figure 2 · p. 3
• Prior work largely overlooks chart spatial structure, while START integrates spatial and textual perspectives for chart understanding.Source: §2 · p. 3

3.1. Method

• Spatial learning grounds chart elements to capture layout, while textual learning treats rendering Python code as the data representation linking elements, values, and formulas.The framework adds these tasks alongside chart question answering in supervised fine-tuning and reinforcement learning.Source: §3.1 · pp. 3–4
• SFT and RL combine spatial-textual learning with chart question answering; RL rewards accuracy and output format, using task-specific checks for answers, boxes, and code.CQA uses string matching, spatial samples use IoU, and generated code is graded by an LLM; format is checked with regular expressions.Source: §3.1 · p. 4

3.2. START-Dataset

• START-Dataset reconstructs Python code from real chart images, evolves that code to emit chart-element locations, and converts the locations into grounding training pairs.The pipeline aims to retain real-chart visual complexity while recovering the underlying data representation.Source: §3.2 · p. 5
• Chart QA pairs rely on chart content rather than external domain knowledge, are filtered for hallucinated or unanswerable questions, and contribute to SFT and difficulty-sampled RL datasets.The QA data are combined with spatial and textual learning data in START-Dataset-SFT; START-Dataset-RL is sampled from it by question difficulty.Source: §3.2 · pp. 5–6

3.3. CS-Bench

• CS-Bench uses held-out evolved-code charts for direct grounding and content-QA-plus-grounding questions, with manually verified pairs and recall@0.3 plus auxiliary QA accuracy.The benchmark addresses limits of earlier spatial evaluation, including simple component types and a single-subplot focus.Source: §3.3 · p. 6

4.1. Results on Chart Understanding

• The evaluation spans chart perception and reasoning, chart-to-code translation, and spatial understanding, using CharXiv, ChartQA, ChartQAPro, ChartMimic, and CS-Bench.Scores are reported for CharXiv and ChartMimic, accuracy for ChartQA and ChartQAPro, and recall@0.3 and accuracy for CS-Bench.Source: §4.1 · p. 7
• START-RL-7B surpasses Chart-R1-7B by 14.7 on CharXiv descriptive, 1.5 on CharXiv reasoning, 2.1 on ChartQAPro, 42.7 on ChartMimic, and 35.7 on CS-Bench, but lags on ChartQA.ChartQA was not used during training, so this is zero-shot performance.Source: Table 1 · p. 7, §4.1 · p. 8

4.2. The Ablation Study.

• Adding chart-to-code improves ChartQAPro and ChartMimic, while adding grounding improves CharXiv and CS-Bench; grounding also further improves ChartQAPro and ChartMimic in RL.The authors interpret these results as evidence that spatial and textual learning complement each other.Source: Table 2 · p. 7, §4.2 · p. 8
• Applying think-before-answer across tasks consistently improves benchmark performance, including on the grounding benchmark CS-Bench.The authors note consistency with prior works [26] [58], suggesting that thinking can also enhance low-level visual understanding.Source: §4.2 · p. 8, Table 3 · p. 8

4.3. Analysis and Visualization.

• Visualizations compare START with Qwen2.5-VL on chart question answering, chart-element grounding, and chart-to-code predictions.The reported examples show a corrected subplot-grounding error, improved bounding-box prediction, and a better re-rendered chart.Source: Figure 6 · p. 7, §4.3 · p. 8

5. Conclusion and Discussion

• START couples chart-element grounding with chart-to-code generation to learn chart spatial structure and textual details.The framework reflects charts’ structured visual layout and underlying textual content.Source: §5 · p. 8
• START-Dataset construction transcribes real-world charts into Python plotting code with an MLLM, then evolves the code with an LLM to obtain chart-element locations.Source: §5 · p. 8
• The paper proposes CS-Bench to evaluate MLLMs’ spatial understanding of charts and reports substantial benchmark improvements over strong baselines.Source: §5 · p. 8

A.1. Chart-to-code

• The chart-to-code exploration found that directly converting chart images with a proprietary MLLM produced the most authentic reproduced charts and preserved most details.The alternatives included direct conversion with Qwen2.5-VL and a caption-then-code pipeline.Source: §A.1 · p. 12
• The pipeline filters non-chart images from ArxivQA, converts the remaining charts to Python, renders them, and filters distorted reproductions.Source: §A.1 · p. 12
• The resulting chart images and Python code form textual-learning data for SFT and RL, using fixed templates for SFT examples and generated code as RL ground truth.Source: §A.1 · p. 12

A.2. Location Generation

• Standardized sample evolved code extracts chart-element locations from rendered charts and stores them in JSON, improving code-generation success and annotation uniformity.The described extraction covers elements including titles, axis labels, and ticks.Source: §A.2 · pp. 12–13
• The pipeline executes evolved code, uniformly samples locations by element type, and converts them into grounding annotations with fixed templates.Source: §A.2 · p. 13

A.3. QA generation

• For QA generation, the authors compare an MLLM using chart images and code with an LLM using code alone, and select the MLLM approach because it consistently produced higher-quality pairs.The MLLM can use visual and code inputs for spatial questions, whereas the code-only LLM suits fine-grained details such as exact point counts or data values.Source: §A.3 · p. 13
• A strong MLLM checks whether generated questions are chart-groundable and answerable and verifies answers, filtering hallucinated or incorrect QA pairs.The generation process uses few-shot examples to produce ten QA pairs per chart image.Source: §A.3 · p. 13

A.4. Curate the SFT and RL data splits

• START-Dataset-SFT combines chart question answering, spatial learning, and textual learning; START-Dataset-RL samples from it using question difficulty estimated by Qwen2.5-VL’s probability of answering correctly.The estimated difficulty is used as the sampling weight.Source: §A.4 · p. 13

B. CS-Bench

• CS-Bench contains 613 images rendered from a holdout subset of code evolved during START-Dataset construction.The chart-element locations are generated from the evolved Python code when it is run.Source: B. CS-Bench · p. 13
• The benchmark includes 350 grounding questions and 342 QA grounding questions, which combine chart-content questions with locating referenced elements.Grounding questions ask directly for an element’s location; QA grounding questions pair a chart-content question with locating referenced chart elements.Source: B. CS-Bench · p. 14
• QA grounding questions are generated by prompting an MLLM and manually verifying the question and answer; location-only questions follow the chart-element location pipeline.For QA grounding items, the location is selected from the question or answer.Source: B. CS-Bench · p. 14
• Recall@0.3 over 692 ground-truth bounding boxes is the main metric, with answer accuracy on 342 QA grounding questions as an auxiliary metric.The authors use recall@0.3 because ground-truth locations include many small items such as tick values and axis names.Source: B. CS-Bench · p. 15

C.1. Benchmarks.

• Evaluation uses CharXiv, ChartQA, ChartQAPro, ChartMimic, and CS-Bench, with accuracy, GPT-4o-judged scores, or bounding-box recall@0.3 as task-specific metrics.CharXiv and ChartMimic use GPT-4o as judge; CS-Bench uses recall@0.3 as its main metric.Source: §C.1 · p. 15
• CharXiv covers descriptive and reasoning questions on 1,000 arXiv chart images, while ChartQA focuses mostly on line, bar, and pie charts across 1,509 images and 2,500 questions.ChartQAPro covers 1,341 images from 157 online platforms and 1,948 questions; ChartMimic evaluates chart-to-code conversion using 600 human-curated figure-instruction-code triplets.Source: §C.1 · p. 15

Training details in Supervised Finetuning (SFT)

• SFT initializes Qwen2.5-VL-3B and 7B checkpoints [1] and trains for one epoch at 1e-6 learning rate, using 0.1 warm-up, cosine decay, and global batch size 128.Training data mixes the START-SFT portions corresponding to the selected tasks.Source: §C.2 · p. 15

Training details in Reinforcement Learning (RL)

• RL starts from the task-matched SFT checkpoint and runs 100 steps at 1e-6 learning rate with rollout batch size 512, five rollouts, micro-batch size 4, and 8 A100 GPUs.Task-specific START-RL questions prompt rollouts, with ground-truth answers or bounding boxes used as reward references.Source: §C.2 · p. 15

Reward design and reward calculation

• Formatting rewards are binary regex checks, grounding rewards use bounding-box IoU, and chart-to-code rewards aggregate normalized LLM scores across five code-quality perspectives.The code perspectives are data, plot type structure, axes scales and limits, text elements, and styling; each is graded 0–5 before normalization.Source: §C.2 · p. 15

Chart-element grounding prompt

• Grounding instructions request pixel-coordinate bounding boxes for subplot elements, including axes and ticks, legends, line intersections, pie portions, and bar values.The instructions also cover chart-specific elements such as 3D axes, nodes, radar-chart labels, tree blocks, and funnel-polygon text.Source: §C.2 · p. 16

Grounding output format

• The grounding output stores locations in location.json under full_chart and subplot structures, with conditional fields for elements such as line intersections, pie portions, and subplot row and column.Locations for elements that do not exist are saved as None.Source: §C.2 · p. 16

Question-generation prompt

• The question-generation prompt requests 10 diverse questions spanning Text-in-Chart, Text-in-General, Number-in-Chart, and Number-in-General, including trends, comparisons, extrema, thresholds, ranking, and frequency.Questions are generated from the chart image and Python code and should go beyond surface elements such as titles and axis labels.Source: §C.2 · p. 17

• Generate at least one question for each of the four categories: Text-in-Chart, Text-in-General, Number-in-Chart, and Number-in-General.Source: §C.2 · p. 17
• For every question, provide an answer, a qa_category selected from the four defined categories, and a single short explanation of how to find the answer from the chart.The answer must give only the final answer, without explanation or filler, and be grounded in the chart's visual information.Source: §C.2 · p. 17
• Questions may go beyond the examples, but must fit one of the four category definitions, and no rationale or explanation should appear outside the required JSON fields.Source: §C.2 · p. 17