ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning

1 Introduction

• Chart reasoning by MLLMs remains under-explored despite their progress in multimodal applications and their ability to generalize across tasks.The paper notes that complicated reasoning from chart data has not been fully explored.Source: §1 · p. 1
• ChartX collects 48K multimodal chart samples across 22 topics, 18 chart types, and 7 tasks, with image, CSV, code, and text modalities.The tasks comprise structural extraction, chart type classification, title extraction, QA, description, summarization, and redrawing.Source: §1 · p. 2
• ChartVLM first predicts chart structure, then uses that representation for cognition tasks to support interpretable reasoning.An instruction adapter dynamically selects tasks according to user instructions.Source: §1 · p. 2
• ChartVLM is compared with current MLLMs on ChartX, with the authors reporting its superiority across chart understanding abilities.Its training uses existing open-source chart datasets and excludes ChartX evaluation data.Source: §1 · p. 2

Chart Perception

• Chart perception methods transform charts into tables, often relying on external modules such as GPT for downstream tasks, whose scalability is difficult to guarantee.Examples include Deplot [Liu et al., 2022a] and StructChart [Xia et al., 2023].Source: §2 · pp. 2–3

Chart Cognition

• Chart cognition combines chart-related and common-sense knowledge, including numerical queries that require mathematical or logical reasoning.Prior approaches include Pix2Struct [Lee et al., 2023] and end-to-end MatCha [Masry et al., 2022].Source: §2 · p. 3

Multi-Modal Chart Generation and Benchmark

• Earlier benchmarks often covered only line, pie, and bar charts or a few tasks, while newer benchmarks still have limited data and metric diversity.ChartBench uses yes-or-no judgments, and the chart types and data in MMC are described as insufficient for assessing off-the-shelf MLLMs.Source: §2 · p. 3

3.1 Coverage Analysis of the Evaluation Set

• ChartX covers general, fine-grained, and domain-specific chart types, with fine-grained types balanced to limit long-tail distribution.Its coverage includes more than 18 types, including line, bar, pie, radar, box plot, heatmap, funnel, and candlestick charts.Source: §3.1 · pp. 3–4
• ChartX spans 22 sub-disciplines grouped under commerce, industry, society, culture, and lifestyle.The topic distribution is presented in Figure 1.Source: §3.1 · p. 4
• ChartX defines three perception tasks and four cognition tasks, including chart-only QA, descriptions, summaries, and redrawing.Each image has two QA samples and one each for description, summarization, and redrawing; the multi-task samples total 48K.Source: §3.1 · p. 4

3.2 Distribution Analysis of the Evaluation Set

• ChartX uses plotting-package and hyperparameter variation to diversify chart styles, and has greater CSV-length and token-distribution diversity than existing benchmarks.Figure 2 compares chart images, CSV data, QA questions, and CSV data length across benchmarks.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Two-stage Chart Data Generation

• ChartX generates chart perception data from GPT-4-selected and human-refined titles and types, plus data distributions tailored to chart type and topic.The perception data then underlies task-specific cognition examples generated with tailored instructions; redrawing code renders chart images.Source: §3.3 · p. 5

3.4 Task Evaluation Metrics

• Structural Extraction is evaluated with SCRM after predicted CSV tokens are converted to triplets; QA uses GPT-acc with a 5% numerical margin.GPT-score uses GPT-4 to rate summarization, description, and redrawing outputs on a 0–5 scale.Source: §3.4 · pp. 5–6

4.1 Overall Model Design

• ChartVLM routes instructions to a base decoder for perception tasks and activates an auxiliary decoder for more intricate generative tasks, using CSV predictions as input.The architecture also includes a pixel-level encoder; the auxiliary decoder uses both base-decoder outputs and user instructions.Source: §4.1 · p. 6, §4.3 · pp. 7–8

4.2 Instruction Adapter: Instruction Selection

• The instruction adapter maps user instructions to one of seven chart-task categories and is designed to select the decoder dynamically.It has three linear layers and achieved 100% accuracy on the constructed validation subset; Figure 5 shows zero-shot results for four cognition tasks.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3 Cascaded Decoders Design

• For basic perception, the fine-tuned encoder-decoder transforms chart images into CSV; for intricate cognition, LoRA fine-tunes pre-trained Vicuna-7B and Vicuna-13B as auxiliary decoders using text-text pairs.The variants are ChartVLM-Base-7.3B and ChartVLM-Large-8.3B.Source: §4.3 · p. 8

5.1 Evaluation Settings

• ChartX evaluation uses 4,848 validation samples and 1,152 manually inspected test samples, with task-specific post-processing and prompts.Figure 6 presents class-wise results, omitting six chart types whose AP metrics are zero for almost all models.Source: §5.1 · pp. 8–9, Figure 6 · p. 9, §5.2 · p. 10

5.2 Baseline Models and Main Results

• ChartVLM-B and ChartVLM-L consistently outperform most evaluated models across ChartX tasks, with GPT-4V the exception on a few cognition tasks.ChartVLM’s stronger chart-structure extraction is reported to partly contribute to 40.71% accuracy on chart QA.Source: §5.2 · p. 10

5.3 Insightful Analyses

• In the cascaded decoder, ChartVLM-L outperforms ChartVLM-B on structural extraction and intricate cognition tasks, with cognition performance peaking when structural-extraction accuracy reaches 100%, corresponding to a golden table in Table 5.Source: §5.3 · pp. 10–11
• ChartVLM’s cognition tasks use perception-derived text representations as conditional input; reported QA accuracy is 36.46% despite SPHINX-V2’s similar structural extraction result of 32.07% versus ChartVLM’s 32.65%.The authors attribute stronger downstream reasoning partly to the cascaded decoder incorporating perceived chart results.Source: §5.3 · p. 11
• ChartVLM’s lightweight base decoder yields faster inference on perception tasks, while cognition-task inference is comparable across models.The speed comparison used one Tesla A100, batch size 1, and a consistent maximum token count.Source: §5.3 · p. 11, Table 6 · p. 11
• Entity replacement improves structural-extraction evaluation results across all baseline models, alleviating assessment bias for percentage-related chart types.For those chart types, the value-column label may be invisible and is replaced with “value” or “percentage.”Source: §5.1 · p. 9, §5.3 · p. 12, Table 7 · p. 12
• Open-source MLLMs perform poorly on specialized chart types, including rose, area, 3D-bar, bubble, multi-axes, and radar charts, in perception and cognition tasks.The class-wise presentation omits six difficult chart types with zero-value AP metrics for almost all models.Source: §5.2 · p. 10, §5.3 · p. 12

5.4 Out-of-Distribution (OOD) Evaluation

• ChartX is treated as an out-of-distribution evaluation because it spans 18 chart types and seven tasks, with little overlap with ChartVLM’s training distribution.Only 20% of ChartVLM’s training data are synthetic, and more than 99% belong to line, bar, or pie charts; ChartVLM nevertheless shows some performance on 15 other chart types.Source: §5.4 · p. 12

• ChartX provides a broad chart evaluation set, while ChartVLM uses a cascaded decoder mechanism to boost interpretability in scientific chart-data applications.Source: §6 · p. 12

A.1 Introduction of Chart Topics

• The 22 chart topics span commerce, industry, lifestyle, society, and culture, and are described as evenly distributed.Source: §A.1 · p. 16, Figure A.1 · p. 16

A.2 Overall of Data Generation Pipeline

• The pipeline samples chart types and topics to generate CSV data and titles, then uses task- and type-specific prompts to create cognition samples.Perception prompts constrain data magnitude and length for chart visibility; cognition prompts guide task-specific labels.Source: §A.2 · pp. 16–17

A.3 Manual Quality Inspection

• Four validation steps check file formats, cross-modal alignment, QA reasoning and answers, and chart summaries and descriptions.The final three stages use human review, with GPT-4 involved in QA-pair and summary-description validation.Source: §A.3 · p. 17

A.4 Examples of ChartX

• Examples show generated data related to assigned chart types and topics, with QA pairs tailored to chart characteristics and summaries and descriptions conveying chart content.The metadata examples include chart type, title, topic, CSV data, QA pairs, summaries, descriptions, and redrawing code.Source: §A.4 · pp. 17–18, Figure A.4 · p. 20

• The appendix provides class-wise mean precision for Structural Extraction evaluated using SCRM, with relatively high tolerance used for some hard fine-grained classes.Source: Table A.1 · p. 21
• The tolerance indicators distinguish strict, slight, and high tolerance levels.Source: B Experimental Details · p. 21, Table A.1 · p. 21
• The appendix reports class-wise accuracy for Question Answering evaluated using GPT-acc.Source: Table A.2 · p. 21

• The chart extraction model and LLM decoder are fine-tuned on ChartQA, Chart2text, PlotQA, and SimChart9K, using CSV ground truth for extraction and available QA, summarization, and code data for the decoder.The decoder instruction-following data includes QA from ChartQA and PlotQA, summaries from Chart2text and SimChart9K, and redrawing code from SimChart9K.Source: §B.1 · p. 21
• The instruction adapter is trained on 7K instruction-category pairs generated using GPT-3.5-turbo.Source: §B.1 · p. 21

B.2 Evaluation Settings

• GPT-acc evaluates QA, while GPT-score evaluates description, summarization, and redrawing.The complete prompts and manual criteria are referenced in Figures A.5 and A.6.Source: §B.2 · p. 21
• SCRM uses strict, slight, and high tolerances, with different thresholds for normal and difficult chart types.The string threshold is an edit distance and the numeric threshold is relative error between prediction and ground truth.Source: §B.2 · pp. 21–22
• Description, summarization, and redrawing class-wise scores use GPT-score on an integer scale from 0 to 5.Source: Table A.3 · p. 22, Table A.4 · p. 22, Table A.5 · p. 22

B.3 Maximum number of generate token settings

• Maximum generated tokens are fixed by task: 1,280 for SE, 100 for title, 20 for type, 100 for QA, 512 for description and summarization, and 1,024 for redrawing code.The same setting is maintained for inference-speed testing.Source: §B.3 · p. 22

B.4 Quantitative Results for Each Chart Type

• ChartVLM shows comprehensive superiority over existing models on most chart types and tasks, although GPT-4V scores above the average in summarization and description and reasoning remains difficult for all models.The authors identify learning multiple chart-related tasks without sacrificing generalization to new chart domains as a common challenge.Source: §B.4 · p. 23

B.5 Visualization Results of Perception Tasks

• ChartVLM performs well on title and chart-type prediction; its radar-chart structural extraction is slightly wrong, while funnel, histogram, and line-chart extraction is strong.Source: §B.5 · p. 23

C More Comprehensive Analysis

• ChartX varies chart representations through multiple modalities and chart types, whereas real-world datasets face acquisition and annotation costs, domain and representation biases, and limited scalability.Its synthetic charts are clearer and more controllable; controllable generalization tools and random prompt tuning support diverse distributions at lower cost.Source: C More Comprehensive Analysis · p. 23
• Models that perform better on ChartX also perform better on ChartQA, supporting transfer of ChartX conclusions to real-world chart distributions.Source: C More Comprehensive Analysis · p. 24
• The cascade uses a lightweight base decoder for perception and both base and heavier auxiliary decoders for cognition; extracted chart representations provide interpretability for downstream tasks.The authors report faster inference, especially for perception, and better cognition-task performance as chart-representation accuracy increases.Source: C More Comprehensive Analysis · pp. 25–27
• The instruction adapter classifies commands into seven task labels plus a special out-of-capability label, which serves as a safeguard against erroneous outputs or hallucinations.Fixed task-specific instructions to the auxiliary decoder improve performance in a supplementary experiment.Source: C More Comprehensive Analysis · p. 27