ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

• Charts require specialized knowledge, spatial reasoning, and numerical understanding because their visual marks, implicit values, axes, and labels interact.General-purpose multimodal models trained on natural images and document models struggle with chart-related tasks.Source: §1 · pp. 1–2
• Prior chart models lack chart-table alignment and sufficient visual-element, mathematical-reasoning, and specialized-chart annotations, limiting generalization and prompting task-specific fine-tuning.The authors identify limited task coverage as a weakness of MatCha and insufficient specialized-chart data in existing training sets.Source: §1 · p. 2
• ChartSFT adds chart-to-table instruction data, numerical-QA chain-of-thought annotations, referring QA, and specialized chart types including radar and box plots.The authors describe its corpus as larger, broader in task and chart-type coverage, and more comprehensively annotated than previous benchmarks.Source: §1 · p. 2
• ChartAssistant uses chart-to-table pre-training to align charts with structured text, followed by multitask instruction tuning for downstream chart tasks.The pre-training task parses a chart into a Markdown table.Source: Figure 1 · p. 1, §1 · p. 2
• ChartAssistant surpasses UniChart by 50.0% on numerical QA and 28.1% on ChartQA, and beats Chartllama by 23.6% on ChartLLM in zero-shot evaluation.The paper also reports a 29.5% performance gain over UniChart on RealCQA in the zero-shot setting.Source: §1 · p. 2

2.1. Multimodal Foundation Model

• Multimodal foundation and document models often fall short on charts because real-world chart images and queries involving mathematical calculations pose distinct challenges.The section notes SPHINX's multiple visual encoders and Donut's OCR-free document-understanding design, while distinguishing chart processing from ordinary images and documents.Source: §2.1 · p. 3

2.2. Chart-specific Vision-Language Model

• Chart-specific approaches include modifying vision-language models or adding LLM plugins, while MatCha and UniChart address chart QA and other tasks but still struggle with computation and chart-type range.ChartSFT is presented as a broader dataset, and ChartAssistant uses it with two-stage training for diverse chart tasks.Source: §2.2 · p. 3

3. ChartSFT

• 39M chart-text pieces make ChartSFT 4.75 and 5.62 times larger than MatCha and UniChart, respectively, and it includes base and specialized chart types.It covers nine chart types.Source: §3 · p. 3

3.1. Chart with Base Types

• Five tasks use base chart types: chart-to-table translation, numerical QA, referring QA, open-ended QA, and chart summarization.The base types are bars, lines, dot-lines, and pies.Source: §3.1 · p. 4

3.1.1 Chart-to-Table Translation

• Chart-to-table data expands ChartQA and PlotQA with style variations and arXiv tables, yielding 220,050 augmented PlotQA examples and 132,719 arXiv chart-text examples.Style augmentation uses five Python APIs and more than 20 parameter variations.Source: §3.1.1 · p. 4

3.1.2 Chart Numerical Question Answering

• Numerical QA adds 101 question templates and chain-of-thought annotations, with ordered function calls used to break solutions into steps and execute calculations.The task collects more than 24M QA pairs; the authors say execution also mitigates calculation errors.Source: §3.1.2 · pp. 4–5

3.1.3 Chart Referring Question Answering

• Referring QA adds marked chart items, including arrows and bounding boxes, to ask about visual elements and their relationships; the dataset contains 5,899,842 examples.Multiple marks can appear in one chart to describe relationships.Source: §3.1.3 · p. 5

3.1.4 Chart Open-ended QA

• Open-ended QA combines existing benchmarks with arXiv table data and ChatGPT-generated questions, totaling 7,075,243 examples across diverse topics.Three QA pairs are generated per table from the table and its descriptive information.Source: §3.1.4 · p. 5

3.1.5 Chart Summarization

• Chart summarization includes 1,006,738 examples, incorporating a large-scale dataset generated through UniChart knowledge distillation.The authors report that ChartSFT exceeds UniChart’s dataset by 4.6 times in total and supports more chart tasks and types.Source: §3.1.5 · p. 5, Figure 3 · p. 6

3.2. Charts with Specialized Types

• For specialized chart types—including radar, area, histogram, bubble, and box-plot charts—the authors use synthetic data to build a diverse collection of complex charts.This addresses the reported poor performance of previous chart models on such types.Source: §3.2 · p. 6

4.1. Architecture

• ChartAst-D is a 260M-parameter Donut-based model with a Swin-Base visual encoder and BART decoder, while ChartAst-S has 13B parameters and uses Sphinx.Their input resolutions are 224 × 224 and 448 × 448, respectively.Source: §4.1 · p. 6
• ChartAst-D sends visual tokens to a BART decoder through cross-attention, whereas ChartAst-S appends visual tokens from multiple encoders to text tokens for LLM processing.The S variant preserves high-resolution information through sampling and partitioning.Source: §4.1 · p. 6

4.2. Training

• Stage I trains on chart-to-table translation, predicting a text-form table from a chart and instruction to align chart content with structured text.The paper frames understanding chart values and their meanings as a prerequisite for downstream chart tasks.Source: §4.2 · p. 6
• Stage II jointly tunes one model on instruction-following data from five ChartSFT tasks, initializing its learnable weights from the Stage I checkpoint.The authors report strong performance across all tasks after multitask instruction tuning.Source: §4.2 · p. 7

5.1. Baselines and Evaluation

• Evaluation covers Chart-to-Text, OpenCQA, ChartQA, MathQA, and ReferQA, with relaxed correctness, BLEU, or RMSF1 depending on the task.Chart-to-Text and OpenCQA use BLEU; ChartQA, MathQA, and ReferQA allow a 5% numerical error tolerance; chart-to-table translation uses RMSF1.Source: §5.1 · p. 7
• Baselines include chart-specific models such as ChartLLaMa and UniChart, general vision-language models, and document-understanding models fine-tuned on evaluation datasets.The paper identifies ChartLLaMa and UniChart as strong chart-task baselines and also includes Matcha, Pix2Struct, and other models.Source: §5.1 · p. 7

5.2. Main Results

• ChartAssistant is reported to lead across chart-related tasks on base-type charts, including 17% and 2.5% gains on ChartQA-human and ChartQAaugment, respectively.Unichart and Matcha use task-specific fine-tuning, whereas ChartAssistant uses a single model after training, except for Chart-to-Text.Source: §5.2 · pp. 7–8, Table 4 · p. 8
• ChartAssistant demonstrates an absolute advantage over chart-specific models on all five specialized-chart tasks.Table 5 compares models on five specialized-chart tasks; BLEU is used for summarization and open-ended QA.Source: §5.2 · p. 8, Table 5 · p. 8
• ChartAssistant-S significantly outperforms existing models across all tasks in the zero-shot scenario.Table 6 covers chart-to-table translation, chart-based question answering, and summarization.Source: Table 6 · p. 8

6. Zero-shot Study

• ChartAssistant shows superior zero-shot performance across chart-to-table translation, chart-based QA, and summarization on samples from StructChart, RealCQA, and ChartLLM.RealCQA uses accuracy within a 5% error margin, ChartLLM uses GPT-4 scoring, and StructChart uses RMSF1.Source: §6 · p. 8
• In zero-shot comparisons, Chartllama performs poorly on precise numerical QA but excels at summarization, while ChartAssistant surpasses existing models on precise numerical QA and long-text summarization.The authors attribute Chartllama's summarization strength to robust language capabilities.Source: §6 · p. 8
• The authors report that weaker decoders are more prone to errors on long zero-shot outputs, including summaries and chain-of-thought answers.They state that large language models can significantly alleviate this issue.Source: §6 · p. 8

The impact of alignment pretraining

• Without alignment pre-training, MathQA and ChartQA-human performance drops by 9.8% and 3.2%, respectively, relative to the two-stage strategy.The authors associate alignment pre-training with learning chart-table correlations before multitask instruction tuning.Source: §7 · p. 9, Table 7 · p. 9

The impact of referring question answering task

• Adding referring QA improves average ChartQA performance by 3.1% and MathQA performance by 11.9%, with gains across almost all tasks.The authors suggest the task strengthens understanding of chart visual elements and their relationships.Source: §7 · p. 9, Table 7 · p. 9

The impact of arXiv data

• Removing arXiv data lowers performance across tasks, especially math QA and referring QA, while its open-ended QA data is reported to support multitask tuning.The authors point to the dataset's diversity of style and context and its precise numerical values as possible contributors.Source: §7 · pp. 9–10, Table 8 · p. 9

COT answer vs. Direct answer for numerical question answering

• Using chain-of-thought answers raises numerical QA accuracy from 51.9% to 72.1%, with gains across chart types, especially dot-line and line charts.The authors say this format teaches reasoning steps and offloads calculations to a backend system.Source: §7 · p. 9, Figure 5 · p. 9

ChartSFT analysis

• The comparison with ChartAst-D trained using Donut is presented as evidence for the superiority of ChartSFT.The supplied passage gives no numerical comparison values.Source: §7 · p. 10

The impact of each multitask instruction tuning component

• Omitting any multitask instruction-tuning component reduces ChartQA performance, and removing numerical QA causes a significant decline.The authors report the smallest contribution from chart summarization, possibly because ChartQA focuses on data extraction and numerical QA.Source: §7 · p. 10, Table 10 · p. 10

8. Conclusion

• The work aims to develop a generalized multimodal model for chart-related tasks and proposes ChartSFT.The supplied conclusion passage ends as it begins describing ChartSFT.Source: §8 · p. 10

A.1. Details of Chart Data Generation in ChartSFT

• ChartSFT generation proceeds from distribution-based table creation to varied chart rendering and then instruction-data generation.Tables use over 20 probability distributions; charts vary styles and layout, while instruction data covers summarization, open-ended QA, numerical QA, and referring QA.Source: §A.1 · p. 14

A.2. Numerical QA Templates

• The numerical QA set contains 101 general templates, with additional templates tailored to box plots, bubbles, histograms, and pies.Not every general template applies to every chart type.Source: §A.2 · p. 14

A.3. Details of Referring QA in ChartSFT

• Referring QA uses 114 templates for label recognition and calculations, adapted for continuous x-ticks in line and area charts.Base-chart referring markers come from PlotQA bounding boxes, while specialized charts are generated with integrated markers.Source: §A.3 · p. 14

B.1. Experimental Setups

• ChartAssistant is pre-trained on chart-to-table translation for 65k steps before multitask instruction tuning.ChartAst-D trains for four epochs and ChartAst-S for one; inference takes an image and textual instruction and generates a textual answer.Source: §B.1 · pp. 14–15
• Adding equivalent questions to training enhances robustness on math questions.Figure 7 compares training with and without the new question-answer pairs.Source: Figure 7 · p. 15
• ChartQA, PlotQA, Chart-to-Text, OpenCQA, MathQA, and ReferQA provide evaluation across translation, summarization, open-ended, numerical, and referring QA tasks.MathQA and ReferQA test sets are sampled from datasets constructed by the authors.Source: §B.1 · p. 15
• Evaluation uses relaxed correctness with 5% numerical tolerance for ChartQA, MathQA, and ReferQA; BLEU for Chart-to-Text and OpenCQA; and RMSF 1 for chart-to-table translation.For table transpositions, both the original and transposed tables are evaluated and the higher RMSF 1 is selected.Source: §B.1 · p. 15
• ChartAssistant has a more significant chart-to-table translation advantage on PlotQA than on ChartQA.Table 11 covers ChartAssistant and baseline performance on PlotQA; the supplied text does not give its cell values.Source: §B.1 · p. 16, Table 11 · p. 16

C. Ablation Study

• Overall accuracy rose from 71.8% to 76.2% when equivalent math questions were included in training.The added questions were generated from template questions and table information to address the rigidity of template-only generation.Source: C. Ablation Study · p. 16

D. Some demos from Out of Distribution

• ChartAssistant shows generalization on randomly selected out-of-distribution chart screenshots for chart-to-table translation, summarization, and question answering.Figures 8–10 present demonstrations spanning these tasks, including mathematical and referring QA.Source: D. Some demos from Out of Distribution · p. 16, Figure 8 · p. 17, Figure 9 · p. 17
• In demonstrations, UniChart and MatCha repeat or hallucinate in summarization, while ChartLlama makes factual errors and ChartAssistant accurately constructs the chart table values.The text also attributes ChartAssistant's mathematical QA performance to incorporating chain-of-thought-format QA training data.Source: D. Some demos from Out of Distribution · p. 16