Source-linked AI summary
ChartBench: A Benchmark for Complex Visual Reasoning in Charts
Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, Jian Guo
TL;DR
Chart-comprehension benchmarks have limited chart-type coverage and often use metrics vulnerable to lucky guesses. ChartBench evaluates MLLMs on diverse annotated and unannotated charts, with Acc+ and chain-of-thought and supervised fine-tuning baselines. Evaluations reveal that current MLLMs struggle with complex charts, especially those without data annotations.
Problem
Existing benchmarks have limited chart-type coverage, and current metrics can overestimate performance through lucky guesses.
Source: §1 · pp. 1–2Method
Results
Evaluations of 18 open-source and 3 closed-source MLLMs show that they struggle to understand complex charts, especially those without data annotations.
Source: §1 · p. 2Takeaways & Limitations
ChartBench highlights MLLMs’ ability to understand charts without data annotations and offers insights for further research directions.
Source: §7 · p. 9Models are highly sensitive to prompt templates, and finding the best template for each model requires further exploration.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in image understanding and generation. However, current benchmarks fail to accurately evaluate the chart comprehension of MLLMs due to limited chart types and inappropriate metrics. To address this, we propose ChartBench, a comprehensive benchmark designed to assess chart comprehension and data reliability through complex visual reasoning. ChartBench includes 42 categories, 66.6k charts, and 600k question-answer pairs. Notably, many charts lack data point annotations, which requires MLLMs to derive values similar to human understanding by leveraging inherent chart elements such as color, legends, and coordinate systems. We also design an enhanced evaluation metric, Acc+, to evaluate MLLMs without extensive manual or costly LLM-based evaluations. Furthermore, we propose two baselines based on the chain of thought and supervised fine-tuning to improve model performance on unannotated charts. Extensive experimental evaluations of 18 open-sourced and 3 proprietary MLLMs reveal their limitations in chart comprehension and offer valuable insights for further research. Code and dataset are publicly available at https://chartbench.github.io.
1 Introduction
Existing chart benchmarks underrepresent complex chart types, rely heavily on datapoint annotations, and use metrics vulnerable to lucky guesses. ChartBench broadens chart coverage, emphasizes unannotated charts, and introduces an evaluation metric intended to make assessment more robust.
1 Introduction
- Earlier benchmarks focus on three common chart types, often provide datapoint annotations that allow OCR-based answers, and use metrics vulnerable to lucky guesses.These limitations leave complex chart formats and visual reasoning without annotations underrepresented.Source: §1 · pp. 1–2
- ChartBench covers 42 chart subcategories, five question-answering tasks, and more than 600k instruction examples, including unannotated charts across all categories.The paper describes the benchmark as including over 68k charts and says over 80% of its training set is unannotated.Source: §1 · p. 2
- Acc+ evaluates whether models judge both positive and negative assertions correctly, with negative assertions using another value from the same chart to reduce lucky guesses.The paper also refines value matching and the query and answer-extraction process for numerical questions.Source: §1 · p. 2
- Evaluations of 18 open- and 3 closed-source MLLMs find that models struggle with complex charts, especially unannotated ones, motivating further work on chart comprehension.The paper also proposes chain-of-thought and supervised fine-tuning baselines to improve performance.Source: §1 · p. 2
- The benchmark and its baselines target chart reasoning from visual elements such as colors and legends rather than relying on datapoint annotations.The proposed baselines use chain of thought and supervised fine-tuning.Source: §1 · p. 2
2 Related Works
Prior work advances multimodal models and chart understanding, but specialized chart comprehension and its evaluation—especially on unannotated charts—remain limited. Existing chart benchmarks tend to cover fewer chart types and provide datapoint annotations.
2.1 Multimodal LLMs
- Chart-understanding approaches include two-stage methods that turn multimodal queries into text question-answer pairs using extracted tables [35, 43, 42, 74] and end-to-end methods tuned on chart-question pairs [26] [9] [51] [44] [77] [47] [69] [86] [76] [10] [81].The authors note that suitable benchmarks, particularly for unannotated charts, remain limited.Source: §2.1 · p. 3
2.2 Multimodal Benchmarks
- Mainstream chart-comprehension benchmarks such as ChartQA [50] and PlotQA [52] focus on three common chart types, while prior benchmarks generally supply datapoint annotations that permit simple OCR-based answers.Chartllama [26] and ChartX [74] broaden chart types, and ChartY [10] expands regular chart types; ChartBench is distinguished by its scale, chart diversity, plot styles, and unannotated charts.Source: §2.2 · p. 3
3 ChartBench
ChartBench generates diverse charts and instructions across 42 chart subcategories, including unannotated charts, and assesses them with five question-answering tasks and revised evaluation metrics.
3.1 Data Processing Pipeline
- ChartBench generates varied charts from standardized data, with some charts lacking data-point markers and 50 samples per chart type selected for the benchmark.Its pipeline filters data against JSON formats for 42 chart types, then creates instructions for five tasks.Source: §3.1 · p. 3
3.2 Automatic Instructions Generation
- The five tasks assess chart-type recognition, value extraction, value comparison, global conception, and numerical answers, using graphical elements rather than metadata for reasoning.Value extraction on unannotated charts relies on legends, axes, and graphical elements; NumberQA uses tolerance evaluation for numerical answers.Source: §3.2 · p. 4
3.3 Dataset Analysis
- ChartBench’s chart-feature distribution resembles ChartX’s, uses more plot styles, and supplements ChartQA with online-plotted charts, while its query style is consistent with ChartQA and ChartX.The analysis compares CLIP-encoded chart, CSV, and query features after sampling 10,000 data points for each.Source: §3.3 · pp. 4–5
3.4 Evaluation Metrics
- Acc+ evaluates paired correct and incorrect assertions, counting a query only when both receive appropriate answers; incorrect values are randomly selected from metadata.CoR measures the tendency to give identical answers to both assertions, which approaches 100% when models fail to use chart visuals; GPT-acc adds 5% error-margin numerical evaluation with LLM response filtering and excludes years and months from NumberQA.Source: §3.4 · p. 5
4 Baselines
The paper proposes chain-of-thought prompting and supervised fine-tuning as baselines for improving MLLMs’ performance on unannotated charts.
ChartCoT
- ChartCoT decomposes chart questions into prompts that mimic human visual reasoning, with chains generated by a model or GPT as alternatives to a shared template.CoT-GPT ensures chain quality and adapts chains to question types, and the paper reports it performs best.Source: §4 · p. 5, Figure 4 · p. 6
Supervised Fine-tuning
- Two-stage fine-tuning aligns chart-CSV pairs through connector updates, then applies LoRA to instruction-chart pairs while keeping the visual encoder frozen.The baselines are reported to improve MLLM performance; the SFT experiments use Qwen-VL-Chat and Internlm-XComposer-v2.Source: §4 · pp. 5–6
5 Experiments
The experiments compare 21 MLLMs on ChartQA and ChartBench, finding persistent difficulties with complex and unannotated charts. They also examine task-specific performance, answer reliability, and CoT and SFT baselines.
Results on ChartQA and ChartBench
- MLLMs show consistent trends across ChartQA and ChartBench, but generally perform better on regular than extra charts; ChartBench is more challenging for charts with more unannotated data.Models with chart pre-alignment also tend to perform better on regular charts, which the authors relate to their alignment data.Source: §5 · p. 6, Table 3 · p. 7
Results w.r.t. Task Types
- CR is easiest and VE is most challenging; strong text recognition alone is insufficient for chart reasoning, while closed-source models outperform open-source models in part using supplementary recognition tools.The authors describe VE as requiring visual and textual logical reasoning beyond basic OCR.Source: §5 · p. 6, Table 4 · p. 7
Results w.r.t. Datapoint annotations
- Internlm-XComposer-v2 (+18.36%) and GPT-4V (+34.40%) show larger annotated-versus-unannotated performance gaps as MLLM capabilities increase.The authors attribute the gap to the relative ease of OCR on annotated charts and call for more unannotated chart data.Source: §5 · p. 7, Table 6 · p. 8
CoT Performance
- CoT generally improves performance without parameter updates, and question-specific CoT-GPT performs best among the tested prompting approaches.The fixed prompt helps across tasks, while self-generated CoT is less effective because its quality and length are uncontrolled.Source: §5 · pp. 7–8
Error analysis
- 41.78% CoR for Internlm-XComposer-v2 is the lowest reported, yet the authors find random guessing without chart information common among open-source models.CoR generally correlates negatively with Acc+, but the authors note exceptions, including Qwen's higher CoR despite better Acc+ than MiniGPT-v2.Source: §5 · p. 6, Table 5 · p. 8
SFT Performance
- Qwen-VL-Chat (+13.01%) and Internlm-XComposer-v2 (+15.62%) improve on ChartBench with SFT; the latter reaches performance on par with GPT-4o after alignment and SFT.The authors report gains on ChartQA as well; training uses two alignment epochs and one SFT epoch, with the visual encoder frozen.Source: §5 · p. 8
6 Discussion
The discussion identifies instruction-following problems and unreliable numeric outputs, and contrasts how CoT and SFT affect models. It calls for more evaluation, prompt exploration, and work on training methods and architectures.
Instruction following
- Some models produce overly detailed, irrelevant, or hallucinatory responses, and different prompt templates can lead the same model to substantially different results.Examples include repeating the question, misinterpreting long extracted dictionaries, and answering unrelated to the query.Source: §6 · pp. 8–9
MLLM performance
- Models favor descriptive chart responses over numbers, may fail to follow complex instructions, and can hallucinate data in VE and NQA, making specific numeric answers unreliable.The authors note that instruction-following failures hinder the application of intricate CoT strategies.Source: §6 · p. 9
CoT v.s. SFT
- CoT benefits weaker MLLMs more, while SFT yields larger gains for Internlm-XComposer-v2 than Qwen-VL-Chat and comparable gains across annotated and unannotated charts.The reported SFT average gains are 15.65% versus 13.01%, respectively; the annotated-versus-unannotated changes are ∆-0.66% versus +0.60%.Source: §6 · p. 9
Limitations
- The authors identify three open needs: evaluate more models, explore each model's best prompt template, and further investigate chart perception and reasoning methods and architectures.They note that the field is rapidly evolving and that models are highly sensitive to prompt templates.Source: §6 · p. 9
7 Conclusion
The conclusion presents ChartBench as an evaluation of MLLM chart comprehension, emphasizing charts without data annotations and visual-cue-based value inference. It also summarizes Acc+, two baselines, and the evaluation's findings about model limitations.
Conclusion
- ChartBench evaluates MLLM chart comprehension across expanded chart types, with values inferred from visual cues such as color and legends.The conclusion frames understanding charts without data annotations as a central focus.Source: §7 · p. 9
- Acc+ is presented as an improved metric for automated assessment that avoids manual effort or costly LLM evaluations.Source: §7 · p. 9
- The authors report that CoT and SFT baselines improve chart performance, while evaluation of 21 MLLMs reveals limitations in chart interpretation.The paper presents these findings as insights for further research.Source: §7 · p. 9
A.1 Design Principle
ChartBench is designed to cover diverse chart types and assess visual reasoning on unlabeled charts. The appendix describes its chart taxonomy, data splitting, and evaluated model architectures.
A.1 Design Principle
- ChartBench expands beyond line, bar, and pie charts to nine major categories and 42 subcategories, with unlabeled charts comprising 84.96% of training and 76.20% of test data.The design emphasizes extracting values from visual cues such as color or line shape, rather than relying on OCR-generated answer candidates.Source: §A.1 · p. 15
A.2 Chart Taxonomy
- The taxonomy covers nine categories, including bar, line, pie, radar, box, scatter, area, node, and combination charts, with multiple chart forms within each category.Combination examples include dual-coordinate line-and-bar charts and bar-and-pie combinations.Source: §A.2 · pp. 16–17
A.3 Data Splitting
- Each chart receives questions for five tasks; test-set subclasses are balanced at 50 data points, and annotation-version comparisons use versions from the same table.The train and test distributions differ slightly because of the per-subclass test-set target.Source: §A.3 · p. 17
B.1 Architecture
- The evaluation covers 18 open-source and 3 closed-source MLLMs; some closed-source models are sampled because their APIs are not efficient.The appendix summarizes model visual and language architectures, memory use, and inference latency.Source: §B.1 · p. 17
B.2 Model Performance Explaination
The model analyses link benchmark-specific failures to long prompts, answer-format control, and prompt sensitivity, while evaluations used zero-shot prompts selected for each model.
B.2 Model Performance Explaination
- OneChart's inaccurate chart-to-CSV dictionaries produce longer prompts, and it often fails instructions on those prompts, even for simple yes-or-no questions.ChartBench metadata is longer than ChartQA's, and its charts lack data-point annotations.Source: §B.2 · p. 18
- ChartVLM often returns numbers or phrases instead of yes-or-no answers, which the metric cannot parse, although some outputs match the correct answer.The authors retain the original results for fair comparison.Source: §B.2 · p. 18
- ChartLlama leads on ChartQA but trails LLaVA-v1.5 on ChartBench; it does better on NQA yet poorly on Acc+, with numerical answers linked to reduced instruction-following.The authors suggest supervised fine-tuning may cause the model to answer yes-or-no questions numerically.Source: §B.2 · pp. 18–19
- mPLUG-Owl-bloomz struggles with ChartQA and NQA's concise-answer requirement because it tends to generate lengthy descriptions.The authors attribute this limitation to training that prioritizes detailed explanations over succinct answers.Source: §B.2 · p. 19
C.1 Evaluation Implementation
- Evaluations used one GPU per open-source model and zero-shot prompts, with several prompt variants tested and the best-performing one selected.The authors note that prompt choice substantially influences responses.Source: §C.1 · p. 19, §C.2 · p. 19
C.2 Zero-shot Prompt
- For judgment questions, models may favor the first option when unsure; the authors varied yes/no order and used ICL-style templates without actually providing in-context examples.The LLaVA-style templates differed only in option order.Source: §C.2 · p. 19
C.3 Supervised Fine-tuning Implementation
The appendix describes balanced SFT data and training setups, then examines chart-type performance, fixed CoT guidance, and image-resolution effects.
C.3 Supervised Fine-tuning Implementation
- The SFT baseline manually balances Acc+ and NQA instruction data to prevent prediction bias..Source: §C.3 · p. 19
- For Qwen-VL-Chat, training freezes the vision encoder and uses LoRA on the LLM branch for three epochs, requiring 12 A100 GPU days.The connector is updated along with the LLM parameters rather than separately aligned.Source: §C.3 · p. 19
- InternLM-XComposer-v2 receives two epochs of chart-CSV connector alignment, followed by one epoch of instruction SFT updating the connector and LoRA-trained LLM branch.The full experiment takes approximately 15 A100-GPU days.Source: §C.3 · pp. 19–20
- Table 12 maps prompt templates to MLLMs and reports the best results for each template; its colors identify system prompts, Acc+ instructions, and chart judgments.The ground-truth judgment is bolded.Source: Table 12 · p. 20
D Additional Results
- Additional results cover accuracy metrics, ChartQA by image type, human evaluation, evaluation samples, and GPT-4 sample analyses..Source: D Additional Results · p. 20
D.1 Further Study
- Some MLLMs achieve close to 0% Acc+ on radar or combination charts, and models generally perform better on regular than intricate graphics.The authors note that Acc+ approaches 0% under random guessing; annotated charts make up about 20% of ChartBench.Source: §D.1 · p. 20, Table 13 · p. 21, Table 14 · p. 21
- Fixed chart CoT guides models to identify queried objects and focus on their locations and spatial relations, using common reasoning patterns without specifying chart-specific logic..Source: §D.1 · p. 21
- As resolution increases, scaled annotations become less readable to OCR and model performance declines; Qwen-VL-Chat drops more than LLaVA-v1.5.The comparison uses annotated regular charts adjusted to five resolutions while keeping font size unchanged.Source: §D.1 · p. 21
D.2 Results of Accuracy Metric
The appendix compares accuracy with Acc+, evaluates ChartBench against ChartQA and human ratings, and examines model behavior, chart-type results, and evaluation practices.
D.2 Results of Accuracy Metric
- 55.70% accuracy versus 15.49% Acc+ for InternLM-Xcomposer indicates many correct answers may be guesses; accuracy also stays near 50% across five tasks.Acc+ requires correct judgments for positive and negative assertions; random guessing tends toward 0% when CoR approaches 100%. Even 16.8K judgment QA pairs cannot fully eliminate small-sample effects.Source: §D.2 · pp. 21–22, Table 15 · p. 22, Table 16 · p. 23
D.3 Results of ChartQA
- ChartQA results show chart-tuned models outperforming most MLLMs, but annotated numerical answers can favor OCR and open-ended responses complicate verification.ChartBench instead uses yes/no questions, while ChartQA’s open-ended answers can require human or GPT intervention to verify.Source: Table 17 · p. 23, §D.3 · pp. 23–24
D.4 Results of Human Evaluation
- Human ratings find value extraction harder than other tasks, fine coordinate granularity can lower confidence, and performance varies little across chart types.The authors note that imperfect scores even on straightforward tasks such as chart-type identification may reflect survey descriptions or hurried responses.Source: Table 19 · p. 24, §D.4 · p. 24
D.5 Case study of ChartBench
- Figures 7–8 show NQA examples for unannotated and annotated charts, marking incorrect answers red and correct answers green.Source: Figure 7 · p. 25, Figure 8 · p. 25
D.6 Case study of GPT-4
- GPT-4V combines APIs and can acknowledge limitations, but unavailable OCR weakens visual interpretation and multi-step human guidance constrains zero-shot performance.Source: §D.6 · p. 26, Figure 9 · p. 26
E Ethical Statement
- The authors report de-identifying data, obtaining informed consent, and finding no academic ethical issues or misconduct.Source: E Ethical Statement · p. 26
F Leaderboards
- The appendix provides leaderboards across chart categories and tasks, including Acc+ and CoR scores and results split by annotation status.Source: F Leaderboards · p. 27
F.1 Leaderboards on Chart Type
- Some models approach 0% Acc+ on radar or combination charts, while MLLMs generally perform better on regular charts than on intricate graphics.The authors also note that chart-tuned models’ ChartBench results remain below their ChartQA performance, attributing the gap to OCR reliance rather than robust visual reasoning.Source: §F.1 · p. 27
F.2 Leaderboards on Task Type
The leaderboards show task-specific strengths and weaknesses, substantial annotation-related performance gaps, and ChartBench’s broad chart-type coverage.
F.2 Leaderboards on Task Type
- Most models succeed at chart recognition, but value extraction is the hardest task; closed-source models outperform open-source counterparts, partly due to size and data coverage.LLaVA-v1.5, mPLUG-Owl-bloomz, and Qwen-VL-Chat show advantages on VC and GC, while BLIP2 and ChartLlama struggle with VE.Source: §F.2 · p. 27, Table 21 · p. 29
F.3 Leaderboards on CoR Metric
- 55.06% CoR for MiniGPT-v2 accompanies its top-performing status, while Qwen-VL-Chat has the highest Acc+ and lower CoR than MiniGPT-v2.GPT-4V exceeds ERNIE on Acc+ despite similar CoR; ERNIE performs better on the challenging VE task.Source: §F.3 · p. 27, Table 22 · p. 29
F.4 Leaderboards on with/without Annotated Charts
- Models perform significantly better on annotated than unannotated charts; reported performance gaps are Qwen-VL-Chat (+16.00%) and GPT-4V (+31.39%).The authors say annotated charts elevate Acc+, while unannotated charts intensify ChartBench’s challenge.Source: §F.4 · p. 27, Table 23 · p. 29
G Chart Type Thumbnails
- ChartBench covers 9 major categories and 42 chart subcategories, with thumbnails illustrating chart types; charts avoid direct data labeling to encourage visual reasoning.Source: G Chart Type Thumbnails · p. 30, Figure 10 · p. 30, Figure 11 · p. 31