PaperlayerView exact revision on arXiv

Source-linked AI summary

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, Danqi Chen

arXiv:2406.18521v1cs.CLcs.CV

TL;DR

Existing chart benchmarks often rely on oversimplified, homogeneous charts and template-based questions, limiting realistic evaluation of MLLM chart understanding. CharXiv introduces a human-curated suite of 2,323 diverse arXiv charts with descriptive and reasoning questions. GPT-4o achieved 47.1% reasoning accuracy versus 29.2% for InternVL Chat V1.5, while humans achieved 80.5%.

  • Problem

    Existing chart benchmarks have limited chart diversity and often use template-based questions, making chart-understanding evaluation less realistic.

  • Method

    CharXiv evaluates MLLMs on 2,323 natural arXiv charts using human-curated descriptive and reasoning questions with hand-validated answers.

  • Results

    47.1% versus 29.2%: GPT-4o outperformed InternVL Chat V1.5 on reasoning questions, while both remained below human accuracy of 80.5%.

  • Takeaways & Limitations

    CharXiv exposes substantial gaps in chart understanding across human, proprietary, and open-source systems and provides a more realistic evaluation of MLLMs.

  • Takeaways & Limitations

    Human-expert-curated evaluation benchmarks are expensive to construct and difficult to update or extend.

Abstract

from arXiv · show

Chart understanding plays a pivotal role when applying Multimodal Large Language Models (MLLMs) to real-world tasks such as analyzing scientific papers or financial reports. However, existing datasets often focus on oversimplified and homogeneous charts with template-based questions, leading to an over-optimistic measure of progress. We demonstrate that although open-source models can appear to outperform strong proprietary models on these benchmarks, a simple stress test with slightly different charts or questions can deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a comprehensive evaluation suite involving 2,323 natural, challenging, and diverse charts from arXiv papers. CharXiv includes two types of questions: 1) descriptive questions about examining basic chart elements and 2) reasoning questions that require synthesizing information across complex visual elements in the chart. To ensure quality, all charts and questions are handpicked, curated, and verified by human experts. Our results reveal a substantial, previously underestimated gap between the reasoning skills of the strongest proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%. All models lag far behind human performance of 80.5%, underscoring weaknesses in the chart understanding capabilities of existing MLLMs. We hope CharXiv facilitates future research on MLLM chart understanding by providing a more realistic and faithful measure of progress. Project page and leaderboard: https://charxiv.github.io/

1 Introduction

CharXiv addresses weaknesses in existing chart benchmarks by evaluating MLLMs on diverse, challenging charts and questions, revealing that apparent open-source advantages can disappear under modest changes. Its results show substantial gaps among proprietary models, open-source models, and humans, especially on reasoning questions.

  • Benchmark limitations: Existing chart benchmarks use limited chart types, homogeneous visual sources, or procedurally generated question templates.FigureQA and DVQA rely on generated templates, while ChartQA draws from only a few visually homogeneous sources.
  • Stress-test evidence: 34.5%: SPHINX V2 accuracy dropped from 63.2% to 28.6% when questions were slightly modified for the same charts.The stress test found that open-source models were less robust than proprietary models to small visual or textual changes.
  • CharXiv: 2,323: CharXiv contains natural charts handpicked from scientific papers across eight major arXiv subjects.It separates descriptive questions about basic chart elements from reasoning questions involving comparisons, approximations, and fine-grained analysis.
  • Results: 47.1% versus 29.2%: GPT-4o outperformed InternVL Chat V1.5 on CharXiv reasoning questions.The reported reasoning-performance gap between the strongest proprietary and open-source models was 17.9 percentage points.
  • Results: 80.5%: human reasoning accuracy exceeded both GPT-4o and InternVL Chat V1.5 on CharXiv.The benchmark also reports human descriptive accuracy of 92.1%, while GPT-4o reached 84.5% and InternVL Chat V1.5 reached 58.5% on descriptive questions.

2 Existing Benchmarks Overestimate Chart Understanding Capabilities

Existing chart benchmarks rely heavily on synthetic or visually narrow charts, templated or constrained questions, and limited answer formats. Stress tests show that these design choices can substantially overestimate open-source models’ chart-understanding capabilities.

  • Chart sources: Synthetic benchmarks restrict charts to a narrow distribution, while ChartQA draws from only four visually limited websites.FigureQA, DVQA, and PlotQA use predefined chart types with stylistically similar elements; ChartX procedurally generates charts in bulk.
  • Question types: Existing benchmarks provide limited question variation through fixed templates or predefined tasks, while some reasoning questions require external domain knowledge.FigureQA, DVQA, and PlotQA use fixed templates, and ChartBench uses four predefined tasks.
  • Answer formats: Binary yes/no evaluation in FigureQA and ChartBench does not faithfully reflect natural free-form chart question answering.Both benchmarks evaluate model performance only with yes/no questions.
  • Benchmark validity: Evaluation data can resemble training data too closely, causing benchmark scores to misrepresent general chart-understanding capabilities.Many open-source models were trained on existing benchmark data, and simple evaluation modifications can produce drastic performance changes.
  • Stress tests: 34.5%: SPHINX V2 accuracy fell from 63.2% to 28.6% when questions were slightly modified for the same charts.The stress test demonstrates that open-source models can lag behind proprietary models in robustness to small textual or visual changes.

3 CharXiv: A Real-World and Challenging Chart Understanding Benchmark

CharXiv is a manually curated benchmark of diverse, naturally occurring arXiv charts paired with descriptive and reasoning questions. Its construction emphasizes visual complexity, question ambiguity control, and evaluation procedures compatible with automatic grading.

  • Dataset scope: 2,323 charts and more than 10K questions form CharXiv’s real-world chart-understanding benchmark.The charts come solely from diverse, naturally occurring, complex figures in arXiv preprints; 1,000 charts are sampled for validation and the remainder used for testing.
  • Chart curation: CharXiv charts are extracted from arXiv figures and curated through visual filtering, expert selection, similarity removal, and labeling checks.The selection pipeline uses SigLIP-based filtering, graduate-student review, pairwise similarity removal, and removal of unclear or inappropriate charts.
  • Chart complexity: CharXiv charts are compositionally complex, often combining multiple chart types or subplots within a single figure.Examples include plots that combine lines and bars, making the visual structure more challenging than existing datasets.
  • Question construction: Descriptive questions test extraction and aggregation of basic chart information, including labels, ticks, counts, and patterns.The dataset uses 19 descriptive templates spanning information extraction, enumeration, pattern recognition, counting, and compositionality.
  • Question construction: Reasoning questions are manually selected or authored per chart, must have definite answers, and cover text- and number-based visual reasoning.Each chart receives one reasoning question chosen or modified by graduate-student annotators from GPT-4V samples or created directly.
  • Evaluation: GPT-4o assigns binary correctness scores after extracting answers, accommodating alternate renderings of symbols and mathematical notation.Exact matching is avoided because equivalent expressions such as Greek symbols and LaTeX forms can be typed differently.

4 Experiments

CharXiv evaluates diverse chart understanding with descriptive and reasoning questions, revealing substantial weaknesses across multimodal models. Proprietary models outperform open-source models, while performance deteriorates on compositional tasks, unanswerable questions, additional subplots, and some subject domains.

  • Experimental setup: 13 open-source and 11 proprietary multimodal models are evaluated alongside text-only and human baselines on CharXiv.Validation results are reported in Table 3, with additional test results and failure cases provided elsewhere.
  • Overall results: 25.95% is the performance drop of InternVL Chat V1.5 relative to GPT-4o on descriptive questions, while most open-source models answer fewer than 50% correctly.GPT-4o lags human descriptive performance by 7.65%.
  • Compositionality: 92.86%, 59.82%, and 5.80% accuracy are achieved by the leading open-source model, GPT-4o, and humans respectively when counting labeled axis ticks.Twenty of 24 models score below 10% on this task, close to the 5.35% random baseline.
  • Robustness and domain variation: Models show idiosyncratic failures on unanswerable questions and different subject domains, including weaker descriptive performance on physics-related charts.InternVL Chat V1.5 matches GPT-4V at 39.26% on math reasoning questions but has gaps greater than 20% in physics and electrical engineering and systems science.
  • Subplots: 30%–50% deterioration occurs for open-source models and 10%–30% for proprietary models on descriptive questions with 6+ subplots.Reasoning performance shows no clear correlation with the number of subplots.

5 Conclusion

The paper presents chart understanding as a crucial visual reasoning capability for MLLMs and introduces CharXiv to measure it with realistic charts and human-curated questions. It reports gaps among humans, proprietary models, and open-source models while noting construction and evaluation limitations.

  • CharXiv pairs charts collected from arXiv papers with human-curated questions and answers.
  • The results expose clear performance gaps across human, proprietary-model, and open-source-model performance.
  • Human-expert-curated evaluation benchmarks are expensive to construct and difficult to update or extend.
  • Automatically generated benchmarks often overestimate existing MLLMs’ capabilities.
  • Chart understanding supports increasingly difficult real-world tasks such as reading and summarizing scientific papers.

C Evaluation Results on Test Set

The test-set evaluation reports model results in task categories and identifies best and second-place performance within open-source and proprietary groups. Input resolution constrains some models.

  • Table 4 reports model evaluation results on the test set.
  • Bold numbers indicate the best in-class performance, while underlined numbers indicate second-place performance.
  • Models marked with an asterisk have performance constrained by input resolutions.
  • The evaluation categories include information extraction, enumeration, pattern recognition, counting, and compositionality.

D.1 Descriptive Question Results on Validation Set

The validation-set analysis reports descriptive-question results by subject, distinguishing best performance within open-source and proprietary model groups.

  • Table 5 reports results by subject on descriptive questions.
  • Bold numbers represent the best performance within the open-source or proprietary class.
  • Electrical Engineering and Systems Science is abbreviated as Elec. Eng. & Sys. Sci.

D.2 Reasoning Question Results on Validation Set

The validation-set analysis reports reasoning-question results by subject, distinguishing best performance within open-source and proprietary model groups.

  • Table 6 reports results by subject on reasoning questions.
  • Bold numbers represent the best performance within the open-source or proprietary class.
  • Electrical Engineering and Systems Science is abbreviated as Elec. Eng. & Sys. Sci.

E.1 Descriptive Question Results on Validation Set

The validation results are organized by publication year for descriptive tasks, with best in-class performance highlighted across open-source and proprietary models.

  • The table reports descriptive-task results broken down by year.
  • Bold numbers identify the best performance within each model class.
  • “Elec. Eng. & Sys. Sci.” abbreviates Electrical Engineering and Systems Science.

E.2 Reasoning Task Results on Validation Set

The validation reasoning results are organized by publication year, with bold values marking the best performance among open-source and proprietary models.

  • The table reports reasoning-question results broken down by year.
  • Bold numbers identify the best performance within each model class.
  • The reported breakdown concerns reasoning questions rather than descriptive tasks.

F Descriptive Question Results by Question Number on Validation Set

Descriptive validation results are reported both by question number and through supporting details on response lengths, model configurations, inputs, and licensing.

  • Table 9 reports descriptive-task results for question numbers Q1–Q9.The mapping from question numbers to contents is provided in Table 14.
  • Table 10 reports descriptive-task results for question numbers Q10–Q19.The mapping from question numbers to contents is provided in Table 14.
  • Figure 6 relates model response generation length to correctness on reasoning questions.Response lengths are calculated with the GPT-4o tokenizer, and colors aggregate points across neighboring bins.
  • Run configurations document model settings, including temperature and model-specific API constraints.For Qwen models, the configured Top-P value is 0.99999 rather than exactly 1.
  • The model summary lists visual and language components, input resolutions, and CharXiv chart dimensions.Charts average 996 × 702 pixels and have a maximum size of 1024 × 1024.
  • Table 13 summarizes licenses for the evaluated models.“Not Applicable” indicates that no explicit code license was displayed in the codebase or model checkpoint page.

K Visualization of Sample Charts

The appendix presents sample-chart visualization and descriptive-question instructions, alongside response-generation and GPT-4o-based grading procedures.

  • Visualization: Thirty charts sampled from different evaluation suites are visualized for comparison.
  • Question instructions: Descriptive questions use subplot prefixes for multi-panel charts and append the corresponding instruction after each question.
  • Question instructions: Instructions specify concrete chart operations such as counting lines, reading ticks, identifying legends, and describing trends.
  • Question instructions: Answers are normalized with task-specific rules, including ordered legend labels, “Not Applicable” cases, and n by m subplot layouts.
  • Grading: Model responses are grouped by question number and graded by GPT-4o using response–ground-truth pairs, answer extraction, and binary correctness scores.The grading rubric accepts equivalent notation and requires “Not Applicable” when that is the ground truth.
  • Question instructions: The question set covers OCR, quantitative, Boolean, enumeration, and trend answer formats.Question 1 concerns titles; Questions 2–7 use OCR; Questions 8–10, 12, 14, 15, 17, and 19 are quantitative; Question 11 is Boolean; Question 13 is enumerative; and Question 16 concerns trends.

O Data Annotation Platform

CharXiv’s annotation workflow uses LabelStudio and human review to select charts and curate descriptive and reasoning questions with verified answers.

  • Chart selection: Annotators select one chart from 10 candidates pre-filtered by cosine similarity above 0.65 to the average MathVista chart embedding.The selection process narrows candidates before human choice.
  • Descriptive annotation: For descriptive tasks, annotators review a shuffled list of 18 GPT-generated questions, selecting answerable and unanswerable items with ground-truth answers.Question Q19 can be automatically converted from subplot-count metadata.
  • Reasoning annotation: For reasoning tasks, annotators choose or create a final question from GPT-4V-generated candidates and record its answer type.Answer types distinguish text or number answers located in the chart or general knowledge.
  • Example questions: The annotation examples include questions and answers from DVQA, FigureQA, and ChartQA, covering counts, comparisons, and numerical aggregation.These examples illustrate the source formats reviewed during annotation.

R Common Failure Cases of Descriptive Questions

Descriptive-question failures span subplot localization, chart-element selection, OCR, counting, hallucination, and unconventional cases, with examples showing varied model outputs.

  • Common failure patterns: Models fail to localize the requested subplot when charts contain many subplots.The failure inventory identifies this issue across multiple descriptive examples.
  • Common failure patterns: Models use incorrect chart elements, make OCR mistakes, or miss relevant elements that are distant from the target subplot.These errors affect both extracting explicit labels and identifying the relevant visual region.
  • Common failure patterns: Models hallucinate answers, fail on tricky or unconventional scenarios, and make counting errors.The reported failure cases include both answerable and unanswerable descriptive questions.
  • Label and title extraction: For subplot titles, model answers vary among “Avg Price ($/Bbl),” “FOMC ann.,” “Powell Q&A,” and other responses for the same target.The examples demonstrate inconsistent extraction of explicitly written subplot titles.
  • Label and title extraction: For axis labels, models return conflicting values such as “Not Applicable,” “Hα,” “Ha,” “BJD-245000,” and “U” for specified subplots.Some responses follow the explicit-label instruction while others provide different labels.
  • Instruction following: Several models incorrectly answer a y-axis-label question because they do not follow the instruction to use the left label when both sides are labeled.The reported example contrasts the required left-side label with multiple alternative model answers.

R.8 Example 8

The example asks for the rightmost labeled x-axis tick in subplot row 1, column 5, and the reported model answers differ substantially.

  • Question and answer: The annotation specifies “Concept” as the answer for the rightmost labeled x-axis tick in subplot row 1, column 5.The instruction says to use the explicitly written tick and ignore separately written units or scales.

R.9 Example 9

These examples test extraction of y-axis ticks and x-axis tick spacing, with model responses often disagreeing with the annotated answers or with one another.

  • X-axis tick spacing: The annotated x-axis spacing answer for subplot row 4, column 2 is 0.001, whereas model responses include 0.01, 0.1, and 0.02.The question requires a constant difference between consecutive numerical ticks.
  • Reasoning example: In another reasoning example, MGM HD Yi 34B states that the two lines in subplot row 1, column 5 do not intersect because one remains above zero and the other below.The explanation identifies the sign separation as the basis for the comparison.

S Common Failure Cases of Reasoning Questions

The failure cases show that reasoning errors are primarily factual mistakes, with models struggling to extract and synthesize chart text, numbers, and visual relationships. Examples span text-in-chart, text-in-general, number-in-chart, and number-in-general instructions, producing divergent answers across models.

  • Failure categories: Models rarely make OCR or instruction-following mistakes; instead, they more often make factual mistakes with or without Chain-of-Thought reasoning.The analysis distinguishes factual mistakes without CoT from factual mistakes with CoT.
  • Failure categories: 30 examples are organized into four failure categories: text-in-chart, text-in-general, number-in-chart, and number-in-general.These categories are abbreviated TC, TG, NC, and NG.
  • Chart-reading disagreements: Answers diverge on chart-reading tasks, including identifying the smallest-total column, with models selecting Cooperative, DICR, and DICE.Other responses include a numerically worked comparison for DACE and DACR, illustrating inconsistent column aggregation.
  • Chart-reading disagreements: Models also disagree when locating visual patterns, such as the subplot where blue dots form the smallest clusters, producing labels including CaII, Hz, RV [m/s], Ha, and Call.The differing labels correspond to different subplot selections or interpretations of the same visual question.
  • Visual and numerical reasoning: For curve-comparison questions, models give substantially different frequency estimates, including 300 cm−1, 700cm−1, 480cm−1, and 600cm−1.The question asks for the first frequency where the Theory curve approximately matches the Experiment curve’s intensity at 320 cm−1.
  • Visual and numerical reasoning: One model identifies q approximately 3 as the highest value where cooperation remains near 0 across the entire p range in subplot (a).The response uses the dark red color as the visual indication of near-zero cooperation.

T Datasheets for Datasets

CharXiv is a benchmark of real-world arXiv charts and curated question-answer pairs designed to evaluate multimodal models on diverse chart understanding tasks. Its documentation describes the dataset’s composition, sampling, splits, acquisition, and known constraints.

  • Dataset composition: The dataset uses charts from arXiv preprints and includes raw chart images with curated question-answer texts.Charts are re-rendered from vector-based source files to JPEG where possible and resized to a 1024-pixel longer side.
  • Dataset composition: CharXiv contains 2,323 charts, 19 unique descriptive questions, and 2,323 unique reasoning questions, with four descriptive and one reasoning question paired to each chart.Every question has a clear short answer.
  • Splits: The dataset is split into 1,000 validation charts and 1,323 test charts, with no training set because it is intended as a benchmark rather than training data.The validation-set size is intended to keep performance-comparison variance small.
  • Data governance: The dataset is self-contained and contains no confidential, offensive, sensitive, or directly identifying information, but chart assets remain subject to their original copyrights.Questions and answers created by the authors are licensed under CC BY-SA 4.0.
  • Sampling and acquisition: Chart selection follows manual rules for eligible preprints and chart properties, followed by random sampling with a reproducibility seed.The charts come from preprints published between 2020 and 2023; collection occurred in November 2023.
Loading 2406.18521v1…