CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

• Existing evaluations such as FigureQA [26], DVQA [25], and ChartQA [48] use template-based or visually homogeneous charts, limiting diversity and potentially exaggerating progress.The paper positions CharXiv as a more realistic, multifaceted evaluation of chart understanding in MLLMs.Source: §1 · pp. 1–2
• 34.5% accuracy loss shows that SPHINX V2 is highly sensitive to slight question changes, dropping from 63.2% to 28.6% on the same charts.This stress test indicates that conventional benchmark results can overstate open-source models’ robustness relative to proprietary models.Source: §1 · p. 2
• CharXiv contains 2,323 handpicked arXiv charts spanning eight subjects, with manually curated questions and human-validated ground-truth answers.Its short-answer format supports straightforward LLM-based automatic grading without requiring advanced domain knowledge or contextual information.Source: §1 · p. 2
• CharXiv distinguishes descriptive questions about titles, labels, and ticks from reasoning questions requiring comparisons, approximations, and fine-grained visual analysis.This design explicitly disentangles basic visual understanding from higher-level reasoning over complex chart elements.Source: §1 · p. 2
• 47.1% versus 29.2% reasoning accuracy separates GPT-4o from InternVL Chat V1.5, while humans reach 80.5%; descriptive accuracy is 84.5%, 58.5%, and 92.1%, respectively.The reasoning gap between the strongest proprietary and open-source models is 17.9%.Source: §1 · p. 2, Figure 2 · p. 3

2.1 Related Works

• Existing benchmarks generate artificial charts from few predefined types or visually narrow sources, failing to represent real-world chart diversity.FigureQA, DVQA, and PlotQA synthesize stylistically similar charts; ChartQA draws from four visually limited websites, and ChartX procedurally generates predefined chart types.Source: §2.1 · p. 3
• Fixed templates, four predefined ChartBench tasks, and external-knowledge requirements in some MMC questions limit question diversity and chart-grounded reasoning.MMC questions may require mapping legend acronyms to algorithms, so they cannot always be solved from the chart alone.Source: §2.1 · p. 3
• Yes/no-only evaluation in FigureQA and ChartBench does not faithfully reflect natural free-form chart question answering.Binary answers can inadequately represent performance in the general free-form question-answering setting.Source: §2.1 · p. 3

2.2 Open-Source MLLMs Are Sensitive to Perturbations

• The robustness test used 174 original samples plus manually annotated modified-question and arXiv-based modified-chart sets, with balanced yes/no answers and the MathVista protocol.The selected open-source models were trained on DVQA and ChartQA, enabling a direct test of whether benchmark familiarity inflates evaluation scores.Source: §2.2 · pp. 3–4
• 34.5%: SPHINX V2 fell from 63.2% to 28.7% on modified questions, while proprietary models remained near the diagonal and most open-source models degraded.The evaluation compared original, modified-question, and modified-chart sets; the largest drop occurred for SPHINX V2 under modified questions.Source: §2.2 · p. 4, Figure 3 · p. 4

3 CharXiv: A Real-World and Challenging Chart Understanding Benchmark

• CharXiv contains 2,323 real-world charts and more than 10K questions, with 1,000 charts for validation and the remainder for testing.All validation QA pairs are released, while test answers remain private to prevent data leakage.Source: §3 · p. 4

3.1 Chart Curation

• A four-step pipeline combines SigLIP filtering, graduate-student selection, near-duplicate removal, and quality checks to curate the final chart set.Charts are sourced from arXiv preprints across eight subjects and rendered as high-resolution JPEGs; many combine multiple chart types or subplots.Source: §3.1 · p. 5

3.2 Question Construction

• CharXiv pairs each chart with descriptive questions and one manually crafted reasoning question spanning text- and number-based answers requiring visual and numerical reasoning.Descriptive questions use 19 templates across extraction, enumeration, pattern recognition, counting, and compositionality, including one intentionally unanswerable question per chart.Source: §3.2 · pp. 5–6

3.3 Evaluation Metrics

• GPT-4o extracts answers and assigns binary correctness scores instead of exact-match grading, accommodating alternative representations of Greek symbols and mathematical notation.The unambiguous answer design makes CharXiv amenable to automatic grading.Source: §3.3 · p. 6

4.1 Experimental Setup

• The evaluation covers diverse MLLMs requiring at least 448 × 448 input resolution and a MathVista testmini score of at least 36, alongside random and human baselines.The random baseline uses GPT-4o prompted to guess without charts, while human participants receive identical questions and instructions and are evaluated identically to models.Source: §4.1 · pp. 6–7

4.2 Experimental Results

• 47.1% reasoning accuracy for GPT-4o trails humans by 33.4 percentage points, while InternVL Chat V1.5 reaches only 29.2% and every other open-source model remains below 25%.These results expose substantial weaknesses in reasoning over CharXiv’s diverse and challenging charts despite models’ stronger performance on existing benchmarks.Source: Table 3 · p. 7, §4.2 · p. 7
• GPT-4o trails humans by 7.65% on descriptive questions, whereas InternVL Chat V1.5 performs 25.95% worse than GPT-4o and most open-source models answer under 50% correctly.The descriptive gap is smaller for GPT-4o than for open-source models, but descriptive performance remains broadly weak.Source: §4.2 · pp. 7–8

4.3 Analysis

• Models with strong reasoning generally have strong descriptive skills, but lengthy chain-of-thought responses help reasoning mainly when descriptive accuracy is already high.MoAI and Qwen VL Plus score 28.70% and 28.93% descriptively, whereas Mini-Gemini HD Yi 34B and Reka Flash score 52.68% and 56.45% and improve with lengthy responses.Source: §4.3 · p. 8
• 92.86% versus 59.82% versus 5.80% accuracy on counting labeled axis ticks for humans, GPT-4o, and the leading open-source model shows a severe compositional-understanding failure.Twenty of 24 models score below 10% on this task, near the 5.35% random baseline.Source: §4.3 · p. 8
• 25% of descriptive questions are unanswerable, yet models below 80% accuracy show idiosyncratic failures that depend on the requested chart element.For example, IDEFICS 2 Chatty nearly always fails on some title and axis-label questions but exceeds 90% on some line-intersection and legend questions, while IXC 2 shows the reverse pattern for certain tick-value differences.Source: §4.3 · pp. 8–9
• With 6+ subplots, descriptive performance deteriorates by 30%–50% for open-source models and 10%–30% for proprietary models, with no clear reasoning correlation.The authors hypothesize that open-source models are disadvantaged because common chart instruction-tuning datasets lack subplots.Source: §4.3 · p. 9
• Descriptive performance is weakest on physics charts and strongest on electrical engineering and systems science, quantitative finance, and economics, while reasoning varies idiosyncratically by subject.InternVL Chat V1.5 matches GPT-4V at 39.26% on math-domain reasoning but trails by more than 20% in physics and electrical engineering and systems science.Source: §4.3 · p. 9

5 Conclusion

• CharXiv pairs natural charts from arXiv papers with human-curated questions and answers to expose performance gaps among humans, proprietary models, and open-source MLLMs.The benchmark was designed in response to stress-test evidence that existing benchmarks overestimate chart-understanding capabilities.Source: §5 · p. 9
• Human accuracy reaches only 80.5% on reasoning and 92.1% on descriptive questions, although grading issues or participant mistakes may partly explain these limits.Despite these uncertainties, the large gap between humans and existing MLLMs supports CharXiv as an informative measurement of chart understanding.Source: §5 · p. 9
• Expert-curated benchmarks are expensive to construct and difficult to update, whereas automatically generated benchmarks can overestimate existing MLLM capabilities.This creates a tradeoff between evaluation quality and scalability for future benchmark development.Source: §5 · p. 9

A More Related Works

• Although open-source MLLMs have reported performance comparable to or better than proprietary models on chart understanding, CharXiv and modified MathVista subsets show that they generalize poorly and retain a performance gap.The evaluation tests robustness beyond the original chart and question distributions.Source: A More Related Works · p. 19
• Earlier chart benchmarks commonly used synthetic or stylistically consistent charts, while other recent benchmarks are unavailable or widely adopted; CharXiv instead uses more natural, diverse, challenging charts with human-curated QA pairs.Its design is most similar to ChartQA [48] but aims to better reflect general chart-understanding capabilities.Source: A More Related Works · p. 19

B Broader Impacts

• Chart understanding is crucial for MLLMs applied to real-world tasks such as reading scientific papers and interpreting trends in business, healthcare, and economics.Strong chart understanding can help non-experts quickly understand and operationalize information from graphs.Source: B Broader Impacts · p. 19
• Because existing MLLMs struggle to answer chart-related questions reliably, faithful benchmarking is needed to identify and rectify their weaknesses.CharXiv’s complex, real-world charts are intended to guide future MLLM development and benchmarking.Source: B Broader Impacts · p. 19

C Evaluation Results on Test Set

• The test set contains 1,323 charts, compared with 1,000 charts in the validation set.Practitioners are advised to evaluate their models independently on the validation set, while test-set answers remain private.Source: C Evaluation Results on Test Set · p. 20

D Evaluation Results by Subject

• The supplied passages do not report the evaluation results broken down by subject.No passage labeled for the subject-wise evaluation is included in the provided input.Source: C Evaluation Results on Test Set · p. 20

C Evaluation Results on Test Set

• Table 4 reports model evaluation results on the 1,323-chart test set, whose ground-truth answers are privately held.The table distinguishes best and second-place performance within open-source and proprietary models.Source: Table 4 · p. 20, C Evaluation Results on Test Set · p. 20
• Table 4 separates best-in-class performance for open-source and proprietary models and marks second-place results separately.Bold numbers indicate the best result within each class, while underlined numbers indicate second place.Source: Table 4 · p. 20
• The evaluation includes information extraction, enumeration, pattern recognition, counting, and compositionality categories.Category details are provided in Figure 4 and Section 3.2.Source: Table 4 · p. 20
• Some model performances are constrained by input resolution, and Table 4 identifies those models with an asterisk.Further details on the resolution constraint are referenced in Table 12.Source: Table 4 · p. 20

D.1 Descriptive Question Results on Validation Set

• Table 5 compares descriptive-question performance across subjects and marks the best in-class result separately for open-source and proprietary models.The subject label includes Electrical Engineering and Systems Science, abbreviated as Elec. Eng. & Sys. Sci.Source: Table 5 · p. 21

D.2 Reasoning Question Results on Validation Set

• Table 6 presents reasoning-question results broken down by subject.Source: Table 6 · p. 21
• Bold numbers identify the best in-class performance for either open-source or proprietary models.Source: Table 6 · p. 21
• “Elec. Eng. & Sys. Sci.” denotes Electrical Engineering and Systems Science.Source: Table 6 · p. 21

Descriptive Question Results

• Table 7 presents descriptive-task results by year, with bold numbers marking the best performance among open-source or proprietary models.The table uses “Elec. Eng. & Sys. Sci.” as an abbreviation for Electrical Engineering and Systems Science.Source: Table 7 · p. 22

• Table 8 reports reasoning-question results by year and bolds the best in-class performance for open-source or proprietary models.Source: Table 8 · p. 22

F Descriptive Question Results by Question Number on Validation Set

• Tables 9 and 10 report model evaluation results for descriptive questions Q1–Q19 on the validation set, with best in-class performances highlighted separately for open-source and proprietary models.Table 9 covers Q1–Q9, while Table 10 covers Q10–Q19; question-number mappings are provided in Table 14.Source: Table 9 · p. 23, Table 10 · p. 23

G Relationship Between Response Length and Correctness

• Figure 6 analyzes the relationship between models’ generation length and correctness on CharXiv reasoning questions using GPT-4o tokenization and neighboring-bin color encoding.The color encoding aggregates applicable data points from each bin together with the preceding and following two bins.Source: Figure 6 · p. 24

H Run Configurations

• Table 11 specifies run configurations for all evaluated models, including temperature and model-specific settings, while leaving unset values at their defaults.For Qwen models, the evaluation uses Top-P 0.99999 rather than exactly 1 because of API constraints, and model-page code is used whenever possible.Source: Table 11 · p. 25

I Open-Source Model Components

• Table 12 summarizes the visual and language components, input resolutions, and additional vision encoders used by the open-source models evaluated in CharXiv.LLaVA 1.6 supports dynamic aspect-ratio resolution, MoAI uses additional vision encoders as verbalizers, and CharXiv charts average 996 × 702 pixels with a maximum of 1024 × 1024.Source: Table 12 · p. 25

J Model License

• Table 13 summarizes the licenses of the models evaluated in CharXiv, marking entries as “Not Applicable” when no explicit code license appears in the codebase or checkpoint page.The table distinguishes license availability across the evaluated models.Source: Table 13 · p. 26

K Visualization of Sample Charts

• Thirty charts from different evaluation suites are sampled and visualized to illustrate the charts used for model evaluation.Source: K Visualization of Sample Charts · p. 27

L.1 Response Generation

• Descriptive questions target chart elements such as axes, lines, legends, trends, ticks, and subplot layouts, with instructions defining formats and Not Applicable conditions.Examples require ordered legend labels, n by m subplot layouts, exact tick differences, or concise trend descriptions while ignoring irrelevant auxiliary markings.Source: Table 14 · p. 28, §L.1 · pp. 29–30

L.2 Grading

• The grader groups response–ground-truth pairs by question and uses GPT-4o with question-specific templates and in-context examples to assign binary correctness scores.Question categories include OCR, quantitative, Boolean, enumeration, trend, and layout; scores require matching terms or exact numerical values, with Not Applicable handled explicitly.Source: §L.2 · pp. 31–34

M.1 Response Generation

• Response generation applies question-type-specific constraints requiring chart text or numbers when relevant, option compliance, full subplot or label names, and exact integer or decimal formats.Number-in-general answers use the specified decimal-place requirement when applicable, following the design of MathVista [45].Source: §M.1 · p. 39

M.2 Grading

• The grading pipeline extracts each model’s final answer and assigns a binary score against the ground truth, with missing responses receiving 0.For text-in-chart questions, answers must refer to the same term, while text-in-general answers require exact option matching or equivalent semantic meaning when options are absent.Source: §M.2 · pp. 40–42
• Number-in-chart and number-in-general grading gives 1 only when the predicted and ground-truth values are numerically equal, allowing equivalent notation such as 0.01 and 10^-2 or 1500 and 1.5e3.Different numeric values receive 0, including 0.012 versus 0.01.Source: §M.2 · pp. 43–44

N Chart-Free Random Baseline Prompts

• The chart-free random baseline instructs models to guess a reasonable answer from the question alone, sampling numbers within a reasonable range or relevant terms as needed.This baseline does not use chart content in its prompting.Source: N Chart-Free Random Baseline Prompts · p. 44

O.1 Chart Selection

• Annotators select one chart from 10 cosine-similar candidates, then verify descriptive questions and answers, including subplot coordinates and answerability.Candidates are pre-filtered using cosine similarity greater than 0.65 against the average MathVista chart embedding; 18 descriptive tasks are randomly shuffled, excluding automatically converted Q19.Source: Figure 8 · p. 45, Figure 9 · p. 46

O.3 Reasoning Question Annotation

• Reasoning-question annotation combines GPT-4V-generated candidates with human selection or rewriting, final answers, and answer types distinguishing chart and general text or numbers.The answer type guides additional instructions during response generation.Source: Figure 10 · p. 47

R Common Failure Cases of Descriptive Questions

• Models commonly fail to localize subplots, select correct chart elements, perform OCR and counting, identify distant relevant elements, avoid hallucinations, and handle unconventional scenarios.These failures are documented across 30 examples for each descriptive question, covering answerable and unanswerable cases.Source: R Common Failure Cases of Descriptive… · p. 53, Table 15 · p. 53

R.1 Example 1

• Models give conflicting answers on basic subplot titles and axis labels, including row-column localization, absent labels, shared axes, and left-versus-right axis instructions.Examples include disagreement over subplot titles, x-axis labels, y-axis labels, and whether labels are explicitly present.Source: §R.1 · p. 54, §R.2 · p. 55, §R.3 · p. 56

R.5 Example 5

• In one y-axis case, the reference answer is LLC (Ways), while several models instead report MBW (%), LLC (MB/s), Load (Gbps), Freq (GHz), or Load & QoS.The example notes that models incorrectly followed the wrong side when labels appeared on both left and right.Source: §R.5 · p. 58

R.7 Example 7

• For the bottom-right subplot, the reference tick is Mα+, whereas GPT-4o answers -200, illustrating failures on explicit tick-value extraction.The instruction requires using the bottom x-axis and ignoring separately written units or scales.Source: §R.7 · p. 60
• Claude 3 Sonnet identified the leftmost labeled x-axis tick as -Mb.Source: §R.7 · p. 60
• InternVL Chat V1.5 and Reka Core identified the leftmost labeled x-axis tick as -200.Source: §R.7 · p. 60
• MGM HD Yi 34B identified the leftmost labeled x-axis tick as -50.Source: §R.7 · p. 60

Question

• The question targets the explicitly labeled rightmost x-axis tick for subplot row 1, column 5, using the bottom axis when axes are shared or duplicated.Units and separately written scales should be ignored.Source: §R.8 · p. 61

Reference answer

• The reference answer is Concept.Source: §R.8 · p. 61

Model responses

• In row 1, column 5, the rightmost x-axis labels are “Setref Concept” for GPT-4o, “Setref(Concept)” for Claude 3 Sonnet, and “Ref Concept” for IDEFICS 2; Reka Core, InternVL Chat V1.5, and MGM HD Yi 34B show 0.15, 0.05, and 0.0, respectively.Source: §R.8 · p. 61

• The expected answer is “CC daily 4.00PM,” based on the explicitly written y-axis tick and the stated axis-selection rules.The instructions specify using the left axis when both sides are present and ignoring units or scales written separately from the tick.Source: §R.9 · p. 62
• GPT-4o, Claude 3 Sonnet, InternVL Chat V1.5, and MGM HD Yi 34B instead answer 0.2, -1.00, -1.00, and -0.50, respectively.Source: §R.9 · p. 62

R.10 Example 10

• Models disagreed on the spatially highest labeled y-axis tick for the subplot at row 2 and column 1, reporting 0.1, 4, 5, 100, or 2.85 m.Source: §R.10 · p. 63

R.11 Example 11

• For the row 4, column 2 subplot, reported differences between consecutive numerical x-axis ticks varied among 0.01, 0.1, and 0.02 under the stated tick-value conditions.Source: §R.11 · p. 64

R.15 Example 15

• The cited responses reported one line-count answer of 8, while GPT-4o, Claude 3 Sonnet, Reka Core, and InternVL Chat V1.5 each reported five lines; another answered Not Applicable.Source: §R.15 · p. 68, §R.16 · p. 69

R.17 Example 17

• Intersection judgments were similarly inconsistent: R.17’s answer was yes although models answered No, while R.18’s answer was Not Applicable despite Claude 3 Sonnet identifying an intersection.MGM HD Yi 34B explained R.17 by distinguishing a blue line above zero from an orange line below zero, whereas it classified R.18 as a histogram.Source: §R.17 · p. 70, §R.18 · p. 71

R.19 Example 19

• Legend-counting examples produced divergent answers: R.19’s answer was 9 while models reported 6 or 8 labels, and R.20’s answer was Not Applicable while models reported 2 or 3.Models differed on whether externally located or inferred legend entries should count as relevant labels.Source: §R.19 · p. 72, §R.20 · p. 73

R.21 Example 21 / R.22 Example 22

• Models disagree substantially on legend extraction, including whether legends exist and which labels are relevant, as shown across Examples 21 and 22.For Example 21, GPT-4o omits “Setting” while Claude 3 Sonnet, Reka Core, and InternVL include it; Example 22 produces answers ranging from “Not Applicable” to different label sets.Source: §R.21 · p. 74

R.23 Example 23 / R.24 Example 24 / R.25 Example 25 / R.26 Example 26

• Colorbar reasoning varies across models and subplots: reported differences include 1.5 for a 0.8–2.3 range, 2.3 using 0 as the minimum, and 200 for a 0–200 range.Source: §R.23 · p. 76, §R.24 · p. 77

R.29 Example 29 / R.30 Example 30

• Responses for one chart report 10, 11, and 12 subplots, while responses for another report layouts of 4 by 1 and 6 by 1.Source: §R.29 · p. 82, §R.30 · p. 83

S Common Failure Cases of Reasoning Questions

• Representative models rarely make OCR or instruction-following mistakes; most failures are factual errors with or without chain-of-thought reasoning.The analysis categorizes failures as FACT, RSN, OCR, or INST and reports that factual mistakes dominate.Source: S Common Failure Cases of Reasoning… · p. 84
• Zero-shot chain-of-thought behavior differs by model: GPT-4o and Claude 3 Sonnet use it about half the time, Reka Core and MGM always, and InternVL Chat V1.5 and IDEFICS 2 almost never.Reka Core and MGM HD Yi 34B sometimes produce highly similar chain-of-thought prefixes.Source: S Common Failure Cases of Reasoning… · p. 84
• The failure-case overview covers 30 reasoning examples organized into four instruction categories: text-in-chart, text-in-general, number-in-chart, and number-in-general.Table 16 defines the four categories as TC, TG, NC, and NG.Source: Table 16 · p. 84, S Common Failure Cases of Reasoning… · p. 84

S.1 Example 1

• In a column-summation example, models select different minima, with GPT-4o and Reka Core answering DICE, Claude 3 Sonnet answering DICR, and InternVL Chat V1.5 answering Cooperative.Reka Core explicitly computes DICE’s total as 3.8, the smallest among its listed column totals.Source: §S.1 · p. 85
• The Cooperative column appears to have the smallest total value, but illegible cell values prevent exact calculation and require visual estimation from color intensity.The answer is based on the lightest blue shades and the chart’s scale rather than summing precise cell values.Source: §S.1 · p. 85

S.2 Example 2

• Models disagree on the y-axis label for the subplot with the smallest blue-dot clusters, giving CaII, Hz, RV [m/s], Ha, and Call.Claude 3 Sonnet identifies Hz, while the other models provide different labels.Source: §S.2 · p. 86

S.3 Example 3

• Models estimate the first matching Theory frequency as 300 cm−1, 700cm−1, 480cm−1, or 600cm−1 after rounding or visual comparison.GPT-4o reports an unrounded estimate around 270 cm−1 and rounds it to 300 cm−1; other models give different visual estimates.Source: §S.3 · p. 87

S.4 Example 4

• Models identify different subplots as showing the greatest agreement among the three quartile curves, including row 3, column 2; row 2, column 2; row 1, column 1; and row 2, column 3.The provided answer is row 2, column 4, while model responses vary across the grid.Source: §S.4 · p. 88

S.5 Example 5

• Four models report a maximum decreasing PACF sequence of 3, whereas GPT-4o reports 2 between the first and second lags.The models otherwise place the sequence at different data points, including the third-to-fifth and tenth-to-twelfth points.Source: §S.5 · p. 89

S.6 Example 6

• Most models identify λL = 0.30 as the line furthest from its fi value on the WH axis, but MGM HD Yi 34B instead identifies the red ni line.Reka Core characterizes λ = 0.30 as a green dashed line and distinguishes it from the λ = 0.13 and λ = 0.20 lines.Source: §S.6 · p. 90

S.7 Example 7

• Models disagree on the highest q value maintaining near-zero cooperation, answering approximately 3, 4.5 or higher, 5, and 2.5.GPT-4o gives approximately 3, whereas Claude 3 Sonnet, Reka Core, InternVL Chat V1.5, and MGM HD Yi 34B provide different thresholds.Source: §S.7 · p. 91

S.8 Example 8; S.11 Example 11

• Categorical bar-chart questions also produce conflicting answers: CT/SDBN’s least category is variously Score = 0 or Score = 5, and the highest Posterior environment is Maze, Forest, BLOB, or TwoWall.These responses show disagreement across both category identification and bar-length comparison.Source: §S.8 · p. 92, §S.11 · p. 95

S.9 Example 9

• Fashion-MNIST’s highest reported accuracy varies from 93.4% to 99.4% across model answers, with GPT-4o reporting 99.1%.Reka Core reports 95.9%, while InternVL Chat V1.5 and MGM HD Yi 34B report 99.4%.Source: §S.9 · p. 93

S.10 Example 10

• For TIMIT training loss below 10^-1, MomentumLSTM is consistently named first, but the second variant differs between AdamLSTM, RMSPropLSTM, and SRLSTM.Several answers identify MomentumLSTM and AdamLSTM, while GPT-4o, InternVL Chat V1.5, and IDEFICS 2 pair it with RMSPropLSTM.Source: §S.10 · p. 94

S.12 Example 12; S.16 Example 16

• At objectness threshold 0.9, models estimate IA-Yolo’s ExDark false-negative detections between approximately 2000 and 3000; for the 0%-5% category, most name the 1st decile.Claude 3 Sonnet instead selects the 10th decile, while GPT-4o, Reka Core, and MGM HD Yi 34B select the 1st.Source: §S.12 · p. 96, §S.16 · p. 100

S.13 Example 13

• For the category with the lowest win rate in most subplots, answers split among Object Recognition, Reasoning, Others, and Attribute Description.GPT-4o supports Reasoning with subplot values of 37.1%, 22.1%, and 56.5%, whereas other models select different categories.Source: §S.13 · p. 97

S.14 Example 14

• Estimated interquartile ranges range from 0.16 to 0.4, reflecting divergent readings of the box-plot quartiles.GPT-4o computes 0.16 after rounding, Claude 3 Sonnet and Reka Core report 0.20, and MGM HD Yi 34B reports 0.4.Source: §S.14 · p. 98

S.15 Example 15

• For ΔNPS values worse than -9%, GPT-4o and Claude 3 Sonnet count 1 point, while Reka Core and MGM HD Yi 34B count 3 and 2.The disagreement concerns which plotted points fall strictly below the -9% threshold.Source: §S.15 · p. 99

S.17 Example 17; S.20 Example 20

• Some examples showed convergence on a chart answer, such as Italy’s post-April search bounce and h5’s estimated curve remaining above the true curve from x = 400 to 450, although MGM HD Yi 34B answered h10.Claude 3 Sonnet, Reka Core, and other models identified Italy or h5, while one model diverged on the curve question.Source: §S.17 · p. 101, §S.20 · p. 104

S.18 Example 18; S.19 Example 19

• Answers varied substantially on which chart element was largest or most divergent, with models naming IT, US, or ALL for S.18 and Linear, Log, or Quad-DLM for S.19.These disagreements concern greatest cumulative-response divergence and the largest United States percentage across models.Source: §S.18 · p. 102, §S.19 · p. 103

S.19 Example 19

• For the S.19 question, the answer should be “Quad,” and excluding “Misalignment” still yields 0% accuracy among all models.Source: §S.19 · p. 103

S.21 Example 21; S.24 Example 24; S.25 Example 25

• Models also conflicted on relational chart-reading tasks, including the majors concentrated in fifth-year completion, the number of arrows positive in both indices, and the plot where all lines decrease.Reported answers include Economics and Law versus alternatives, 3–5 arrows, and several competing plot labels.Source: §S.21 · p. 105, §S.24 · p. 108, §S.25 · p. 109

S.22 Example 22

• Models disagreed on the first Survey of Consumer Finances data year, answering approximately 1989, 2020, or 1910 despite post-analysis identifying 1989 as the chart’s starting year.The post-analysis notes that one chart starts in 1989 based on the SCF website and that models still ignored chart contents.Source: §S.22 · p. 106

S.26 Example 26

• For S.26, Reka Core and MGM HD Yi 34B identify the Positive, Low λ curve as having the smallest welfare-increase rate between t=1 and t=2.Both responses justify the answer by comparing curve slopes; IDEFICS 2 provides only “Inst. High A.”Source: §S.26 · p. 110

S.27 Example 27

• For S.27, answers to the histogram-peak question range from 1 to 4 sources, with GPT-4o and Claude 3 Sonnet reporting 1.Reka Core reports 3, InternVL Chat V1.5 reports 2, and MGM HD Yi 34B reports 4.Source: §S.27 · p. 111

S.28 Example 28

• For S.28, models disagree sharply on the number of Serviced K users with meters values less than 5, giving 3, 4, 10, or 18.GPT-4o counts 18 blue crosses, Claude 3 Sonnet counts 4 blue markers, Reka Core counts 3, and MGM HD Yi 34B counts 10.Source: §S.28 · p. 112

S.29 Example 29

• For S.29, four models place the third all-curve intersection around ω tC = 6, whereas InternVL Chat V1.5 answers 0.5.GPT-4o, Claude 3 Sonnet, Reka Core, and MGM HD Yi 34B converge on the labeled value 6.Source: §S.29 · p. 113

S.30 Example 30

• For S.30, four models identify [0, 100] as the interval with the largest area, while InternVL Chat V1.5 says all intervals are equal.The [0, 100] responses attribute the result to the signal’s initial overshoot and sharp rise.Source: §S.30 · p. 114

T Datasheets for Datasets / T.1 Motivation

• CharXiv addresses the lack of diverse chart-question evaluation data by providing a realistic benchmark for multimodal large language models, following Datasheets for Datasets by Gebru et al..Graduate students from Princeton University curated the chart selections and question-answer annotations.Source: §T.1 · p. 115, T Datasheets for Datasets · p. 115

T.2 Composition

• CharXiv contains 2,323 arXiv charts, 19 unique descriptive questions, and 2,323 unique reasoning questions, with four descriptive and one reasoning question paired to each chart.Charts are sampled from 2020–2023 arXiv figures, re-rendered to JPEG with a 1024px long side, and split into 1,000 validation and 1,323 test charts without a training set.Source: §T.2 · pp. 115–116

T.3 Collection

• The data were collected from publicly available arXiv preprints, manually selected and randomly sampled under reproducible rules, then annotated with human-created questions and answers.Chart data were collected in November 2023, and questions were annotated in April 2024; human validation was used for all question-answer pairs.Source: §T.2 · p. 116, §T.3 · p. 117

T.4 Preprocessing / Cleaning / Labeling

• Figures were re-rendered, resized, and manually screened as charts, while questions were manually curated using LabelStudio.Original raw data remain available on arXiv, with relative paths provided for each chart.Source: §T.4 · pp. 118–119

T.5 Uses

• CharXiv is intended solely to evaluate models in open-vocabulary chart understanding and should not be used to train models.Because its charts come from 2020–2023 preprints, visual representations may become outdated if charts change substantially.Source: §T.5 · p. 119

T.6 Distribution / U Misc.

• The dataset is publicly distributed for research evaluation, with question-answer pairs on GitHub and charts on Hugging Face, while chart copyrights remain with the original preprint authors.The created questions and answers use the CC BY-SA 4.0 license, and the code uses Apache 2.0.Source: §T.6 · pp. 119–120, U Misc · p. 121

T.7 Maintenance

• The CharXiv authors host and maintain the dataset, plan updates every 3–6 months, and provide mechanisms for public extension and adaptation under CC BY-SA 4.0.The initial release will be updated with future errata through the CharXiv project page.Source: §T.7 · p. 120

U Misc.

• CharXiv stores 2,323 JPEG charts separately from validation and test JSON files containing metadata, questions, answers, and grading instructions.The metadata maps charts to their years, subjects, original arXiv paths, captions, preprint identifiers, and titles; the benchmark and Croissant metadata are publicly linked.Source: U Misc · p. 121