Source-linked AI summary
ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering
Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
TL;DR
Existing chart QA benchmarks lack real-world diversity and have shown performance saturation, leaving it unclear whether chart understanding is solved and whether open-source models match closed-source systems. ChartQAPro addresses this gap with a diverse benchmark of complex charts and varied question types, plus broad model evaluation and analyses. Models show substantial performance drops on ChartQAPro, indicating that chart reasoning remains challenging.
Problem
Existing chart QA benchmarks lack diverse sources, complex layouts, and varied real-world question types, limiting evaluation of chart understanding and reasoning.
Method
ChartQAPro compiles diverse charts and human-verified questions, including multi-chart, text-linked, hypothetical, conversational, and unanswerable queries, and evaluates LVLMs with error analyses and ablation studies.
Results
ChartQAPro reveals substantial performance declines across evaluated models; Claude Sonnet 3.5 falls from 90.50% on ChartQA to 55.81% on ChartQAPro.
Takeaways & Limitations
ChartQAPro shows that chart reasoning remains an unsolved challenge and identifies areas for improving multimodal chart understanding.
Takeaways & Limitations
The benchmark focuses on chart question answering and excludes other chart tasks such as chart-to-summary generation and chart-to-code translation.
Abstract
from arXiv · showhide
Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 157 diverse sources, spanning various chart types, including infographics and dashboards, and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
1 Introduction
Existing chart QA benchmarks offer limited diversity and simple question types, while CHARTQAPRO targets more realistic visual and analytical complexity. Evaluations show that this expanded challenge exposes substantial weaknesses in current LVLM chart reasoning.
- Benchmark limitations: Existing benchmarks mainly use charts from limited sources and focus on bar, line, and pie charts with directly labeled numeric values.These design choices reduce the need for visual reasoning.
- Benchmark limitations: Prior chart QA evaluations largely emphasize factoid questions involving simple data extraction or basic arithmetic.Real-world chart use also includes hypothetical, multiple-choice, conversational, and unanswerable questions.
- CHARTQAPRO: CHARTQAPRO contains 1,341 charts from 157 diverse online platforms and 1,948 human-written, human-verified questions spanning multiple real-world query types.The benchmark includes factoid, multiple-choice, conversational, hypothetical, multi-chart, and unanswerable queries.
- CHARTQAPRO: Its complex visualizations include multi-chart layouts, infographics, and dashboards, with some questions requiring multi-turn interaction or accompanying-text references.These features broaden the multimodal reasoning skills being evaluated.
- Evaluation: Claude Sonnet 3.5’s accuracy falls from 90.50% on ChartQA to 55.81% on CHARTQAPRO, while open- and closed-source models show sharp performance drops.The evaluation indicates substantial room for improvement in LVLM chart reasoning.
- Contributions: The benchmark combines broad evaluation with qualitative error analyses and ablation studies to identify challenges and future directions for LVLM chart reasoning.These analyses are presented as part of the paper’s contributions.
2 Related Work
Related chart-understanding benchmarks span question answering, summarization, fact-checking, and explanation generation, but chart QA remains the dominant evaluation task. Existing QA datasets still have limited sources and question types, motivating CHARTQAPRO’s broader design.
- Chart Understanding Datasets: Chart-understanding benchmarks cover question answering, chart summarization, fact-checking, and explanation generation.Chart question answering is the most commonly used evaluation task among them.
- Chart Understanding Datasets: Early benchmarks such as STL-CQA and Leaf-QA relied on synthetic charts and templated questions.Later datasets introduced real-world charts and more complex visual reasoning questions.
- Chart Understanding Datasets: Existing real-world chart benchmarks extract charts from limited sources, cover few question types, and have reached performance saturation with strong LVLMs.CHARTQAPRO instead uses 157 diverse online domains and human-written, verified questions across multiple types.
- Vision-Language Models for Charts: Closed-source models lead recent chart benchmarks, while open-source general multimodal models are rapidly narrowing the gap and chart-specific models perform strongly on standard benchmarks.The related work notes that chart-specific models’ generalization to real-world settings remains an open concern.
3 THE CHARTQAPRO BENCHMARK
ChartQAPro is constructed to broaden chart question answering beyond limited chart sources, formats, and simple factoid questions. It combines diverse real-world charts with human- and VLM-assisted question creation, review, and linguistic analysis.
- Dataset Construction: The dataset construction pipeline has three stages: chart image collection, question-answer annotation, and question-answer review.
- Visual Diversity: ChartQAPro contains 1,341 chart images from 157 online platforms, including dashboards, infographics, multi-series line charts, and grouped or stacked bar charts.
- Question Types: Its 1,948 human-written, human-verified question-answer pairs cover reasoning, conversational, multiple-choice, hypothetical, fact-checking, multi-chart, and unanswerable queries.
- Question Types: Question creation combines annotator-written seed pairs, expansion by GPT-4o, Gemini, and Claude, and manual refinement to remove simple or unclear questions.
- Visual Diversity: The benchmark spans topics including politics, economy, health, environment, and technology, with web-crawled charts comprising 74% and Tableau charts 14%.
- Linguistic Diversity: CHARTQAPRO is linguistically richer than comparison datasets, with 6,638 unique question tokens, 1,496 answer tokens, and the highest reported text type-token ratio of 0.15.
4 Experiments
Experiments evaluate models on multimodal chart question answering across prompting strategies, question types, and chart conditions. Results show substantial performance gaps and distinct weaknesses involving visual perception, reasoning, instruction following, and context.
- Main Results: Table 3 evaluates models across Direct, Chain-of-Thought, and Program-of-Thought prompting and five question types using accuracy.
- Main Results: 55.81% is Claude Sonnet 3.5’s highest accuracy, achieved with Chain-of-Thought prompting on CHARTQAPRO.
- Main Results: Closed-source models consistently outperform open-source counterparts, while Chain-of-Thought and Program-of-Thought generally improve their overall accuracy.
- Main Results: Conversational, hypothetical, and factoid questions are most difficult, whereas fact-checking and multiple-choice questions achieve relatively higher accuracy.
- Main Results: Open-source models below 7B parameters often fall below 20% overall accuracy, while Qwen2-VL-7B and InternVL-2.5-8B exceed 30%.
- Main Results: None of the evaluated models reaches near-human-level chart understanding, leaving considerable room for improvement.
- Qualitative Analysis: Error analysis identifies visual perception, instruction following, and math reasoning as three major failure categories, including runtime errors in generated programs.
- Ablation Results: Ablations show closed-source models are more robust to dashboards and infographics and use added paragraph context more effectively than smaller open-source and chart-specific models.
5 Conclusion
CHARTQAPRO is a diverse, challenging benchmark designed to test real-world chart reasoning beyond existing evaluations. Its evaluation exposes substantial model weaknesses and identifies future directions, while planned extensions target dynamic chart settings and reasoning-oriented training data.
- CHARTQAPRO contains 1,341 charts from 157 sources and covers factoid, multiple-choice, fact-checking, conversational, and hypothetical questions.
- The benchmark reveals significant performance gaps between existing models and human-level chart understanding.
- Even the strongest closed-source models experience substantial performance drops, showing that chart reasoning remains an unsolved challenge.
- Detailed error analyses and ablation studies identify key areas for improving LVLM chart reasoning.
- Future work will add dynamic and interactive charts and dashboards and curate a large-scale reasoning-format training dataset.
Limitations
The benchmark evaluates chart question answering comprehensively but leaves other chart tasks unexplored, and its static screenshots and prompt choices constrain evaluation scope. These limitations define boundaries for interpreting results and motivate broader future evaluation settings.
- CHARTQAPRO primarily evaluates chart question answering, leaving chart-to-summary generation and chart-to-code translation unexplored.
- Performance may vary slightly with further prompt engineering, although the authors do not expect substantial improvements or changed overall findings.
- Dashboards are represented as static screenshots rather than interactive elements such as hover, filtering, or dynamic data manipulation.
- Because interactivity is absent, models are evaluated only on the static visual and textual information shown in the images.
- Despite these limitations, CHARTQAPRO provides a rigorous and diverse benchmark highlighting key challenges in chart reasoning.
Ethical Considerations
The authors describe dataset safeguards centered on manual review, public sourcing, and author-led quality control. These procedures address harmful content, proprietary data, and annotation accuracy within the reported collection process.
- All collected images underwent manual review to filter content considered harmful or offensive.
- The benchmark excludes proprietary data because its charts were sourced from publicly available online platforms.
- Authors with expertise in chart understanding generated the question-answer data, with LVLMs used only as assistance during QA expansion.
- All questions and answers were manually reviewed and refined for accuracy, coherence, and ethical neutrality.
A.1 Dataset Construction
CHARTQAPRO constructs its dataset from diverse real-world chart sources and deliberately prioritizes visual and topical coverage. The collection combines curated source-specific data with a large web-scale corpus filtered and manually selected for diversity.
- Stage 1 - Chart Images Collection: Pew and Tableau images were sourced from larger corpora, with Tableau selection focusing on complex dashboards presented in paginated stories.
- Stage 1 - Chart Images Collection: OWID selection emphasized complex charts such as multi-series line charts and multi-column bar charts.
- Stage 1 - Chart Images Collection: PPIC selection excluded simple statistical charts and basic single-bar or single-line visualizations, favoring visually diverse topics and charts.
- Stage 1 - Chart Images Collection: WebCharts used a 157-website seed list, image search, ViT filtering, manual removal of non-charts, and selection of 800 charts from a 41,000-image pool.
- Stage 1 - Chart Images Collection: The expanded source pool provides broader domain coverage and richer stylistic representation than prior datasets concentrated in a few sources.
A.2 Complex Visualizations
ChartQAPro includes multi-chart images, infographics, and dashboards as distinct forms of data visualization, each serving different purposes.
- Multi-chart images combine multiple charts in one visual to enable comparison or present different aspects of a dataset.
- Infographics integrate text, images, and charts to explain concepts or tell a story, emphasizing clarity and engagement.
- Dashboards organize charts, tables, and key metrics in a structured layout to provide an overview of important information.
A.3 Dataset Analysis
ChartQAPro is analyzed through diverse chart sources, linguistic and visual diversity measures, evaluation procedures, and error categories. The analysis reports richer language and visuals than prior benchmarks, while lower model accuracy and observed errors indicate challenging chart reasoning.
- Linguistic Diversity: ChartQAPro uses Type-Token Ratio to quantify lexical diversity, with T denoting total tokens and U denoting unique tokens.
- Linguistic Diversity: 0.1516 TTR for ChartQAPro exceeds 0.1377 for ChartQA and 0.1189 for Chartxiv.
- Linguistic Diversity: 0.84 semantic diversity for ChartQAPro is the highest reported among the compared datasets, based on average pairwise cosine distance between text embeddings.
- Error Analysis: Reported error analyses cover visual perception, instruction following, mathematical reasoning, complex visuals, unanswerable questions, and charts with accompanying paragraphs.
- Benchmark Evaluation: 55.81% accuracy for Claude Sonnet 3.5 on CHARTQAPRO is lower than 90.50% on ChartQA.