Source-linked AI summary
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, Enamul Hoque
TL;DR
ChartQA addresses the limited coverage of complex visual and logical chart reasoning in datasets dominated by template-based questions and fixed-vocabulary answers. It builds a benchmark with human-written and generated questions and uses transformer-based models that combine extracted chart tables with visual features. The models achieve strong results on prior datasets and ChartQA, while evaluation exposes difficulties with nested operations and automatic data extraction.
Problem
Existing chart-QA datasets largely omit complex visual and logical questions because they rely on templates and fixed-vocabulary answers.
Method
ChartQA combines extracted chart data tables and visual features in transformer-based QA models, using human-authored questions and generated questions from chart summaries.
Results
ChartQA models achieve state-of-the-art results on prior datasets and the benchmark, while VL-T5 accuracy rises from 41.56% to 51.84% after PlotQA pretraining.
Takeaways & Limitations
The benchmark supports evaluation of chart QA under richer language and visual-logical reasoning demands than earlier datasets.
Takeaways & Limitations
The models struggle with nested arithmetic operations, and performance decreases when the gold data table is unavailable because chart data extraction is challenging.
Abstract
from arXiv · showhide
Charts are very popular for analyzing data. When exploring charts, people often ask a variety of complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to answer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.
1 Introduction
ChartQA targets complex chart questions that combine arithmetic or logical reasoning with references to visual attributes. It introduces a large benchmark and a unified approach combining chart data tables with visual features for transformer-based question answering.
- Chart questions often require multiple operations, such as retrieving values, comparing them, and calculating sums or differences.
- Visual references can identify chart marks by attributes such as color and peak value, alongside trends and outliers.
- Existing datasets commonly use template-generated questions, automatically created charts, and fixed-vocabulary answers.
- The benchmark contains 9,608 human-written questions, 23,111 generated questions, and 20,882 charts curated from four online sources.
- The proposed pipeline extracts a structured data table and visual features from each chart, then feeds both into transformer-based QA models.
- ChartQA contributes a large-scale dataset, a multimodal QA pipeline, and extensive model evaluation.
2 Related Work
Prior chart-QA datasets largely rely on synthetic charts, templates, or fixed vocabularies, while ChartQA emphasizes human-authored questions and real-world charts. Earlier extraction systems also struggled to recover fully structured chart data.
- Earlier datasets mostly use template-based questions, fixed-vocabulary answers, and automatically generated charts.
- PlotQA supports open-vocabulary aggregation questions but lacks visual reasoning questions and still uses template-based questions and software-plotted charts.
- ChartQA differs through human-authored question types and real-world chart sources rather than template-based questions and generated charts.
- Existing chart-QA models commonly use classification architectures that handle only fixed-vocabulary questions.
- Earlier automatic chart-data extraction pipelines relied on heuristics, while Luo et al. extracted raw mark values without linking them to axes or legends.
3 ChartQA Datasets
ChartQA combines real-world charts with human-authored and machine-generated questions designed to capture linguistic diversity and complex visual or compositional reasoning. The dataset uses multiple sources, annotation procedures, and quality checks to broaden coverage while filtering invalid generated questions.
- Dataset construction: 20,882 charts were collected from four sources to provide diverse topics and chart styles.The sources include Statista, Pew Research, Our World in Data, and another source described in the dataset construction.
- Data annotation: Human-authored questions target compositional reasoning with multiple operations and visual attributes such as color, height, and length.Workers answered two questions per chart, with another annotator checking agreement and disagreements manually reviewed.
- Data augmentation: 23,111 generated questions were produced from human-written chart summaries using a fine-tuned T5 model rather than fixed templates.This augmentation strategy was intended to preserve richer linguistic variation than template-based generation.
- Quality control: 86.64% of 1,250 manually analyzed generated QA pairs were complete, answerable, and correct after filtering questions whose answers were absent from the chart table.The test set was additionally cleaned by removing invalid machine-generated questions.
- Dataset analysis: 76.33% of sampled questions were compositional or both visual and compositional, with references most often involving color and length.The benchmark includes bar, line, and pie charts, along with varied language, informal phrasing, and typos.
4 Method
The method answers chart questions by combining chart images, questions, and structured data tables, using extracted tables when gold tables are unavailable. It adapts table-question-answering transformers, including a visual TaPas variant, and adds operations needed for chart reasoning.
- Problem setup: ChartQA predicts answers from a chart image, its data table, and a natural-language question.When the table is unavailable, ChartOCR is adapted to extract structured values and labels from the image.
- Base models: T5 and TaPas provide the table-question-answering foundations, while VL-T5 and VisionTaPas incorporate chart image features.T5 generates answers from flattened table tokens and the question; TaPas uses table-specific positional embeddings and operation heads.
- VisionTaPas: VisionTaPas extends TaPas by combining visual features from ViT with the outputs of TaPas encoders.TaPas separately encodes tabular structure through positional embeddings, while the cross-modality encoder performs multimodal integration.
- VisionTaPas: VisionTaPas combines a ViT image encoder, a TaPas question-and-table encoder, and a cross-modal encoder.The cross-modality encoder integrates visual features with textual-tabular representations before TaPas aggregation and cell-selection heads produce the answer.
- Operation extensions: Subtraction and ratio operations were added to TaPas because many ChartQA questions require them.The extension uses direct, potentially noisy cell-level supervision generated with heuristics rather than only weak supervision from final answers.
5 Evaluation
The evaluation compares the models with prior baselines across earlier chart-QA datasets and ChartQA, then analyzes transfer, ablations, chart and question types, and qualitative failure modes. Results show strong performance on prior datasets but greater difficulty on ChartQA, especially without gold tables, for complex visual and nested reasoning.
- Datasets, Baselines & Metrics: The evaluation covers FigureQA, PlotQA, DVQA, and ChartQA, comparing the proposed models with PREFIL and a reimplementation of PlotQA.The evaluation also uses extracted tables for the PLOTQA* comparison because PlotQA’s original extraction method does not generalize well to real-world charts.
- Results: 94.54% accuracy on the DVQA test set was achieved by VisionTaPas, a 14.5% margin over PReFIL.VisionTaPas also showed a 0.92% decrease with OCR outputs instead of ORACLE, compared with PReFIL’s 16.49% drop.
- Results: Performance was generally lower on ChartQA than on previous datasets, indicating greater difficulty from its human-written visual and logical reasoning questions.The models also performed worse without gold data tables, highlighting the challenge of extracting data from diverse real-world charts.
- Transferability: 51.84% accuracy was reached by VL-T5 after PlotQA pretraining and ChartQA fine-tuning, up from 41.56%, while VisionTaPas improved by only 1.56%.The authors suggest that PlotQA pretraining helps VL-T5 mainly on compositional questions without nested operations.
- Transferability: 45.52% to 31.96%: VisionTaPas performance decreased when trained on PlotQA and evaluated directly on ChartQA without fine-tuning.This supports the benchmark’s greater challenge from visual, compositional, and lexical variation.
- Ablation Studies: Removing supervision for difference and ratio operations reduced overall accuracy by 1.80% and ChartQA-H accuracy by 4.76%.The larger ChartQA-H decrease indicates that these operations matter for questions containing them frequently.
6 Conclusion
ChartQA combines human-written questions targeting visual and logical reasoning with a chart-answering approach that uses visual features and extracted data tables. The evaluation finds promise in this approach while exposing challenges from the informal, intricate, and nuanced language of human questions.
- ChartQA is a large-scale benchmark with human-written questions focused on visual and logical reasoning.
- The proposed approach combines visual features with an extracted chart data table to answer questions.
- The evaluation highlights the approach’s promise but also reveals challenges in human questions involving informal, intricate, and nuanced language.
Ethical Considerations
The paper describes safeguards concerning chart-source rights, annotator compensation and privacy, reproducibility, and possible misuse of the models. It also cautions that model outputs are not guaranteed to be correct.
- The dataset uses publicly available charts subject to source-specific terms and conditions.
- Mechanical Turk annotators were paid $0.6 USD per task based on a $7.25 USD hourly minimum-wage reference and a 3–5 minute estimated task duration.
- Annotator annotations were anonymized to protect privacy.
- Hyperparameter settings are provided in Appendix A.5 to support reproducibility.
- The models can be abused to mislead the public about chart content and implications.
- State-of-the-art results on existing datasets do not guarantee correct outputs in every case.
A.1 Additional Details on Data Annotation
The annotation process combines verification of existing questions with creation of new question-answer pairs, using worker screening and a pretest to support quality.
- Each HIT asks workers to verify two existing questions and provide two new question-answer pairs.
- Workers were selected using a 95% acceptance rate, at least 5000 completed HITs, and a pretest.
- The data collection interface is shown in Figure 5.
A.2 Dataset Analysis
The dataset analysis reports topic distributions across chart sources and examines how questions refer visually to chart elements. It also points readers to benchmark linguistic statistics and visual-reference usage tables.
- Politics is common across all four sources and accounts for 45.4% of charts in the Pew dataset.
- Society is the most frequent topic in the OECD and OWID sources, at 34.0% and 26.0%, respectively.
- Table 7 reports linguistic statistics for the ChartQA benchmark.
- Table 8 reports visual-reference usage in a randomly selected set of 300 question-answer pairs.
A.3 Automatic Chart Data Extraction
The chart image is parsed into an underlying data table through visual mark detection and subsequent extraction stages, then evaluated by matching predicted and ground-truth values. Results show extraction is less reliable for difficult chart types and logarithmic-scale data, while the metric omits extracted textual tokens.
- Extraction pipeline: ChartOCR-based parsing begins by detecting chart visual marks such as bars, plot areas, and line points.The method uses key-point detection networks adapted from prior work to locate chart structures.
- Evaluation: The extraction evaluation computes value distances, builds a cost matrix, and solves a linear sum assignment problem to obtain minimum matching costs.The final overall score aggregates the resulting costs across charts.
- Evaluation findings: Accuracy is lower on line and dot-line charts in FigureQA and PlotQA, while logarithmic-scale charts produce noisy extracted tables because ChartOCR does not support them.PlotQA charts with very large E-notation values also make extraction errors more consequential.
- Limitations: The extraction metric does not account for textual tokens, limiting its adequacy for noisy real-world figures.The authors identify the need for better chart data-extraction metrics.
A.4 Visual Features Extraction in VL-T5
The models combine chart data tables with visual information, while the visual extraction pipeline detects chart objects and supports downstream chart-question answering. Experiments report results by chart type and question type, alongside qualitative examples and generated questions.
- Visual feature extraction: The object detector is trained to identify 15 chart components, including axes, labels, legends, plot areas, bars, pies, lines, and dot lines.Bounding-box annotations are used, while masks for pie elements are generated by projecting radii along slice perimeters.
- Experimental setup: VisionTaPas, VL-T5, T5, TaPas, PlotQA, and PReFIL are fine-tuned with dataset-specific training settings for the benchmark experiments.The reported configurations vary in epochs, batch sizes, learning rates, and inference procedures.
- Analysis: Results are analyzed by chart type for VisionTaPas and VL-T5 and by question type on 200 randomly sampled ChartQA-H examples.The benchmark also includes machine-generated question-answer pairs and sample VisionTaPas predictions.
- Qualitative examples: Figure 10 presents VisionTaPas test-set predictions, distinguishing correct answers in green from incorrect answers in red.The figure provides qualitative examples of model outputs.