Source-linked AI summary
Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis
Daniel J. Evans, J. P. Naiman, JooYoung Seo, J. Stephen Downie
TL;DR
Scientific charts are difficult to search and interpret, and systematic evaluation of LMMs on multi-panel chart questions remains limited. The paper builds a synthetic chart-VQA benchmark and evaluates several models; accuracy declines as figure and question complexity increase.
Problem
Systematic evaluation of LMMs on task-oriented questions about multi-panel charts remains limited, despite their prevalence in scientific figures and the accessibility challenges of chart content.
Source: §1 · p. 1Method
The authors create synthetic multi-panel chart VQA questions across description levels and evaluate several LMMs on them.
Source: §1 · p. 2Results
LMM accuracy decreases as the number of panels, plot elements, and question difficulty increase.
Source: Conclusions and Future Work · p. 9Takeaways & Limitations
The complexity that challenges LMMs is also the complexity most needed for multi-panel figures to be usable by blind and low-vision readers.
Source: Conclusions and Future Work · p. 9The study benchmarks zero-shot model accuracy but leaves direct evaluation of model outputs with blind and low-vision users to future work.
Source: Conclusions and Future Work · p. 9
Abstract
from arXiv · showhide
Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large language and multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/datasets/ReadingTimeMachine/visual\_qa\_multipanel
1 Introduction
Scientific charts are important but difficult to parse and access, while systematic evaluation of LMMs on multi-panel visual questions remains limited. This study addresses that gap with synthetic chart questions organized by figure-description level.
1 Introduction
- Only 2.4% of audited scientific PDFs met basic accessibility criteria, and figure and chart content was among the least accessible elements.Source: §1 · p. 2
- Systematic evaluation of cross-panel reasoning remains limited, although multi-panel charts comprise approximately 30–60% of scientific charts.Source: §1 · p. 2
- The study evaluates LMMs on synthetic multi-panel chart VQA, scaffolding questions from factual identification to interpretive and analytical reasoning.Source: §1 · p. 2
1.2 Four-Level Figure Description Model
- The analysis uses Levels 1–3 of a four-level description model; Level 3 questions address visual trends, including comparisons of overall trends across panels.Source: §1.2 · p. 3, Table 1 · p. 3
1.1 Related Work
- Existing multi-panel VQA datasets often ask about individual charts rather than cross-panel reasoning, and the authors found none including contour plots.Source: §1.2 · p. 3
2 Methods
The methods generate synthetic plots with saved ground-truth metadata, then create and validate questions about the figures or individual panels. Questions rely on visual content alone and do not require synthesizing values across panels.
2.1 Figure Dataset Generation
- Synthetic plots span line, histogram, contour, and scatter types, with randomized data relationships, noise, labels, scales, styles, and layouts.The pipeline saves generation parameters and chart-mark bounding boxes; the authors note synthetic figures do not fully capture real publications’ visual complexity.Source: §2.1 · pp. 3–5
- Approximately 41% of generated figures have multiple panels, after the pipeline checks for and corrects overlapping figure boxes or labels.Source: §2.1 · p. 5
- The dataset includes histograms and contour plots, which the authors identify as underrepresented in existing datasets, alongside complete generation metadata and bounding boxes.Source: §2.1 · p. 5
2.2 Question Data Generation
- Questions target either the whole figure or a specified panel and use only visual information; contextual and domain-dependent Level 4 analysis is left to future work.Source: §2.2 · p. 5
- Level 1–2 answers are computed from generation metadata or data arrays, Level 3 answers from known distribution types, and sampled pairs were manually reviewed.Neither figure-level nor plot-level questions currently require synthesizing values across panels; the authors identify that as a benchmark extension.Source: §2.2 · p. 6
3 Results and Analysis
Across 200 figures, LMM performance varied by task: Gemini led plot-type identification and label extraction, while histogram-bar estimates worsened as bar counts increased. Higher-reasoning models generally improved mean estimation, but accuracy came with substantial cost differences.
3 Results and Analysis
- The evaluation submitted one question per independent API call to three proprietary LMM families, yielding 33,560 answered questions across 200 figures after 130 outputs were rejected.The tested model variants represented lower- and mid-to-high-level reasoning or cost choices.Source: §3 · p. 6
- Gemini tracked ground-truth panel counts most closely among lower-level models, while GPT tended to overestimate; histogram-bar estimates diverged and trended toward underestimation as bar counts increased.Higher-level models had comparable panel-count performance, and GPT-HRM performed best on histogram-bar estimates.Source: §3 · pp. 6–7, Figure 2 · p. 7
- Gemini achieved the lowest character error rate overall, with GPT-HRM close behind; title extraction generally had lower error than axis-label extraction, and y-axis labels were harder than x-axis labels.The figure reports mean CER by ground-truth panel count; titles were included with 50% probability, while x/y labels were always present.Source: §3 · p. 7, Figure 3 · p. 8
- Higher-level reasoning models generally improved mean estimation over lower-level counterparts, while Gemini showed the tightest zero-centered clustering among lower-level models.For histograms, the requested mean was that of the underlying distribution, a different and more difficult task than estimating the mean along the axis.Source: Figure 4 · p. 8, §3 · p. 8
3.1 Cost Analysis
- At ∼$55, GPT-HRM had the highest spend to answer all questions; Claude-T cost ∼$36 and did not match Gemini at ∼$25, while GPT cost ≲$1 and was least accurate overall.These API costs reflect pricing at the time of writing and are intended as relative comparisons, not absolute benchmarks.Source: §3.1 · pp. 8–9
Conclusions and Future Work
LMM accuracy decreases as charts and questions become more complex, while Gemini currently has the highest overall accuracy per cost. The authors also identify metric limitations, accessibility implications, and directions for future validation.
Conclusions and Future Work
- Accuracy decreases as panel count, plot elements, and question difficulty increase, although models describe simple features such as panel count.The authors report that Gemini currently provides the highest overall accuracy per cost, a ranking they expect may change rapidly as other LMMs evolve.Source: Conclusions and Future Work · p. 9
- Binary comparisons and Levenshtein edit distance suit figure text, but free-form answers need more sophisticated metrics and accuracy should be compared with human performance across backgrounds.Source: Conclusions and Future Work · p. 9
- Because BLV readers rely most on Levels 2 and 3, declining accuracy on these tasks carries direct accessibility implications.The study leaves direct evaluation with BLV users for future work.Source: Conclusions and Future Work · p. 9
- Future work will train models using synthetic plots and validate benchmark accuracy against real multi-panel scientific figures.The authors also expect training and more complex prompting to decrease reported error rates.Source: Conclusions and Future Work · p. 9