Source-linked AI summary
PlotQA: Reasoning over Scientific Plots
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar
TL;DR
PlotQA addresses the gap between synthetic plot datasets and real-world reasoning by introducing a large dataset and a hybrid model for fixed-vocabulary and OOV questions. The model combines classification with visual extraction, OCR, structured-table generation, and table question answering, achieving state-of-the-art results on DVQA and PlotQA. The paper identifies visual element detection as a key area for further improvement.
Problem
Existing synthetic plot datasets lack variability in labels and real-valued data, while their models are not suited to complex questions with OOV answers.
Method
The paper introduces PlotQA and a hybrid model that routes fixed-vocabulary questions to classification and OOV questions through visual detection, OCR, structured tables, and table question answering.
Results
22.52% accuracy on PlotQA and 58% on DVQA establish state-of-the-art results on both datasets.
Takeaways & Limitations
PlotQA exposes the need for models that combine perception, language understanding, numeric reasoning, and OOV answer handling for realistic plot question answering.
Takeaways & Limitations
The pipeline needs more accurate visual element detection because bounding-box errors can produce incorrect values and incomplete tables for downstream question answering.
Abstract
from arXiv · showhide
Existing synthetic datasets (FigureQA, DVQA) for reasoning over plots do not contain variability in data labels, real-valued data, or complex reasoning questions. Consequently, proposed models for these datasets do not fully address the challenge of reasoning over plots. In particular, they assume that the answer comes either from a small fixed size vocabulary or from a bounding box within the image. However, in practice, this is an unrealistic assumption because many questions require reasoning and thus have real-valued answers which appear neither in a small fixed size vocabulary nor in the image. In this work, we aim to bridge this gap between existing datasets and real-world plots. Specifically, we propose PlotQA with 28.9 million question-answer pairs over 224,377 plots on data from real-world sources and questions based on crowd-sourced question templates. Further, 80.76% of the out-of-vocabulary (OOV) questions in PlotQA have answers that are not in a fixed vocabulary. Analysis of existing models on PlotQA reveals that they cannot deal with OOV questions: their overall accuracy on our dataset is in single digits. This is not surprising given that these models were not designed for such questions. As a step towards a more holistic model which can address fixed vocabulary as well as OOV questions, we propose a hybrid approach: Specific questions are answered by choosing the answer from a fixed vocabulary or by extracting it from a predicted bounding box in the plot, while other questions are answered with a table question-answering engine which is fed with a structured table generated by detecting visual elements from the image. On the existing DVQA dataset, our model has an accuracy of 58%, significantly improving on the highest reported accuracy of 46%. On PlotQA, our model has an accuracy of 22.52%, which is significantly better than state of the art models.
1. Introduction
PlotQA addresses limitations of synthetic plot datasets by combining real-world plots, diverse labels and values, complex questions, and a hybrid approach for fixed-vocabulary and OOV answers.
- 28.9 million question-answer pairs are grounded over 224,377 plots in the PlotQA dataset.
- 80.76% of questions have answers absent from both the plot and a fixed vocabulary.
- PlotQA uses real-world data sources and 74 templates derived from 7,000 crowd-sourced questions over 1,400 sampled plots.
- Existing models achieve less than 8% accuracy on PlotQA because they cannot answer OOV questions, although SAN performs well on fixed-vocabulary questions.
- The hybrid model classifies questions as fixed-vocabulary or OOV, then uses classification or visual extraction and structured-table question answering accordingly.
- The model reaches 22.52% accuracy on PlotQA and 58% on DVQA, improving on SANDY's reported 46% DVQA result.
3. The PlotQA dataset
PlotQA is built from real-world data sources, varied plot designs, and crowd-sourced question templates to support realistic plot reasoning. Its large-scale dataset combines 224,377 plots with 28,952,641 questions and broad question types.
- Dataset construction: PlotQA combines data curation, plot generation, crowd-sourced questioning, and template instantiation to build its dataset.The construction process uses online statistics, varied chart designs, human-written questions, and manually analyzed templates.
- Plot generation: 224,377 plots cover bar, line, and scatter plots, with bar plots divided into horizontal and vertical orientations.The dataset also provides annotations for visual elements including legends, axes, bars, lines, and titles.
- Plot generation: Plot variation includes grid lines, font sizes, tick notation, line styles, marker styles, legend positions, and 73 line or bar colors.These randomized parameters are intended to increase visual variety across generated plots.
- Question template extraction and instantiation: 74 templates were derived from 7,000 crowd-sourced questions and manually paraphrased before instantiation with curated indicators and entities.This process generated 28,952,641 questions, with longer questions, a larger unique-answer set, and a more realistic vocabulary than FigureQA or DVQA.
- Question types: Questions are organized into Structural Understanding, Data Retrieval, and Reasoning categories covering structural, single-element, comparative, and multi-element queries.Reasoning questions may require numeric reasoning, comparative analysis, or both.
4. Proposed Model
The proposed model uses a hybrid architecture that routes simple fixed-vocabulary questions to classification and complex questions through visual perception, OCR, table extraction, and structured-table QA.
- Hybrid architecture: A binary classifier routes questions to either a small fixed-vocabulary classifier or a multi-staged reasoning pipeline.The multi-staged path contains visual element detection, optical character recognition, structured table extraction, and structured table question answering.
- Multi-staged pipeline: The model detects plot elements, reads their text, organizes values into a semi-structured table, and answers questions through semantic parsing.Visual element detection uses Faster R-CNN with Feature Pyramid Network; OCR processes cropped, grayscale, resized, and deskewed regions, while table QA ranks logical forms.
- Visual element detection: The pipeline models ten visual-element classes, including titles, axis labels, tick labels, legend markers and names, bars, and lines.These elements are detected with bounding boxes and classified into their appropriate classes.
- Structured table extraction: Structured tables use x-axis ticks as rows, legend elements as columns, and each cell as the value associated with a tick–legend combination.OCR supplies tick-label values and legend names, while visual relationships provide the mapping needed for table construction.
- Dataset construction: PlotQA contains 224,377 plots generated from 841 indicator variables and 160 entities, with question–answer pairs instantiated from 74 templates.The dataset is split into 70% training, 15% validation, and 15% test data.
6. Observations and Results
Existing plot-QA models perform poorly on PlotQA, especially for open-vocabulary and numerical reasoning questions, while the proposed hybrid model achieves the strongest reported PlotQA result.
- Existing models: BAN and LoRRA achieve less than 1% accuracy on PlotQA, while existing models overall remain below 8%.Their fixed-vocabulary or text-extraction designs do not address open-vocabulary and complex numeric reasoning questions.
- Overall results: 22.52% accuracy is achieved by the proposed model on PlotQA, the best performance reported for the dataset.The result is reported against existing models whose designs do not support open-vocabulary questions.
- Pipeline analysis: VED errors sharply reduce detection accuracy at higher IoU thresholds, with dotline accuracy under 20% at IoU 0.9.A predicted box with IoU 0.58 estimates a bar as 760 instead of the ground-truth 680, affecting downstream table QA.
- Pipeline analysis: OCR accuracy drops from 97.06% in oracle mode to 93.10% after VED, whereas table extraction reaches only 0.68 F1.The results indicate that imperfect visual-element detection propagates into erroneous structured tables.
7. Conclusion
PlotQA combines separate pipelines for fixed-vocabulary and OOV questions, achieving state-of-the-art results while exposing visual detection as a key bottleneck. Pipeline errors can propagate from detection and OCR into table generation and question answering.
- Conclusion: PlotQA’s hybrid model handles simpler fixed-vocabulary questions separately from complex OOV questions.The OOV pipeline combines visual element detection, OCR, structured table extraction, and table question answering.
- Conclusion: The proposed model achieves state-of-the-art results on both the DVQA and PlotQA datasets.
- Conclusion: More accurate visual element detection is needed to improve reasoning over plots.
- Failure cases: VED errors can produce incorrect extracted values, incomplete tables, and multiple incorrect answers in PlotQA.At IOU 0.5, boxes may fit reasonably well but still yield incorrect values; overlapping points can also be missed.
- Failure cases: Slight OCR bounding-box misalignments can corrupt labels and cause incorrect answers for all questions involving those labels.Examples include “Indoo” for “Indoor” and malformed axis tick values.
- Failure cases: Errors propagated from VED and OCR lead to incorrect structured-table generation.
10. Samples from the PlotQA dataset
The PlotQA dataset includes plot, question, and answer triplets, with question phrasing varying according to each plot’s context.
- Samples from the PlotQA dataset: PlotQA samples are represented as {plot, question, answer} triplets.
- Samples from the PlotQA dataset: Most question templates apply to each plot, but their language varies with the plot’s context.
- Samples from the PlotQA dataset: The dataset presents examples of questions and answers grounded in individual plots.
11. Question Templates
PlotQA uses 74 context-dependent question templates spanning structural understanding, data retrieval, and reasoning over plots. The examples include counting, comparison, aggregation, ratio, monotonicity, and axis-interpretation questions.
- Question categories: The question templates cover structural understanding, data retrieval, and reasoning over plots.
- Template variation: Template language varies according to the context of each plot, and not every template applies to every plot type.
- Reasoning questions: The templates include comparisons, averages, threshold counts, ratios, monotonicity checks, and differences between axis ticks.Examples ask for gasoline-price differences, average air quality, neonatal-death counts, sodium ratios, monotonic increase, and Y-axis tick differences.
- Structural understanding: Structural templates ask about zero values, grids, legend placement and counts, label stacking, plot counts, colors, and group counts.
- Structural understanding: Other templates ask whether bar counts match legend-label counts and how many bars occur at indexed ticks from different directions.
11. Are the number of bars in each group equal?
These templates ask for bar counts at indexed ticks, using positions measured from the left, right, top, or bottom of the plot.
- Indexed bar counts: One template asks how many bars appear on the ith tick from the left.
- Indexed bar counts: Another template asks how many bars appear on the ith tick from the right.
- Indexed bar counts: Additional templates measure the ith tick from the top or bottom when asking for bar counts.
17. How many lines intersect with each other?
This section presents structural and data-retrieval question templates for interpreting plot organization, labels, axes, and individual values.
- Structural Understanding: Structural templates ask about plot organization, including whether line counts match legend labels and how bars map to legend entries.
- Structural Understanding: Axis-oriented templates ask about consecutive Y-axis tick differences, scientific E-notation, and axis titles.
10. What is the title of the graph ?
This section lists templates for retrieving plot metadata and answering extrema, aggregation, comparison, threshold, and ratio questions over plotted values.
- Structural Understanding: Metadata templates ask whether a legend label appears and request the X-axis or Y-axis title.
- Reasoning: Extrema templates ask for maximum or minimum values and the X-axis positions or legend entries where they occur.
- Reasoning: Aggregation templates ask for sums, averages, medians, totals, and differences across ticks or legend entries.
- Reasoning: Additional templates count values above thresholds and test whether one value is less than or greater than another.
25. Is the <Y label> of/in <legend label> in < ith
These templates require higher-order comparisons involving differences, extrema, averages, sums, and repeated conditions across plotted categories or series.
- Reasoning: Some questions compare a selected difference with differences between any two X-axis categories.
- Reasoning: Extrema templates ask for gaps between the highest and second-highest values or between the highest and lowest values.
- Reasoning: Other templates compare adjacent-value sums with global maxima or compare changes across two legend series.
- Reasoning: Multi-condition templates count values above their series average and test whether sums satisfy inequalities across every category.