Source-linked AI summary
FigureQA: An Annotated Figure Dataset for Visual Reasoning
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, Yoshua Bengio
TL;DR
FigureQA addresses the challenge of teaching machines to reason intuitively about structured information in scientific figures rather than merely reconstructing source data. It introduces a large synthetic, annotated corpus with relational visual questions and evaluates neural baselines, finding that the task remains challenging and that the Relation Network reaches 72.40% and 76.52% on two test settings. The dataset is intended as a controlled first step toward reasoning over real scientific figures.
Problem
Machine understanding of scientific figures remains an open visual-reasoning problem because figures encode trends, rates, and proportions spatially, while many existing approaches reconstruct the source data.
Method
FigureQA synthesizes over one million question-answer pairs over 100,000 figures, generates relational questions across five plot types, and provides source data and bounding-box annotations for auxiliary supervision.
Results
72.40% and 76.52% accuracy are achieved by the Relation Network on FigureQA test sets with alternated and unswapped colors, respectively.
Takeaways & Limitations
FigureQA provides a controlled benchmark for studying visual reasoning over scientific figures, with extensible data and annotations supporting future model and task development.
Takeaways & Limitations
Because FigureQA is synthetic, its data may not exhibit the same richness as figures in the wild; future work will test transfer to real scientific figures.
Abstract
from arXiv · showhide
We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as a strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.
1 INTRODUCTION
FigureQA introduces a large synthetic corpus for machine visual reasoning over scientific figures, designed around questions requiring relationships among multiple plot elements. Its controlled generation provides reliable answers, auxiliary annotations, and balanced answer frequencies for training and evaluation.
- Dataset and task: FigureQA contains over one million question-answer pairs grounded in over 100,000 figures across five common plot types.The represented types are line, dot-line, vertical bar, horizontal bar, and pie plots.
- Dataset and task: Its 15 question types test magnitude, extrema, medians, area-under-the-curve, smoothness, and intersections using yes-or-no answers.The questions address one-to-all and one-to-one relationships among plot elements.
- Dataset and task: Synthetic generation provides reliable ground-truth answers, auxiliary supervision signals, and control over task complexity, while limiting richness relative to figures in the wild.The authors propose iteratively increasing complexity and evaluating transfer to real figures.
- Dataset and task: The corpus exposes numerical source data and bounding boxes for plot elements, including points, axes, labels, ticks, and legend tokens.A two-stage process samples constrained numerical data and renders it with a modified Bokeh backend.
- Dataset and task: Balancing yes and no answers for each question type and figure makes answer-frequency shortcuts less useful to models.The balancing is intended to make visual content necessary for solving the task.
- Initial evaluation: The Relation Network reaches 72.40% and 76.52% accuracy on test sets with alternated and unswapped colors, respectively.The corpus also includes an official benchmark, generation scripts, and baseline implementations.
2 RELATED WORK
Prior figure-understanding work emphasizes extraction, chart parsing, or limited annotations, whereas FigureQA isolates visual-linguistic reasoning over scientific figures. Its questions require integrating multiple plot elements and spatially distributed information without simply reconstructing source data.
- Visual reasoning context: Existing VQA systems often exploit linguistic priors, motivating balanced or synthetic settings that isolate visual reasoning from common-sense knowledge.Real images can entangle visual-linguistic reasoning with common-sense concepts that are difficult to learn from VQA corpora alone.
- Prior figure-understanding work: Earlier figure datasets include 60,000 FigureSeer images, but only 600 figures have richer annotations of axes, legends, and plot data.FigureQA provides similar annotations across its full corpus.
- Prior figure-understanding work: Chart-recovery systems detect visual encodings, classify chart types, recover text, or extract data through type-specific tools.These tasks differ from FigureQA’s emphasis on answering reasoning questions about figures.
- FigureQA distinction: FigureQA focuses on reasoning rather than subtasks that existing OCR tools can solve with high accuracy.It follows the VQA setup while adding numerical data and dense bounding-box supervision.
- FigureQA distinction: Its questions often require synthesizing information from multiple plot elements distributed spatially throughout a figure.The intended system should understand figures intuitively instead of inverting the visualization pipeline.
3 DATASET
FigureQA is a large synthetic corpus of scientific-style figures, generated from constrained numerical data and paired with templated questions, answers, and auxiliary annotations. Its design supports controlled visual reasoning evaluation while enabling extension through source data and balanced question labels.
- FigureQA contains 100,000 training images with 1.3 million questions, while validation and test sets each contain 20,000 images with over 250,000 questions.
- The corpus covers horizontal and vertical bar graphs, continuous and discontinuous line charts, and pie charts.Figures include common elements such as axes, gridlines, labels, and legends.
- Questions are generated from source data using 15 predefined templates covering relationships and characteristics such as extrema, area-under-the-curve, smoothness, and intersection.Each applicable template produces one yes and one no question before balancing.
- Training uses all 100 colors, whereas validation and testing use unseen color-plot combinations through the alternated color scheme.The color subsets are disjoint and equally sized, with validation and test figures using the subset opposite the training assignment for each figure type.
- Yes and no answers are balanced per question type and figure to reduce exploitable answer-frequency biases.Question-answer pairs are discarded until the answer distribution is balanced.
- The generation process provides numerical source data and bounding boxes for plot elements, making arbitrary additional questions synthesizable.This side data supports auxiliary supervision and future dataset extension.
4 MODELS
FigureQA evaluates four neural baselines, including a Relation Network that processes visual object relationships alongside question representations. The models use convolutional or pretrained visual features, recurrent question encodings, and supervised classification.
- Four baselines are evaluated: a text-only LSTM, CNN+LSTM, CNN+LSTM using VGG-16 features, and a Relation Network.The text-only model serves as a sanity check for question-answer biases.
- The text-only baseline encodes questions word by word with an LSTM and classifies its final state using a two-hidden-layer MLP.Words use learned 32-dimensional embeddings, and the LSTM has 256 hidden units.
- The CNN+LSTM baseline concatenates an LSTM question encoding with a learned CNN visual representation before classification.Its CNN has five convolutional layers followed by a 512-unit fully connected layer.
- The FigureQA Relation Network follows the CLEVR-from-pixels architecture with an added convolutional layer and more channels for higher-resolution inputs.Random rotations are omitted because they could distort figures and change correct answers.
- The Relation Network forms representations for pairs of CNN-derived object features, sums the relational features, and maps the result to predicted outputs.The question encoding is concatenated with object pairs before processing by gθ and fφ.
5 EXPERIMENTAL RESULTS
The experiments evaluate baseline models on FigureQA and compare their performance with human accuracy. The Relation Network benefits from relational reasoning but remains below human-level performance.
- Experimental setup: All baselines were trained and evaluated with the alternated color scheme, using early stopping based on an exponential moving average of validation accuracy.The moving average used decay 0.9, and early stopping began after the 100th update.
- Model comparison: The Relation Network significantly outperforms the simpler CNN+LSTM model because FigureQA questions have relational structure.The text-only versus CNN+LSTM comparison also indicates that visual input contributes to learning.
- Evaluation coverage: The reported comparisons include baseline performance on validation and test sets, a human-versus-model subset, and accuracy broken down by figure and question type.Tables 3, 4, 5, and 6 organize these evaluations under the alternated color scheme.
6 CONCLUSION
FigureQA is introduced as a synthetic visual-reasoning corpus with extensive question-answer data and auxiliary annotations. The authors conclude that stronger models are needed and plan to test transfer to real figures and expand the dataset.
- Contribution: FigureQA contains over one million question-answer pairs grounded in over 100,000 synthetic figures.Questions address extrema, area-under-the-curve, smoothness, and intersection while integrating information distributed across figures.
- Contribution: The corpus provides numerical source data and bounding-box annotations for all plot elements to support machine-learning training.These resources are intended to facilitate auxiliary objectives.
- Conclusion: Results indicate that more powerful models must be developed to reach human-level performance.This conclusion follows the paper’s baseline and human comparisons.
- Future work: Future work will test transfer to real scientific figures and iteratively extend the dataset with more templates or crowdsourced questions.The authors describe FigureQA as a first step toward models that extract knowledge from scientific figures.
A DATA SAMPLES
The data samples illustrate FigureQA across five plot types, pairing figures with questions, answers, and selected annotations. The examples cover comparisons, extrema, medians, area under curves, smoothness, and intersections.
- Figure types: The samples span vertical and horizontal bar graphs, line graphs, dot-line graphs, and pie charts.Each plot type is shown with question-answer pairs and some bounding boxes or label annotations.
- Data format: The examples demonstrate that FigureQA pairs visual plot elements with explicit natural-language questions and binary answers.The sample figures also expose annotations for interpreting plot structure.
- Question examples: Example questions ask whether a named element is the maximum, minimum, high median, or low median.The answers are binary yes-or-no judgments tied to named colors such as Aqua, Purple, and Cadet Blue.
- Question examples: Line-plot examples test area under the curve, smoothness, highest or lowest value, pairwise comparison, and intersection.Examples include Olive’s smoothness and intersection with Midnight Blue.
- Question examples: The samples include pie-chart questions about maximum area, minimum area, roughness, and lowest value.They also show comparisons such as whether Lime Green is less than Web Gray.
B.1 PERFORMANCE BY FIGURE TYPE
Human accuracy varies by figure type, with bar graphs easiest and line, dot-line, and pie charts more difficult. Similar-sized pie slices and overlapping dot-line elements are identified as frequent sources of error.
- Accuracy by figure type: Human performance was exceptionally strong on bar graphs but lower on line plots, dot-line plots, and pie charts.The comparison is based on accuracy stratified by figure type.
- Error analysis: Pie charts with similarly sized slices led most frequently to human mistakes.The ambiguity arises when slices have similar visual extent, as illustrated by the Sandy Brown, Web Gray, and Tan slices.
- Error analysis: Dot-line plot accuracy was lower because plot elements sometimes obscure each other.The paper references Figure 21 as an example of this occlusion.
B.2 PERFORMANCE BY QUESTION TYPE
Human accuracy varies substantially by question type: annotators perform best on minimum, maximum, and greater/less-than queries, but struggle more with median and smoothness judgments.
- Human annotators perform best on minimum, maximum, and greater/less-than questions.
- Accuracy is generally higher for categorical figures than for continuous figures.
- Median questions become harder with more unordered plot elements, which may also induce optical illusions.
- Smoothness judgments require weighing both the number and size of curve deviations, which challenged annotators.
B.3 UNKNOWN ANSWERS
Annotators could answer nearly all test questions, selecting unknown rarely and mainly when visual evidence was difficult to distinguish or interpret unambiguously.
- Only 0.34% of test questions received the unknown response.
- Unknown responses accounted for 3.91% of all incorrect answers.
- Annotators generally used unknown when two colors were difficult to distinguish from each other.
C PERFORMANCE OF THE RELATION NETWORK WITH AND WITHOUT
The Relation Network baseline was trained with early stopping under two color-scheme validation settings, then evaluated across corresponding test color schemes.
- The RN baseline used early stopping on two validation sets with different color-scheme conditions.
- One validation set matched the training color scheme, while the other used alternated color-set-to-plot assignments.
- Table 7 compares both models on test sets for each color scheme.