Source-linked AI summary
DVQA: Understanding Data Visualizations via Question Answering
Kushal Kafle, Brian Price, Scott Cohen, Christopher Kanan
TL;DR
Bar charts communicate precise information, but existing algorithms struggle with their visual variation and chart-specific words. DVQA addresses this gap with a large bar-chart question-answering dataset and two systems designed for image-specific vocabulary. VQA algorithms handle mainly simple structure questions, while SANDY and MOM better answer data retrieval and reasoning questions; synthetic training also transfers to real charts.
Problem
Existing VQA systems and fixed-vocabulary classification methods struggle with varied bar charts, numeric answers, and words unique to each chart.
Method
DVQA provides over 3 million bar-chart image-question pairs, and SANDY and MOM handle chart-specific words through chart reading or dynamic local dictionaries.
Results
About 59% accuracy was achieved on real-chart structure questions without fine-tuning, a 15% absolute improvement over the QUES baseline; SANDY and MOM also better answered data retrieval and reasoning questions.
Takeaways & Limitations
DVQA supports study of visual attention, memory, reasoning, and out-of-vocabulary handling while targeting automatic querying of chart repositories.
Takeaways & Limitations
SANDY’s dynamic encoding can suffer cascading failure when real OCR misdetects a single word, and the dataset currently contains only bar charts.
Abstract
from arXiv · showhide
Bar charts are an effective way to convey numeric information, but today's algorithms cannot parse them. Existing methods fail when faced with even minor variations in appearance. Here, we present DVQA, a dataset that tests many aspects of bar chart understanding in a question answering framework. Unlike visual question answering (VQA), DVQA requires processing words and answers that are unique to a particular bar chart. State-of-the-art VQA algorithms perform poorly on DVQA, and we propose two strong baselines that perform considerably better. Our work will enable algorithms to automatically extract numeric and semantic information from vast quantities of bar charts found in scientific publications, Internet articles, business reports, and many other areas.
1. Introduction
DVQA studies open-ended question answering over bar charts, whose concise visual information is not designed for machine interpretation. It introduces a varied dataset and systems for handling chart-specific words, while showing that existing VQA methods struggle with these demands.
- Motivation: DVQA targets open-ended questions about bar charts so repositories of scientific, web, and business visualizations can be queried automatically.The task is motivated by the information density of visualizations and their practical applications.
- Motivation: Answering chart questions can require locating colors, hatching, bar groups, and heights, then measuring or interpolating values.The example question requires multiple attention, memory, measurement, and reasoning steps.
- Challenges: DVQA challenges fixed-vocabulary VQA because chart labels and answers may be out-of-vocabulary and vary across images.Chart words can refer to bars with different positions, sizes, textures, and colors.
- Challenges: Small visual changes can alter a bar chart’s information, unlike many natural-image changes that primarily affect local image content.Legend-color changes are given as an example of a change that can completely alter chart meaning.
- Contributions: The paper contributes a dataset with over 3 million image-question pairs and two systems that handle image-specific words, including chart-read answers and dynamic local dictionaries.The dataset tests structure understanding, data retrieval, and reasoning.
2. Related Work
Prior chart-extraction and natural-image VQA methods rely on assumptions or capabilities that do not cover DVQA’s varied charts, numeric answers, and chart-specific vocabulary. DVQA therefore combines visual parsing with language understanding, attention, memory, and reasoning.
- Chart Parsing: Earlier bar-chart extraction methods used fixed heuristics, strong appearance assumptions, and evaluations on as few as 41 charts.Examples exclude textures, gradients, or stacked bars.
- Chart Parsing: DVQA varies chart appearance beyond those methods and requires language understanding, attention, short-term memory, and reasoning beyond data extraction.The dataset tests more than recovering values from individual bars.
- Natural-Image VQA: VQA 1.0 suffered from language bias, and VQA 2.0 reduced that bias but retained a skewed distribution of question types.These dataset properties affect the questions represented in natural-image VQA.
- Natural-Image VQA: Natural-image VQA datasets and algorithms commonly use classification with fixed answer categories, while DVQA requires reading chart-specific text.This creates an out-of-vocabulary requirement absent from existing natural-image VQA datasets.
- Related Chart QA: FigureQA differs from DVQA by limiting questions to yes/no, excluding numeric answers, fixing labels across figures, and avoiding out-of-vocabulary words.The comparison identifies capabilities emphasized by DVQA.
3. DVQA: The Dataset
DVQA is a synthetic bar-chart dataset generated with controlled visual styles, data, and question templates. It spans structure understanding, data retrieval, and reasoning while varying chart appearance, labels, scales, and bias-related question distributions.
- Dataset Construction: DVQA contains 3,487,194 question-answer pairs for 300,000 images across structure understanding, data retrieval, and reasoning.Synthetic generation provides chart metadata such as drawing-element positions, underlying data, and text locations.
- Dataset Construction: Charts are generated by selecting a visual style, choosing chart data, and creating questions from fixed templates conditioned on titles and labels.The templates produce the three major question types.
- Visual Styles: DVQA varies bar count, grouping, grid lines, color, width, spacing, orientation, texture, and label placement.Matplotlib provides programmatic control over the visual elements.
- Vocabulary: Labels use frequent nouns for training and familiar testing, while Test-Novel introduces 500 words unseen during training.This split tests scaling to unknown answers.
- Question Types: Structure questions inspect chart organization, retrieval questions target individual components or scales, and reasoning questions combine components with operations.Reasoning examples include maxima, differences, sums, comparisons, and threshold counts.
- Bias Control: Randomized chart generation removes correlations between styles, colors, and labels, while balancing yes/no and related structure and scale questions.These measures address statistical biases that can inflate VQA evaluation.
4. DVQA Algorithms & Models
DVQA introduces models that read chart-specific text alongside generic answers, including MOM’s dual-network design and SANDY’s dynamic encoding approach.
- The proposed algorithms read bar-chart text to answer questions requiring chart-specific information or answers.
- MOM: MOM combines a classification sub-network for generic answers with an OCR sub-network for chart-specific answers.
- MOM: MOM’s OCR branch predicts a label bounding box, extracts and resizes its image patch, and decodes characters with spatial attention.
- MOM: MOM trains a binary classifier to select between its generic-answer and chart-specific OCR branches.The classifier uses LSTM question features and predicts whether the answer is generic or chart-specific.
- SANDY: SANDY uses a dynamic local dictionary to encode chart-specific question words and generate chart-specific answers.The dictionary augments global word and answer dictionaries with chart-specific entries.
- SANDY: SANDY is evaluated with oracle annotations and with Tesseract OCR output, using filtering based on character content, confidence, and word length.
5. Experiments
Experiments compare baseline and proposed systems on familiar and novel chart labels, showing that chart-specific handling is central to DVQA performance.
- Evaluation setup: Models are evaluated on Test-Familiar charts with seen labels and Test-Novel charts containing novel labels.
- Evaluation setup: Exact-match accuracy counts a question correct only when the generated string matches the ground truth.MOM (±1) additionally accepts answers within one edit distance.
- Overall results: SANDY performs best overall on both Test-Familiar and Test-Novel, followed closely by SANDY-real.
- Question types: Structure-question performance rises substantially for IMG+QUES over IMG or QUES, indicating that these questions require both image and question features.
- Question types: SANDY and MOM outperform all baselines on data retrieval and reasoning questions, while SANDY variants outperform MOM by a larger margin on Test-Novel.
- Chart-specific words: IMG+QUES and SAN-VQA fail when chart-specific labels appear in answers, while other models struggle with chart-specific labels in questions.These systems do not meaningfully outperform QUES on the latter questions.
- Chart-specific words: SANDY has the highest chart-specific-answer accuracy and similar performance across Test-Novel and Test-Familiar.MOM outperforms baselines but drops substantially on Test-Novel, consistent with small string-generation errors.
- Chart-specific words: SANDY improves encoding of chart-specific question labels, but precise measurement questions may remain beyond the SAN architecture.
6. Discussion
DVQA exposes failures of conventional VQA on chart-specific language and answers, while SANDY and MOM improve handling of novel chart content and transfer to real charts. SANDY remains limited by cascaded OCR-dependent dynamic encoding.
- SANDY and MOM answer data-retrieval and reasoning questions better than VQA algorithms, which mainly handle simple structure questions.
- SANDY and MOM generate answers novel to the test set, unlike traditional VQA algorithms; SANDY also encodes novel chart words in questions.
- SANDY’s dynamic encoding is vulnerable to one OCR error, which can propagate through the position-based encoding chain and invalidate the whole image representation.The passage identifies reducing this cascaded dependence as a route to better imperfect-OCR performance.
- Existing compositional VQA models such as NMNs cannot produce chart-specific answers without suitable modifications.
- SANDY and MOM extend SAN in distinct ways: SANDY uses DEM and OCR to encode chart text, while MOM predicts the needed text location.
- The dataset currently contains bar charts, while a follow-up version is planned to include pie charts, plots, and other visualizations.
- Without fine-tuning, SAN-based models achieve about 59% accuracy on real-bar-chart structure questions, a 15% absolute improvement over QUES.The result indicates positive transfer from synthetic to real-world bar charts.
7. Conclusion
The paper introduces DVQA for bar-chart understanding and proposes algorithms that handle chart-specific words in questions and answers. It positions DVQA as a benchmark for visual attention, memory, reasoning, and out-of-vocabulary handling.
- DVQA is a dataset for understanding bar charts and querying human-generated data visualizations.
- The paper reports that VQA algorithms cannot answer simple DVQA questions, while SANDY and MOM handle chart-specific words in questions and answers.
- DVQA targets issues often ignored in natural-image VQA, including out-of-vocabulary words and dynamic question encoding.
- The authors propose DVQA as a proxy task for studying visual attention, memory, and reasoning capabilities.
A. Additional details about the dataset
The dataset is generated from bar charts with varied visual styles, data patterns, and title-label combinations. Its construction aims to reflect common chart appearances while retaining important less frequent cases.
- Question generation: DVQA varies questions for the same chart by changing its title and labels, which determine the meaning of chart entities.
- Question generation: Question templates cover values, accuracies, units sold, and percentages tied to chart-specific labels, legends, datasets, stores, categories, and objects.
- Chart generation: The dataset generation process uses downloaded Google image-search bar charts to loosely model real-world distributions while retaining uncommon but important chart elements.
- Chart generation: Chart variation includes bar counts, grouped or stacked layouts, grid lines, textures, label orientation, colors, legend placement, bar width, spacing, titles, labels, and orientation.
- Chart generation: Stacked charts include additive stacking, where bars represent individual values, and fractional stacking, where each bar represents a fraction of the whole.
- Chart generation: Less common styles are applied to fewer charts, but every style choice appears at least 1000 times in training; 70% of charts use vertical bars.
A.3.2 Distribution of data-types
DVQA combines linear, percentage, and exponential data scales with varied chart rendering and color naming procedures. The dataset also records question distributions and MOM localization performance.
- Data scales: DVQA contains three major data scales: linear, percentage, and exponential.
- Data scales: Linear values range from 1–10, percentage values from 10–100, and exponential values span 1–10^10 with logarithmic axes.
- Question templates: Questions include chart-specific formulations about legend values, algorithm accuracy, item sales, and object preferences.
- Evaluation metadata: The supplementary material includes statistics by question type and MOM localization evaluations using IOU and center-distance measures.
- Data scales: The dataset distribution is 70% linear, 25% percentage, and 5% exponential; 10% of linear charts allow negative values.
- Rendering: Final chart images use a fixed 448 × 448 pixel size to simplify processing and avoid stretching or aspect-ratio changes.
- Rendering: Labels are initially horizontal but may rotate by 45 or 90 degrees when they overlap or consume too much chart space.
- Color naming: Chart colors are named by matching rendered colors to 138 CSS3 X11 colors in CIE L*a*b* space using CIE 2000 delta E.
B. Analysis of MOM’s localization performance
MOM often predicts bounding boxes near the ground truth, but imperfect localization can prevent OCR from extracting chart text exactly. The results suggest that tighter box regression could substantially improve chart-specific answer accuracy.
- MOM predictions are often close to the ground-truth answer strings but not exactly identical.Edit-distance analysis corroborates this pattern.
- MOM accurately predicts boxes for text with different orientations and positions, but some boxes remain offset from the text area.Correct and incorrect examples illustrate both behaviors.
- Because OCR requires total text coverage, boxes that are merely nearby can still produce erroneous predictions.The analysis identifies imperfect enclosure—not just large spatial error—as a source of chart-specific answer mistakes.
- More than half of MOM’s predicted boxes lie within 32 pixels of the ground-truth boxes.The images measure 448×448 pixels, providing scale for this localization error.
- Additional fine-tuning to regress more exact bounding boxes could considerably increase MOM’s chart-specific answer accuracy.The proposed refinement uses features surrounding the initial prediction.
C. Additional examples
The examples show strong structure-understanding performance across visual styles, while chart-specific labels and exact values remain major sources of error. SANDY handles chart-specific answers better by dynamically mapping labels to their positions in the image.
- SAN-VQA, MOM, and SANDY achieve high accuracy on structure-understanding questions across different visualization styles.The models share the SAN architecture for these questions.
- SAN cannot answer chart-specific questions reliably, producing the same answer regardless of the question even on the Test-Familiar split.This failure occurs despite answer-words appearing in the training set.
- SANDY correctly parses chart-specific labels by using a dynamic local dictionary, unlike SAN and MOM.The examples cited include Figures 10c, 11c, and 11e.
- Exact data-value questions commonly fail because they require precise measurement extraction and arithmetic across multiple values.The models can perform some measurement, producing values close to the correct answers.
- MOM’s chart-specific errors arise from OCR decoding, incorrect box prediction, or both, whereas SANDY’s dynamic dictionary reduces additional sources of error after position prediction.The figure examples illustrate these distinct failure modes.