Source-linked AI summary
LEAF-QA: Locate, Encode & Attend for Figure Question Answering
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, Ajay Joshi
TL;DR
Existing chart-QA datasets use synthetic data and restrict chart values or answers, unlike real-world charts with potentially unseen vocabularies. LEAF-QA pairs varied real-world charts with a chart-element-based attention model; LEAF-Net performs consistently across question types and outperforms prior work on FigureQA and DVQA.
Problem
FigureQA and DVQA use synthetic charts with limited data ranges, and restrict answers or use color labels that may encourage models to associate colors rather than chart semantics.
Source: §1 · p. 1Method
Results
LEAF-Net performs consistently across question types and significantly outperforms prior work on both DVQA and FigureQA.
Source: §6 · p. 7, §5.2 · p. 7, §7 · p. 8Takeaways & Limitations
LEAF-QA advances reasoning from figures and charts that emulate real-world data.
Source: §7 · p. 8At inference, LEAF-Net returns only one answer even for questions with multiple answers; multiple-answer generation is left for future work.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
We introduce LEAF-QA, a comprehensive dataset of $250,000$ densely annotated figures/charts, constructed from real-world open data sources, along with ~2 million question-answer (QA) pairs querying the structure and semantics of these charts. LEAF-QA highlights the problem of multimodal QA, which is notably different from conventional visual QA (VQA), and has recently gained interest in the community. Furthermore, LEAF-QA is significantly more complex than previous attempts at chart QA, viz. FigureQA and DVQA, which present only limited variations in chart data. LEAF-QA being constructed from real-world sources, requires a novel architecture to enable question answering. To this end, LEAF-Net, a deep architecture involving chart element localization, question and answer encoding in terms of chart elements, and an attention network is proposed. Different experiments are conducted to demonstrate the challenges of QA on LEAF-QA. The proposed architecture, LEAF-Net also considerably advances the current state-of-the-art on FigureQA and DVQA.
1. Introduction
LEAF-QA addresses the limits of chart-QA datasets built from synthetic data and fixed question templates by providing varied real-world charts, dense annotations, diverse questions, and a chart-aware QA architecture.
1. Introduction
- Synthetic chart data, narrow answer vocabularies, limited chart types, and fixed question templates constrain prior chart-QA datasets relative to real-world charts.FigureQA and DVQA use synthetic data; FigureQA has binary answers, while DVQA is limited to bar charts and has limited label variation.Source: §1 · p. 1
- LEAF-QA combines real-world charts from varied sources with dense chart-element annotations and structural and relational question-answer pairs.It includes bar, pie/donut, box, scatter, and line charts, with paraphrased questions intended to prevent template memorization.Source: §1 · pp. 1–2
- A novel test set from unseen data sources is provided to evaluate question-answering generalization.The training and standard test sets use charts from real-world sources; the additional test set uses unseen sources.Source: §1 · p. 2
- LEAF-Net parses chart elements, encodes questions and answers in terms of those elements, and uses the encodings to support question answering with chart-specific vocabulary.The need arises because answers can include chart strings and may not appear in training, limiting fixed-answer-dictionary VQA architectures.Source: §1 · p. 2
2. Related Work
Prior work spans chart extraction, figure understanding, and visual question answering, while this paper focuses on question answering that jointly reasons over chart images and their text.
Figure Understanding
Visual Question Answering
Object Detection
- Chart elements include rectangular boxes and pie wedges, motivating the use of Mask R-CNN to parse them.Source: §2 · p. 2
3. LEAF-QA dataset
LEAF-QA is built by transforming public real-world data tables into varied charts, annotating their elements, and pairing them with paraphrased structural and relational questions; its corpus includes standard and unseen-source test sets.
3. LEAF-QA dataset
- LEAF-QA uses public online data tables and preprocesses them to support chart generation and address missing data.Source: §3.1 · p. 3, §3.2 · p. 3
3.1. Data Sources
- Sources include World Bank indicators, Indian government open data, UN commodity trade statistics, US Census data, and stock and ETF prices.Source: §3.1 · p. 3
3.2. Data pre-processing
- Preprocessing merges tables with shared label spaces, decomposes multi-indicator tables along legend or axis labels, imputes missing values, and removes aggregated rows and complex alphanumeric columns.Source: §3.2 · p. 3
3.3. Figure/Chart Generation
- Charts are generated from selected table columns and randomized rows, covering pie/donut, bar, box, line, and scatter plots.Source: §3.3 · p. 3
3.4. Figure/Chart Style Variations
- Style variations alter plot colors, titles, fonts, markers, grids, legends, bar widths, pie radii, and error-bar presence, while columns of similar magnitude help prevent skewed plots.Source: §3.4 · p. 3
3.5. Annotation Generation
- Annotations assign 20 chart-element classes and use masks when boxes do not represent plot elements adequately, including pie/donut elements and line segments.For line plots, multiple boxes between consecutive x-labels represent the plot.Source: §3.5 · pp. 3–4
3.6. Question-Answer Generation
- GRE-inspired structural and relational questions are paraphrased, and answers include chart vocabulary, chart type, and common vocabulary; value-based questions are excluded.Structural questions target individual chart elements; relational questions target relations between elements.Source: §3.6 · p. 4
3.7. Training/Test Corpus
- The training corpus has around 200,000 images and 12,000 unique answers, while standard testing uses 40,000 images and the novel test set has 8,150 images from unseen sources.Training includes around 1.5 million questions; standard testing uses charts from the training sources.Source: §3.7 · p. 4
4. LEAF-Net baseline
LEAF-Net combines chart parsing, chart-element-based question and answer encoding, and attention to answer questions without a fixed vocabulary. At inference, it selects the chart string associated with the highest-scoring output dimension.
4. LEAF-Net baseline
- LEAF-Net integrates chart parsing, chart-element question and answer encoding, and attention-based learning to address multimodal QA without a fixed vocabulary.The pipeline combines encoded questions with chart images to calculate spatial attention and predict answers in terms of detected chart elements.Source: §4 · p. 5, Figure 4 · p. 6
4.1. Chart Parsing
- Mask R-CNN with a ResNet-101 backbone detects and classifies chart elements from images annotated with bounding boxes or masks.The detector is trained on 200,000 images with element labels and annotations.Source: §4.1 · p. 5
4.2. Question-Answer Encoding
- Chart text is ordered by element type and position, matched longest-first in questions, and represented in answer vectors with dimensions for chart elements and other answer types.The answer representation has 75 dimensions, including chart text elements, chart types, yes/no/none/negative/positive, and numeric values 1–15.Source: §4.2 · p. 5
4.3. Attention Network
- The attention network combines a ResNet-152 encoding of the chart image with an LSTM question encoding, then predicts an answer vector.The image is represented at 14×14×2048 before combination with the question encoding.Source: §4.3 · pp. 5–6
4.4. Inference
- At inference, the highest-valued output dimension selects one chart-string answer, so questions with multiple correct answers still receive only one prediction.Text extraction uses Oracle or OCR, and multiple-answer generation is left for future work.Source: §4.4 · p. 6
5. Evaluation
LEAF-Net performs consistently across question sets while the QUES, QUES-ENC, and IMG baselines do not generalize well; performance drops on the novel test set for some question types. Detection and question-answering results vary across chart elements and question types.
5.1. Chart Parsing Evaluation
- Chart titles and axis labels show high detection precision and recall, while legend subelements and pie-element precision are weaker.Legend boxes are detected accurately; pie values and labels have high recall but lower precision.Source: §5.1 · p. 6
5.2. Question Answering Evaluation
- Question-answering accuracy counts a prediction correct when it matches any ground-truth answer; QUES omits the chart image, IMG omits the question, and QUES-ENC omits attention.The evaluation breaks performance down by question and answer type.Source: §5.2 · pp. 6–7
- The QUES, QUES-ENC, and IMG baselines do not generalize well across question sets, while LEAF-Net performs consistently; some question types drop on the novel test set.The authors identify the novel-test drop as a challenge and call for future study of adaptation.Source: §5.2 · p. 7
- Chart-type questions have high accuracy across test sets, while common-vocabulary structural answers lose accuracy on the novel test set with similar Oracle and OCR performance.The novel-test decline for common-vocabulary answers is reported for both text-extraction settings.Source: §5.2 · p. 7
- Relational questions with common-vocabulary answers perform consistently for Oracle and OCR, but complex relational questions trail structural questions and OCR struggles more with chart-vocabulary answers.The OCR decline for chart-vocabulary answers is attributed to parsing longer chart labels.Source: §5.2 · p. 7
6. Comparisons on DVQA and FigureQA
LEAF-Net outperforms prior work on DVQA and FigureQA, with particularly strong results on complex data and reasoning questions. Its chart-element localization also limits the OCR performance drop relative to oracle input.
6. Comparisons on DVQA and FigureQA
- For FigureQA, LEAF-Net uses a fixed yes/no answer dictionary without encoding; for DVQA, it is fine-tuned with ground-truth bar-chart boxes and chart-specific strings.Source: §6 · p. 7
- LEAF-Net significantly outperforms SANDY [13] across all question types, particularly complex data and reasoning questions.The authors attribute generalizability on varied vocabularies to consistent chart element-wise encoding.Source: §6 · p. 7
- LEAF-Net shows only a slight OCR performance drop from oracle input, unlike SANDY’s drastic reduction.The authors link the difference to precise chart-element localization, which improves OCR compared with processing the entire chart.Source: §6 · pp. 7–8
- LEAF-Net significantly outperforms prior work on both DVQA [13] and FigureQA [14].Tables 4 and 5 report the comparisons.Source: §6 · p. 7, Table 4 · p. 8, Table 5 · p. 8
7. Conclusion and Future Direction
The paper presents LEAF-QA, a varied real-world chart dataset with dense element annotations and a test set from unseen data sources, alongside LEAF-Net. It identifies user query behavior, new architectures, and broader evaluation metrics as future directions.
7. Conclusion and Future Direction
- LEAF-QA combines varied real-world charts and question-answer pairs with dense plot-element masks; LEAF-Net parses charts, encodes questions and answers using chart elements, and applies attention.Charts are randomized visually and questions paraphrased to discourage template memorization; an unseen-source test set probes generalizability.Source: §7 · p. 8
- Future work includes studying how users query charts and incorporating those behaviors into the question-answer framework.Source: §7 · p. 8
- The paper calls for novel deep-learning architectures for complex reasoning and information-retrieval metrics for a more holistic performance view.Source: §7 · p. 9