Source-linked AI summary
A Diagram Is Worth A Dozen Images
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi
TL;DR
Diagram understanding requires identifying constituents, relationships, and their semantics beyond the scope of conventional natural-image analysis. The paper introduces DPGs, LSTM-based parsing, and DPG-based question answering, evaluates them on AI2D, and reports improvements over baselines and standard VQA techniques.
Problem
The paper addresses limited diagram understanding by studying how to identify diagram structure, constituent relationships, and their semantic interpretation for reasoning.
Method
The paper represents diagrams with DPGs, uses Dsdp-Net to infer parses, and uses Dqa-Net to attend to DPG relations for question answering.
Results
The proposed models improve diagram parsing over strong baselines and diagram question answering over standard VQA techniques, using AI2D with over 5000 diagrams and more than 15000 questions.
Takeaways & Limitations
DPGs provide a basis for evaluating syntactic parsing, semantic interpretation, and reasoning in diverse science diagrams.
Takeaways & Limitations
The representation assumes separating arrow heads from arrow tails to represent arrows uniformly across head configurations.
Abstract
from arXiv · showhide
Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation and reasoning, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for over 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs.
1 Introduction
The paper frames diagram understanding as interpretation and reasoning over structured visual entities and relationships, addressing a gap left by natural-image research. It introduces DPG-based parsing and question answering, supported by the AI2D dataset.
- Diagrams represent complex concepts, events, and systems that can be difficult to express with text or natural images.
- Unlike natural images, diagrams suppress irrelevant visual signals while depicting higher-order relations and phenomena.
- Diagram interpretation comprises syntactic parsing of constituents and relationships plus semantic interpretation linking them to real-world entities and events.
- DPGs encode diagram constituents and relationships, while Dsdp-Net infers parses and Dqa-Net attends to useful relations for question answering.
- AI2D contains over 5000 science diagrams, over 150000 annotations, and more than 15000 multiple-choice questions, and the proposed models outperform reported baselines.
- The contributions define diagram interpretation and reasoning tasks, introduce DPG-based models, and provide a dataset with baselines for evaluation.
2 Background
Prior diagram-understanding methods were often rule-based, dependent on manual visual primitives, restricted to specific diagram types, or limited to vector representations. The paper proposes a unified approach for diverse diagrams.
- Earlier diagram-understanding techniques often used handwritten rules, manually identified primitives, or specific diagram sets.
- Some prior methods analyzed graphs and finite-automata sketches but operated only on vector representations.
- The paper proposes a unified diagram-understanding approach that builds on graphic-representation language and works across diverse diagrams.
- Abstract-image methods reduce low-level visual noise but still depict real-world scenes, unlike diagrams that can represent more complex phenomena.
3 The Language of Diagrams
The paper models diagrams as composite graphics containing a space, constituents, and relationships, then extends this language with diagrammatic objects and Diagram Parse Graphs.
- Graphic representations include structural, semiotic, and classification aspects applicable to diagrams, maps, and complex visualizations.
- A diagram consists of a graphic space, constituents, and relationships involving those constituents.
- Constituents can be illustrative, textual, diagrammatic, informative, or decorative, while relationships capture spatial and attribute-based properties.
- The representation is recursive because an individual constituent may itself be a composite graphic.
- DPGs add diagrammatic objects as primary entities and represent constituents as nodes and relationships as edges.
4 Syntactic Diagram Parsing
Syntactic parsing maps diagrams to DPGs by selecting a coherent subset of noisy node and relationship proposals. Dsdp-Net performs this selection sequentially using local features and long-range context.
- Syntactic Diagram Parsing: Syntactic parsing detects and recognizes diagram constituents and relationships, then finds the DPG that best explains the diagram.
- Syntactic Diagram Parsing: The parser generates node proposals with constituent detectors and edge proposals with relationship classifiers before selecting a subset using local and global cues.
- Syntactic Diagram Parsing: Arrow heads and tails are separated so arrows with single heads, multiple heads, or no heads can share one representation.
- Syntactic Diagram Parsing: The proposal space contains several hundred constituent proposals and several thousand relationship proposals per diagram, making exhaustive graph search impractical.
- Syntactic Diagram Parsing: Dsdp-Net sequentially presents relationship candidates to an LSTM, which predicts whether each relationship and its constituents should enter the DPG.
- Syntactic Diagram Parsing: The model is trained on sampled relationship sequences from diagrams, using proposal scores as sampling weights.
- Syntactic Diagram Parsing: Dsdp-Net models dependencies across time steps, rejects redundant or structurally incorrect relationships, handles variable candidate counts, and stops after image-space coverage.
- Syntactic Diagram Parsing: At test time, candidates are sorted by proposal score and selected relationships are added sequentially to form the final DPG.
5 Semantic Interpretation
The paper addresses semantic interpretation of diagrams through question answering, where DPG structure is used to connect diagram relations with answer choices. Dqa-Net encodes DPGs as facts and attends to relations relevant to each question.
- Semantic Interpretation: DPGs capture syntactic relationships but not the semantic meanings of constituents and relationships, which can vary across diagram types.For example, an inter-object linkage may denote consuming in food webs or evaporation in water cycles.
- Semantic Interpretation: Diagram question answering evaluates semantic interpretation by asking a model to select one correct choice from a diagram, question, and multiple-choice options.This formulation supplies a defined problem, evaluation criteria, and metrics.
- Semantic Interpretation: Dqa-Net encodes DPG relations into facts and uses attention to identify the fact most relevant to a question and answer choice.The model embeds questions and choices, embeds diagram relations, and compares their similarity.
- Semantic Interpretation: The model computes probabilities over answer choices from the best statement–relation similarity scores and trains with cross-entropy loss.The probability distribution is produced through a softmax layer.
6 Dataset
AI2D is a large dataset of grade-school science diagrams with rich structural annotations and associated multiple-choice questions. Its annotations cover constituents, relationships, and diagram-canvas connections.
- Dataset: AI2D contains more than 5,000 grade-school science diagrams annotated with constituent segmentations and relationships to each other and the diagram canvas.The dataset includes more than 118K constituents and 53K relationships.
- Dataset: AI2D includes more than 15,000 multiple-choice questions associated with its diagrams.The questions support evaluation of diagram interpretation and reasoning.
- Dataset: Figure 4 illustrates an AI2D diagram with rich annotations alongside a multiple-choice question.The figure provides a visual example of the dataset’s annotation and question format.
- Dataset: Images were collected using Google Image Search with seed terms derived from Grade 1–6 science textbook chapter titles and annotated through sequential Amazon Mechanical Turk phases.The phases label constituents, relationship categories, linkage endpoints, and other diagram relationships.
7 Experiments
The experiments evaluate proposal generation, DPG inference, and diagram question answering using constituent and relationship models. Dsdp-Net and Dqa-Net outperform their respective baselines, while relationship proposals remain challenging because local spatial cues are ambiguous.
- Proposal generation: AP 0.7829 versus 0.02: the constituent proposal approach substantially outperforms Edge Boxes for detecting blobs.The approach combines multiscale segmentation with canvas probability maps and Random Forest features.
- Proposal generation: AP 0.6748: the arrow-tail proposal model captures many targets but misses heavily overlapping and very thick arrows.The reported misses include arrows overlapping with more than three other arrows and arrows confused with blobs.
- Proposal generation: Relationship proposal AP is low for several categories because local spatial decisions cannot resolve inherent relationship-classification ambiguity.Relationship classifiers use spatial-layout and diagram-space features together with constituent detection probabilities.
- DPG inference: Dsdp-Net sequentially adds candidate relationships and constituents with an LSTM, outperforming Greedy Search and A* Search on graph similarity.The Jaccard Index for Graphs measures overlap between predicted and ground-truth node and edge sets.
- Diagram question answering: Dqa-Net outperforms the baseline on question answering when trained on both VQA and AI2D, indicating that DPG relations encode useful diagram semantics.The evaluation reports test-set accuracy and compares relation-aware Dqa-Net with standard visual question answering methods.
8 Conclusion
The paper frames diagram interpretation and reasoning as a research area extending beyond natural-image understanding. It releases AI2D and baselines for further work, with future work incorporating diagrammatic and commonsense knowledge into DQA.
- Conclusion: The paper concludes that diagram interpretation and reasoning raises research questions beyond natural image understanding.The conclusion positions diagram understanding as a distinct research direction.
- Conclusion: AI2D and the authors’ baselines are released to facilitate further research in diagram understanding and reasoning.The stated future direction is incorporating diagrammatic and commonsense knowledge in diagram question answering.