PaperlayerView exact revision on arXiv

Source-linked AI summary

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, Song-Chun Zhu

arXiv:2110.13214v4cs.CVcs.AIcs.CLcs.LG

TL;DR

VQA has largely focused on natural images, leaving abstract diagrams underexplored despite their semantic richness. This paper introduces IconQA and Icon645, benchmarks existing methods, and develops Patch-TRM; Patch-TRM outperforms current state-of-the-art VQA models across all three sub-tasks. The dataset is designed to test diagram understanding alongside diverse visual reasoning skills.

  • Problem

    Abstract diagrams remain understudied in VQA, while existing datasets mainly focus on natural images or restricted diagram types and templates.

  • Method

    The paper releases IconQA and Icon645, benchmarks eight VQA methods, and develops Patch-TRM with icon-pre-trained diagram embeddings and hierarchical patch processing.

  • Results

    Patch-TRM outperforms current state-of-the-art VQA models in all three IconQA sub-tasks.

  • Takeaways & Limitations

    IconQA provides a benchmark for abstract diagram understanding and visual language reasoning across perception and diverse cognitive skills.

  • Takeaways & Limitations

    IconQA is built entirely from abstract icon images and does not contain sensitive personal information such as gender or race.

Abstract

from arXiv · show

Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

1 Introduction

IconQA addresses the limited study of abstract diagrams in VQA by introducing a large-scale icon question-answering benchmark designed for perception and diverse cognitive reasoning. It also provides icon data for pre-training and a Patch-TRM baseline for benchmarking.

  • Existing VQA datasets mainly target natural images, while prior abstract-diagram datasets focus on domain-specific content or limited templates.
  • 107,439 questions span multiple-image-choice, multiple-text-choice, and filling-in-the-blank sub-tasks.
  • IconQA requires diagram recognition, text understanding, and reasoning skills including counting, comparison, arithmetic, logic, spatial reasoning, and commonsense.
  • Icon645 contains 645,687 colored icons across 377 classes for pre-training diagram representations.
  • Patch-TRM benchmarks existing VQA methods with a pyramid cross-modal Transformer and icon-pre-trained diagram embeddings.

2 Related Works

Prior VQA research largely emphasizes natural images, whereas diagram QA work addresses abstract visual reasoning but often relies on restricted diagram types or templates. These limitations motivate broader attention to abstract diagrams with informative semantics and practical applications.

  • VQA Datasets: Many VQA datasets use natural images and questions requiring visual and textual content understanding.
  • VQA Datasets: Natural-image VQA datasets pay little attention to abstract diagrams, despite their informative semantics and wide applications.
  • Diagram QA Datasets: Abstract diagram QA datasets include abstract scenes, NLVR, FigureQA, and DVQA, but diagrams or questions are often generated from limited templates.
  • VQA Methods: Early VQA methods combine image and question inputs with attention, while later approaches use object proposals or Transformer-based visual representations.

3 The IconQA Dataset

IconQA is a large-scale icon-based VQA dataset built from real-world math word problems, spanning three task formats and diverse reasoning skills. Its 107,439 questions cover icon objects, visual and textual understanding, commonsense, arithmetic, and other cognitive reasoning abilities.

  • Data analysis: IconQA uses icon images covering diverse real-world objects and supports recognition, spatial, counting, algebraic, commonsense, and other reasoning processes.The dataset comparison describes 388 object classes, while the released icon collection contains 377 retrieved classes.
  • Dataset construction: IconQA is built from real-world math word problems collected from textbooks and annotated by trained crowd workers.The source material comes from IXL Math Learning resources aligned with California Common Core Content Standards.
  • Dataset construction: 107,439 questions are divided into multi-image-choice, multi-text-choice, and filling-in-the-blank sub-tasks.The dataset is split into train, validation, and test sets with a 6:2:2 ratio.
  • Reasoning skills: Questions require multiple reasoning abilities, averaging 1.63 skills per question across 13 categories.Filling-in-the-blank questions require the most skills on average, at 1.81, and 9.25% require three skills.
  • Motivation and impact: IconQA targets a gap in existing diagram QA datasets, which often focus on domain-specific diagrams or limited generation templates.The dataset is intended to support follow-up research and applications such as K-6 tutoring assistants.

4 The Icon645 Dataset

Icon645 is a large-scale icon dataset created to pre-train vision backbones for IconQA diagrams. It addresses the mismatch between natural-image pretraining and icon-image representation learning.

  • Motivation: Icon645 was created because natural-image-pretrained backbones may inadequately represent IconQA’s abstract icon images.The authors retrieve 388 classes from Flaticon and remove 11 classes that cannot be retrieved.
  • Dataset construction: 645,687 colored icons across 377 classes comprise the Icon645 pretraining dataset.The icons range from 64×64 to 256×256 pixels and vary in colors, formats, and styles.

5 Benchmarks

The benchmarks compare attention-based, Transformer-based, blind-study, and human systems on IconQA. Patch-TRM uses hierarchical image patches, icon-pretrained visual features, and cross-modal Transformer fusion.

  • Patch-TRM: Patch-TRM parses diagrams into ordered hierarchical-pyramid patches and encodes them with a ResNet pretrained on Icon645.Question tokens and patch embeddings are fused through cross-modal Transformers and attention before classification.
  • Benchmark methods: The benchmark includes four attention-based VQA models and four Transformer-based VQA models.Attention baselines include Top-Down, BAN, DFAF, and MCAN; Transformer baselines include ViLBERT and UNITER among others.
  • Bias analysis: Three blind-study models test whether IconQA can be answered through random guessing, question-only bias, or image-only information.These models are designed to examine possible dataset shortcuts and biases.
  • Human evaluation: Human performance is evaluated across age groups on all three sub-tasks and thirteen reasoning skills.The study uses IconQA test questions presented through Amazon Mechanical Turk and records participants’ age groups.

6 Experiments

Experiments benchmark existing VQA methods and Patch-TRM across IconQA’s three sub-tasks and reasoning skills, alongside human, blind-study, attention, and ablation analyses. Patch-TRM consistently outperforms benchmark models, while domain-specific pre-training and hierarchical attention are identified as important design choices.

  • Analysis by Task Types: Patch-TRM outperforms current state-of-the-art VQA models in all three IconQA sub-tasks.The analysis attributes these improvements mainly to icon-image pre-training and hierarchical attention.
  • Analysis by Task Types: Humans outperform all benchmark models consistently across the three sub-tasks and most reasoning skills.Humans also perform more consistently overall, although models exceed them on several numerical reasoning skills.
  • Analysis by Task Types: Q-only and I-only blind-study results are close to random, indicating robustness against shortcut biases in IconQA.The studies support the dataset’s reliability in distribution.
  • Analysis by Reasoning Types: Patch-TRM exceeds average human performance on estimation, measurement, and probability reasoning skills.The authors interpret this as evidence of neural networks’ potential for basic mathematical reasoning.
  • Quantitative Analysis: Patch-TRM’s cross-modal attention assigns higher weights to image patches corresponding to the input question.Figure 9 visualizes this question-conditioned attention behavior.
  • Ablation Study: 0.95-2.49% performance decreases when icon-specific pre-training is removed from Patch-TRM.Using natural-image pre-training instead reduces performance, while attention improves results by up to 7% over simple concatenation.

7 Conclusion

The paper concludes that IconQA provides an open-source benchmark for abstract diagram understanding and visual language reasoning, while Icon645 supports diagram-encoding pre-training. It positions the resources for research spanning multimodal reasoning and smart education applications.

  • Conclusion: IconQA contains 107,439 questions, three sub-tasks, and thirteen cognitive reasoning skill types for evaluating abstract diagram understanding.The benchmark is evaluated with user studies, blind studies, and multiple attention-based and Transformer-based approaches.
  • Conclusion: Patch-TRM parses diagrams hierarchically and uses cross-modal Transformers with attention to learn joint diagram-question features.Icon645 provides icon data for pre-training the diagram encoding network used by Patch-TRM.
  • Conclusion: IconQA is envisioned to facilitate computer vision, natural language processing, and smart education applications such as tutoring systems.The stated vision includes applications related to science education.

Checklist

The checklist records the dataset’s purpose, provenance, access, licensing, ethical review, reproducibility information, and human-subject documentation. It also identifies IconQA as a broad abstract-diagram resource designed to address limitations of natural-image, template-generated, and domain-specific datasets.

  • Reproducibility and Access: The authors report providing code, data, training details, compute information, and asset licensing through the project materials and paper sections.The dataset download is available through the IconQA website.
  • Ethics: The datasets do not contain identifiable or offensive content, according to the checklist’s ethics documentation.The social-ethics discussion also states that IconQA lacks sensitive personal information such as gender and race.
  • Skill Categorization: Question metadata are mapped to one to three of thirteen skill categories using trigger-word rules designed after exploration by informed individuals.The trigger words are listed in Table 9.
  • Dataset Description: IconQA contains 107,439 instances in PNG and JSON formats under a CC BY-NC-SA license.Its original use case is training VQA systems.
  • Dataset Purpose: IconQA was created to provide broad VQA data for the abstract image domain and address limitations in existing datasets.The checklist contrasts it with natural-image datasets, template-generated diagrams, and resources requiring substantial domain-specific knowledge.
  • Provenance: The dataset was created by researchers from UCLA, Sun Yat-sen University, East China Normal University, and Columbia University, without funding or an associated grant.These provenance details are reported in the dataset documentation.

A.6 Composition

IconQA is a large-scale, self-contained icon-question dataset designed to test abstract diagram understanding and diverse visual reasoning. Its composition includes multiple task formats, metadata, quality checks, and documented scope boundaries.

  • Dataset composition: 107,439 instances are distributed across multi-image-choice, multi-text-choice, and filling-in-the-blank questions.The sub-task counts are 57,672, 31,578, and 18,189, respectively.
  • Dataset composition: Each IconQA instance includes a textual question, an image, optional image or text choices, and a correct answer.The dataset also associates questions with reasoning-skill types and grade levels.
  • Quality and preprocessing: Quality checks found one error among 1,000 multi-image-choice questions and none among 2,000 questions from the other two sub-tasks.Redundant instances and invalid diagrams, answers, or choices were filtered during preprocessing.
  • Scope and future use: IconQA covers 13 cognitive reasoning skills, while the authors plan greater diversity in diagram formats, grade levels, icon classes, and skill types.The dataset currently focuses on colored abstract diagrams and questions for third grade and below.
  • Scope and future use: Object-level parsing annotations and textual explanations are identified as beneficial additions for semantic diagram parsing and transparent visual reasoning.The current release contains questions, diagrams, and answers but not these fine-grained annotations.

B.1 Dataset Statistics

Icon645 is a large, labeled icon-image dataset created to pre-train image encoders for abstract icon classification and support IconQA modeling. It contains 645,687 instances across 377 classes, with preprocessing, labeling, and usage constraints documented.

  • Purpose: Icon645 was created to pre-train image encoders for abstract icon image classification.The authors use it to pre-train the abstract image encoder serving as Patch-TRM’s backbone.
  • Dataset composition: 645,687 colored icon images are categorized into 377 classes.Each image has a text label specifying its class.
  • Source and scope: The dataset is a sample of the Flaticon library containing only 377 icon classes that meet the paper’s requirements.It is not intended to represent all possible instances from the larger library.
  • Usage: The dataset has no recommended split, so users choose how to divide it for their applications.The dataset is described as self-contained, with all related information included.
  • Preprocessing: Preprocessing crops white space, removes black-and-white icons, and eliminates redundant instances using diagram similarity.The raw data was not saved in addition to the processed dataset.
  • Distribution constraints: Icon645 is free for non-commercial use and distributed under the CC BY-NC-SA license.The source, Flaticon, does not allow the data to be used commercially.

D.1 Crowd Sourcing Method

The human-performance study collected IconQA test answers through structured AMT tasks with demographic checks, attention checks, worker-quality thresholds, and post hoc filtering. After filtering, 54,896 effective answers remained from 64,467 collected test answers.

  • Collection procedure: 64,467 IconQA test answers were collected through AMT tasks assigned to three workers each.Each task contained 20 test questions and included age-group information.
  • Task design: Each AMT HIT included 2 age questions, 4 attention checks, and 20 real IconQA questions.The attention checks tested instruction comprehension and included a fake choice question for multiple-choice tasks.
  • Quality assurance: 54,896 effective question answers remained after filtering failed checks, inconsistent age responses, and implausibly fast or slow completion times.The authors characterize this as a decently large sample for human-performance evaluation.

E.3 Quantitative Analysis

Patch-TRM produces promising predictions on most IconQA examples but remains challenged by dense objects and multi-hop reasoning. The qualitative examples span all three IconQA sub-tasks and mark correct and incorrect predictions distinctly.

  • Quantitative Analysis: Patch-TRM achieves promising results for most IconQA problems but fails on some complicated cases.The reported failures involve both perception of dense objects and multi-hop reasoning.
  • Quantitative Analysis: Dense-object identification and multi-hop reasoning remain difficult for Patch-TRM.These difficulties are illustrated among the model’s IconQA test-set prediction examples.
  • Quantitative Analysis: Figure 18 presents five Patch-TRM prediction examples for each IconQA sub-task.The examples cover multi-image-choice, multi-text-choice, and filling-in-the-blank questions.
  • Quantitative Analysis: Correct predictions are highlighted in green and incorrect predictions in red.The color coding applies to the displayed test-set examples.
Loading 2110.13214v4…