PaperlayerView exact revision on arXiv

Source-linked AI summary

InfographicVQA

Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, C. V Jawahar

arXiv:2104.12756v2cs.CVcs.CL

TL;DR

InfographicVQA addresses automatic understanding of infographics, whose textual and visual elements must be jointly interpreted. It introduces a diverse VQA dataset emphasizing elementary reasoning and arithmetic, then evaluates two multimodal baselines that perform poorly on the new task. The dataset establishes a benchmark for infographic question answering and motivates research on better infographic-specific feature extractors.

  • Problem

    Infographics combine textual, graphical, and visual elements whose layout and relationships make automatic understanding challenging.

  • Method

    The paper introduces InfographicVQA, annotating infographic images with questions and answers involving text, tables, figures, visualizations, multiple cues, and discrete operations.

  • Results

    30,035 questions over 5,485 images form the dataset, while adapted M4C and LayoutLM baselines perform poorly on the new task.

  • Takeaways & Limitations

    InfographicVQA establishes a benchmark for progress at the meeting point of vision, language, and document understanding.

  • Takeaways & Limitations

    The visual object detectors used for visual tokens are not suitable for infographics and detect only a few object instances.

Abstract

from arXiv · show

Infographics are documents designed to effectively communicate information using a combination of textual, graphical and visual elements. In this work, we explore the automatic understanding of infographic images by using Visual Question Answering technique.To this end, we present InfographicVQA, a new dataset that comprises a diverse collection of infographics along with natural language questions and answers annotations. The collected questions require methods to jointly reason over the document layout, textual content, graphical elements, and data visualizations. We curate the dataset with emphasis on questions that require elementary reasoning and basic arithmetic skills. Finally, we evaluate two strong baselines based on state of the art multi-modal VQA models, and establish baseline performance for the new task. The dataset, code and leaderboard will be made available at http://docvqa.org

1. Introduction

InfographicVQA frames infographic understanding as a multimodal VQA problem requiring joint reasoning over text, layout, graphics, visualizations, and elementary numerical operations. It introduces a diverse benchmark and evaluates multimodal and layout-aware baselines.

  • Infographic understanding requires jointly reasoning over textual content, document layout, graphical elements, data visualizations, color schemes, and visual art.
  • InfographicVQA contains 30,035 questions over 5,485 images, including questions grounded on tables, figures, visualizations, and multiple cues.
  • Questions target elementary reasoning skills such as counting, sorting, and arithmetic operations, reflecting the numerical content common in infographics.
  • The dataset is positioned as a benchmark at the intersection of vision, language, and document understanding.
  • The study adapts M4C and LayoutLM as strong baselines for VQA on infographics.

2. Related works

InfographicVQA extends multimodal question answering to born-digital infographics with dense embedded text, diverse layouts, and interactions among textual and visual elements. Unlike related specialized or single-source collections, it draws infographics from thousands of sources without topic specialization.

  • Unlike textbook and recipe QA, InfographicVQA uses text embedded in images rather than machine-readable text supplied separately.
  • InfographicVQA differs from scene-text datasets because its born-digital images contain an order of magnitude more text tokens per image and richer layout interactions.
  • Chart VQA datasets use chart images rendered by plotting libraries and template questions, whereas InfographicVQA targets broader infographic understanding.
  • InfographicVQA is extractive except for numerical answers produced by discrete operations, distinguishing it from abstractive VisualMRC.
  • Existing infographic datasets are specialized or single-source, while InfographicVQA contains diverse layouts and designs from thousands of sources without topic specialization.

3. InfographicVQA

InfographicVQA is built through a two-stage annotation process that labels answer sources, evidence, and required operations. The dataset contains diverse infographic questions, with substantial numerical and OCR content reflected in its statistics and visual analyses.

  • Data collection and annotation: InfographicVQA was assembled from Internet infographics, cleaned for duplicates, and annotated in two stages with questions, answers, and additional validation/test answers.The second stage collected independently entered answers and retained unique lowercase answer strings.
  • Question-answer types: Question-answer types include four answer sources: Image-span, Question-span, Multi-span, and Non-extractive.Multi-span answers may contain unordered lists of image-text spans, while numerical answers may be non-extractive.
  • Question-answer types: Evidence types identify whether an answer relies on Text, Figure, Table/List, Map, or Visual/Layout information.The operation type separately records Counting, Arithmetic, or Sorting when such discrete operations are required.
  • Question-answer types: Counts in the validation distributions do not sum to 100% because questions can have multiple answer-source, evidence, or operation labels, and many require no listed operation.The distributions are therefore overlapping rather than mutually exclusive categories.
  • Dataset statistics: 30,035 questions span 5,485 images from 2,594 web domains, split into 23,946 training, 2,801 validation, and 3,288 test questions.The corresponding image counts are 4,406 train, 500 validation, and 579 test images.
  • Dataset statistics: Questions frequently ask about counts or percentages, answers are commonly numerical, and images contain an average of 217.89 OCR text tokens.The answer and recognized-text word clouds show similar common terms.

4. Baselines

The paper evaluates heuristic and state-of-the-art baselines for InfographicVQA, including M4C and a multimodal LayoutLM adaptation. The LayoutLM-based model combines textual, visual, and layout information for answer-span prediction, while upper bounds measure answer recoverability from vocabulary and OCR.

  • Baselines: The evaluated baselines include heuristics, upper bounds, and state-of-the-art models for visual question answering and document understanding.Heuristics predict random or majority training answers or a random OCR token.
  • Upper bounds: Vocab UB measures whether the answer appears in the common training vocabulary, while OCR UB measures whether it appears as a subsequence of serialized OCR tokens.Vocab+OCR UB counts questions covered by either upper bound.
  • M4C: M4C fuses question, OCR-token, and image representations with Transformers, then autoregressively decodes answers from a fixed vocabulary or detected OCR tokens.Its visual features come from ROI-pooled Faster-RCNN Box-head features for detected objects and OCR tokens.
  • LayoutLM: The LayoutLM adaptation uses SQuAD-style span prediction and therefore handles only questions whose answers come from image spans.Nearly 70% of validation and test questions have this answer-source type.
  • LayoutLM: LayoutLM inputs contain question and OCR tokens, represented by summed token, segment, 1D position, four 2D position, and visual embeddings.OCR-token coordinates are encoded through four coordinate-specific 2D embedding layers, while visual features are fused early with text input.
  • LayoutLM: The LayoutLM-based model maps textual, visual, and layout modalities into a shared space, adds them, and feeds the result into stacked Transformer layers.The model is pretrained with a masked visual-language task and then fine-tuned for span prediction.

5. Experiments and results

Experiments evaluate heuristic baselines, upper bounds, human performance, M4C, and LayoutLM on InfographicVQA. Results show that answer availability is relatively high, while detector-based visual features provide limited benefit and current baselines remain challenged by infographic reasoning.

  • Experimental setup: InfographicVQA evaluates ANLS and Accuracy, using Textract OCR transcriptions and bounding boxes for image text.Human test answers were collected from two volunteers, with each question answered by one volunteer.
  • Heuristics and upper bounds: More than 75% of questions have answers present in a fixed vocabulary or as OCR-token spans in default reading order.The Vocab + OCR upper bound indicates substantial answer extractability despite the task’s reasoning requirements.
  • M4C results: M4C performs best without detector finetuning and with only one full-image ROI feature, rather than the original object-feature configuration.The reported M4C comparison covers detector finetuning, object features, and the single full-image feature setting.
  • LayoutLM results: LayoutLM benefits from in-domain pretraining, while the best validation model uses a Full Img feature and the best test model uses no visual feature.The comparison varies visual-feature type, OCR visual features, and where the full-image feature is added.
  • Analysis: OCR-token visual features do not improve results, suggesting that relatively noise-free OCR token embeddings already provide sufficient information in this setting.The paper attributes the limited gain to the additional token-region visual information contributing little to performance.
  • Analysis: The object detectors identify few infographic objects, limiting the suitability of bottom-up visual features for this domain.Table 3 compares a Visual Genome detector with a document-layout detector and reports fewer detected instances on infographics.
  • Analysis: The baselines are compared with upper bounds and human performance across Answer-sources, Evidence types, and Operation types.Qualitative examples show isolated successes by LayoutLM and M4C, but both miss an example requiring subtraction of 77 from 100.

6. Conclusion

InfographicVQA introduces a dataset and task for visual question answering on infographics. Baseline results suggest that existing multimodal QA and VQA models perform poorly on this setting.

  • InfographicVQA introduces a dataset and associated visual question answering task for infographics.
  • Existing models designed for multimodal QA or VQA perform poorly on the new dataset.
  • The work aims to stimulate research on images combining layout, graphical elements, and embedded text.

A. Data collection

The dataset was assembled through staged collection and annotation procedures designed to reduce noise and improve answer coverage. Workers received example-based training, forum support, and follow-up annotation for validation and test questions.

  • Data collection: Infographics were collected from internet image searches, then deduplicated with perceptual hashing and commercial OCR.The initial search downloaded more than 10K images, and perceptual hashing removed nearly 2000 duplicates.
  • Annotation scheme and selection of workers: A pilot annotation produced over 40% noisy question-answer pairs, prompting example-based explanations of acceptable questions and answers.
  • Annotation scheme and selection of workers: Workers were selected through outreach and trained in a 90 minute webinar covering annotation requirements with many examples.
  • Annotation scheme and selection of workers: An online forum let workers resolve annotation questions with author feedback while uncertain images remained pending.Workers were encouraged to include screenshots when asking questions.
  • Annotation process: Annotation used two stages: initial question-answer collection followed by additional answers for validation and test questions.The first stage collected more than 30K questions before train, validation, and test splitting; the second stage retained unique answers.

A.3. Data release

The dataset release separates annotation access from image access according to use case. Annotations are offered under CC-BY, while images are downloadable for research and education and otherwise represented by original URLs.

  • Question-answer annotations will be made available under a CC-BY license.
  • Images will be downloadable as a single zip file for research and educational uses.
  • For other uses, the release will provide a list of original URLs for all dataset images.

B. Additional Statistics and Analysis of InfographicVQA dataset

Additional analyses characterize InfographicVQA’s topic coverage and compare question, answer, and OCR-token distributions with related datasets. Topic modeling uses OCR-recognized text from the images.

  • Topic coverage: LDA topic modeling on OCR-recognized image text indicates coverage of topics including energy, war, health, and social media.The analysis uses the Gensim implementation of LDA.
  • Dataset statistics: Question-length distributions are compared across InfographicVQA, TextVQA, ST-VQA, DocVQA, and VisualMRC.
  • Dataset statistics: Additional plots compare answer lengths and numbers of OCR tokens across InfographicVQA and similar datasets.
  • Dataset statistics: The analysis also reports the most frequent questions, answers, and non-numeric answers in InfographicVQA.
  • Analysis methodology: OCR-token statistics use Amazon Textract for InfographicVQA and release-provided OCR results for the comparison datasets.

C.1. Evaluation

The evaluation follows ST-VQA and DocVQA by using ANLS for text-extracted answers, supplemented by exact-match Accuracy. ANLS softens mismatches likely caused by OCR imperfections through a normalized edit-distance threshold.

  • Because over 70% of answers are copied verbatim from image text, evaluation follows the ST-VQA and DocVQA protocol with ANLS as the primary metric.ANLS was introduced for VQA evaluation in ST-VQA and is designed to handle OCR-related answer mismatches softly.
  • ANLS compares each predicted answer with ground-truth answers using lower-cased normalized Levenshtein distance.For each question, the score is based on the closest applicable ground-truth answer among the defined answer set.
  • A normalized distance below τ = 0.5 yields score 1 − NL, while distances at or above the threshold yield zero.The threshold filters mismatches unlikely to be explained by OCR errors.
  • Accuracy is the percentage of questions whose prediction exactly matches at least one ground-truth answer.Ground-truth and predicted answers are lowercased before both ANLS and Accuracy are computed.

C.2. Experimental setup for M4C

The experimental setup specifies implementation and optimization details for M4C, alongside extractive QA preparation for LayoutLM and supplementary qualitative examples. OCR tokens are serialized in reading order to identify answer spans for LayoutLM training.

  • Experimental setup for M4C: M4C uses the official implementation from the MMF multimodal learning framework.
  • Experimental setup for M4C: M4C training used 4 NVIDIA RTX 2080Ti GPUs, 12 decoding steps, a 4-layer transformer with 12 attention heads, and dropout ratio 0.1.
  • Experimental setup for M4C: The M4C models were optimized with Adam using batch size 128 for 24,000 iterations and base learning rate 1e-04.Training also used warm-up learning factor 0.2 and 2,000 warm-up iterations.
  • LayoutLM data preparation: LayoutLM is fine-tuned for SQuAD-style extractive QA by predicting the start and end tokens of an answer span.OCR tokens are serialized in natural reading order, and matching answer subsequences are used to mark spans.
  • Qualitative examples: Figures D.1–D.7 provide qualitative examples of answer types, evidences, and operations discussed in the paper.These examples supplement the qualitative results in Section 5.2.
Loading 2104.12756v2…