PaperlayerView exact revision on arXiv

Source-linked AI summary

DocVQA: A Dataset for VQA on Document Images

Minesh Mathew, Dimosthenis Karatzas, C. V. Jawahar

arXiv:2007.00398v3cs.CVcs.IR

TL;DR

Document image understanding lacks a broad question-answering benchmark that tests both textual content and document structure. The paper introduces DocVQA and evaluates existing VQA and reading-comprehension baselines on it. Baselines perform unevenly, with heuristic methods below 1% accuracy and BERT models outperforming the best M4C baseline, while structural questions remain challenging.

  • Problem

    Holistic document-image understanding is difficult because task-specific extraction modules may not capture the scale and complexity of documents.

  • Method

    The paper introduces DocVQA, a dataset and task using natural-language questions over document images, and evaluates heuristic, VQA, and NLP baselines.

  • Results

    BERT models outperform the best M4C VQA baseline, while heuristic approaches achieve under 1% accuracy on validation and test splits.

  • Takeaways & Limitations

    DocVQA motivates methods that jointly use textual, visual, layout, and higher-level contextual cues to answer practical questions about documents.

Abstract

from arXiv · show

We present a new dataset for Visual Question Answering (VQA) on document images called DocVQA. The dataset consists of 50,000 questions defined on 12,000+ document images. Detailed analysis of the dataset in comparison with similar datasets for VQA and reading comprehension is presented. We report several baseline results by adopting existing VQA and reading comprehension models. Although the existing models perform reasonably well on certain types of questions, there is large performance gap compared to human performance (94.36% accuracy). The models need to improve specifically on questions where understanding structure of the document is crucial. The dataset, code and leaderboard are available at docvqa.org

1. Introduction

DocVQA reframes document analysis as answering natural-language questions over varied document images, requiring models to interpret text together with layout and other visual cues. It introduces a large dataset and evaluates baseline approaches to expose the challenge of holistic document understanding.

  • Motivation: Existing document-analysis processes are often task-specific, and their independent modules may not achieve holistic document-image understanding as document scale and complexity increase.The paper argues that these difficulties motivate a different, purpose-driven perspective.
  • Task formulation: DocVQA treats document visual question answering as a high-level task that conditionally drives document analysis algorithms to interpret document images.The intended system responds to ad-hoc information requests expressed as natural-language questions.
  • Task formulation: Answering DocVQA questions requires extracting and interpreting text while exploiting layout, forms, tables, marks, separators, diagrams, and style.The task therefore combines textual content with numerous non-textual and structural cues.
  • Task formulation: Unlike generic VQA and scene-text VQA, document VQA requires prior knowledge of written communication conventions and supports inherently open-ended answers.Document images convey high-density semantic information that cannot be handled solely through a closed answer dictionary.
  • Motivation: Earlier document VQA work focused on specific elements or collections, whereas DocVQA targets a generic task over a large, varied collection of real documents.The examples include data visualizations and book covers, while DocVQA spans broader document content.
  • Dataset and contributions: DocVQA contains 12,767 varied document images with 50,000 questions and answers categorized by reasoning requirements.The authors also evaluate heuristic, human-performance, scene-text VQA, and NLP baselines.

2. Related Datasets and Tasks

DocVQA extends visual question answering and reading comprehension to real document images rather than computer-readable text or narrowly defined image collections. It differs from prior datasets through varied real documents containing textual, graphical, and structural elements.

  • Reading comprehension and QA: Machine reading comprehension usually answers questions from computer-readable passages, whereas DocVQA uses document images as contexts.Extractive MRC commonly represents answers as spans of the supplied text context.
  • Visual question answering: Visual Question Answering provides natural-language answers to questions about images, with scene-text VQA specifically addressing questions requiring text understanding.ST-VQA and TextVQA are prominent scene-text datasets related to DocVQA.
  • Existing datasets: ST-VQA contains 31,000+ questions over 23,000+ images, while TextVQA contains 45,000+ questions over 28,000+ text-containing images.These datasets use public or OpenImages-derived image collections.
  • Existing datasets: OCR-VQA contains more than 1 million question-answer pairs over 207K+ book-cover images, with domain-specific template questions generated from metadata.Its questions and answers are tied to the book-cover domain.
  • Existing methods: Scene-text VQA methods use pointer mechanisms and OOV-capable embeddings to produce answers from an open answer space.M4C and MM-GNN are identified as recent top-performing methods in this area.
  • Domain-specific tasks: DVQA and FigureQA provide synthetic chart-focused VQA datasets, while TQA asks multiple-choice questions over multimodal contexts with computer-readable text.DVQA has more than three million question-answer pairs over 300,000 images, and FigureQA has over one million questions over 100,000 images.
  • DocVQA distinction: DocVQA differs by covering real document images across many types, including tables, forms, figures, and diverse textual, graphical, and structural elements.This broad collection contrasts with prior datasets centered on real-world images, charts, or book covers.

3. DocVQA

DocVQA is built from diverse document pages and 50,000 annotated questions, with multi-stage verification and question-type labels. Its statistics show dense, open-ended document content that differs from standard VQA datasets and often requires structural understanding.

  • Data Collection: Pages were hand-selected from the UCSF Industry Documents Library, prioritizing tables, forms, lists, and figures while reducing poor-quality binarized images.The collection process aimed to prevent image quality from becoming a bottleneck for VQA.
  • Data Collection: Questions were created and verified in three annotation stages, with ambiguous items flagged and unmatched answers reviewed or edited by the authors.Workers first proposed question-answer pairs, then independently supplied answers and assigned question types.
  • Data Collection: DocVQA contains 50,000 questions on 12,767 document images collected from 6,071 industry documents spanning 1900–2018 and five industries.The images include typewritten, printed, handwritten, and born-digital text.
  • Statistics and Analysis: The dataset categorizes questions by reasoning requirements, including table/list, form, and layout questions that depend on document structure.Layout questions can require understanding titles or headings to locate the answer.
  • Statistics and Analysis: 70.72% of questions are unique, while most begin with “what is the” and request dates, titles, totals, amounts, or names.The average question length is 8.12 words, and the average answer length is 2.17 words; 63.2% of answers are unique.
  • Statistics and Analysis: 182.75 OCR tokens per image is the DocVQA average, exceeding SQuAD 1.1’s 117.23 tokens and the no-more-than-13-token averages of real-world-image VQA datasets.Common OCR tokens substantially overlap with words appearing in answers.

4. Baselines

The baseline suite combines heuristic and upper-bound analyses with text-aware VQA and extractive reading-comprehension models. Experiments vary OCR-copy capacity and visual-object features to better match document images.

  • Heuristics and Upper Bounds: The study evaluates random-answer, random-OCR-token, longest-OCR-token, and majority-answer heuristics.These baselines measure performance from random selection, document OCR selection, or train-set answer frequency.
  • Heuristics and Upper Bounds: Three upper bounds test whether answers are present in a repeated-answer vocabulary, an OCR substring, or an OCR subsequence.OCR tokens are serialized from top-left to bottom-right order for the substring and subsequence bounds.
  • VQA Models: LoRRA and M4C are used as text-aware VQA baselines that incorporate image features and OCR tokens for open-answer prediction.LoRRA combines attention over visual features and OCR tokens, while M4C uses multimodal transformer-based iterative answer prediction.
  • VQA Models: Because DocVQA images contain many text tokens, experiments use larger dynamic OCR vocabularies and also test models without fixed vocabularies.Variants additionally omit object features because conventional visual objects are less applicable to document images.
  • Reading Comprehension Models: BERT extractive question-answering models provide reading-comprehension baselines that predict the start and end indices of an answer span.These models apply pretrained language representations to downstream extractive QA.

5. Experiments

The experiments evaluate heuristic, text-reading VQA, and BERT question-answering baselines on DocVQA, using Accuracy and ANLS alongside human and OCR-based upper bounds. Heuristic methods are weak, while BERT and M4C reveal different strengths across question types and remain imperfect on document understanding.

  • Evaluation metrics: Accuracy and Average Normalized Levenshtein Similarity (ANLS) are used to evaluate DocVQA systems.ANLS is the primary metric because minor OCR-related answer mismatches are not severely penalized.
  • Heuristic baselines: None of the heuristic methods achieves even 1% accuracy on the validation or test splits.The evaluation includes random, majority-answer, and other heuristic or upper-bound approaches.
  • VQA baselines: M4C’s ANLS improves by around 50% when its dynamic vocabulary increases from 50 to 500, whereas LoRRA depends strongly on its fixed vocabulary.For M4C, omitting object features performs slightly better than the original setting; adding more OCR tokens provides little help for LoRRA.
  • BERT baselines: All BERT models outperform the best M4C baseline, with bert-large-squad performing best and matching target answers exactly for around 55% of questions.BERT inputs are formed by serializing OCR tokens in top-left-to-bottom-right order and approximating answer spans using the first answer-string match.
  • Question-type analysis: Qualitative results show complementary errors: M4C succeeds on one layout question, BERT succeeds on one form question, and both fail on a step-by-step illustration.The experiments also compare model performance across question types against human performance.

6. Conclusion

The paper introduces DocVQA as a dataset and VQA task intended to promote purpose-driven document image analysis. Its initial results motivate combining visual and textual cues with both low-level document signals and high-level goals.

  • Contribution: DocVQA is introduced as a dataset and associated VQA task to inspire a purpose-driven approach in document image analysis and recognition.The task is intended to connect document interpretation with practical information-seeking goals.
  • Implications: The initial baselines motivate simultaneous use of visual and textual cues for answering questions about document images.The proposed direction includes text, layout, arrangements, purpose, relationships, and domain knowledge.

A. Screen grabs of Annotation Tool

The annotation process is documented through three stages, with separate screen grabs corresponding to stages 1, 2, and 3.

  • Annotation stages: Figure A.1, Figure A.2, and Figure A.3 show screen grabs from annotation stages 1, 2, and 3, respectively.The figures provide visual documentation of the staged annotation workflow.

B. Examples of Question Types

DocVQA questions are organized into nine types according to the reasoning needed to answer them. The appendix illustrates examples spanning forms, layout, handwriting, running text, photographs, tables, and figures.

  • Question taxonomy: DocVQA defines nine question types based on the kind of reasoning required to answer each question.Question types are assigned during the second annotation stage.
  • Examples: Examples cover form, yes/no, layout, handwritten, running text, photograph, table, and figure questions.The examples are presented in Figures B.1 through B.5, including a handwritten date in a form and questions based on plots.

C. Additional Qualitative Examples

Additional examples illustrate DocVQA’s diverse question types and expose where baseline models succeed, disagree, or fail, especially on document structure, figures, diagrams, and OCR-sensitive answers.

  • Annotation: The annotation pipeline collects questions, verifies answers and question types, and sends cases with no matching answers for author review.Annotators can skip documents, flag invalid questions or language issues, and reviewers may edit questions or answers before acceptance.
  • Question types: DocVQA questions span yes/no forms, spatially grounded dates, handwritten key:value fields, running text, photographs, tables, and plotted figures.Examples include identifying a checkbox state, locating dates, reading names and table cells, and finding a graph’s highest X-axis value.
  • Baseline behavior: BERT answers some non-running-text questions, while M4C outperforms BERT on logo and packaging text questions where BERT makes no prediction.The examples include an exact BERT match for a recipe question and a table answer differing only by an extra space.
  • Baseline behavior: Both models can answer one table question correctly yet repeat the first meeting’s value for a similar last-meeting question whose ground truth differs.They predict 6 for both meetings, although the first answer is 6 and the last answer is 5.
  • Structural reasoning: Both models fail questions requiring organizational-diagram or graph interpretation, including identifying the largest value on a vertical axis.These examples require understanding figures and diagrams rather than extracting text alone.
  • OCR sensitivity: OCR errors produce incorrect answers even when models locate the relevant image information, including fragmented dates and mismatched passenger names.A date is split into multiple tokens, while the passenger-name example also shows disagreement between human and ground-truth spelling.
Loading 2007.00398v3…