PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualMRC: Machine Reading Comprehension on Document Images

Ryota Tanaka, Kyosuke Nishida, Sen Yoshida

arXiv:2101.11272v2cs.CLcs.CV

TL;DR

VisualMRC targets the gap between text-focused machine reading comprehension and understanding the visual layout and content of real-world documents. The paper introduces a generative document-image QA dataset and extends text-pretrained sequence-to-sequence models with visual information. The resulting models outperform base sequence-to-sequence and state-of-the-art text-based VQA models, while diagram-object understanding remains a limitation.

  • Problem

    Existing machine reading comprehension work focuses largely on text-level understanding and does not adequately address the visual layout and content of real-world documents.

  • Method

    VisualMRC defines generative QA over document images and extends text-pretrained encoder-decoder models with ROI, OCR, layout, and saliency information.

  • Results

    The proposed models outperformed BART, T5, and M4C on VisualMRC, including a case where LayoutT5 generated the reference answer of 77.3%.

  • Takeaways & Limitations

    VisualMRC supports research on connecting vision and language understanding for systems that read and reason about real-world document images.

  • Takeaways & Limitations

    The models could not correctly identify diagram objects and their colors in an example, motivating further work on diagrams.

Abstract

from arXiv · show

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.

Introduction

VisualMRC addresses the gap between text-focused machine reading comprehension and understanding the visual layout and content of real-world documents. It introduces a generative document-image QA dataset and a model that transfers sequence-to-sequence language abilities to this setting.

  • Introduction: Current MRC studies mostly focus on text-level understanding, neglecting document layout and visual content such as text appearance, tables, and charts.
  • Introduction: Existing VQA tasks with text in images do not develop the ability to visually read and comprehend real-world documents.
  • Introduction: VisualMRC asks models to read and reason about text in document images before generating natural-language answers.
  • Introduction: The dataset additionally requires understanding document layout, visual text representations, non-text objects, and relations between objects.
  • Introduction: VisualMRC provides generative QA pairs and uses long abstractive answers to evaluate natural-language understanding and generation on document images.
  • Introduction: The proposed model transfers NLU and NLG abilities from text-pretrained sequence-to-sequence models to VisualMRC.
  • Introduction: The model outperformed the base sequence-to-sequence models and an existing state-of-the-art VQA model on VisualMRC.

Existing Vision and Language QA Datasets

Existing vision-and-language QA datasets cover text, diagrams, recipes, and other multimodal contexts, but VisualMRC emphasizes visually organized documents and generative understanding. Its design uses diverse contemporary webpages and long abstractive answers rather than narrowly extractive responses.

  • Existing Vision and Language QA Datasets: Text-based VQA datasets commonly address text in everyday scenes, book covers, diagrams, charts, or other specialized image types.
  • Existing Vision and Language QA Datasets: VisualMRC images contain more words and multiple pieces of text and visual content laid out in documents than the images in the cited VQA datasets.
  • Existing Vision and Language QA Datasets: DocVQA uses document images from a single source, whereas VisualMRC includes images from multiple domains of contemporary born-digital webpages.
  • Existing Vision and Language QA Datasets: VisualMRC provides long abstractive answers, while DocVQA provides short, extractive, SQuAD-like answers from a single document-text span.
  • Existing Vision and Language QA Datasets: Natural Questions provides HTML documents, but state-of-the-art models such as RikiNet do not use visual information.
  • Existing Vision and Language QA Datasets: Generative settings are considered important for open-domain questions because collecting answer candidates is highly costly, despite evaluation difficulty.

The VisualMRC Task and Dataset

VisualMRC defines generative question answering over document images, combining natural-language understanding with visual layout, ROI, and OCR processing. Its dataset uses crowdsourced annotations and spans diverse, information-rich webpage documents.

  • Task Definition: VisualMRC asks a model to generate an answer from a question and document image, rather than restricting answers to context word spans.The task decomposes image understanding into ROI detection and OCR subtasks.
  • Task Definition: ROI detection identifies bounding boxes and semantic classes, while OCR extracts word objects and their locations within each region.The dataset provides ground-truth ROIs, OCR words, and question-relevant ROIs.
  • Dataset Collection: The dataset was built from 10,197 webpage images collected across 35 domains, with crowdworkers validating content and annotating document regions.Images came from 5,599 full webpage screenshots licensed under Creative Commons.
  • Dataset Collection: Crowdworkers annotated nine ROI classes, Tesseract extracted OCR words, and workers created and validated three unique generative question-answer pairs per image.Question creators also selected the relevant ROIs needed to answer each question.
  • Dataset Collection: VisualMRC contains 21,015 training, 2,839 development, and 6,708 test questions split by URL domain.The domain-based split separates webpage sources across the dataset partitions.
  • Statistics and Analysis: Compared with TextVQA and DocVQA, VisualMRC has more diverse and longer questions and answers, substantially more document OCR text, and broader document topics.VisualMRC also provides ROI annotations, and 44.8% of images contain picture or data regions such as tables and charts.

Proposed Model

The proposed model extends a Transformer encoder-decoder with document structure, layout, appearance, and question-relevant token information for generative VisualMRC. It combines these inputs with saliency detection and multi-task training.

  • Architecture: The main module extends pre-trained BART or T5 encoder-decoder models to learn visual document layout and content while retaining their natural-language generation ability.The module uses a Transformer architecture and maps an input sequence to encoder embeddings.
  • Input Sequence: The input sequence concatenates the question with OCR words from detected regions, inserting a semantic ROI token before each region’s OCR sequence.Special tokens distinguish ROI classes, while [S] and [SEP] delimit the question and context for the selected base model.
  • Input Embeddings: Each encoder input embedding fuses token and sequence-position information with segment, relative-location, and visual-appearance embeddings.Segment embeddings identify ROI classes; location embeddings encode normalized bounding-box coordinates; appearance embeddings come from 2048-dimensional Faster R-CNN fc7 features.
  • Saliency Detection: The model predicts question-relevant token saliency from encoder outputs using learnable weights applied to each OCR token’s hidden state.The saliency detector is trained jointly with the sequence-to-sequence task.
  • Saliency Detection: Pseudo saliency labels mark an OCR token as relevant only when it appears in the answer and belongs to a relevant ROI.This supervision addresses the absence of reference labels for individual tokens.
  • Training: The main module minimizes the negative-log-likelihood loss plus a weighted saliency loss, while ROI detection is trained independently and OCR uses an untuned system such as Tesseract.Up to 100 detected objects with the highest score are selected per document image for the independently trained detection sub-module.

Experiments

Experiments show that incorporating document layout, visual content, and saliency into pretrained sequence-to-sequence models improves VisualMRC performance across evaluation settings, while object detection and diagram understanding remain challenging.

  • Main evaluation: LayoutBART and LayoutT5 outperformed BART, T5, and M4C on all reported metrics in the main evaluation.The baselines included text-only fine-tuned models and the state-of-the-art text-based VQA model M4C.
  • Ablation studies: Saliency-detector multitask training improved performance over models without the detector, except for LayoutT5 on METEOR.The improvement was smaller for LayoutT5, whose pretraining included a related machine reading comprehension task.
  • End-to-end evaluation: LayoutBART and LayoutT5 also outperformed the baselines on all metrics in the end-to-end setting.End-to-end evaluation included ROI detection with Faster R-CNN and OCR extraction with Tesseract.
  • End-to-end evaluation: ROI detection preserved performance relative to the main setting because it helps models use visual layout when OCR misorders tokens in complicated multi-column documents.ROI detection achieved 7.86% mean Average Precision, leaving room for improvement.
  • Ablation studies: Adding region labels, segment and location embeddings, and appearance embeddings generally improved text-only baselines, although appearance embeddings reduced BLEU-4.The appearance-embedding result was consistent with prior studies cited by the authors.
  • Semantic classes: LayoutT5 outperformed T5-Text across all metrics and semantic classes, with especially large improvements for Picture, Caption, Sub-Data, and Other regions.Both systems underperformed on Picture and Data classes relative to the other classes.
  • Human comparison: The best models achieved the highest BLEU-4, but humans scored significantly higher on the other metrics.The comparison indicates a remaining gap between machine and human performance.

Related Work and Discussion

The paper builds on vision-language, document-layout, and text-to-text modeling, using pretrained BART and T5 because VisualMRC requires abstractive natural-language generation alongside visual understanding.

  • Vision and language models: LayoutLM models text-layout interactions for document understanding but cannot consider visual content such as charts, which M4C and the proposed models can handle.LayoutLM performs well in form and receipt understanding and document image classification.
  • Discussion: In a Cape Verde example, LayoutT5 selected the relevant table value, while T5-Text was distracted by another percentage in the document.LayoutT5 generated the ground-truth answer, 77.3%.
  • Discussion: The models’ advantage depends on handling document layout, since text-only processing can confuse visually unrelated values.The example contrasts the full model with T5-Text, which does not use visual layout.
  • Text-to-text pretraining: The authors use BART and T5 as base models because VisualMRC requires natural-language generation and existing vision-language models are not pretrained on text-to-text tasks.The proposed models modify these pretrained sequence-to-sequence architectures to incorporate visual information.

Conclusion

VisualMRC defines generative reading comprehension over document images and combines a diverse dataset with layout-aware pretrained encoder-decoder models. The results support progress toward document-reading assistants, while data-object understanding remains an open limitation.

  • Conclusion: VisualMRC frames document-image question answering as a vision-language task requiring text reasoning, non-text understanding, and abstractive answer generation.The task extends beyond extracting a word span from an image.
  • Conclusion: The dataset contains 30,000+ questions on 10,000+ contemporary webpage images from multiple domains.Its design emphasizes varied document images and generative question answering.
  • Conclusion: The proposed models transfer pretrained NLU and NLG abilities from BART and T5 while learning document-image layout and content.The approach adds visual representations and auxiliary saliency detection to pretrained encoder-decoder models.
  • Conclusion: The models outperformed text-only BART and T5 and the state-of-the-art text-based VQA model M4C on VisualMRC.The reported comparison supports transferring pretrained generation ability to visual machine reading comprehension.
  • Limitations and future work: Future work targets more effective pretraining and improved understanding of tables, charts, and diagrams.These data objects remain an explicit scope for improvement.
Loading 2101.11272v2…