PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical multimodal transformers for Multi-Page DocVQA

Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny

arXiv:2212.05935v2cs.CVcs.AIcs.CL

TL;DR

Existing DocVQA datasets and methods focus on single-page documents, despite the multi-page documents common in real workflows. The paper introduces MP-DocVQA and Hi-VT5, a hierarchical method that summarizes page information before answer generation. Hi-VT5 processes sequences up to 20,480 tokens without increased model complexity and predicts the answer page alongside the answer.

  • Problem

    Existing DocVQA datasets and methods focus on single-page documents, although real document workflows require processing all pages together and long inputs can be truncated by token limits.

    Source: §1 · p. 1

  • Method

    The paper introduces MP-DocVQA and Hi-VT5, which encodes each page into question-conditioned summaries for answer generation and answer-page prediction.

    Source: §1 · pp. 1–2

  • Results

    Hi-VT5 accepts sequences up to 20,480 tokens without increasing model complexity and answers regardless of answer-page position.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Joint answer and page prediction can help interpret results by identifying the answer page, including cases where the answer is wrong or the predicted page is wrong.

    Source: §5.3 · p. 8

    The dataset’s questions were designed around information located in a specific place; questions requiring answers from different pages were judged unnatural by annotators.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Document Visual Question Answering (DocVQA) refers to the task of answering questions from document images. Existing work on DocVQA only considers single-page documents. However, in real scenarios documents are mostly composed of multiple pages that should be processed altogether. In this work we extend DocVQA to the multi-page scenario. For that, we first create a new dataset, MP-DocVQA, where questions are posed over multi-page documents instead of single pages. Second, we propose a new hierarchical method, Hi-VT5, based on the T5 architecture, that overcomes the limitations of current methods to process long multi-page documents. The proposed method is based on a hierarchical transformer architecture where the encoder summarizes the most relevant information of every page and then, the decoder takes this summarized information to generate the final answer. Through extensive experimentation, we demonstrate that our method is able, in a single stage, to answer the questions and provide the page that contains the relevant information to find the answer, which can be used as a kind of explainability measure.

1. Introduction

DocVQA methods and datasets focus on single pages, although real document workflows require processing all pages together. The paper introduces MP-DocVQA and Hi-VT5 to address this multi-page setting, combining answers with answer-page identification.

1. Introduction

  • MP-DocVQA requires methods to interpret each page’s text, layout, and visual elements to identify the answer page and answer the question.The paper frames DocVQA as reasoning across these document elements and the semantics they convey.Source: Figure 1 · p. 1, §1 · p. 1
  • Single-page DocVQA datasets and methods do not match real workflows, where all pages of a document need to be processed together.Transformer-based methods face quadratic input-sequence complexity, so token limits can truncate substantial portions of long documents.Source: §1 · p. 1
  • MP-DocVQA extends SingleDocVQA with questions on documents containing 1–20 pages.The paper also evaluates existing methods and provides the dataset, baselines, and Hi-VT5 code and weights publicly.Source: §1 · pp. 1–2
  • Hi-VT5 processes pages separately into question-conditioned [PAGE] summaries, which its decoder uses to generate an answer and predict the answer page.The proposed T5-based architecture extends input length to 20,480 tokens without increasing model complexity.Source: §1 · p. 2
  • The paper’s experiments examine framework components and the relationship between answer accuracy and answer-page identification.These are among the stated contributions alongside the dataset and method.Source: §1 · p. 2

2. Related Work

Related work spans document question-answering datasets and NLP or multimodal methods, while long documents motivate approaches that reduce sequence-length constraints. MP-DocVQA contributes a larger, diverse multi-page dataset for training and evaluation.

2. Related Work

  • DocVQA [17], InfographicsVQA, VisualMRC [22], DuReaderVis, DocCVQA, and TAT-DQA [31] cover differing document types and answer formats.The cited work includes screenshot and infographic questions, collection retrieval, and datasets with comparatively few genuinely multi-page documents.Source: dataset · p. 2, §2 · p. 2

dataset.

  • MP-DocVQA contains 46,176 questions across 5,928 multi-page documents and 47,952 page images, providing data for training and evaluating multi-page methods.The paper characterizes it as larger and more diverse than existing datasets.Source: dataset · p. 3

2. Related Work

  • For SingleDocVQA, BertQA uses a BERT-based classification head to predict answer-span boundaries; document-focused models combine OCR text and layout with visual features [7] [16] [28] [29] [18] [9].Donut [10] and Dessurt [6] take document images and questions as input, implicitly learning to read and understand image semantics and layout.Source: dataset · p. 3

dataset.

  • Longformer [1] uses global question-token and local context attention, while Big Bird [30] adds randomly selected global-attention tokens to support long sequences.The cited methods are presented as approaches to long-sequence modeling without increasing model complexity.Source: dataset · p. 3

3. MP-DocVQA Dataset

MP-DocVQA extends single-page DocVQA with questions over multi-page industry documents, using a 20-page limit. Its construction preserves realistic single-page questions while adding surrounding pages, and the final dataset is predominantly multi-page.

3. MP-DocVQA Dataset

  • MP-DocVQA contains 46K questions over 48K scanned page images from 6K industry documents, with varied layouts and text styles.The pages include forms, tables, lists, diagrams, pictures, and handwritten, typewritten, and printed text.Source: §3 · p. 3

3.1. Dataset creation

  • The authors retained realistic questions whose answers lie in a specific document location, while filtering questions that became ambiguous in the multi-page setting.Questions referring to a specific page rather than the whole document could become ambiguous when expanded to multiple pages.Source: §3.1 · pp. 3–4
  • The dataset extends SingleDocVQA by adding preceding and following source-document pages to each image-question pair, with contexts capped at 20 pages.For longer documents, the selected 20 pages include the page containing the answer.Source: §3.1 · p. 3
  • 85.95% of the final 46,176 questions concern multi-page documents; the dataset spans 5,928 documents and 47,952 page images.The dataset also includes single-page documents where applicable.Source: §3.1 · p. 4
  • The train, validation, and test split prevents document overlap by trimming context pages in affected cases, and OCR annotations are provided for all 47,952 images.OCR annotations also cover page images beyond the 20-page limit.Source: §3.1 · p. 4

3.2. Dataset statistics

  • MP-DocVQA has question and answer lengths similar to SingleDocVQA, but documents vary from 1 to 20 pages and contain 1 to 42,313 recognized OCR words.The paper also notes varied content distribution, text density, layouts, and visual elements across pages.Source: §3.2 · p. 4

4. Hi-VT5

Hi-VT5 processes each document page separately, summarizing question-relevant multimodal information into learned page tokens. It combines those summaries for answer generation and predicts the page containing the answer.

4. Hi-VT5

  • Hi-VT5 independently encodes each page’s OCR text and layout, visual patches, question tokens, and learnable [PAGE] tokens into question-conditioned page summaries.The concatenated page summaries form the document representation passed to the decoder for answer generation.Source: §4 · pp. 4–5
  • The model’s OCR-token representation combines semantic and spatial embeddings, while visual features come from projected Document Image Transformer patch embeddings.The paper describes spatial embeddings for word bounding boxes and a trainable projection for the visual features.Source: §4 · pp. 4–5
  • A classification layer uses the contextualized page tokens to predict the page containing the answer, alongside decoder-based answer generation.The paper presents page prediction as a kind of explainability measure.Source: §4 · pp. 4–5
  • Hi-VT5’s layout-aware denoising pretraining masks text spans and predicts them using page summaries, while exposing the encoder to rough masked-token locations.The task is intended to align layout and semantic representations and let [PAGE] tokens attend to other tokens.Source: §4 · p. 5
  • Training first uses two-page contexts containing the answer page and a neighboring page, then fine-tunes on full documents with encoder weights frozen.The authors report that two-page training offers a memory trade-off and comes close to whole-document results, though it lowers overall performance.Source: §4 · p. 6

5. Experiments

The experiments compare adapted DocVQA baselines under three multi-page input setups and evaluate Hi-VT5 on answer generation and page identification. Hi-VT5 leads in the more realistic setups, especially when answers occur late in the document, while its oracle performance trails T5.

5. Experiments

  • The experiments assess answer quality with accuracy and ANLS, and page identification with accuracy, comparing adapted transformer baselines including long-sequence models.The baselines include BERT, Longformer, Big Bird, LayoutLMv3, and T5.Source: §5 · p. 6, §5.1 · p. 6

5.1. Baselines

  • The oracle setup supplies only the answer-containing page; concat joins all page contexts, while max-conf. processes pages separately and selects the answer with the highest logit confidence.Oracle approximates single-page DocVQA and serves as a theoretical maximum; concat is the most realistic setup.Source: §5.1 · pp. 6–7

5.2. Baseline results

  • In oracle, T5 ranks first; in concat, Longformer leads, while T5 struggles when answers exceed its input and Big Bird’s ITC attention makes distant question-answer links harder to model.T5 performs worst in max-conf. because vocabulary softmax logits are unusable as cross-page answer confidence scores.Source: §5.2 · p. 7

5.3. Hi-VT5 results

  • Hi-VT5 outperforms the other methods on answering and page identification in concat and max-conf. setups, with a larger answer-position gap near document ends.In the oracle setup, Hi-VT5 drops below T5 because it decodes from compact page summaries rather than the full page representation.Source: Figure 4 · p. 7, §5.3 · p. 8
  • Page prediction adds an interpretive signal: Hi-VT5 sometimes predicts the correct page but answers incorrectly, and sometimes answers correctly despite predicting the wrong page.The latter cases suggest answers may come from learned prior bias rather than the document input.Source: §5.3 · p. 8, Figure 5 · p. 8

6. Ablation studies

The ablations indicate that hierarchical de-noising pretraining contributes the largest boost, while page prediction, final fine-tuning, and visual features also affect performance. The final fine-tuning phase is especially important for predicting answer pages across the dataset’s document lengths.

6. Ablation studies

  • Hierarchical de-noising pretraining provides the most significant boost by helping [PAGE] tokens better represent document content.Removing the answer page prediction module slightly worsens answering; the authors describe the two tasks as complementary.Source: §6 · p. 8
  • Without final fine-tuning, the page classifier can predict only the first or second document page because it learns only page indexes seen during training.The final phase adapts the decoder and page prediction module to MP-DocVQA’s maximum 20-page length.Source: §6 · p. 8
  • Removing visual features slightly worsens final scores, while the authors note that text and its position convey the most relevant information for grayscale documents.The caption identifies independent removals of 2D position embeddings, visual features, APPM, pretraining, and fine-tuning as the ablation components.Source: §6 · p. 8, Table 3 · p. 8

7. Conclusions

The paper introduces MP-DocVQA and Hi-VT5 to address multi-page document question answering. It reports that Hi-VT5 handles sequences up to 20,480 tokens without increased model complexity and predicts answer pages alongside answers.

7. Conclusions

  • Hi-VT5 accepts sequences up to 20,480 tokens without increasing model complexity and answers questions regardless of answer page location.The conclusion contrasts this with existing methods’ difficulty answering questions on final document pages.Source: §7 · p. 8
  • The work introduces the multi-page DocVQA task and releases the MP-DocVQA dataset.It also reports component studies and analyzes how page prediction can identify answers inferred from learned bias rather than input evidence.Source: §7 · p. 8

A. MP-DocVQA construction process

MP-DocVQA is constructed by extending SingleDocVQA documents with neighboring pages, then filtering questions judged ambiguous in a multi-page setting. The released dataset retains single-page cases, while most questions concern multi-page documents.

A. MP-DocVQA construction process

  • Appending preceding and following pages increased the number of page images from 12,767 to 64,057.The source documents came from SingleDocVQA.Source: A. MP-DocVQA construction process · p. 11
  • The released dataset removes questions containing “document” because repeated page titles can make them ambiguous, while retaining questions with other reviewed keywords.The authors kept questions involving words such as “image,” “appears,” or “graphic” after finding they were not always ambiguous and were relatively few.Source: A. MP-DocVQA construction process · p. 11
  • Questions posed over multiple pages make up 85.95% of the dataset; single-page documents remain included as a real-world case.The released filtered version was used in the experiments.Source: A. MP-DocVQA construction process · p. 11

B. Number of [PAGE] tokens

Hi-VT5 uses [PAGE] tokens to summarize each page, with the token budget shaped by decoder capacity and the number of pages. On MP-DocVQA’s 20-page documents, performance improves as the budget rises to 10 tokens, but not beyond.

B. Number of [PAGE] tokens

  • The token budget M is selected according to page count, ranging from 1,024 tokens for one page to one token per page for 1,024 pages.The study varies M to find an optimal setting.Source: B. Number of [PAGE] tokens · p. 11
  • With 10 [PAGE] tokens, Hi-VT5 performs similarly to 25, while performance decreases with more; the authors attribute this to harder training without added information.They report significant improvement over using one or two tokens and say 10 tokens can fully represent the information extracted from each page.Source: B. Number of [PAGE] tokens · p. 11, Table 5 · p. 11
  • For MP-DocVQA’s 20-page documents, the maximum number of [PAGE] tokens per page is 51.This maximum is constrained by decoder input length and the number of pages.Source: B. Number of [PAGE] tokens · p. 11

C. Document pages during training

Because training on 20-page documents was infeasible under resource limits, the authors examined training with fewer pages and selected two pages for the first stage. The returns diminished beyond two training pages.

C. Document pages during training

  • Training with 20-page documents was infeasible under resource limits; performance was significantly worse with one training page, while returns diminished beyond two.The authors therefore used two pages in the first training stage.Source: Table 7 · p. 12, C. Document pages during training · p. 12

D. Hyperparameters

The hyperparameter table reports settings for the baselines and Hi-VT5, whose three stages use different epoch counts. In page identification accuracy by answer-page position, Hi-VT5 outperforms the baselines by a large margin.

D. Hyperparameters

  • Table 8 lists hyperparameters used to train and evaluate the baselines and Hi-VT5; for Hi-VT5’s pre-training, training, and fine-tuning stages, the epoch counts are 1, 10, and 1.The table uses cross-entropy loss for training.Source: Table 8 · p. 12

E. Page identification accuracy by answer page position

F. Hi-VT5 attention visualization

The attention visualizations show that Hi-VT5’s [PAGE] tokens attend to different parts of the document, including the question and answer. One token consistently focuses on the provided answer.

F. Hi-VT5 attention visualization

Loading 2212.05935v2…