Source-linked AI summary
DocFormer: End-to-End Transformer for Document Understanding
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, R. Manmatha
TL;DR
Visual Document Understanding requires jointly interpreting text, layout, and visual content, but correlating these modalities is difficult. DocFormer combines them with multimodal self-attention, shared spatial embeddings, and unsupervised pre-training, achieving state-of-the-art results across four datasets and sometimes outperforming models nearly 4x larger.
Problem
Visual Document Understanding requires understanding document structure and layout in addition to text, because text or text-plus-spatial features alone do not capture all relevant visual information and cross-modality correlation is difficult.
Method
DocFormer is an end-to-end encoder-only transformer that combines visual, language, and spatial features with multimodal self-attention, shared spatial embeddings, and three unsupervised pre-training tasks.
Results
DocFormer matches or surpasses state-of-the-art results on four varied VDU datasets and sometimes outperforms transformer models nearly 4x larger in parameter count.
Takeaways & Limitations
DocFormer learns generalized features through unsupervised pre-training while remaining end-to-end trainable without a pre-trained object-detection network for visual features.
Takeaways & Limitations
The authors identify improving generalizability to multilingual settings and additional document types such as infographics, maps, and webpages as future work.
Abstract
from arXiv · showhide
We present DocFormer -- a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).
1. Introduction
Visual Document Understanding requires models to interpret varied document tasks using text, spatial structure, and visual content. DocFormer addresses cross-modality correlation with multimodal attention and unsupervised pre-training tasks.
- Motivation: VDU covers token labeling, semantic grouping and relation extraction, and document classification across varied document formats.Examples include linking “DATE PREPARED” as a key to “1/29/74” as a value, while handling other text such as “C-5”.
- Motivation: Text and spatial features alone are insufficient because documents contain information distributed across language semantics, visual layout, and imagery.Visual features can capture font and layout information that text misses, while text captures language semantics that visual features miss.
- Motivation: Cross-modality feature correlation is difficult because text refers to semantic concepts while visual features correspond to image pixels and regions.The paper uses mapping “ITEM 1” to its visual region as an example of this challenge.
- Approach: DocFormer uses an encoder-only transformer with novel multimodal self-attention and shared spatial embeddings to combine document modalities.The architecture follows pre-training and fine-tuning and is designed to correlate text and visual features through spatial information.
- Approach: DocFormer introduces three unsupervised pre-training tasks, including learning-to-reconstruct and multimodal masked language modeling, to encourage multimodal feature collaboration.These two tasks are described as novel unsupervised multimodal tasks.
- Results: DocFormer achieves state-of-the-art results on four varied downstream VDU tasks and sometimes outperforms transformer models nearly 4x larger in parameter count.It is end-to-end trainable, uses no pre-trained object-detection network for visual features, and does not use custom OCR unlike some recent papers.
2. Background
Document understanding research combines image, spatial, and text features for structurally rich documents, while transformer architectures differ in how they combine modalities. Prior methods include grid-based CNNs, LayoutLM variants, and multimodal encoder-decoder models.
- Multimodal document understanding: Existing document understanding methods combine image, spatial, and text features for forms, tables, receipts, and invoices.Finding effective ways to combine these multimodal features remains an active research area.
- Earlier approaches: Grid-based invoice methods encode text pixels with character or word vectors and classify them into field types using convolutional neural networks.Example field types include invoice number, date, vendor name, and address.
- Transformer approaches: LayoutLM adds 2D spatial coordinates, 1D positions, and visual features from Faster-RCNN to BERT text-token representations.LayoutLM was pre-trained on 11 million unlabeled pages and fine-tuned for form processing, classification, and receipt processing.
- Transformer approaches: LayoutLMv2 treats visual features as separate tokens rather than adding them to corresponding text tokens and explores additional pre-training tasks.These changes were intended to make use of unlabeled document data.
- Transformer approaches: Other approaches use BERT encoders with graph-based relation classifiers or T5-based encoder-decoder architectures for document understanding tasks.BROS predicts entity relations, while Layout-T5 and TILT address question answering or downstream document understanding on document images.
3. Approach
DocFormer combines visual, language, and spatial features in an encoder-only transformer with multimodal self-attention and shared spatial weights. It is trained end-to-end with multimodal pre-training tasks that encourage visual and textual interaction.
- Architecture: DocFormer uses an encoder-only transformer and CNN visual backbone, with all components trained end-to-end.Its transformer layers enforce deep multimodal interaction through novel multimodal self-attention.
- Visual features: Visual features are extracted from ResNet50, reduced to the transformer dimension, flattened, and projected into a sequence of 512 embeddings.The representation uses d = 768 and N = 512.
- Language features: Language inputs are word-piece tokenized, limited to 511 tokens plus [CLS], and embedded through a trainable layer.Shorter sequences are padded with [PAD] tokens, which are ignored.
- Spatial features: Spatial embeddings encode word bounding-box coordinates and additional geometric features, with separate visual and language embeddings summed into trainable representations.Absolute 1D positional encodings are also used because transformer layers are permutation-invariant.
- Multimodal attention: Multimodal self-attention combines visual, language, and spatial information while sharing spatial weights across vision and language to support cross-modal correlation.The formulation also clips relative attention to emphasize local features.
- Pre-training: Pre-training includes multimodal masked language modeling and Text Describes Image, using visual features to reconstruct masked text and classify matched text-image pairs.For mismatched pairs, the learning-to-reconstruct loss is ignored because the mismatch would make reconstruction loss high.
4. Experiments
DocFormer is evaluated across four visual document understanding tasks using standardized fine-tuning and reporting procedures. It achieves strong results across sequence labeling, classification, and entity extraction, while ablations examine pre-training, spatial sharing, projection heads, and multimodal attention.
- Sequence Labeling: 83.34% F1 on FUNSD makes DocFormer-base state of the art, exceeding LayoutLMv2-base by 0.58 and using only 5M pre-training documents.LayoutLMv2-base achieves 82.76 after pre-training on more than twice as many documents.
- Sequence Labeling: 80.54 F1 without images surpasses LayoutLMv1’s 78.66%, suggesting multimodal pre-training transfers visual information to text-and-spatial fine-tuning.The DocFormer-base T+S model is pre-trained with I+T+S but fine-tuned without images.
- Document Classification: 96.17% accuracy on RVL-CDIP gives DocFormer-base the highest reported classification performance, exceeding TILT-large by 2.15 percentage points.TILT-large uses 780M parameters, more than four times the model size, and reaches 94.02%.
- Entity Extraction: 96.33% F1 on CORD lets DocFormer-base match or exceed prior base and large variants, while DocFormer-large reaches 96.99% and achieves state-of-the-art performance.CORD evaluates entity-level F1 on receipt field labeling.
- Ablation Study: Pre-training is especially helpful in low-to-medium data regimes and also improves performance and convergence on RVL-CDIP, while architectural ablations show multimodal self-attention contributes 1.08% to 3.89%.The reported experiments also evaluate shared spatial embeddings and deeper projection heads.
5. Conclusion
DocFormer combines multimodal modeling and unsupervised pre-training for Visual Document Understanding, achieving strong results across varied document tasks. The authors also identify broader generalizability as future work.
- DocFormer is a multi-modal, end-to-end trainable transformer for varied Visual Document Understanding tasks.
- Two novel unsupervised multi-modal pre-training tasks encourage DocFormer to learn generalized features without labeled supervision.
- DocFormer matches or surpasses state-of-the-art results on four datasets covering a variety of document types.
- DocFormer shows superior performance against strong baselines despite being one of the smallest models in its class by parameter count.
- Future work targets improved generalizability to multilingual settings and additional document types, including infographics, maps, and web pages.
6. Supplemental
The supplemental material provides implementation details, pseudo-code, complexity information, architectural illustrations, and qualitative visualizations for DocFormer. It reports examples of both successful predictions and observed errors across FUNSD and CORD.
- The supplement directs readers to the main paper for model formulation, performance numbers, further analysis, and ablations.
- Training used 100 epochs for FUNSD and Kleister-NDA, 200 for CORD, and 30 for RVL-CDIP.
- Pseudo-code: The pseudo-code describes multimodal self-attention with separate text and image attention computations, shared spatial embeddings, and a multimodal attended output.
- Downstream architecture: DocFormer uses a trainable dataset-specific linear head after pre-training, while the backbone including the visual branch is fine-tuned downstream.
- Architecture visualizations: The multimodal self-attention visualization contrasts traditional self-attention with shared spatial weights across text and vision.
- Visualizations and results: Qualitative examples show learned repetition and regularity, correct CORD entity-region predictions, and errors associated with annotation or token-classification cases in FUNSD.