Source-linked AI summary
Unified Pretraining Framework for Document Understanding
Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Nikolaos Barmpalios, Rajiv Jain, Ani Nenkova, Tong Sun
TL;DR
Document pretraining has largely remained language-dominated despite documents combining text, visual information, and spatial layout. UDoc integrates textual and visual features from semantic regions and uses three self-supervised tasks; pretraining improves results across downstream tasks, with reported gains on document benchmarks.
Problem
Many document pretraining methods are language-dominated despite documents containing visual information and spatial layout as well as text.
Source: §1 · p. 2Method
UDoc combines hierarchical text encoding with visual features from semantic regions and uses Masked Sentence Modeling, Visual Contrastive Learning, and Vision-Language Alignment.
Source: §1 · p. 2Results
Pretraining consistently improves all three downstream tasks; RVL-CDIP rises from 93.96 (UniDoc) to 95.05↑ (UniDoc∗) when the sentence encoder is unfrozen during fine-tuning.
Source: §4.3 · pp. 8, 10Takeaways & Limitations
The experiments indicate that multimodal inputs and proxy tasks can aggregate and align visual and textual information in document images.
Source: §5 · p. 10The sentence encoder is frozen during pretraining and fine-tuning due to computational constraints.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Document intelligence automates the extraction of information from documents and supports many business applications. Recent self-supervised learning methods on large-scale unlabeled document datasets have opened up promising directions towards reducing annotation efforts by training models with self-supervised objectives. However, most of the existing document pretraining methods are still language-dominated. We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. Each input element is composed of words and visual features from a semantic region of the input document image. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses, encouraging the representation to model sentences, learn similarities, and align modalities. Extensive empirical analysis demonstrates that the pretraining procedure learns better joint representations and leads to improvements in downstream tasks.
1 Introduction
Document pretraining must account for hierarchical text, visual content, and spatial layout, while existing methods are described as language-dominated. UDoc combines hierarchical text encoding, visual and textual features, gated cross-modal interaction, and three pretraining tasks.
1 Introduction
- Document pretraining faces word-level input constraints despite documents’ hierarchical structure, and text-rich or multipage documents can exceed input lengths.Word and sentence importance also depends on context.Source: §1 · pp. 1–2
- Documents require visual features and spatial layout as well as words, but many BERT-based methods omit multimodal content and alignment within semantic regions.The passage notes that words in forms and receipts are often related to local surroundings.Source: §1 · p. 2
- UDoc uses a hierarchical transformer to form sentences from words and documents from sentences, and combines convolution with self-attention for local and global dependencies.The paper says the hierarchy reduces computation complexity exponentially and increases the number of input words.Source: §1 · p. 2
- A gated cross-attentional transformer dynamically controls textual and visual influence for each semantic region, supporting cross-modal connections.The model jointly trains its CNN-based visual backbone and gated cross-attention encoder during pretraining and fine-tuning.Source: §1 · p. 2
- UDoc introduces Masked Sentence Modeling, Visual Contrastive Learning, and Vision-Language Alignment as pretraining tasks for learning textual, visual, and cross-modal information.The paper reports extensive experiments and analyses of these tasks and downstream performance.Source: §1 · p. 2
2 Related Work
Document-image pretraining differs from NLP and vision-language pretraining because text and visual information co-occur in spatially distributed semantic regions. UDoc differs from prior approaches by learning visual features from semantic regions and updating its image encoder during pretraining.
2 Related Work
- Document-image inputs combine spatially distributed text and visual information within semantic regions, unlike pure-text NLP inputs or vision-language objects paired with captions.Document images also include spatial layouts such as bounding boxes.Source: §2 · pp. 2–3
- LayoutLM adds 2D positions to OCR words but incorporates visual information only at fine-tuning, while LayoutLMv2 adds an image encoder and image-text pretraining tasks.SelfDoc uses pre-extracted RoI features and sentence embeddings with a cross-modality encoder.Source: §2 · p. 3
- Unlike SelfDoc’s fixed detector and LayoutLMv2’s fixed image regions, UDoc updates its image encoder and extracts visual features from semantic regions such as paragraphs, titles, figures, and tables.It also masks latent visual information and uses a contrastive task over quantized visual embeddings to learn contextualized visual representations.Source: §2 · p. 3
3 Method
UDoc combines text and visual features from document regions in a gated cross-attention Transformer. It pretrains their joint representations with sentence modeling, visual contrastive learning, and vision-language alignment.
3.1 Model Architecture
- UDoc pairs sentence embeddings with visual RoI features and position information, then feeds the multimodal elements to a Transformer encoder.OCR groups text into sentences; a sentence encoder and a ConvNet with RoI Align extract the respective embeddings.Source: §3.1 · pp. 3–4
- Product quantization discretizes visual RoI features into a finite set of representations, constraining the visual representation space for end-to-end image-encoder learning.The model selects codebook entries to form each quantized representation.Source: §3.1 · p. 4
- Gated cross-attention exchanges information between visual and textual branches, while modality-specific weights dynamically control each branch’s influence.The encoder uses position information and masked visual and textual features to produce contextualized multimodal representations.Source: §3.1 · pp. 4–5
3.2 Training Tasks and Objectives
- Masked Sentence Modeling masks whole sentences and predicts their embeddings from surrounding sentences and image-region context.The objective minimizes smooth L1 loss on the masked sentence embeddings.Source: §3.2 · p. 5
- Visual Contrastive Learning predicts the true quantized representation for a masked RoI from candidate representations, encouraging agreement for the same region.Its second term encourages more equal use of codebook entries.Source: §3.2 · p. 5
- Vision-Language Alignment aligns pairwise sentence and image-region similarities for text and image features belonging to the same region.UDoc’s full pretraining objective sums the MSM, VCL, and VLA losses.Source: §3.2 · pp. 5–6
4 Experiment
The experiments evaluate UDoc’s region-based pretraining and multimodal representations across document understanding tasks. UDoc improves over prior models on official FUNSD and CORD test sets, with results also reported for classification and object detection.
4.1 Pretraining UDoc
- UDoc pretrains on 1M IIT-CDIP images after excluding RVL-CDIP, using paragraph-level regions as semantic units that reduce word-level input-length pressure.The corpus is sampled from IIT-CDIP, which contains more than 11M scanned document images; some semi-structured documents still yield many single-word regions.Source: §4.1 · p. 6
4.2 Finetuning Tasks
- Fine-tuning covers form understanding on FUNSD, receipt understanding on CORD, document classification on RVL-CDIP, and object detection on PubLayNet.The evaluations use entity-level F1 for FUNSD and CORD, classification accuracy for RVL-CDIP, and MAP for PubLayNet.Source: §4.2 · pp. 7–8
4.3 Results and Discussion
- Without pretraining, vision-language models outperform single-stream baselines, while task results indicate that visual and language information contribute differently across tasks.Visual information dominates document classification, while language contributes substantially to form and receipt understanding; gated cross-attention also outperforms its non-gated version.Source: §4.3 · p. 8
- Pretraining improves all three evaluated tasks; combining MSM, VCL, and VLA yields further gains, while batch-wide negatives and image-text matching hurt performance.The ablation uses FUNSD, CORD†, and RVL-CDIP; the reported gains vary by task.Source: §4.3 · pp. 8–9, Table 2 · p. 8
- Adding MLM lowers entity-level F1: FUNSD falls from 87.38 to 83.76, and CORD falls from 98.75 to 98.63.The authors suggest tiny token-level RoI boxes may yield insufficiently discriminative visual features.Source: §4.3 · p. 9
- UniDoc outperforms previous models on official FUNSD and CORD test sets and achieves promising document-classification results.Table 3 reports the state-of-the-art comparison.Source: Table 3 · p. 9, §4.3 · p. 9
- UniDoc’s RVL-CDIP accuracy is 93.96 with EasyOCR versus 93.42 with Tesseract; unfreezing the sentence encoder raises it to 95.05 but increases training demands.Google OCR yields 94.10; the authors also report improvements on several downstream applications with UniDoc∗.Source: §4.3 · p. 10
- On PubLayNet detection, UDoc’s pretrained visual backbone outperforms an ImageNet-pretrained backbone.Table 4 reports MAP @ IOU [0.50:0.95] on the validation set.Source: Table 4 · p. 10, §4.3 · p. 10
5 Conclusion, Limitations, and Future Works
UDoc combines visual and textual information through self-supervised proxy tasks and is evaluated on form understanding, receipt understanding, and document image classification. The authors identify limits and future directions involving training constraints, spatial modeling, broader document coverage, OCR standardization, and privacy.
5 Conclusion, Limitations, and Future Works
- UDoc is evaluated on form understanding, receipt understanding, and document image classification; its proxy tasks aggregate and align visual and textual information.The authors conclude that task-specific fine-tuning may improve results while reducing annotation costs.Source: §5 · p. 10
- Pretraining documents may contain sensitive information, so the authors suggest exploring privacy-preserving learning alongside self-supervised learning.Source: §5 · p. 10
- Computational constraints require freezing the sentence encoder during pretraining and fine-tuning, while jointly training it with the visual backbone and cross-attention encoder is proposed as future work.The authors also identify better spatial-relation modeling for semi-structured forms as a research direction.Source: §5 · p. 10
- The authors propose extending pretraining to multipage and multilingual documents and unifying pure-text 1D and image-based 2D document pretraining.They also raise transferring knowledge from text-based document tasks such as summarization to image-based documents.Source: §5 · p. 10
- The authors call for standardized document-image datasets with preprovided OCR results and identify digital PDFs as a possible source of useful metadata.They note that OCR-tool differences are a major source of inconsistency among existing document-pretraining work.Source: §5 · p. 10