Unifying Vision, Text, and Layout for Universal Document Processing

Zineng Tang; Ziyi Yang; Guoxin Wang; Yuwei Fang; Yang Liu; Chenguang Zhu; Michael Zeng; Cha Zhang; Mohit Bansal

Source-linked AI summary

Unifying Vision, Text, and Layout for Universal Document Processing

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal

arXiv:2212.02623v3Dec 5, 2022cs.CVcs.AIcs.CLcs.LG

TL;DR

Document AI requires unified handling of visually situated text, images, layout, and diverse task formats. UDOP addresses this with layout-induced representations, a Vision-Text-Layout Transformer, and unified generative pretraining across unlabeled and labeled data. It achieves state-of-the-art performance on 8 tasks, ranks first on the Document Understanding Benchmark leaderboard, and supports customizable document generation and editing.

Problem
Document AI must model strong text-image-layout correlations while supporting diverse tasks across domains and paradigms.
Method
UDOP uses layout-induced vision-text representations, a Vision-Text-Layout Transformer, and unified generative pretraining combining self-supervised objectives with supervised datasets.
Results
UDOP achieves SOTA on 8 tasks and ranks first on the Document Understanding Benchmark leaderboard, while enabling customizable document generation and editing.
Takeaways & Limitations
UDOP provides one foundation model for jointly processing and generating document text, vision, and layout across diverse Document AI tasks.
Takeaways & Limitations
Applying UDOP to non-English data, especially non-Latin writing systems, may require additional character embeddings and model modifications.

Abstract

from arXiv · show

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 8 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark.

1. Introduction

Document AI must jointly model text, visual content, and 2D layout while supporting diverse task formats. UDOP addresses this with a unified representation, Vision-Text-Layout Transformer, generative pretraining, and broad task integration.

Motivation: Document AI combines text with visually situated content and 2D spatial layout, creating stronger crossmodal interactions and diverse processing challenges.These challenges span information extraction, layout detection, classification, question answering, and related tasks.
Approach: UDOP integrates text and image features according to token locations, producing a layout-induced representation that strengthens vision-text interaction.The representation adds text-token embeddings to features from the image patches where those tokens occur.
Approach: UDOP uses a Vision-Text-Layout Transformer and homogeneous text-layout vocabulary to jointly encode and decode vision, text, and layout.Its modality-agnostic encoder, text-layout decoder, and vision decoder support a unified sequence-to-sequence generation framework.
Pretraining: UDOP combines holistic self-supervised objectives with supervised datasets during unified document pretraining.The objectives include layout modeling, text-layout reconstruction, and vision recognition, while supervised tasks cover layout analysis, extraction, classification, question answering, and NLI.
Capabilities: UDOP processes and generates text, vision, and layout together, including customizable document editing and generation.Masked autoencoding reconstructs document images from text and layout modalities.
Results: UDOP achieves SOTA on 8 tasks and ranks first on the DUE-Benchmark leaderboard, while also achieving SOTA on CORD.The evaluation covers FUNSD, CORD, RVLCDIP, DocVQA, and DUE-Benchmark.

2. Related Work

Prior multimodal research has explored unified architectures and generative task formulations, while Document AI adapts vision-language, joint-encoder, and text-only paradigms.

Unifying Model Architectures in Multimodal Learning: Multimodal models commonly use concatenated text and image embeddings or separate modality-specific towers with projection or fusion layers.
Unifying Tasks with the Generative Framework: Generative frameworks unify diverse tasks by converting objectives, images, and bounding boxes into sequence-generation targets.
Document Artificial Intelligence: Document AI systems include vision-language models, joint text-image encoders, and approaches that represent documents as text only.

3. Universal Document Processing

UDOP unifies document vision, text, and layout through spatially aligned representations and a Vision-Text-Layout Transformer. Its encoder fuses modalities, while its decoders generate text, layout, and image content within a sequence-to-sequence framework.

3. Universal Document Processing: OCR supplies document text tokens and their bounding boxes, while the document image is partitioned into encoded image patches.The input combines the document image, OCR tokens, and extracted layout structure.
Layout-Induced Vision-Text Embedding: UDOP adds each token embedding to the feature of the image patch containing that token’s bounding-box center.The layout indicator is 1 when the token-box center lies within the patch and 0 otherwise.
Layout-Induced Vision-Text Embedding: Patches without text retain their image features, while patches containing text use fused vision-text representations in the encoder.The resulting joint representations explicitly leverage spatial correlations between vision, text, and layout.
Vision-Text-Layout Representation: UDOP discretizes normalized bounding-box coordinates into vocabulary tokens so layout can be inserted into text context and generated.For vocabulary size 500, (0.1, 0.2, 0.5, 0.6) becomes <50><100><250><300>.
Position Bias: UDOP uses two-dimensional relative attention bias without one-dimensional position embeddings because joint embeddings and spatial bias encode document layout.
Vision-Text-Layout Transformer: The VTL Transformer uses a modality-agnostic encoder, text-layout decoder, and vision decoder within a unified generative architecture.The text-layout decoder generates text and layout tokens, while the vision decoder generates image pixels and cross-attends to the encoder.

4. Unified Generative Pretraining

UDOP uses a universal prompt-based sequence-to-sequence format to combine self-supervised and supervised document tasks. Its objectives jointly model text, layout, and vision, including masked image reconstruction from text and layout.

Unified task format: UDOP converts diverse document tasks into a universal generative format with task prompts and targets.The framework includes self-supervised and supervised objectives in a shared sequence-to-sequence scheme.
Self-supervised objectives: Joint text-layout reconstruction masks text and trains the model to recover both missing tokens and their bounding boxes.The target combines reconstructed text with discretized layout tokens.
Self-supervised objectives: Layout modeling predicts the positions of text-token groups from the document image and contextual text.The objective represents predicted positions as layout tokens.
Self-supervised objectives: Visual text recognition identifies text at specified image locations to learn vision-text correspondence.The task masks text spans and uses their locations as part of the input specification.
Self-supervised objectives: Masked image reconstruction adapts MAE to reconstruct document images using text and layout signals.UDOP adds character cross-attention and a vision-decoder design based on trainable placeholder embeddings.
Supervised pretraining: Supervised pretraining covers classification, layout analysis, information extraction, question answering, and document natural language inference.The labeled tasks use dataset-specific prompts and generated targets.

5. Experimental Setup

UDOP is evaluated as a pretrained generative model across document understanding datasets spanning extraction, classification, question answering, and table reasoning. The experiments include DUE-Benchmark and standard datasets with task-specific metrics and reported variability.

Model and data: UDOP uses a 794M-parameter configuration based on T5-large components and an MAE-large vision decoder.The model uses extended vocabulary tokens for sentinel and layout representations.
Evaluation datasets: The evaluation covers FUNSD, CORD, RVL-CDIP, DocVQA, and seven datasets in DUE-Benchmark.DUE-Benchmark spans document question answering, key information extraction, and Table QA/NLI.
Evaluation tasks: FUNSD and CORD evaluate information extraction with F1, while RVL-CDIP evaluates document classification with accuracy.CORD contains 30 labels across four categories and uses 800/100/100 train, validation, and test samples.
Reported results: UDOP achieves state-of-the-art performance on all seven DUE-Benchmark tasks and on CORD using one open-vocabulary generative model.The comparison contrasts UDOP with predominantly task-specific classification-based baselines.
Resolution study: 63.9, 64.3, and 65.1 are UDOP’s average DUE-Benchmark performances at 224, 512, and 1024 image resolution, respectively.At 224 resolution, UDOP already exceeds the previous best reported average of 62.9.

6. Analysis

UDOP’s analyses examine document-image reconstruction, content and layout editing, pretraining objectives, architecture variants, auxiliary training, and the role of vision. The results support high-quality generation and improvements from unified objectives and additional supervised training.

Visualization Analysis: Masked image reconstruction produces clear document contents that are nearly identical to originals on unseen documents despite high masking ratios.The reconstruction uses text and layout signals alongside masked image patches.
Visualization Analysis: UDOP performs title replacement, text addition, text replacement, and tilted text replacement in one model run.The generated content preserves contextual font, size, style, and orientation.
Ablation Analysis: Pretraining objectives designed for layout, text, and vision outperform a masked-language-modeling-only baseline.Masked image reconstruction and supervised learning provide further performance improvements.
Visualization Analysis: UDOP also edits document layout by changing line breaks and rearranging text in one model run.The figure compares the original document with the customized-layout output.
Ablation Analysis: UDOP-Dual replaces the unified encoder with separate text-layout and vision encoders to test modality-specific architecture.The variant has 1098M trainable parameters.
Ablation Analysis: Auxiliary training on QA datasets further improves UDOP and outperforms TILT on DocVQA and InfographicsVQA.The comparison uses an additional supervised training stage before evaluation.

7. Conclusion

UDOP unifies document vision, text, layout, and task formats in a generative foundation model. It achieves state-of-the-art performance across eight tasks and supports customizable document generation and editing.

UDOP unifies document vision, text, and layout through layout-induced vision-text representations and a Vision-Text-Layout transformer.
UDOP applies a generative framework to self-supervised and supervised document tasks.
UDOP achieves state-of-the-art results on 8 tasks and ranks first on the Document Understanding Benchmark Leaderboard.
UDOP enables customizable realistic document generation and editing.The appendix includes additional demonstrations, including neural editing examples.
The appendix reports additional performance, curriculum-learning, variance, and limitation analyses.It includes results for UDOP-Dual, image-resolution curriculum learning, standard deviations, and limitations and societal impacts.

B. Visualization Analysis

UDOP supports controllable document image generation and editing, including customized content additions. It also performs document question answering while predicting answer locations.

UDOP generates and edits document images with customized content at high quality.Additional examples include documents with vertical content.
UDOP maintains high generation quality when document content is vertical.
UDOP performs question answering while predicting the location of answers in documents.
UDOP-Dual performance is reported on FUNSD, CORD, and RVL-CDIP.

D. Supervised Pretraining Tasks

The supervised pretraining and evaluation tasks cover document classification, layout analysis, information extraction, question answering, and table reasoning across varied document domains.

RVL-CDIP provides 320k training, 40k validation, and 40k test document images across 16 categories.Categories include invoices, scientific publications, and forms.
PubLayNet contains over 360k medical-publication images labeled with document layout elements such as titles and paragraphs.
DocBank contains 500K pages with 12 semantic structure labels and bounding boxes for each token.Its splits are 400K training, 50K validation, and 50K testing pages.
Kleister Charity contains 2.7k invoice images and 21.6k entities for extracting fields such as dates, numbers, amounts, and vendor names.
PWC contains 2,291 leaderboards and asks for information such as task, dataset, and metric from complete papers.
The QA and table-reasoning datasets span web pages, visual documents, infographics, HTML tables, and open-domain table statements.WebSRC, VisualMRC, DocVQA, InfographicsVQA, WTQ, and TabFact provide these task formats and domains.

D.6. Finetuning Experiment Setting

Finetuning experiments use fixed optimizer settings for benchmark and dataset-specific tasks. Curriculum learning evaluates image resolutions from 224 to 1024, with performance increasing at higher resolutions.

DUE-Benchmark finetuning uses Adam with learning rate 5e-5, 1000 warmup steps, batch size 16, and weight decay 1e-2.The optimizer uses β1 = 0.9 and β2 = 0.98.
FUNSD and CORD use learning rate 3e-4, while RVL-CDIP uses 1e-3 under the same remaining settings.
Curriculum learning evaluates input image resolutions of 224, 512, and 1024 on DUE-Benchmark validation sets.
Performance further increases at 512 and 1024 resolution despite already competitive results at 224.
Reported results include standard deviations computed from 5 runs with different parameter-initialization seeds.

G. Limitations and Societal Impact

UDOP’s document generation capabilities create potential risks of malicious counterfeit, while applying the model to non-English data may require further modifications. The paper proposes access controls for sensitive documents and limited release of the vision generation model.

Societal impact: Document generation could be abused for signature forgery, check tampering, or fake medical and financial record generation.The authors identify malicious document counterfeit as a potential misuse of UDOP’s generation capability.
Mitigation: The planned mitigation is limited access to the vision generation model, potentially through an API.Sensitive submissions, including checks and personal IDs, are intended to be denied.
Limitations: Applying UDOP to non-English data, especially non-Latin writing systems, may require further model modifications.The paper specifically notes the need to include more character embeddings for the vision decoder to attend to.
Capabilities: UDOP’s document editing demonstrations include title replacement, text addition, text replacement, and tilted text replacement in one model run.The figure presents these edits as document-generation capabilities that motivate the counterfeit-risk discussion.
Capabilities: For VisualMRC, UDOP generates answers and predicts their regions of interest through layout tokens.The labeled regions in the dataset are at paragraph level.