Source-linked AI summary
Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, Gabriela Pałka
TL;DR
Document understanding requires handling layout and visual information in addition to textual semantics, especially for richly formatted and scanned documents. TILT combines contextualized image features and layout-aware attention within a pretrained encoder-decoder Transformer, achieving strong results across document tasks while simplifying the workflow.
Problem
Document understanding must handle text, spatial layout, and visual structure because richly formatted and scanned documents do not provide sufficient plain-text context.
Method
TILT uses a pretrained encoder-decoder Transformer with 2D layout attention bias, contextualized visual embeddings, and case augmentation for multimodal document processing.
Results
TILT achieved state-of-the-art results on three of four considered tasks, including DocVQA and CORD, while matching previous best scores on SROIE and RVL-CDIP with a simpler workflow.
Takeaways & Limitations
The end-to-end framework handles key information extraction, question answering, and document classification without complicated preprocessing or postprocessing.
Takeaways & Limitations
Sequence labeling can require manual annotation or heuristic tagging when OCR errors, absent exact matches, or multiple matching values make span selection ambiguous.
Abstract
from arXiv · showhide
We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.
1 Introduction
Document understanding must combine text with layout and visual information because scanned and richly formatted documents do not expose reliable reading order or complete textual context. TILT addresses this challenge with an end-to-end encoder-decoder framework that unifies extraction, question answering, and classification.
- 1 Introduction: Document understanding spans NLP, computer vision, and layout analysis because contracts, forms, applications, and invoices contain complex spatial layouts.
- Importance of Spatio-Visual Relations.: Scanned and analog documents lack readily accessible layout and reading-order information, requiring systems to interpret structure and images alongside text.
- Limitations of Sequence Labeling.: Sequence labeling is constrained when expected values are absent from the document text or when OCR errors and repeated matches make span tagging ambiguous.
- Resorting to Encoder-Decoder Models.: Encoder-decoder models can generate values not explicitly present in the input, remove dependence on detected word order, and unify question answering and classification.
- Resorting to Encoder-Decoder Models.: The decoder-based approach simplifies processing by avoiding human-made heuristics and time-consuming rule engineering, while autoregressive decoding remains a known drawback.
2 Related Works
Related work incorporates spatial information and additional modalities into Transformer-based text processing, while TILT extends encoder-decoder language modeling to multimodal document inputs. Its design combines 2D attention bias, contextualized visual features, and order-independent decoding.
- Spatial-aware Transformers.: Transformer models for 2D documents improve on sequential models by incorporating layout through positional embeddings or spatially contextualized representations.
- Spatial-aware Transformers.: TILT adds a self-attention bias generalized to 2D distances and introduces word-centric masking across image and text modalities.
- Resorting to Encoder-Decoder Models.: Encoder-decoder models unify language tasks, and TILT extends this text-to-text approach to multimodal inputs.
- Multimodal Transformers.: Unlike earlier multimodal approaches, TILT uses visual features contextualized across multiple image-resolution levels before adding them to word embeddings.
3 Model Architecture
TILT extends a pretrained Transformer encoder-decoder with relative spatial attention biases and contextualized image embeddings, preserving a separation between semantic content and positional or spatial information.
- Transformer foundation: TILT uses a Transformer encoder-decoder as its core architecture for generative natural-language tasks.The model starts from the Transformer and uses a decoder-oriented framework.
- Spatial bias: Relative attention biases replace positional embeddings and incorporate sequential, horizontal, and vertical relationships between tokens.T5 supplies sequential bias, which TILT extends to two-dimensional spatial distances.
- Spatial bias: Spatial distances are grouped into 32 logarithmically growing buckets, with larger buckets representing greater token-pair distances.The bucketing groups similarly distanced token pairs while preserving coarser resolution at larger distances.
- Contextualized image embeddings: Contextualized image embeddings represent image-region semantics in the context of the page’s visual neighborhood.A convolutional U-Net processes the full page, and ROI pooling extracts features aligned with each token’s bounding box.
- Contextualized image embeddings: The visual encoder processes a 512×384 page into a 64×48×128 feature map before projecting token-region features to the model embedding dimension.U-Net skip connections provide multi-scale information from nearby and distant page regions.
- Multimodal input: Visual information is injected by adding contextualized image-region embeddings to semantic embeddings at the Transformer input.This preserves the distinction between semantic embeddings and structural biases while adding image semantics.
4 Regularization Techniques
The regularization strategy addresses overfitting risks in sequence-to-sequence extraction and improves robustness to casing, spatial scaling, and visual deformation in formatted documents.
- Motivation: Sequence-to-sequence extraction may overfit because documents generate one higher-dimensional training instance and many tokens are irrelevant.The authors therefore introduce regularization separately for each modality.
- Text regularization: The method addresses subword tokenization failures on unusually cased text, a recurring issue in formatted document datasets.Casing can be an important visual aspect in FUNSD, CORD, and DocVQA.
- Spatial regularization: Spatial-bias augmentation randomly scales horizontal and vertical token distances by factors sampled uniformly from [0.8, 1.25].The transformation simulates horizontal and vertical stretching or squeezing of document pages.
- Image regularization: Image augmentation applies affine transformations that alter position, angle, size, and shear while updating every token bounding box.The augmentation preserves parallel lines and uses a 0.9 probability of application.
5 Experiments
TILT is evaluated across information extraction, visual question answering, document classification, and layout-rich question answering using datasets spanning receipts, documents, and document images.
- Evaluation scope: The evaluation covers Key Information Extraction, Visual Question Answering, rich-document classification, and Question Answering from layout-rich texts.The experiments are designed to represent a broad spectrum of document-understanding tasks.
- Datasets: CORD contains Indonesian receipt images for information extraction across four categories and thirty subclasses.The dataset consists of receipts collected from shops and restaurants.
- Datasets: SROIE extracts company, date, address, and total values from scanned receipts, while DocVQA focuses on visual question answering.These datasets target extraction and question answering over document images.
- Datasets: RVL-CDIP classifies grayscale document images into 16 categories, including letters, forms, invoices, news articles, and scientific publications.OCR sources differ across datasets: Textract for DocVQA, Azure OCR for RVL-CDIP, and original OCR for SROIE and CORD.
- Dataset comparison: Table 2 compares the supervised-pretraining and evaluation datasets using statistics reported in thousands of documents or questions.The table is intended to provide additional dataset statistics alongside the task descriptions.
- Training setup: Training uses vanilla T5 initialization, unsupervised document pretraining, selected supervised-task training, and downstream finetuning.The Base and Large variants contain 230M and 780M parameters after adding visual-encoder weights.
Unsupervised Pretraining.
Unsupervised pretraining builds a general-purpose layout-aware model from a large document corpus, using salient-span text masking and corresponding image-region masking before downstream finetuning.
- Pretraining objective: Pretraining uses a T5-like masked-language objective that preferentially masks named entities rather than random tokens.This is described as salient span masking.
- Pretraining objective: Image regions corresponding to selected text tokens are masked with 80% probability during pretraining.The paired masking scheme links textual spans with their corresponding visual regions.
- Pretraining impact: Figure 6 compares CORD, DocVQA, SROIE, and RVL-CDIP against a baseline without supervised pretraining using metric differences.Orange labels identify datasets selected for the final supervised-pretraining run.
- Supervised pretraining: The supervised-pretraining dataset renders layout and word bounding boxes for diverse tasks, including originally plain-text datasets.Datasets with HTML use their original structure, while others are rendered with assumed fonts and document dimensions.
- Training schedule: Each supervised-pretraining dataset is trained for 10,000 steps or five epochs, with the latter condition intended to avoid rapid overfitting.The final downstream model is finetuned for two epochs using AdamW and task-specific hyperparameters.
- Supervised pretraining: WikiTable, WikiOps, SQuAD, and infographicsVQA were selected as a foundation because they performed robustly in downstream-value estimation.The investigation used fixed pretraining steps followed by finetuning.
5.2 Results
TILT achieved state-of-the-art results on three of four evaluated tasks and improved performance particularly on layout-dependent categories.
- TILT achieved state-of-the-art results on three out of four considered tasks.The evaluation covered Key Information Extraction, Visual Question Answering, rich-document classification, and layout-rich Question Answering.
- The results confirmed that layout- and vision-aware pretraining supports downstream comprehension of tables and other document structures.The model also leveraged supervised training from both plain-text and layout-based datasets.
- 0.33 points improved SOTA results on this dataset, with the largest gains in forms, tables, and yes/no questions.Forms increased from 89.5 to 94.6, tables from 87.7 to 89.8, and yes/no results from 55.2 to 69.0.
DocVQA.
On DocVQA, TILT used image tokens for documents lacking readable text and achieved a highly competitive accuracy while improving information extraction performance.
- DocVQA: TILT inserted at least 16 image tokens to carry general visual information when documents lacked readable text.These tokens covered adjacent document regions and received U-Net representations like regular text tokens.
- DocVQA: The model achieved SOTA results and improved the previous best score by 0.3 points when missing entities were generated as None.Manual review found some outputs differed from references only insignificantly, such as “2.00 ITEMS” versus “2.00”.
CORD.
On CORD, TILT’s corrected evaluation reached performance indistinguishable from the state of the art, while parameter counts were not directly comparable across architectures.
- CORD: 98.10 versus 98.17 produced results indistinguishable from the state of the art.The evaluation excluded OCR mismatches and corrected total entity annotation discrepancies using the established procedure.
- CORD: Parameter counts cannot directly compare TILT’s encoder-decoder with language models such as LayoutLMs and LAMBERT.The reported parameter number does not reflect computational cost, which may be similar for larger encoder-decoder models.
6 Ablation study
Ablations on DocVQA showed that spatial bias and visual embeddings are important, while case augmentation produced the strongest regularization effect and unsupervised pretraining reduced scores.
- Significance of Modalities: Removing the 2D layout positional bias caused a larger ANLS decrease than removing the U-Net visual feature extractor.The results identify spatial relations as crucial and contextualized image embeddings as beneficial to the encoder-decoder.
- Justifying Regularization: Removing case augmentation decreased performance most severely, while bounding-box and image alterations had comparatively small effects.The baseline was almost one point better than the equivalent non-augmented model.
- Impact of Pretraining: Unsupervised-only pretraining achieved significantly lower scores, with an impact comparable to removing spatial bias or visual embeddings.The authors note that this gap might vanish in larger TILT variants not examined in the paper.
7 Summary
The paper introduces an end-to-end encoder-decoder framework for layout-aware document understanding, combining text, layout, and image information across multiple tasks. It achieves state-of-the-art results on DocVQA and CORD, matches previous best scores on SROIE and RVL-CDIP, and uses a simpler workflow.
- The proposed encoder-decoder framework supports Key Information Extraction, Question Answering, and Document Classification end to end.It eliminates the need for complicated preprocessing and postprocessing steps.
- State-of-the-art results were achieved on DocVQA and CORD, while performance matched previous best scores on SROIE and RVL-CDIP.The paper reports these results alongside a much simpler workflow.
- TILT combines textual semantics, document layout, and image information through spatial and image enrichment of a Transformer.The layout-aware architecture represents spatial information while incorporating visual context.
- Regularization methods significantly improve the model’s results.