Source-linked AI summary
DocLLM: A layout-aware generative language model for multimodal document understanding
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, Xiaomo Liu
TL;DR
Visual documents pose a challenge because their meaning depends on both text and complex spatial layouts. DocLLM adds bounding-box-based spatial modeling to LLMs through disentangled attention and block infilling, then adapts the model with instruction tuning. The paper reports strong performance across document intelligence tasks and highlights scope boundaries involving OCR and zero-shot evaluation refinement.
Problem
Visual document understanding requires handling textual semantics together with complex spatial layouts, while conventional LLMs and vision-based multimodal models have important architectural limitations.
Method
DocLLM extends causal LLMs with bounding-box spatial representations, disentangled spatial attention, block-text infilling pre-training, and instruction tuning for document intelligence tasks.
Results
DocLLM demonstrates that spatial layout structure can support document intelligence tasks without a vision encoder, including form understanding, table alignment, and visual question answering.
Takeaways & Limitations
DocLLM provides a compact approach to multimodal document reasoning that preserves causal decoding while incorporating spatial layout information.
Takeaways & Limitations
The choice of OCR engines and layout parsers remains open, and the paper leaves practical comparisons for future work.
Abstract
from arXiv · showhide
Enterprise documents such as forms, invoices, receipts, reports, contracts, and other similar records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets.
1 Introduction
Visual document understanding requires jointly modeling text semantics and spatial layout, because enterprise documents have complex, irregular formats. DocLLM addresses this with lightweight spatial modeling, disentangled attention, block infilling, and instruction tuning for document intelligence tasks.
- Enterprise documents combine rich textual content with complex layouts, creating challenges for accuracy, contextual understanding, reliability, and generalization.
- DocLLM is a lightweight extension to standard LLMs that models text semantics and spatial layouts for visually rich document understanding.
- Bounding box coordinates provide spatial information without a vision encoder, preserving the causal decoder architecture while reducing model size and processing complexity.
- Disentangled spatial attention captures cross-alignment between text and layout modalities by extending transformer attention with cross-modal relationships.
- Block-level infilling addresses heterogeneous content and irregular layouts, while instruction tuning adapts DocLLM to document intelligence tasks.
2 Related Work
Related work spans text-based and multimodal language models, autoregressive infilling, and disentangled attention. DocLLM builds on these directions by modeling text semantics with spatial layout while avoiding expensive vision encoders.
- Text-based LLMs: Text-based LLMs evolved from transformer-based pretrained models such as BERT, GPT, and T5 toward larger models with stronger zero-shot generalization.The passage attributes this shift to increases in model parameters and training-data size.
- Multimodal LLMs: Multimodal LLMs extend language models to visual inputs, but general-purpose systems have not been rigorously evaluated on visually rich document understanding tasks.The literature distinguishes general-purpose multimodal models from models tailored to visually rich documents.
- Multimodal LLMs: DocLLM addresses multimodal document understanding by combining text semantics with explicit spatial-layout modeling instead of relying on complex, memory-intensive open-domain vision encoders.This design extends a unimodal architecture by adding spatial signals to text semantics.
- Autoregressive Infilling: Autoregressive infilling includes fill-in-the-middle with one sampled span and blank infilling with multiple spans, with the OpenAI FIM method reorganizing prefix, suffix, and middle segments for prediction.FIM uses [PRE], [SUF], and [MID] tokens to structure the reordered sequence.
- Disentangled attention: DocLLM applies disentangled attention to separate text and spatial information, motivated by prior work that computes attention from content and position through independent matrices.The figure describes masking text segments, prepending them with [S], replacing them with [M], and predicting them autoregressively while accounting for text–spatial cross-attention.
3 DocLLM Framework
DocLLM extends a causal transformer with separate text and spatial representations, using bounding boxes and disentangled attention to model their interactions. It is pretrained with block-level text infilling and instruction-tuned on diverse document-intelligence tasks.
- 3.1 Model Architecture: DocLLM integrates OCR-derived token bounding boxes as a distinct spatial modality alongside text in a causal decoder architecture.The model uses separate vectors for text and spatial information rather than additive positional encoding.
- 3.2 Disentangled Spatial Attention: Its attention mechanism decomposes interactions into text-to-text, text-to-spatial, spatial-to-text, and spatial-to-spatial scores.Separate spatial projection matrices and weighting hyperparameters control these cross-modal attention components.
- 3.3 Pretraining: DocLLM is pretrained on unlabeled documents with block-level masking and infilling rather than relying only on next-token prediction.Blocks can represent coherent text chunks or linear spans, and infilling conditions predictions on preceding and following blocks.
- 3.3 Pretraining: Block information is used only to construct the pretraining masking objective, not as an input feature or during instruction tuning and downstream tasks.The model is not told the number of tokens in a masked block, avoiding leakage of useful information.
- 3.4 Instruction Tuning: Instruction tuning uses varied templates and datasets spanning visual question answering, natural language inference, key information extraction, and document classification.The data mix includes both single- and multi-page documents, with layout hints such as separators, titles, and captions available during tuning.
4 Experiments
The experiments evaluate DocLLM on same-domain splits and held-out datasets using instruction-tuned and zero-shot baselines. DocLLM performs strongly across both settings, especially on layout-intensive tasks, while classification generalization remains weaker.
- Experimental settings: The evaluation covers same-dataset splits across 16 datasets and held-out datasets with unchanged tasks but different domains and layouts.SDDS tests train–test performance when tasks and domains remain similar; STDD tests transfer across domains and layouts.
- Baselines: The baselines include comparably sized and state-of-the-art LLMs under zero-shot prompting, plus recent document-intelligence LLMs in SDDS.Zero-shot prompts use OCR-extracted text without spatial information.
- SDDS results: 14 out of 16 datasets favor DocLLM over equivalent models in SDDS, with particularly strong performance on layout-intensive KIE and CLS tasks.DocLLM-7B excels on 12 of 16 datasets against the broader comparison set, while DocLLM-1B performs close to the larger model.
- STDD results: 4 out of 5 held-out datasets favor DocLLM over Llama2 in STDD, with the best overall score on two datasets, including a KIE task.DocLLM also beats specified multimodal baselines on DocVQA and KLC, despite their instruction tuning on those datasets.
- STDD results: Classification accuracy is notably lower in STDD, possibly because training used only one classification dataset.The paper presents this as a limitation on generalization to new classification datasets.
5 Ablation Studies
The ablations test disentangled spatial attention, block infilling, and decoder masking using out-of-sample next-token accuracy. Spatial-to-spatial attention and autoregressive block infilling perform best, while causal and prefix decoders are broadly comparable.
- Ablation design: The ablations evaluate disentangled spatial features, block infilling, and the masking strategy used for decoding.Configurations are compared with out-of-sample NTP accuracy using sampled pre-training data and unseen documents.
- Disentangled Spatial Attention: Spatial-to-spatial interaction yields the highest NTP accuracy, while vanilla text-only self-attention yields the lowest.Differences among other text–spatial interaction configurations are subtle.
- Autoregressive Block Infilling: Autoregressive block infilling achieves the best performance among causal learning, causal learning with spatial modality, and block infilling with spatial modality.The comparison isolates the contribution of the proposed infilling objective and spatial modality.
- Prefix Decoder and Causal Decoder: Causal and prefix decoders show marginally different performance across five configurations, with a slight edge for the causal decoder.The paper therefore uses a causal decoder for the remaining experiments.
6 Discussion and Findings
DocLLM is presented as a compact approach for structured document understanding and potentially layout-rich pre-training. The discussion also identifies OCR quality and zero-shot evaluation procedures as practical boundaries.
- Broader implications: DocLLM could incorporate e-books, e-publications, and other richly laid-out documents into generative pre-training without extensive preprocessing.The paper frames spatial-aware reading as treating documents as structured knowledge.
- Broader implications: Multi-page awareness handles page breaks and document boundaries across documents of varying lengths.This is contrasted with smaller multimodal models focused mainly on single-page documents and multimodal LLMs designed primarily for images.
- Limitations: The choice of OCR engine for cohesive infilling blocks remains open, and comparisons across OCR engines or layout parsers are left for future work.The discussion points to more accurate OCR as a possible source of improvement when available.
- Limitations: The reported zero-shot results may still be refined despite prompt engineering, repeated refinement, and post-processing across three independent prompt engineers.The authors explicitly acknowledge potential for further improvement in the zero-shot methodology.
- Limitations: The training observations on weight decay, learning rate, and schedulers lack robust validation.The paper reports these as internal training experiences rather than rigorously validated findings.
7 Conclusions
DocLLM combines bounding-box spatial information, disentangled text–spatial attention, and block-text infilling for document reasoning, then evaluates broadly across document intelligence tasks.
- DocLLM omits costly image encoders and uses bounding-box information to capture document spatial structure.
- Disentangled attention decomposes transformer attention to align text and spatial modalities in structured documents.
- A block-text infilling pre-training objective targets irregular layouts and heterogeneous document content.
- DocLLM surpassed equivalent models on 14 of 16 datasets for known tasks.
- The model generalized to previously unseen datasets in 4 of 5 settings.