Source-linked AI summary
SelfDoc: Self-Supervised Document Representation Learning
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, Hongfu Liu
TL;DR
Document understanding requires contextual, multimodal representations, but existing approaches can be word-level, modality-limited, or dependent on annotated data. SelfDoc pre-trains component-level textual, visual, positional, and cross-modal representations from unlabeled documents, then adaptively fuses modalities for downstream tasks. It reports superior performance across multiple tasks with significantly fewer pre-training documents, while assuming identically ordered language and vision inputs for cross-modal encoding.
Problem
Document understanding must capture sequential context and complementary textual, visual, and layout information without relying solely on fully annotated training data.
Method
SelfDoc uses semantically meaningful components, separate language and vision encoders, cross-modal pre-training, and modality-adaptive attention for multimodal fusion.
Results
SelfDoc achieves superior performance on document entity recognition, classification, and clustering, using significantly fewer documents for pre-training than other methods.
Takeaways & Limitations
Component-level contextualization and cross-modal pre-training provide a generic document representation that supports multiple downstream applications.
Takeaways & Limitations
Cross-modal encoding assumes that language and vision inputs have identical ordering, with corresponding indices representing the same proposal.
Abstract
from arXiv · showhide
We propose SelfDoc, a task-agnostic pre-training framework for document image understanding. Because documents are multimodal and are intended for sequential reading, our framework exploits the positional, textual, and visual information of every semantically meaningful component in a document, and it models the contextualization between each block of content. Unlike existing document pre-training models, our model is coarse-grained instead of treating individual words as input, therefore avoiding an overly fine-grained with excessive contextualization. Beyond that, we introduce cross-modal learning in the model pre-training phase to fully leverage multimodal information from unlabeled documents. For downstream usage, we propose a novel modality-adaptive attention mechanism for multimodal feature fusion by adaptively emphasizing language and vision signals. Our framework benefits from self-supervised pre-training on documents without requiring annotations by a feature masking training strategy. It achieves superior performance on multiple downstream tasks with significantly fewer document images used in the pre-training stage compared to previous works.
1. Introduction
Document understanding must model contextual relationships across sequentially organized content while integrating textual, visual, and layout information. SelfDoc addresses these challenges with component-level, multimodal, task-agnostic pre-training and reports strong performance across downstream applications.
- Motivation: Document parsing is challenging because interpreting content depends on surrounding context and document organization.Headings can summarize subsequent text blocks, and captions can clarify related figures.
- Motivation: Documents combine textual and visual information whose meaning is complemented by layout structure.This multimodal organization distinguishes documents from plain text and ordinary images.
- SelfDoc: SelfDoc represents semantically meaningful components, such as text blocks, headings, and figures, rather than individual words.Component-level contextualization is intended to avoid redundant whole-document analysis for every word while preserving localized context.
- SelfDoc: SelfDoc separately learns textual and visual representations, then encourages cross-modal learning and adaptively fuses language and vision features.The framework uses a cross-modality encoder and modality-adaptive attention for downstream multimodal fusion.
- Evaluation: SelfDoc is pre-trained on unlabeled documents and fine-tuned for document entity recognition, classification, and clustering.The authors report leading performance on these applications and superior results using fewer pre-training samples.
2. Related Work
Prior document-analysis methods are often task-specific and annotation-intensive, while document pre-training has begun incorporating layout and multimodal information. SelfDoc extends this direction by introducing cross-modality learning during pre-training for contextualized document-component representations.
- Document Image Understanding: Earlier document-analysis approaches applied neural, grammatical, statistical, graph, and multimodal methods to segmentation, recognition, and classification.Several approaches model document structures or combine language and vision, but they target specific tasks.
- Document Image Understanding: Task-specific document models commonly require exhaustive annotations for document understanding.This motivates learning general representations from unlabeled document collections.
- Self-Supervised Learning: Self-supervised language pre-training established generic representations learned from large unlabeled corpora for downstream linguistic tasks.BERT and related models provide the conceptual foundation for document pre-training.
- Document Pre-training: LayoutLM adds two-dimensional positional information to BERT and introduces image embeddings during fine-tuning, while other work extends pre-training to heavily worded or multi-page documents.These methods establish document-specific pre-training but differ in when and how visual information is incorporated.
- Document Pre-training: SelfDoc introduces cross-modality learning during pre-training to contextualize document components across language and vision without annotation.This design targets multimodal information in document images before downstream task fine-tuning.
3. Methodology
SelfDoc represents document proposals with textual, visual, and positional features, then models within- and across-modality context before adaptively fusing language and vision features for downstream use.
- Input Modeling: Document object proposals from detected text blocks, titles, lists, tables, and figures form the framework’s basic input units.Textual features come from OCR and Sentence-BERT, while visual features come from Faster R-CNN RoI heads.
- Input Modeling: Component-level input reduces sequence length relative to word-level input, lowering the cost of quadratic contextualized attention.This is especially relevant for text-heavy documents such as scholarly articles.
- Single-Modality Encoder: Textual and visual proposals are processed by separate BERT-style encoders with multi-head attention, feed-forward layers, residual connections, and layer normalization.The encoders create contextual embeddings from surrounding proposals within each modality.
- Cross-Modality Encoder: The cross-modal encoder alternates two cross-attention functions with self-attention to connect language and vision representations.One function identifies agreement between modalities, while the other propagates contextual clues between corresponding proposal sequences.
- Modality-Adaptive Attention: Modality-adaptive attention assigns sample-dependent weights to language and vision features before residual and additive fusion.The weights are predicted from concatenated modality features by a nonlinear mapping network.
4. Experiments
Experiments evaluate SelfDoc on document entity recognition, classification, and clustering, using datasets, implementation comparisons, and ablations. Results show strong performance across tasks, with benefits from fewer pre-training documents and fewer fine-tuning labels.
- Results: SelfDoc outperforms the listed baselines on all three downstream tasks, except LayoutLM’s reported result obtained with cleaner data and a deeper CNN.The comparison includes task-specific and task-agnostic models, including LayoutLM.
- Results: Document clustering results indicate that proposal-level embeddings are more informative than word-level features for whole-document representation without fine-tuning.The observation compares SelfDoc with BERT and LayoutLM representations.
- Ablation and analysis: Adding whole-image VGG-16 features improves document classification by around 1%.The improvement addresses empty detections and uninformative OCR on low-quality or obscured documents.
- Ablation and analysis: With an 8 times reduction in fine-tuning data, SelfDoc surpasses LayoutLM by a larger margin than when fine-tuning uses much more data.The reduced-label setting uses the RVL-CDIP validation set for fine-tuning.
5. Conclusion
SelfDoc is a task-agnostic document-image representation framework that models semantic components with linguistic, visual, and layout information. It combines contextualized learning, cross-modal learning, and modality-adaptive fusion, achieving superior performance on multiple tasks with significantly fewer pre-training data.
- SelfDoc learns document-image representations at the semantic-component level while incorporating linguistic, visual, and structural layout information.
- The framework uses contextualized learning over sequential proposals, cross-modal language–vision learning, and modality-adaptive attention for multimodal fusion.
- SelfDoc achieves superior performance on multiple tasks with significantly fewer data for pre-training.
A. Visualization of Modality-Adaptive Attention
Figure 4 visualizes modality-adaptive attention for document classification, with blue and red covering areas representing language and vision weights. The visualization indicates that weighting shifts across document types according to their available textual and visual cues.
- The displayed cluster label set for 10 clusters includes email, file folder, form, handwritten, letter, news article, questionnaire, resume, scientific publication, and specification.
- Documents with heavy words or fewer visual clues receive larger wlang values.
- Forms with multiple font styles or unrecognizable handwriting receive larger wvisn values.
- Blue and red covering areas represent the values of wlang and wvisn, respectively.
B.1. Evaluative metrics
The evaluation uses cluster-assignment quantities based on co-occurrences between obtained and ground-truth clusters. Cluster labels are mapped to ground-truth labels before cluster sizes and total instance counts are interpreted.
- The metric uses nij, ni+, n+j, and n to represent cluster co-occurrences, cluster sizes, and the total number of instances.
- map(ŷi) maps each obtained cluster label to a ground-truth label using linear sum assignment.
B.2. Label sets
The experiments evaluate document clustering with label sets containing 4, 6, and 8 clusters. As the cluster count increases, additional document categories are included in the listed sets.
- The 4-cluster label set contains email, form, handwritten, and letter.
- The 6-cluster label set adds news article and resume to the 4-cluster categories.
- The 8-cluster label set further adds questionnaire and scientific publication.