Source-linked AI summary
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, Xiang Bai
TL;DR
Text-centric document understanding must handle small text, high-resolution images, and the limitations of external OCR or existing visual encoders. TextMonkey addresses these challenges with shifted-window high-resolution processing, similarity-based token compression, and spatially grounded multimodal outputs. Across 12 benchmarks, it reports improvements in scene text, document, and key information extraction tasks, with 561 on OCRBench.
Problem
Document and scene-text understanding is limited by external OCR errors and costs, low-resolution visual encoders, and crop-induced loss of context for small text.
Method
TextMonkey combines Shifted Window Attention with zero initialization, similarity-based token resampling, and text grounding with positional outputs for high-resolution text-centric multimodal understanding.
Results
Across 12 benchmarks, TextMonkey improves scene text-centric tasks by 5.2%, document-oriented tasks by 6.9%, and key information extraction tasks by 2.8%, while reaching 561 on OCRBench.
Takeaways & Limitations
TextMonkey extends OCR-free multimodal document understanding to text spotting, grounding, positional responses, and screenshot-clicking tasks while improving benchmark performance.
Takeaways & Limitations
Requiring positional answers benefits image-based datasets such as DocVQA and SROIE but can hurt reasoning datasets such as ChartQA and InfoVQA.
Abstract
from arXiv · showhide
We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. Our approach introduces enhancement across several dimensions: By adopting Shifted Window Attention with zero-initialization, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability. It also learns to perform screenshot tasks through finetuning. Evaluation on 12 benchmarks shows notable improvements: 5.2% in Scene Text-Centric tasks (including STVQA, TextVQA, and OCRVQA), 6.9% in Document-Oriented tasks (such as DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, and WikiTableQuestions), and 2.8% in Key Information Extraction tasks (comprising FUNSD, SROIE, and POIE). It outperforms in scene text spotting with a 10.9\% increase and sets a new standard on OCRBench, a comprehensive benchmark consisting of 29 OCR-related assessments, with a score of 561, surpassing previous open-sourced large multimodal models for document understanding. Code will be released at https://github.com/Yuliang-Liu/Monkey.
1 INTRODUCTION
TextMonkey targets text-heavy document and scene-text understanding without relying on the conventional external-OCR pipeline. It combines high-resolution cross-window processing, token compression, spatially grounded outputs, and broad benchmark evaluation.
- External OCR pipelines can accumulate text-reading errors, add engineering and computational costs, and weaken connections between text and surrounding context.
- Pretrained visual encoders often use 224- or 336-pixel inputs, limiting recognition of small text; cropping can split related words and hinder position-sensitive tasks.
- TextMonkey uses sliding-window high-resolution processing with Shifted Window Attention and zero initialization to establish cross-window connections while stabilizing early training.
- Similarity-based token resampling identifies significant tokens, reducing token length while improving performance compared with random queries.
- Text grounding and positional information extend TextMonkey beyond question answering, improving interpretability and supporting screenshot-clicking finetuning.
- 5.2% improvement appears on scene text-centric tasks, 6.9% on document-oriented tasks, and 2.8% on key information extraction tasks across 12 benchmarks.
- 10.9% improvement is reported for scene text spotting, while OCRBench reaches 561 across 29 OCR-related evaluations.
2 RELATED WORKS
Related work divides text-image understanding systems into OCR-Model-Driven and OCR-Free methods. OCR-driven approaches integrate extracted text and layout but incur external-tool costs and inherited OCR errors, while OCR-free methods avoid OCR engines but often require task-specific finetuning.
- Text-related image-understanding methods are broadly categorized as OCR-Model-Driven or OCR-Free.
- OCR-Model-Driven methods obtain text and bounding boxes with OCR tools, then integrate text, layout, and visual data through model-based processing.
- OCR-driven systems require additional computational resources and longer processing durations, and may inherit OCR inaccuracies.
- OCR-Free methods avoid off-the-shelf OCR engines and include end-to-end Transformer-based and variable-resolution approaches.
- Despite avoiding OCR-tool limitations, OCR-Free methods still require finetuning for specific tasks, motivating instruction-tuned multimodal approaches for text-rich images.
3 METHODOLOGY
TextMonkey processes high-resolution document images with shifted-window visual encoding, token resampling, and position-aware task training. Its design preserves cross-window context while reducing redundant visual tokens and supports evidence localization through text-related tasks.
- Image Resampler: 448x448 sliding-window patches are processed with CLIP Transformer blocks, then combined with global features and language-aligned resampling.Shifted Window Attention connects information across window patches while retaining efficient local processing.
- Shifted Window Attention: Zero-initialized shifted-window layers keep image-encoder parameters stable during early training and support smoother optimization.The method uses random Gaussian initialization for A and zero initialization for B in the two-layer MLP modification.
- Overview: The overall framework combines resolution enhancement, cross-window information preservation, token reduction, and multi-task prompting.Figure 2 summarizes these components as a unified TextMonkey pipeline for multiple tasks.
- Token Resampler: 68/256 (26.6%), 571/1024 (55.8%), and 1373/2304 (59.5%) tokens are redundant at resolutions 448, 896, and 1334 using a 0.8 similarity threshold.The observed increase in repeated tokens motivates compression as resolution grows.
- Token Resampler: Similarity-based filtering identifies distinctive image tokens, and cross-attention aggregates all features through a Token Resampler instead of directly discarding tokens.The token filter ranks tokens by 1 minus their maximum similarity to other tokens before selecting and restoring their original order.
- Position-Related Task: The model adds text spotting, text reading, grounding, and positional cues in answers to strengthen connections between recognized text and its visual locations.Original question-answering tasks are retained alongside the position-aware modifications.
4 EXPERIMENTS
Experiments evaluate TextMonkey across document, scene-text, spotting, and structural tasks, alongside ablations of resolution, token compression, and cross-window attention. The model improves across benchmark groups and shows strong text localization and interpretation across diverse visual inputs.
- OCRBench Results: TextMonkey achieves average improvements of 5.2% on Scene Text-Centric VQA, 6.9% on Document-Oriented VQA, and 2.8% on KIE benchmarks.The evaluated groups include STVQA, TextVQA, OCRVQA, DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, WikiTableQuestions, FUNSD, SROIE, and POIE.
- Document Benchmarks results: TextMonkey leads on additional document benchmarks, scoring 71.5 in DocVQA, 30.6 in WTQ, 65.5 in ChartQA, and 68.0 in TextVQA.These evaluations use the task-specific metrics reported for DeepForm, KLC, WTQ, ChartQA, DocVQA, and TextVQA.
- Text spotting results: The model accurately locates and identifies text across natural images, documents, charts, and tables.Figure 5 visualizes predicted bounding boxes in red and ground-truth locations in green.
- Ablation studies: Zero initialization yields a 0.6% ChartQA gain, while cross-window connections improve SROIE by 0.1%, DocVQA by 1.5%, and TextVQA by 2.4%.Directly increasing resolution without token compression decreases DocVQA performance by 9.2%.
- Ablation studies: Similarity-based token filtering reduces token length and improves performance, while random queries significantly underperform the selected significant tokens.Replacing important tokens with random ones causes an average performance decline of roughly 12.7%.
- Structured prediction: TextMonkey can structure charts and tables into JSON, extending evaluation beyond answering toward structured visual-text interpretation.The paper motivates structuralization as a way to make image data easier for computers to parse and analyze.
5 DISCUSSION
The discussion examines grounding as an interpretability aid, compares positional representations, and identifies conditions where positional answers help or hurt. It also highlights unresolved directions concerning reasoning chains and position encoding.
- 5.1 Interpretability: Grounding information can distinguish hallucination, recognition errors, and correct answers paired with incorrect evidence.Examples include grounding to a white region, recognizing the wrong text at the correct location, and grounding to incorrect text despite a correct answer.
- 5.2 Chain-of-Thought: Requiring answer positions benefits image-based datasets such as DocVQA and SROIE but can hurt reasoning datasets such as ChartQA and InfoVQA.The paper attributes the negative effect to grounding partially interfering with comparison or quantitative reasoning needs.
- 5.2 Chain-of-Thought: Automating the construction of a coherent thinking chain is identified as a future research direction for improving reasoning capabilities.This proposal is presented as a subsequent step rather than an evaluated component of the current system.
- 5.3 Comparison Between Different Representations of Position: Points improve over rectangles by 0.7% on DocVQA and 0.9% on SROIE, while rectangles often outperform polygons.The paper presents points as potentially suitable when inference cost and representation length matter.
6 CONCLUSION
TextMonkey addresses text-heavy document and fine-grained text-analysis tasks through high-resolution modeling, token compression, and spatially oriented outputs. Its benchmark results support the approach while showing that resolution increases require efficient token management.
- 6 CONCLUSION: TextMonkey combines Shifted Window Attention with zero initialization and a Token Resampler to support higher resolution while reducing redundant tokens.The model also uses text-oriented tasks and positional information to improve spatial understanding and interpretability.
- 6 CONCLUSION: TextMonkey achieves excellent results across multiple benchmarks and supports screenshot interaction through its spatially grounded outputs.The conclusion also notes that directly increasing input resolution does not always improve performance, especially for much smaller images.