UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding

1 Introduction

• Existing large multimodal models lack text detection and have not effectively used connections among OCR tasks to improve individual task performance.The introduction also identifies inconsistencies between pre-training and fine-tuning data distributions as a concern.Source: §1 · p. 1
• UniDoc is presented as the first large multimodal model capable of simultaneous text detection, recognition, spotting, and multimodal understanding.The work also contributes a large-scale multimodal instruction-tuning dataset for text detection, recognition, and spotting in text-rich images.Source: §1 · pp. 1–2
• UniDoc unifies detection, recognition, spotting, and multimodal understanding in a natural-language-instruction-driven framework.Spotting is described as end-to-end text detection and recognition; the authors say task interactions elevate existing capabilities.Source: Figure 1 · p. 2
• UniDoc achieves state-of-the-art performance across multiple publicly available benchmark datasets.The authors report extensive quantitative and qualitative experiments validating its effectiveness.Source: §1 · p. 2

2.1 Instruction Tuning

• Instruction tuning aligns language models with human intent by teaching them to follow natural-language instructions and support multitask learning and generalization.The review cites self-instruction for Alpaca and dialogue-based instruction fine-tuning for Vicuna.Source: §2.1 · p. 2

2.2 Multimodal Instruction Tuning

• Multimodal instruction-tuned models combine language models and visual encoders, using approaches such as modality mapping, added visual-feature layers, and adapters.Examples reviewed include MiniGPT-4, LLaVA, mPLUG-Owl, InstructBLIP, and X-LLM.Source: §2.2 · p. 2
• Text-focused approaches add recognition pre-training, document-comprehension data, or visual grounding, but fall short of comprehensive detection, recognition, and spotting.The review also says these approaches do not effectively harness reciprocal enhancement from learning these capabilities together.Source: §2.2 · p. 2

3.1 Model Architecture

• UniDoc extracts image features with CLIP-ViT-L/14, projects them to the LLM embedding dimension, and concatenates them with instruction embeddings before Vicuna generates a response.The architecture incorporates both pre- and post-Transformer-layer grid features; the visual embedding serves as a soft prompt.Source: §3.1 · pp. 2–3

3.2 Unified Multimodal Instruct Tuning

• Pre-training freezes the visual and language models and trains only the projector to align their features, using detection, recognition, spotting, and image captioning instructions.The authors associate the OCR tasks with high-level semantic understanding and captioning with natural-scene understanding.Source: §3.2 · p. 3
• Fine-tuning unfreezes the LLM and projector and adds multimodal understanding of text-rich images to the pre-training tasks.The paper states that the tasks mutually enhance one another.Source: §3.2 · p. 3

4 Dataset Construction

• Pre-training data combines 595K CC3M natural-scene images with captions and 600K image-text pairs collected from PowerPoint presentations.The authors selected slides for their varied combinations of elements and relatively large, legible text.Source: §4 · p. 3
• The proposed dataset examples feature varied font sizes, styles, and colors alongside a rich array of visual elements.Source: Figure 3 · p. 4
• The preparation process filters small-text and noisy images, extracts text and box annotations with an in-house OCR tool, and uses GPT-4 to diversify instructions.Detection, recognition, and spotting data each account for one-third of the total.Source: §4 · p. 4
• Fine-tuning extends 16K LLaVAR instruction-following examples with OCR instructions and adds 150K OCR instruction data from the pre-training stage.Detection, recognition, and spotting each constitute one-third of the OCR data.Source: §4 · p. 4

5.1 Training Details

• Training uses one epoch per phase, batch sizes of 128 for pre-training and 32 for fine-tuning, and 224×224 input images.The study notes that larger resolutions are likely to yield better results because more text is discernible.Source: §5.1 · p. 4

5.2 Evaluation Metrics

• Detection is evaluated with F-score, while recognition and visual question answering use accuracy based on whether the response contains the ground-truth string.The paper denotes these metrics as F and A, respectively.Source: §5.2 · p. 4

5.4 Ablation Studies

• Spotting instructions consistently outperform recognition instructions, and a case study shows spotting mitigates recognition omissions.The instruction comparison spans multiple recognition datasets.Source: Figure 6 · p. 5, Figure 7 · p. 6, §5.4 · p. 7

5.3 Comparison with Other LMMs

• UniDoc reports state-of-the-art recognition scores across benchmarks, state-of-the-art or comparable multimodal results, and detection of irregular scene text.The multimodal comparison covers VQA, KIE, and HMER benchmarks.Source: §5.3 · pp. 5–6, Table 4 · p. 6
• Recognition examples span varied fonts, styles, blur, and non-horizontal text, with UniDoc consistently recognizing the embedded text.Source: §5.3 · p. 6

5.4 Ablation Studies

• Including detection, recognition, and spotting during both training stages yields the best performance across detection, recognition, and multimodal understanding.Adding these tasks in individual phases also improved all three task areas.Source: §5.4 · pp. 6–7, Table 5 · p. 6
• Adding 224 positional-index tokens for bounding-box coordinates did not improve text detection over directly predicting integer coordinates.The default predicts coordinates normalized to [0, 223] for 224×224 images.Source: Table 6 · p. 6, §5.4 · p. 7

• UniDoc combines text detection, recognition, spotting, and understanding through unified multimodal instruction tuning.Source: §6 · p. 7
• The work contributes a large-scale instruction-following dataset and reports state-of-the-art scores across multiple benchmarks.Source: §6 · p. 8
• Current limitations include lacking fine-grained visual features for detection and recognition and limited input-image resolution.Source: §6 · p. 8

• The listed affiliations are the University of Science and Technology of China and ByteDance.Source: haof@mail.ustc.edu.cn, {zhwg,… · p. 10
• The paper is identified as arXiv:2308.11592v2, dated 2 September 2023.Source: haof@mail.ustc.edu.cn, {zhwg,… · p. 10

• Templates were generated using GPT-4 and varied action verbs, image descriptors, and question formats.Source: A. Prompt Template · p. 10
• The detection task has one template, intended to avoid confusion with recognition instructions and adverse effects on network training.Source: A. Prompt Template · p. 10

B. Additional Qualitative Results

• On TotalText, UniDoc reasonably detects text, with returned bounding boxes visualized over the original images.The prompt asks for all text locations in the photo.Source: Figure 1 · p. 10, B. Additional Qualitative Results · p. 10
• Spotting instruction templates use <term> for image-type descriptors such as “image” or “picture,” diversifying the templates.Source: Table 3 · p. 11
• On electronic text-rich images, UniDoc recognizes most text but sometimes conjectures; underlines mark some recognition errors.Source: B. Additional Qualitative Results · p. 10, Figure 2 · p. 11
• On Host line-level images, UniDoc accurately recognizes text despite image text being missing or incomplete.Source: B. Additional Qualitative Results · pp. 10–11
• Across diverse image scenarios, UniDoc uses visual and textual information, instruction context, and language-model world knowledge to generate reasonable responses.Source: B. Additional Qualitative Results · p. 11