UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

• Existing MLLMs show shallow zero-shot text recognition on low-resolution images with salient text, but image diversity and size limit universal visually situated understanding.Prior methods use domain-specific pretraining and finetuning; Donut (Kim et al., 2022) costs more than 192 A100 days.Source: §1 · pp. 1–2
• UReader converts visually situated language tasks into a unified vision-language instruction format and adds text-reading and key-points-generation tasks to strengthen text and semantic understanding.Its instruction-tuning dataset covers five domains: documents, tables, charts, natural images, and webpage screenshots.Source: §1 · p. 2
• A shape-adaptive cropping module splits high-resolution images into local crops, encodes them with a frozen visual encoder and trainable abstractor, then combines them with crop-position encodings and a resized global image.The global image helps mitigate salient-information loss from cropping, while the module avoids blur and distortion from resizing the original image.Source: §1 · p. 2
• UReader achieves state-of-the-art OCR-free performance on 8 out of 10 tasks across five domains: documents, tables, charts, natural images, and webpage screenshots.Source: §1 · p. 2

2 Related Work

• Visually-situated language understanding covers documents, tables, charts, natural images, and webpages, with tasks including question answering, captioning, information extraction, and natural language inference.Examples cited include document, table, chart, natural-image, and webpage datasets.Source: §2 · p. 2
• Prior work uses OCR-dependent two-stage models or OCR-free end-to-end models, with domain-specific pretraining and finetuning that can be costly; Donut training exceeds 192 A100 days.UReader instead proposes instruction tuning across image types, leveraging MLLMs’ shallow text recognition ability to reduce training cost.Source: §2 · p. 2
• MLLMs connect pretrained vision encoders to large language models and show shallow zero-shot text recognition, but their text understanding across domains and high-resolution images remains limited.UReader proposes instruction tuning across five domains and shape-adaptive cropping to address these limitations.Source: §2 · p. 3

3 UReader

• UReader sends cropped sub-images through the visual encoder and abstractor in parallel, then uses crop-position encoding to provide spatial information across them.The paper uses mPLUG-Owl as its basic MLLM, while noting that other MLLMs are not excluded.Source: §3 · p. 3

3.1 Shape-Adaptive Cropping Module

• The shape-adaptive module selects among predefined grids to preserve image resolution and fit the input aspect ratio, maximizing a score that combines resolution-related and resolution-agnostic IoU.It resizes the image into local sub-images and a global image, which are passed in parallel to the visual encoder and abstractor.Source: §3.1 · p. 3
• The visual encoder extracts features from the local and global images, and the visual abstractor uses learnable queries to produce higher-semantic representations in language feature space.Source: §3.1 · p. 4

3.2 Cropped Images Modeling with LLM

• UReader adds two-dimensional row and column crop-position embeddings to visual features so the language model can represent spatial relationships among cropped images.It freezes the original language model and applies LoRA to model multiple images while keeping training costs low.Source: §3.2 · p. 4

4 Instruction Tuning

• Joint instruction tuning combines multiple downstream datasets without large-scale pretraining and adds text-reading and key-points tasks.The tasks use a unified instruction format (Dai et al., 2023).Source: §4 · p. 4

4.1 Tuning Tasks

• Visual question answering, information extraction, natural-language inference, and image captioning are all reorganized into instruction format.Prompts ask for a QA answer, a category’s value (or “None”), a yes/no entailment judgment, or a brief caption; caption prompts are randomly selected from 11 LLaVa prompts (Liu et al., 2023a).Source: §4.1 · p. 4
• Text Reading trains recognition and continuation by splitting text in common reading order into an input prefix and a target suffix.The split position is sampled from {0, L/6, 2L/6, …, 5L/6}; starting-from-zero samples are chosen at rate 0.5 and other positions at 0.1, with distinct prompts for starting and continuing.Source: §4.1 · p. 5
• Key Points Generation converts image question-answer pairs into declarative key points using Vicuna.Templates prompt the model to identify key points about the image.Source: §4.1 · p. 5

4.2 Instruction Data Resources

• Document resources include DocVQA’s 50k QA pairs on 12k images, InfoVQA’s 30k pairs on 5k infographics, and DeepForm and KLC extraction datasets.DeepForm covers 1.1k election-spending documents, while KLC contains 2.7k charity reports.Source: §4.2 · p. 5
• Table and chart resources include WTQ’s 2.1k table images and 23k question-answer pairs, TabFact’s 112k statements, and ChartQA’s 21k images and 32k pairs.WTQ questions demand comparison and arithmetic; TabFact statements concern 16k Wikipedia tables.Source: §4.2 · p. 5
• Natural-image resources include TextVQA’s 45k QA pairs on 28k text-containing images and TextCaps’ 145k captions based on TextVQA.TextCaps supports image captioning with reading comprehension.Source: §4.2 · p. 5
• VisualMRC contributes 30k fluent-sentence QA pairs from 5k webpage screenshots across 35 websites.Its answers average 9.53 words and are longer than those in the other mentioned QA datasets.Source: §4.2 · p. 5

5.1 Implementation Details

• Instruction tuning used 514,764 samples for 20k steps over 10 epochs, taking 16 A100 days.The model uses mPLUG-Owl (Ye et al., 2023); the shape-adaptive cropping module allows up to 20 cells, and LoRA uses rank 8.Source: §5.1 · pp. 5–6

5.3 Main Results

• UReader achieves state-of-the-art performance on 8 of 10 tasks across 5 domains and outperforms Pix2Structlarge on InfoVQA, ChartQA, and TextCaps with 86M versus 1.3B trainable parameters.UReader requires no task-specific finetuning stage, while Pix2Structlarge was trained for more than 170k steps with batch size 1024 on 128 TPUs.Source: Table 1 · p. 6, §5.3 · p. 6

5.2 Evaluation

• Models were tuned on official training splits and evaluated on test splits, using task-specific metrics including ANLS, F1, accuracy, relaxed accuracy, and CIDEr.TextVQA and TextCaps were evaluated on their official challenge website.Source: §5.2 · p. 6

5.4 Ablation Study

• Removing Key Points Generation reduces performance across domains, removing Text Reading causes larger declines, and freezing either the visual abstractor or LoRA also hurts performance.These ablations support both auxiliary tasks and finetuning the vision and language components.Source: §5.4 · p. 6, Table 2 · p. 7
• Removing the four document datasets worsens results on table, natural-image, and webpage domains, while the document-free model still scores 46.2 on DocVQA.The authors interpret the cross-domain results as evidence that domains share characteristics and joint training improves universal performance.Source: §5.4 · pp. 6–7, Table 2 · p. 7
• Shape-adaptive cropping significantly improves UReader performance over directly tuning mPLUG-Owl without revisions (r7 vs r6); more crops improve performance, a resized global image slightly helps most datasets, and removing crop-position encoding hurts.Source: §5.4 · p. 7

5.5 Qualitative Results

• Qualitative examples show document extraction, region-based question answering, table comprehension, webpage reading, and top-left-to-bottom-right text reading, but dense-text continuation and chart key-point generation remain unreliable.The chart example includes a mistake about the lowest line, and the authors describe key-point listing as a superficial attempt at open-ended generation.Source: §5.5 · p. 8

• The approach leverages existing multimodal large language models for universal OCR-free visually-situated language understanding through low-cost instruction tuning.Source: §6 · p. 8
• UReader achieves state-of-the-art OCR-free performance on 8 out of 10 datasets spanning documents, tables, charts, natural images, and webpage screenshots.Source: §6 · p. 8
• UReader organizes downstream tasks in a unified instruction-tuning format and adds Text Reading and Key Points Generation tasks to strengthen text recognition and vision-language semantic comprehension.Source: §6 · p. 8
• A shape-adaptive cropping module divides high-resolution images into local images based on their raw aspect ratio and resolution, enabling use of the pretrained vision encoder.Source: §6 · p. 8

Limitations

• UReader struggles with multi-page documents because it cannot correlate different pages and has a limited decoder sequence length.This affects books and papers.Source: Limitations · p. 9
• Text reading helps recognition, but performance with the LLM decoder remains unsatisfactory because of hallucinations and difficulty following images strictly.The authors identify strict image-grounded reading as a challenging topic.Source: Limitations · p. 9
• UReader allocates an equal number of features to every local image, although some crops contain little visual or textual information.The authors plan to explore more efficient encoding of different crops.Source: Limitations · p. 9
• Open-ended visually situated language generation remains under-studied; UReader explores key-point generation but not more difficult tasks such as chain-of-thought answers.How to simulate such abilities through instruction tuning remains an open topic.Source: Limitations · p. 9

Ethics Statement

• UReader is further finetuned on publicly available datasets for visually situated language understanding, where the authors say toxicity and bias have minimal impact.The underlying multimodal large language models are trained on large-scale web image and text data and may be subject to toxic language and bias.Source: Ethics Statement · p. 9

A Grid Distribution on Downstream Datasets

• The shape-adaptive cropping module selects grids across all ten datasets, whose varied image shapes highlight the module’s importance.The paper visualizes the frequency distribution of selected grids in Figure 6.Source: A Grid Distribution on Downstream… · p. 12

B.1 Underperforms Ocr-Free Baselines on DocVQA and DeepForm

• UReader underperforms OCR-free baselines on DocVQA and DeepForm, where large-scale document pretraining and layout understanding benefit competing models.The paper contrasts Donut’s pretraining on 11M IIT-CDIP document images with UReader’s instruction finetuning on fewer than 0.5M assorted images, and describes Pix2Struct’s layout-focused pretraining.Source: §B.1 · p. 12
• Donut’s pretraining on 11M IIT-CDIP document images matches DocVQA and DeepForm’s domain, whereas UReader has no pretraining and is instruction-finetuned on fewer than 0.5M assorted images.The passage states that training with more document images brings better performance.Source: §B.1 · p. 12
• Pix2Struct’s masked-web-screenshot HTML DOM prediction pretraining develops layout understanding, while UReader’s top-to-bottom, left-to-right text reading requires weaker layout understanding.The paper identifies layout-understanding pretraining as another factor improving performance on DocVQA.Source: §B.1 · p. 12
• UReader outperforms baselines on InfoVQA and KLC, where poster-style text-object relationships or the simpler first-page report-cover layout matter more than DocVQA and DeepForm layouts.OCR-free models use only the first page on KLC; the passage characterizes that page as always the report cover.Source: §B.1 · p. 12
• UReader is better at understanding cross-modal image relationships but weaker at text-layout comprehension without large-scale document pretraining and specific layout-understanding tasks.The paper attributes this contrast to UReader’s open-domain MLLM pretraining compared with Donut and Pix2Struct.Source: §B.1 · p. 12

B.2 Compared with Pipeline Methods

• UReader performs comparably to or slightly worse than pipeline methods on TextVQA, ChartQA, InfoVQA, TextCaps, and TabFact, but shows a clear gap on DocVQA, DeepForm, KLC, WTQ, and VisualMRC.Table 4 presents the performance comparison; the supplied passages do not provide its cell values.Source: §B.2 · p. 12, Table 4 · p. 13
• Similar pipeline and OCR-free results reflect challenges in cross-modal relations, while simple formats such as TabFact’s binary classification can narrow performance gaps.TextVQA, TextCaps, and InfoVQA require relating text to visual objects; ChartQA requires line-trend comprehension.Source: §B.2 · p. 12
• Pipeline methods’ advantage on document, table, and webpage tasks stems from stronger text and layout extraction and, on multi-page datasets, access to pages UReader omits.UReader and Donut can struggle to recognize all text; UReader processes only the first page, while 98% of KLC examples exceed four pages and 75% of DeepForm examples exceed one page.Source: §B.2 · p. 12

B.3 Zero-shot Performance

• On unseen OCR-VQA, UReader scores 41.1 versus mPLUG-Owl’s 28.6 and UniDoc’s 34.5 (Feng et al., 2023), which was trained with layout prediction.The comparison uses the same evaluation metrics.Source: §B.3 · p. 12

C More Qualitative Results

• Figures 7–11 show qualitative results across natural images, charts, tables, documents, and webpage screenshots.Source: §C.1 · p. 13

C.1 Downstream Results

• On a VisualMRC webpage example, UReader can read text in other regions with suitable instructions, but unrestricted reading may drift into vision-irrelevant text.The passage attributes this hallucination to the decoder following nearby text context rather than the image after reading the beginning text.Source: §C.1 · p. 13

C.2 Open-domain Results

• UReader retains some interactive ability in open-domain use and shows stronger visually situated language understanding than mPLUG-Owl.The comparison spans examples involving small text, spatial references, conversation history, common sense, and forms.Source: §C.2 · p. 16, Figure 12 · p. 17
• UReader accurately reads small text in a natural image and answers questions about it, whereas mPLUG-Owl fails to give the correct name even after a follow-up prompt.The example asks for the passenger name and identifies it as MORRIS/KARLA.Source: §C.2 · p. 13, Figure 12 · p. 17
• UReader completes a cascaded query requiring both recognition of referenced objects and extraction of corresponding fields, while mPLUG-Owl errs on both object recognition and price extraction.Source: §C.2 · p. 13, Figure 12 · p. 17
• In multi-turn screenshot questions that depend on conversation history and common sense, UReader answers well, while mPLUG-Owl produces unrelated or erroneous content.The examples connect account creation with joining Twitter and infer that an inactive Follow button means the user has not followed Iron Musk.Source: §C.2 · pp. 13–14, Figure 12 · p. 17
• For a form with multiple tables, UReader finds the relevant form and filters out zero-priced components, but includes a header and omits individual component prices.The authors contrast this with mPLUG-Owl responses described as illusory because textual information in the image is lost.Source: §C.2 · pp. 14–15, Figure 12 · p. 17

D Instruction Templates

• Table 5 presents instruction templates for beginning text reading, continuing reading, and key-points generation; continuing-reading instructions randomly combine prompts from parts A and B.Source: D Instruction Templates · p. 16, Table 5 · p. 18