PaperlayerView exact revision on arXiv

Source-linked AI summary

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, Fei Huang

arXiv:2310.05126v1cs.CVcs.AI

TL;DR

Visually-situated language understanding requires reading diverse text-rich images, while existing approaches rely on costly domain-specific training. UReader uses unified instruction tuning, auxiliary text and semantic tasks, and shape-adaptive cropping with an MLLM. It achieves state-of-the-art OCR-free performance in 8 out of 10 tasks across five domains, while remaining limited on multi-page documents and richer open-ended generation.

  • Problem

    Existing MLLMs have shallow visual text recognition, but diverse image types and sizes remain challenging for universal visually-situated language understanding.

  • Method

    UReader jointly instruction-tunes an MLLM across tasks, adds text reading and key points generation, and uses shape-adaptive cropping for high-resolution images.

  • Results

    UReader achieves state-of-the-art OCR-free performance in 8 out of 10 tasks across documents, tables, charts, natural images, and webpage screenshots.

  • Takeaways & Limitations

    The results support using open-domain MLLMs and low-cost instruction tuning for universal OCR-free visually-situated language understanding.

  • Takeaways & Limitations

    UReader struggles with multi-page documents and its open-ended generation ability remains far from well studied.

Abstract

from arXiv · show

Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.

1 Introduction

Existing MLLMs show shallow zero-shot visual text recognition, but diverse image types and sizes leave universal visually-situated language understanding unresolved. UReader addresses this gap through unified instruction tuning, auxiliary tasks, and shape-adaptive cropping, achieving state-of-the-art OCR-free performance across five domains.

  • MLLMs exhibit shallow zero-shot visual text recognition, yet remain far from universal understanding across diverse image types and sizes.
  • Prior visually-situated language understanding methods use domain-specific pretraining and finetuning, incurring high training costs such as more than 192 A100 days for Donut.
  • UReader applies low-cost instruction tuning to MLLMs while directly leveraging diverse visually-situated language understanding datasets without pretraining tasks.
  • UReader converts tasks into a unified vision-language instruction format and adds text reading and key points generation tasks to improve text recognition and semantic understanding.
  • UReader covers document, table, chart, natural image, and webpage screenshot domains and achieves state-of-the-art OCR-free performance in 8 out of 10 tasks.

2 Related Work

Visually-situated language understanding spans diverse text-rich image domains and includes OCR-dependent two-stage and end-to-end approaches. MLLMs offer shallow zero-shot text recognition, but remain insufficient for universal understanding.

  • Visually-situated language understanding covers documents, tables, charts, natural images, webpage screenshots, and other text-rich image types.
  • Existing methods are divided into two-stage and end-to-end models according to whether they use off-the-shelf OCR models or APIs.
  • Two-stage methods commonly design pretraining tasks to learn cross-modality alignment between visual and textual inputs.
  • MLLMs generally connect a pretrained vision encoder with a strong language model and exhibit shallow zero-shot text recognition ability.
  • Despite this emergent ability, MLLMs remain far from universal visually-situated language understanding.

3 UReader

UReader adapts an MLLM to high-resolution, multi-image understanding by cropping images according to aspect ratio and resolution, encoding each crop, and adding spatial information before language decoding.

  • UReader preprocesses an image with shape-adaptive cropping, then passes the resulting sub-images through a visual encoder and visual abstractor.
  • 3.1 Shape-Adaptive Cropping Module: The cropping module selects among predefined grids to preserve image resolution and fit the input aspect ratio.
  • 3.1 Shape-Adaptive Cropping Module: The matched grid maximizes a score combining resolution-related and resolution-agnostic intersection-over-union measures.
  • 3.1 Shape-Adaptive Cropping Module: The selected grid produces multiple local images plus a global resized image, which are processed in parallel by the visual encoder and abstractor.
  • 3.2 Cropped Images Modeling with LLM: The visual encoder extracts features, while the visual abstractor summarizes them into language-space visual representations using learnable queries.
  • 3.2 Cropped Images Modeling with LLM: Two-dimensional crop position embeddings encode each cell's row and column, are added to visual features, and help the language model correlate cropped images.
  • 3.2 Cropped Images Modeling with LLM: The original language model is frozen and LoRA is used to keep training costs low while modeling multiple images.

4 Instruction Tuning

UReader converts diverse visually-situated language understanding tasks into a unified instruction format and jointly tunes them without large-scale pretraining. Text Reading and Key Points Generation provide auxiliary supervision for visual text recognition and semantic comprehension across five domains.

  • 4 Instruction Tuning: UReader jointly trains multiple downstream datasets after reorganizing visually-situated language understanding tasks into a unified instruction format.The covered task families include visual question answering, information extraction, natural language inference, and image captioning.
  • 4.1 Tuning Tasks: Visual question answering uses the question as the instruction, while information extraction asks for category-value pairs and returns ‘None’ for absent categories.
  • 4.1 Tuning Tasks: Natural language inference is reformulated as a Yes-or-No instruction, and image captioning uses randomly selected descriptive prompts.
  • 4.1 Tuning Tasks: The Text Reading auxiliary task trains the model to recognize text in reading order and to continue from partial text inputs.Sampling emphasizes reading from the beginning because initial text often contains key image information, such as a chart title.
  • 4.1 Tuning Tasks: Key Points Generation is added as an auxiliary language-modeling task to strengthen vision-and-language semantic comprehension.
  • 4.2 Instruction Data Resources: The instruction-tuning resources span document datasets including DocVQA, InfoVQA, DeepForm, and KLC.These resources include question answering, infographic understanding, and information extraction data.
  • 4.2 Instruction Data Resources: Table resources include WikiTableQuestions for comparison and arithmetic questions and TabFact for entailment and refutation statements about tables.
  • 4.2 Instruction Data Resources: ChartQA, TextVQA, TextCaps, and VisualMRC extend the data to charts, text-rich natural images, and webpage screenshots.The webpage screenshot resource contains fluent-sentence answers averaging 9.53 words.

5 Experiments

Experiments evaluate UReader across ten datasets and show strong cross-domain performance, while ablations examine auxiliary tasks, trainable components, joint training, and shape-adaptive cropping. Qualitative results demonstrate broad image understanding but expose difficulties with long, text-dense images and open-ended key-point generation.

  • 5.1 Implementation Details: The evaluation uses ANLS, F1, accuracy, relaxed accuracy, and CIDEr according to the task and dataset.
  • 5.3 Main Results: UReader achieves state-of-the-art performance in 8 of 10 tasks across five domains, covering question answering, information extraction, natural language inference, and image captioning.
  • 5.3 Main Results: 86M trainable parameters versus 1.3B enables UReader to outperform Pix2Structlarge on InfoVQA, ChartQA, and TextCaps without a specific downstream finetuning stage.The comparison contrasts UReader’s lower trainable-parameter count with Pix2Structlarge’s larger training setup.
  • 5.4 Ablation Study: Removing Key Points Generation decreases performance across all dataset domains, while additionally removing Text Reading causes a larger degradation.These ablations support complementary semantic-understanding and text-recognition roles for the auxiliary tasks.
  • 5.4 Ablation Study: Freezing either the visual abstractor or LoRA reduces performance, indicating that both visual and language components are tuned for the target tasks.
  • 5.4 Ablation Study: Removing four document datasets worsens performance on table, natural-image, and webpage domains, while the document-free model scores 46.2 on DocVQA.The result also indicates potential out-of-domain understanding under cross-domain joint training.
  • 5.4 Ablation Study: Shape-adaptive cropping improves performance over direct tuning, more crops help, a resized global image provides slight gains, and crop-position encoding is effective.Cropping preserves constant 224x224 local resolution while increasing overall image resolution; the module selects grids according to raw aspect ratio.
  • 5.5 Qualitative Results: Qualitative examples show document extraction, regional instruction following, table layout and arithmetic reasoning, webpage paragraph localization, and ordered text reading.For text-rich book pages, the model may continue decoding after reading only the beginning; key-point generation can also misidentify chart properties.

6 Conclusion

UReader applies low-cost instruction tuning to MLLMs for universal OCR-free visually-situated language understanding. It unifies downstream tasks and adds text reading, key points generation, and shape-adaptive cropping, achieving state-of-the-art performance across diverse domains.

  • UReader leverages existing Multimodal Large Language Models for universal OCR-free visually-situated language understanding through low-cost instruction tuning.
  • All downstream tasks are reorganized into a unified instruction-tuning format, supplemented by Text Reading and Key Points Generation auxiliary tasks.
  • The shape-adaptive cropping module cuts images into multiple local images according to aspect ratio and resolution, enabling a frozen low-resolution vision encoder to process high-resolution images.
  • UReader achieves state-of-the-art OCR-free performance in 8 out of 10 datasets spanning documents, tables, charts, natural images, and webpage screenshots.

Limitations

UReader has limitations in multi-page document understanding, crop efficiency, open-ended generation, and text reading fidelity. These limitations reflect challenges in correlating pages, allocating features, modeling complex reasoning, and controlling decoder hallucination.

  • UReader struggles with multi-page documents because it cannot correlate different pages and has limited decoder sequence length.
  • UReader feeds equal numbers of features for each local image even though crops vary in visual and textual information richness.
  • More difficult open-ended generation tasks, such as generating chain-of-thought answers, are not currently considered.
  • Text Reading improves text recognition, but performance remains unsatisfactory because the language decoder hallucinates and does not reliably follow image content.

Ethics Statement

UReader relies on MLLMs trained on large-scale web image-text data, which may introduce toxic language and bias. The authors state that public-dataset finetuning and domain-specific use minimize these issues.

  • UReader inherits potential toxic-language and bias issues from MLLMs trained on large-scale image and text data from the web.
  • The authors state that further finetuning on publicly available datasets and use in visually-situated language understanding minimize these ethical risks.

A Grid Distribution on Downstream Datasets

The shape-adaptive cropping module responds to diverse image shapes across downstream datasets. Results indicate that UReader favors cross-modality relationships but is weaker on text layout without document pretraining and layout-specific tasks.

  • A Grid Distribution on Downstream Datasets: The shape-adaptive cropping module selects grid configurations across all ten datasets, reflecting the wide variety of downstream image shapes.
  • B.1 Underperforms Ocr-Free Baselines on DocVQA and DeepForm: UReader underperforms OCR-free baselines on DocVQA and DeepForm.
  • B.1 Underperforms Ocr-Free Baselines on DocVQA and DeepForm: The comparison attributes UReader's DocVQA and DeepForm weakness partly to Donut's 11M-image IIT-CDIP pretraining and Pix2Struct's layout-focused HTML prediction task.
  • B.1 Underperforms Ocr-Free Baselines on DocVQA and DeepForm: InfoVQA relies more on relationships between text and visual objects because its poster-style layout is less important than in DocVQA and DeepForm.
  • B.1 Underperforms Ocr-Free Baselines on DocVQA and DeepForm: UReader performs better at cross-modality relationships but weaker at text layout understanding than Donut and Pix2Struct without large-scale document pretraining and layout-specific tasks.

B.2 Compared with Pipeline Methods

UReader is comparable to pipeline methods on several tasks but shows a clear gap on document, table, and webpage-screenshot benchmarks. The authors attribute this gap to text/layout recognition challenges and limited multi-page input handling.

  • UReader achieves comparable or slightly worse results than pipeline methods on TextVQA, ChartQA, InfoVQA, TextCaps, and TabFact.
  • UReader has an obvious performance gap relative to pipeline methods on DocVQA, DeepForm, KLC, WTQ, and VisualMRC.
  • Text recognition and layout extraction create gaps on document, table, and webpage-screenshot datasets because their information is dominated by text and relatively uniform layouts.
  • 98% of KLC examples contain more than four pages, while 75% of DeepForm examples contain more than one page, but OCR-free models input only the first page.
  • UReader outperforms mPLUG-Owl and UniDoc on unseen OCR-VQA, scoring 41.1 versus 28.6 and 34.5, respectively.
  • Qualitative results include text reading and visual question answering examples across natural images, charts, tables, documents, and webpage screenshots.

C.2 Open-domain Results

Open-domain examples show UReader recognizing small text, answering cascaded and multi-turn questions, and extracting information from forms and screenshots. The examples also expose repetition and vision-irrelevant generation in some responses, while contrasting UReader with mPLUG-Owl.

  • UReader accurately recognizes small text in natural images, including “Name of passenger” and “MORRIS/KARLA,” where mPLUG-Owl answers incorrectly.
  • UReader completes a cascaded query requiring object-position understanding and corresponding price extraction, while mPLUG-Owl answers incorrectly on both.
  • UReader answers multi-turn screenshot questions involving conversation history and common-sense references, whereas mPLUG-Owl generates unrelated content in some cases.
  • UReader can list textual content from an image, but one example repeats the same statement about Ferrari’s completed laps three times.
  • For a multi-form image, UReader understands field meanings and filters out components priced at $0, although it wrongly includes the header and omits some prices.
  • The qualitative figures cover text reading, key-point generation, visual question answering, and image examples across documents, webpages, natural images, charts, and tables.
  • The authors conclude that UReader retains interactive MLLM ability in open-domain settings and shows stronger visually-situated language understanding than mPLUG-Owl.
  • The instruction templates for auxiliary text reading and key-point generation tasks are provided in Table 5.
Loading 2310.05126v1…