Source-linked AI summary
OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, Xiang Bai
TL;DR
Text-related visual capabilities of large multimodal models remain relatively unexplored despite their progress in language and vision-language learning. The paper evaluates 14 models across five OCR-related tasks and introduces OCRBench, finding promising recognition performance alongside persistent weaknesses in multilingual, handwritten, non-semantic, and mathematical text.
Problem
The effectiveness of large multimodal models on text-related visual tasks remains relatively unexplored, including recognition of multilingual, handwritten, non-semantic, and mathematical text.
Method
The paper evaluates 14 large multimodal models across five representative OCR tasks and introduces OCRBench using manually filtered and corrected question-answer pairs.
Results
Large multimodal models achieve promising results, especially in text recognition, but remain significantly behind domain-specific methods on various text-related tasks.
Takeaways & Limitations
OCRBench provides an evaluation benchmark for multimodal OCR capabilities, while the findings indicate that specialized techniques tailored to each task remain essential.
Takeaways & Limitations
OCRBench lacks comprehensive coverage of multilingual documents, texts captured in diverse scenarios, and text-detection tasks.
Abstract
from arXiv · showhide
Large models have recently played a dominant role in natural language processing and multimodal vision-language learning. However, their effectiveness in text-related visual tasks remains relatively unexplored. In this paper, we conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). To facilitate the assessment of Optical Character Recognition (OCR) capabilities in Large Multimodal Models, we propose OCRBench, a comprehensive evaluation benchmark. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available. Furthermore, our study reveals both the strengths and weaknesses of these models, particularly in handling multilingual text, handwritten text, non-semantic text, and mathematical expression recognition. Most importantly, the baseline results presented in this study could provide a foundational framework for the conception and assessment of innovative strategies targeted at enhancing zero-shot multimodal techniques. The evaluation pipeline and benchmark are available at https://github.com/Yuliang-Liu/MultimodalOCR.
1 Introduction
Large multimodal models show strong zero-shot potential, but their text-related visual capabilities remain insufficiently understood. This study evaluates their OCR abilities across representative tasks and finds persistent weaknesses alongside opportunities for domain-specific improvement.
- Large multimodal models have extended large-model capabilities into vision-language tasks, including zero-shot OCR without explicit OCR-domain training.
- Improved understanding of textual and visual interplay could support applications such as digital marketing and social media analysis.
- The study evaluates 14 large multimodal models across Text Recognition, Scene Text-Centric VQA, Document-Oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.
- Even state-of-the-art models struggle with blurry, handwritten, multilingual, and handwritten mathematical text, while often favoring common words over random letter sequences.
- The findings reveal significant gaps from domain-specific methods and identify domain-specific adaptation and optimization as an opportunity for improving LMM OCR capabilities.
2 Related Work
Existing large multimodal benchmarks assess broad abilities but provide limited coverage of OCR, while multiple-choice and true-or-false formats cannot accurately test word recognition. The paper therefore evaluates five text-related tasks and introduces OCRBench with manually filtered and corrected question-answer pairs.
- Existing LMM evaluation systems use GPT-4 or human judgments, multiple-choice questions, and true-or-false questions to assess model abilities.
- These benchmarks provide limited OCR testing, and multiple-choice or true-or-false formats cannot accurately assess whether models recognize words in images.
- The paper evaluates LMMs across five prominent text-related tasks and presents OCRBench as a collection of 1,000 manually filtered and corrected question-answer pairs.
3 Experiments
The study evaluates LMM OCR abilities across five text-related visual tasks and finds strong performance on several printed-text settings but persistent weaknesses in semantic-free, handwritten, multilingual, blurry, and mathematical text.
- Evaluation Metric: LMMs use a unified criterion that checks whether the ground truth appears in the generated output, while filtering answers shorter than four symbols.This criterion addresses explanatory responses that make exact matching and ANLS unsuitable for zero-shot LMM evaluation.
- Evaluation Dataset: The evaluation covers Text Recognition, Scene Text-Centric VQA, Document-Oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.The benchmark draws on datasets spanning regular, irregular, occluded, artistic, handwritten, Chinese, digit-string, semantic, and non-semantic text.
- Results: LMMs achieve performance comparable to supervised state-of-the-art models on regular, irregular, occluded, and artistic text recognition.In WordArt, InstructBLIP2 and BLIVA outperform the supervised state-of-the-art model.
- Results: LMMs perform poorly on handwritten, Chinese, handwritten-string, non-semantic, and handwritten mathematical expression recognition.The results also show poorer performance for smaller input resolutions on scene-text VQA, document VQA, and KIE, where fine text details matter.
- Results: On non-semantic text, LMM accuracy decreases by an average of 57.0% relative to semantic text, whereas a scene-text recognition state-of-the-art method drops by around 4.6%.The authors attribute this contrast to LMM reliance on semantic understanding, while the supervised method directly recognizes individual characters.
- OCRBench: OCRBench contains 1000 manually verified question-answer pairs across five components, and Gemini scores highest, followed by GPT4V.GPT4V refused results for 84 images, while both Gemini and GPT4V still struggle with HMER, unclear images, handwritten and non-semantic text, and instruction adherence.
4 Future work
Large multimodal models have made substantial OCR progress, but OCRBench still lacks broad coverage of challenging image types and tasks.
- MiniCPMV-2.6 scored 852 on OCRBench, while Mini-Monkey scored 802 with 2B parameters.
- OCRBench’s current data are not entirely open-source, and many evaluated models remain proprietary.
- OCRBench lacks comprehensive coverage of multilingual documents, texts captured in diverse scenarios, and text detection tasks.
5 Conclusion
The paper evaluates LMM OCR across five task categories and finds promising text-recognition performance alongside substantial gaps requiring task-specific supervised techniques.
- The study evaluates text recognition, Scene Text-Centric VQA, Document-Oriented VQA, KIE, and HMER.
- LMMs achieve promising OCR results, including state-of-the-art performance on some datasets.
- Significant gaps remain compared with domain-specific supervised methods that use less computational resources and data.
- OCRBench provides an evaluation benchmark intended to support development of LMM OCR capabilities.
A Summary of the Evaluation Benchmarks.
OCRBench combines five OCR-related task categories into a 1,000-question benchmark, with manually filtered and corrected answers.
- The Text Recognition component contains 300 images across six subtasks, with 50 images per subtask.
- Scene Text-Centric VQA contains 200 questions drawn equally from four datasets.
- OCRBench encompasses 1,000 questions across five tasks, and all answers were manually filtered and corrected.
B Summary of the Models.
The evaluated LMMs commonly have input resolutions of 224 or 336, and higher-resolution models perform better on Document-Oriented VQA and KIE.
- Most tested LMMs are limited to input resolutions of 224 or 336 because of their visual modules.
- These resolution limits are associated with poor performance on Document-Oriented VQA and KIE tasks.
- Models with higher resolutions exhibit better performance on Document-Oriented VQA and KIE.
C Supervised SOTA
The section surveys supervised state-of-the-art methods across OCR text recognition, scene-text VQA, and document-oriented tasks. These methods use contextual modeling, generative image-to-text learning, layout-aware pretraining, and specialized alignment objectives.
- Text Recognition: PARSeq achieves state-of-the-art performance on widely adopted OCR text-recognition datasets using permutation language modeling and unified non-AR and AR inference.The authors train PARSeq on the ST and MJ synthetic datasets and test it directly on real datasets.
- Text Recognition: Yu achieves state-of-the-art performance on ORAND-CAR-2014 with an efficient text-line recognition method based on prototype learning and feature-level sliding windows.
- Scene Text VQA: GIT leads STVQA and OCR-VQA with a generative image-to-text Transformer combining one image encoder and one text decoder under a single language-modeling task.For TextVQA, Mia uses T5 and aligns object features with scene text through masked language modeling and relative position prediction.
- Document-Oriented Tasks: ERNIE-Layout leads FUNSD through reading-order correction during pretraining, while StrucTexT leads SROIE with segment-token alignment and specialized pretraining tasks.For POIE, Kuang uses contrastive learning to establish the connec.